@wpelevator/tug 0.9.0 → 0.11.0

package/README.md

@@ -1,25 +1,99 @@
 # Tug
 
-The name: tugboats pull ships into place, like the pull-based agents here; `tug` is the CLI.
+Deploy Docker Compose apps to a few ordinary Linux servers, from your laptop or CI, without Kubernetes or a hosted platform.
 
-Deploy Docker Compose applications to one or more ordinary Linux VPSes through a small self-hosted control plane and a pull-based agent on each node, without Kubernetes or a hosted PaaS. Tug runs an agent on each node; the agents drive the node's Docker Engine API directly, and Compose is only used on your machine to read the file. What it is for, the deployment shapes it targets and the planned milestones are in the [Roadmap](docs/roadmap.md).
+You keep your `docker-compose.yml`. Tug installs a small control plane on one server and an agent on each of the others, then `tug deploy` puts your services where you say, waits until they are healthy and rolls back if they are not. It is built for running one to a few dozen sites, WordPress first, on two to ten VPSes, with backups and restores included.
 
-## Features
+## What you can do with it
 
-- One mutual-TLS port: every operator, CI job and node authenticates with a certificate from the control plane's private CA, and control authorizes it by public key. Scoped `read`/`deploy`/`admin` operator credentials for CI and teammates, an audit trail naming who acted, and SQLite metadata through Node's built-in driver.
-- Nodes join over SSH (`node-install`) or enroll themselves with a single `docker run` after `node-prepare-enrollment`, each with its own certificate and an immutable node ID. Agents only dial out, so nodes behind NAT work.
-- Agents pull desired state from control, reconcile containers through the Docker Engine API, run TCP/HTTP/Docker health checks and report status. An outbound WebSocket wakes them as soon as control queues work and carries live `exec`, followed logs, `ps` and `inspect` sessions, with long-poll and periodic-poll fallbacks for ordinary work.
-- `x-tug` placement of services and volumes on nodes, with immutable desired-state records per deployment.
-- Build application images on control from uploaded source, or locally with `deploy --build-local`. Agents download saved images directly from control. Published images are pinned to registry digests at deployment creation.
-- Secrets encrypted at rest, released only to the nodes whose services reference them, with targeted service recreation when only a secret changes.
-- Deployment convergence, per-project operation locks, cancellation, rollback, per-service rollout events and log fetching through the agents.
-- Opt-in zero-downtime rolling rollouts for ingress-fronted services (`x-tug.rollout`) and automatic rollback of failed deployments (`x-tug.rollback: auto`). See [Compose extensions](docs/compose.md#rolling-rollouts).
-- Private published images use encrypted project credentials for digest resolution and Engine API pulls.
-- Optional WireGuard mesh between nodes, health-aware cross-node service names (hosts-file entries injected into containers) and a managed Caddy ingress with automatic HTTPS, each opt-in. See [Networking](docs/networking.md).
-- Private-by-default published ports: generated Compose binds them to the overlay IP, loopback or the node address, never every interface unless you write `0.0.0.0`. See [Networking](docs/networking.md#published-ports-private-by-default).
-- Workload backups of service commands and volumes with restic, a separate control-plane backup and restore, and `tug doctor` diagnostics for certificate expiry, node connectivity and clock skew.
+- **Deploy a Compose file to your servers.** `tug deploy` builds or pulls your images, starts them on the right nodes, waits for health checks and tells you why if something fails. `tug rollback` returns to the previous version, and `x-tug: { rollback: auto }` does it for you.
+- **Choose where things run.** Pin a database and its volume to one node, spread web containers over several, and let Tug's managed Caddy ingress serve them over HTTPS with certificates it obtains for you.
+- **Update without downtime.** Opt-in rolling rollouts start the new version, wait until it is healthy and only then switch traffic.
+- **Keep secrets out of your repository.** `tug secret set DB_PASSWORD` stores them encrypted and sends each one only to the nodes whose services use it.
+- **Back up and restore.** Scheduled backups of databases and volumes to S3-compatible storage with restic, and `tug restore run db` to bring one back.
+- **Look inside running services.** `tug logs`, `tug exec`, `tug run` (for example `wp` CLI), `tug ps` and `tug status` work through the control plane, so you never SSH to a node to see what is going on.
+- **Give CI a limited key.** `tug credential create` makes read-only, deploy-only or admin credentials, and every change records who made it.
 
-## How it fits together
+Nodes only dial out to the control plane, so they work behind NAT or a firewall that allows nothing inbound. Tug does not need a cloud provider's API, a registry or an overlay network.
+
+## Try it
+
+You need Node.js 22.18 or newer and Docker Compose v2 on your machine, and SSH access to one or more Linux servers (Debian, Ubuntu, Fedora or a RHEL-compatible distribution). Tug can install Docker on them for you.
+
+```bash
+npm install -g @wpelevator/tug      # or run it with npx @wpelevator/tug
+
+tug setup root@203.0.113.5          # install the control plane on a server
+tug node add root@198.51.100.7      # add a server that will run your apps
+tug deploy                          # deploy ./docker-compose.yml
+```
+
+In a terminal Tug asks before it changes anything: whether to install Docker, which private address the node should use and whether it should handle ingress. It shows the plan and waits for you to confirm. `tug deploy` asks for any Compose variables, such as `${DB_PASSWORD}`, that no secret or `.env` provides. In CI or any other script nothing is ever asked, so pass the choices as flags (`--address`, `--ingress`, `--install-docker`, `-y`).
+
+## Your Compose file, plus a little `x-tug`
+
+A normal Compose file already works. Add `x-tug` to say where services run, how to tell they are healthy and where traffic comes in:
+
+```yaml
+services:
+  db:
+    image: mariadb:11.4.8
+    volumes: [db-data:/var/lib/mysql]
+    environment:
+      MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
+    x-tug:
+      placement: { nodes: [app1] }
+      health: { type: tcp, port: 3306 }
+
+  wordpress:
+    image: wordpress:6.8.3-apache
+    depends_on: [db]
+    x-tug:
+      placement: { nodes: [app1] }
+      health: { type: http, port: 80, path: /wp-includes/version.php }
+      ingress: { host: blog.example.com, port: 80, tls: auto }
+
+volumes:
+  db-data:
+    x-tug: { placement: { nodes: [app1] } }
+```
+
+```bash
+tug secret set DB_ROOT_PASSWORD
+tug deploy
+tug status       # what is running where, and anything failing
+tug logs wordpress --follow
+```
+
+A complete WordPress site with scheduled database and uploads backups is in the [WordPress example](examples/wordpress/README.md). Everything `x-tug` can express is in [Compose extensions](docs/compose.md).
+
+## Everyday commands
+
+Project commands sit at the top level; everything else reads `tug <subject> <verb>`.
+
+| I want to... | Run |
+| --- | --- |
+| Deploy, or undo a deploy | `tug deploy`, `tug rollback` |
+| See what is running and what failed | `tug status`, `tug events`, `tug ps` |
+| Read logs or open a shell | `tug logs <service>`, `tug exec <service> sh` |
+| Run a one-off command | `tug run <service> -- wp cache flush` |
+| Change the number of replicas | `tug scale <service> 3` |
+| Add, list or replace servers | `tug node add`, `tug node list`, `tug node replace <name>` |
+| Add a server I cannot SSH to | `tug node enroll`, then run the printed `docker run` on it |
+| Manage secrets | `tug secret set`, `tug secret list`, `tug secret remove` |
+| Back up now, or restore | `tug backup run`, `tug restore run <target>` |
+| Give CI a deploy key | `tug credential create ci --role deploy` |
+| Check everything is healthy | `tug doctor` |
+
+Every command and option is in the [CLI reference](docs/cli.md). `tug status --json` prints raw JSON for scripts.
+
+## Is Tug right for you?
+
+Tug fits if you run a handful of Compose apps on VPSes you already pay for, want deploys, rollbacks, secrets and backups handled in one tool, and are happy with a single control server that you back up.
+
+It deliberately does not do Kubernetes-style scheduling, autoscaling, distributed storage or database failover, and it does not provision servers through a provider's API. Node-to-node traffic travels over your own private network (Tailscale, a VPC, a LAN), so you own its firewalling and encryption. If you want automatic rescheduling of containers across a cluster, see how it compares with [Docker Swarm](docs/swarm-comparison.md). The goals, principles and planned milestones are in the [Roadmap](docs/roadmap.md).
+
+## How it works
 
 ```text
   tug CLI / CI (operator credential)
@@ -38,33 +112,23 @@ Deploy Docker Compose applications to one or more ordinary Linux VPSes through a
        |                       |
 +------+------------+   +------+------------------+
 | Node app1         |   | Node app2 (ingress node)|
-|   Agent           |<->|   Agent                 |
+|   Agent           |   |   Agent                 |
 |   Docker Engine   |   |   Docker Engine         |
 |   services        |   |   services              |
 |                   |   |   managed Caddy ingress |
 +-------------------+   +-------------------------+
-        <-> optional WireGuard mesh between nodes
+        \_________________________/
+          workload traffic between node addresses
+          over the operator's private network
 
   SSH is used only by the CLI to install the control plane and nodes.
 ```
 
-Operators and CI talk to the control API. The control plane stores desired state and builds images; each agent dials out to fetch the work for its node, applies it through the Docker Engine API and reports health back, so nodes behind NAT need no inbound ports. SSH is used only to install the control plane and its nodes. The optional WireGuard mesh and Caddy ingress are described in [Networking](docs/networking.md), and every flow in detail in [Architecture](docs/architecture.md).
-
-## Quick start
-
-Install the CLI (Node >= 22.18), or run it without installing through `npx @wpelevator/tug`:
-
-```bash
-npm install -g @wpelevator/tug
-tug control-install root@203.0.113.5 --advertise-url https://203.0.113.5:8443 --install-docker
-tug init tug-root.credential.json
-tug node-install root@198.51.100.7 --name app1 --install-docker
-tug deploy -f docker-compose.yml
-```
+`tug deploy` reads your Compose file on your machine and sends it to the control plane, which stores it as an immutable desired state, builds any images and pins published images to digests. Each agent fetches the work for its node, applies it through the Docker Engine API, runs the health checks and reports back. The control plane marks the deployment done when every node reports it serving. Compose runs only on your machine; nodes need just Docker.
 
-The CLI installs the matching `ghcr.io/wpelevator/tug:<version>` image on your hosts. Control serves mutual TLS directly on its port (`8443`): every operator, CI job and node authenticates with a certificate from a private CA created at install, and clients verify a name generated from that CA, so the address can be a bare IP and can change later. Do not terminate TLS in front of it.
+Control serves mutual TLS directly on one port (`8443`). Every operator, CI job and node authenticates with a certificate from a private CA created at install, so the control address can be a bare IP and can change later. Do not terminate TLS in front of it. A `controlUrl` in `tug.yml` tells the CLI where control is (see [CLI configuration](docs/cli.md#config-file)). [Architecture](docs/architecture.md) covers every flow in detail, and [Security](docs/security.md) the trust boundaries.
 
-The full walkthrough, including nodes you cannot SSH to (`tug node-prepare-enrollment`), CI credentials (`tug credential-create`), moving the control plane and recovery, is in [Getting started](docs/getting-started.md).
+The full walkthrough, including CI credentials, moving the control plane and recovery, is in [Getting started](docs/getting-started.md).
 
 ## Documentation
 
@@ -72,11 +136,11 @@ The full walkthrough, including nodes you cannot SSH to (`tug node-prepare-enrol
 - [Terminology](docs/terminology.md): core concepts, the naming review (which recommendations are implemented), industry comparisons and the remaining proposed interface improvements.
 - [Architecture](docs/architecture.md): components, reconciliation, workflow diagrams, secrets and runtime paths.
 - [Compose extensions](docs/compose.md): `x-tug` placement, health checks, volumes and limitations.
-- [Control plane](docs/control-plane.md): install, mutual TLS and the private CA, image delivery, control-plane backup and restore.
+- [Control plane](docs/control-plane.md): install, mutual TLS and the private CA, image delivery, control-plane backup and recovery on another host.
 - [Backups](docs/backups.md): scheduled workload backups with `x-tug.backup` (restic), restore, recipes for common databases, and a restore drill. See also the [WordPress example](examples/wordpress/README.md).
 - [Access control](docs/access-control.md): certificate identity classes, scoped operator credentials, roles, audit trail and a GitHub Actions deploy example.
 - [Security](docs/security.md): deployment trust boundaries, security audit findings and validation limits.
-- [Networking](docs/networking.md): WireGuard mesh, cross-node service names, ingress and TLS, firewall notes.
+- [Networking](docs/networking.md): bring-your-own private networking, cross-node service names, ingress and TLS, firewall notes.
 - [Nodes](docs/nodes.md): node names and IDs, SSH install, self-service enrollment, certificate renewal and recovery, registration and status.
 - [CLI reference](docs/cli.md): every `tug` command and option.
 - [Configuration](docs/configuration.md): CLI, control and agent environment variables.
@@ -86,6 +150,7 @@ The full walkthrough, including nodes you cannot SSH to (`tug node-prepare-enrol
 - [Releases](docs/releases.md): changesets, npm and Docker image publishing.
 - [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution, with the mTLS dependency findings.
 - [Roadmap](docs/roadmap.md): goals, principles, the reference WordPress topology and planned milestones.
+- [Docker Swarm comparison](docs/swarm-comparison.md): overlapping capabilities, workflow advantages, scheduling and networking limits, failure behavior and workload fit.
 
 ## Development
 

package/dist/agent-credential.js

@@ -5,7 +5,7 @@ import { Agent, fetch as undiciFetch } from 'undici';
 import { clientTlsOptions, createCsr, generateKeyPair, identityUri, parseCredential, validateInstalledCertificate, writeFileDurable, } from "./pki.js";
 export const agentFile = (stateDir) => path.join(stateDir, 'agent.json');
 export const pendingFile = (stateDir) => path.join(stateDir, 'agent-pending.json');
-const NO_AGENT_IDENTITY = 'this node has no identity: run `tug node-install`, `tug node-prepare-enrollment` or `tug node-repair-identity` from an operator machine';
+const NO_AGENT_IDENTITY = 'this node has no identity: run `tug node add`, `tug node enroll` or `tug node repair` from an operator machine';
 async function readJson(file) {
     let text;
     try {
@@ -20,7 +20,7 @@ async function readJson(file) {
         return JSON.parse(text);
     }
     catch (e) {
-        throw new Error(`${file} is corrupt; run \`tug node-repair-identity\` to replace this node's identity`, {
+        throw new Error(`${file} is corrupt; run \`tug node repair\` to replace this node's identity`, {
             cause: e,
         });
     }
@@ -37,7 +37,7 @@ export async function loadAgentDocument(stateDir) {
         return { ...credential, nodeId, nodeName };
     }
     catch (e) {
-        throw new Error(`${agentFile(stateDir)} is invalid (${e instanceof Error ? e.message : String(e)}); run \`tug node-repair-identity\``, { cause: e });
+        throw new Error(`${agentFile(stateDir)} is invalid (${e instanceof Error ? e.message : String(e)}); run \`tug node repair\``, { cause: e });
     }
 }
 async function pendingKey(stateDir) {

package/dist/agent.d.ts

@@ -6,7 +6,7 @@ import type { ContainerSummary, DockerApi } from './docker-api.ts';
 import type { AgentLiveCommand, AgentLiveFrame } from './protocol.ts';
 import type { LiveEngineOptions } from './live.ts';
 import { parseDuration } from './util.ts';
-import type { HostRunner, HttpClient } from './runners.ts';
+import type { HttpClient } from './runners.ts';
 import type { Command, CommandResult, DesiredState, HealthCheck, ProjectStatus, ImageArtifact, PortBinding, ServiceEndpoint } from './protocol.ts';
 export interface AgentContext {
     docker: DockerApi;
@@ -17,10 +17,6 @@ export interface AgentContext {
     controlUrl: string;
     /** The node credential presented to control; absent only in tests with a fake `http`. */
     credential?: Credential;
-    /** Runs `wg` / `ip` for the mesh. */
-    host?: HostRunner;
-    /** Manage the WireGuard mesh (generate a key, report it, reconcile `wg0`). */
-    mesh?: boolean;
     caddyImage?: string;
     /** Host port range (`LOW-HIGH`) for rolling-rollout revisions; default 20000-20999. */
     rolloutPorts?: string;
@@ -96,8 +92,10 @@ export declare function resolveBindAddress(address: string | null, lookup?: (hos
     address: string;
 }>): Promise<string>;
 export declare function reconcile(desiredState: DesiredState, ctx?: AgentContext, endpoints?: Record<string, ServiceEndpoint[]>, ports?: PortBinding, held?: string[]): Promise<ProjectStatus>;
-/** Reports this node's WireGuard public key when control has not seen the current one yet. */
-export declare function reportWgKey(ctx: AgentContext): Promise<void>;
+/** Docker `extra_hosts` values must be IPs, so resolve hostname node addresses on this agent. */
+export declare function endpointAddressesForAgent(endpoints?: Record<string, ServiceEndpoint[]>, localServices?: Set<string>, lookup?: (host: string) => Promise<{
+    address: string;
+}>): Promise<Record<string, ServiceEndpoint[]>>;
 export declare function pollOnce(ctx?: AgentContext): Promise<ProjectStatus[]>;
 /** Loads the persisted identity, or enrolls with a join key. Without either the agent refuses to start. */
 export declare function loadOrEnroll(): Promise<AgentDocument>;

package/dist/agent.js

@@ -13,7 +13,7 @@ import { agentIdentityCsr, createTransport, describeMissingIdentity, installIden
 import { clientTlsOptions, parseCredential } from "./pki.js";
 import { createHash } from 'node:crypto';
 import { addressBoundServices, isRolling, serviceSecretVersions } from "./compose.js";
-import { LABEL_PROJECT, LABEL_SERVICE, containerSpecs, networkName, volumeName, } from "./container-spec.js";
+import { LABEL_PROJECT, LABEL_SERVICE, assertVolumeOwnership, containerSpecs, networkName, volumeName, } from "./container-spec.js";
 import { API_VERSION, DockerEngine, isNotFound } from "./docker-api.js";
 import { JobRunner } from "./agent-jobs.js";
 import { LIVE_MAX_FRAME_BYTES, LIVE_START_TIMEOUT_MS } from "./protocol.js";
@@ -21,17 +21,13 @@ import { LiveOperationEngine } from "./live.js";
 import { activeRollingStates, clearRolloutState, ensureProjectResources, loadRolloutState, projectContainers, reconcileRecreate, reconcileRolling, sweepOrphans, } from "./reconciler.js";
 import { ensureDir, errMessage, isPublicAddress, now, parseDuration } from "./util.js";
 import { openAgentDb } from "./db.js";
-import { createHostRunner } from "./runners.js";
 import { CADDY_CONTAINER, reconcileIngress, removeIngress } from "./ingress.js";
-import { WG_INTERFACE, ensureWgKeys, readReportedKey, reconcileMesh, rotateWgKeys, writeReportedKey, } from "./mesh.js";
 // Connect-only override (for example an SSH tunnel); the certificate is still verified against the credential's public URL.
 const CONTROL_URL_OVERRIDE = process.env.TUG_CONTROL_URL;
 let NODE_NAME = process.env.TUG_NODE_NAME || process.env.HOSTNAME || 'node1';
 const STATE_DIR = process.env.TUG_STATE_DIR || '/var/lib/tug';
 const APPLY = process.env.TUG_APPLY !== 'false';
 const INTERVAL = Number(process.env.TUG_POLL_INTERVAL_MS || 5000);
-// Opt-in WireGuard mesh: needs `wg`/`ip` in the agent image, host networking and NET_ADMIN.
-const MESH = process.env.TUG_MESH === 'true';
 let activeDocument;
 export function defaultContext(doc = activeDocument) {
     if (!doc)
@@ -45,8 +41,6 @@ export function defaultContext(doc = activeDocument) {
         apply: APPLY,
         controlUrl: transport.base,
         credential: doc,
-        host: createHostRunner(),
-        mesh: MESH,
         caddyImage: process.env.TUG_CADDY_IMAGE,
         rolloutPorts: process.env.TUG_ROLLOUT_PORTS,
     };
@@ -506,7 +500,7 @@ export function runsInContainer() {
 export function outsideContainerHint(health, inContainer = runsInContainer()) {
     if (inContainer || health.type === 'docker')
         return '';
-    return '; this agent is not running in a container, so it may be unable to reach container addresses (a host process on macOS cannot); run it with `docker run` as `tug node-install` does';
+    return '; this agent is not running in a container, so it may be unable to reach container addresses (a host process on macOS cannot); run it with `docker run` as `tug node add` does';
 }
 export async function waitForHealth(ctx, project, container, service, health) {
     if (!ctx.apply || !health)
@@ -591,7 +585,7 @@ async function publicPortsCheck(ctx) {
                 name: 'ports.public',
                 ok: true,
                 stdout: stdout.slice(-4000),
-                warning: `${listeners.length} published port${listeners.length === 1 ? ' of Tug projects listens' : 's of Tug projects listen'} on a public address; bind them to 127.0.0.1, the overlay IP or a private address unless that is intended`,
+                warning: `${listeners.length} published port${listeners.length === 1 ? ' of Tug projects listens' : 's of Tug projects listen'} on a public address; bind them to 127.0.0.1 or a private node address unless that is intended`,
             }
             : {
                 name: 'ports.public',
@@ -633,10 +627,6 @@ export async function executeCommand(command, ctx = defaultContext()) {
             }
         }
         const extra = [];
-        if (ctx.mesh && ctx.host) {
-            const host = ctx.host;
-            extra.push(['wireguard.version', () => host('wg', ['--version'])], ['wireguard.interface', () => host('wg', ['show', WG_INTERFACE])]);
-        }
         if (await fs.stat(path.join(ctx.stateDir, 'ingress')).then(() => true, () => false)) {
             extra.push([
                 'ingress.caddy',
@@ -700,28 +690,21 @@ export async function executeCommand(command, ctx = defaultContext()) {
     if (command.type === 'volume.delete') {
         const { project, volume } = payload;
         // Volumes are never removed by reconciliation; this explicit command is the only way one goes away.
-        await ctx.docker.removeVolume(volumeName(project, volume)).catch((e) => {
-            if (!isNotFound(e))
-                throw e;
-        });
+        const name = volumeName(project, volume);
+        const existing = await ctx.docker.inspectVolume(name);
+        if (existing)
+            assertVolumeOwnership(existing, project, volume);
+        if (existing)
+            await ctx.docker.removeVolume(name).catch((e) => {
+                if (!isNotFound(e))
+                    throw e;
+            });
         return {
             commandId: command.id,
             state: 'succeeded',
             result: { project, volume, deleted: true },
         };
     }
-    if (command.type === 'wg.rotate-key') {
-        if (!ctx.mesh)
-            throw new Error('the WireGuard mesh is not enabled on this agent (TUG_MESH=true)');
-        const { publicKey } = await rotateWgKeys(ctx.stateDir);
-        await api(ctx, '/agent/v1/wireguard/key', {
-            method: 'POST',
-            headers: { 'content-type': 'application/json' },
-            body: JSON.stringify({ publicKey }),
-        });
-        await writeReportedKey(ctx.stateDir, publicKey);
-        return { commandId: command.id, state: 'succeeded', result: { publicKey } };
-    }
     return {
         commandId: command.id,
         state: 'failed',
@@ -1051,7 +1034,7 @@ async function warnPublicPorts(ctx, desiredState, serviceNames, bind) {
         ? addressBoundServices(desiredState.compose, serviceNames, bind.loopback)
         : [];
     const message = exposed.length
-        ? `published ports of ${exposed.join(', ')} listen on public address ${bind.address}; enable the mesh, give the node a private address, or write the host IP in ports to make the exposure explicit`
+        ? `published ports of ${exposed.join(', ')} listen on public address ${bind.address}; give the node a private address, or write the host IP in ports to make the exposure explicit`
         : '';
     if ((publicPortWarnings.get(key) ?? '') === message)
         return;
@@ -1070,44 +1053,44 @@ async function warnPublicPorts(ctx, desiredState, serviceNames, bind) {
         }),
     }).catch(() => { });
 }
-/** Reports this node's WireGuard public key when control has not seen the current one yet. */
-export async function reportWgKey(ctx) {
-    const { publicKey } = await ensureWgKeys(ctx.stateDir);
-    if ((await readReportedKey(ctx.stateDir)) === publicKey)
-        return;
-    await api(ctx, '/agent/v1/wireguard/key', {
-        method: 'POST',
-        headers: { 'content-type': 'application/json' },
-        body: JSON.stringify({ publicKey }),
-    });
-    await writeReportedKey(ctx.stateDir, publicKey);
-}
-let lastNetworkError = '';
-async function networkStep(label, fn) {
-    try {
-        await fn();
-        lastNetworkError = '';
-    }
-    catch (e) {
-        const message = `${label}: ${errMessage(e)}`;
-        if (message !== lastNetworkError)
-            console.error(message);
-        lastNetworkError = message;
-    }
+/** Docker `extra_hosts` values must be IPs, so resolve hostname node addresses on this agent. */
+export async function endpointAddressesForAgent(endpoints = {}, localServices = new Set(), lookup = (host) => dns.lookup(host)) {
+    const cache = new Map();
+    const resolve = (address) => {
+        if (net.isIP(address))
+            return Promise.resolve(address);
+        let resolved = cache.get(address);
+        if (!resolved) {
+            resolved = lookup(address).then(({ address: ip }) => ip, (error) => {
+                throw new Error(`cannot resolve node address ${address} for cross-node service names: ${errMessage(error)}`, { cause: error });
+            });
+            cache.set(address, resolved);
+        }
+        return resolved;
+    };
+    return Object.fromEntries(await Promise.all(Object.entries(endpoints)
+        .filter(([service]) => !localServices.has(service))
+        .map(async ([service, list]) => [
+        service,
+        (await Promise.all(list.map(async (ep) => {
+            try {
+                return { ...ep, address: await resolve(ep.address) };
+            }
+            catch (error) {
+                console.error(errMessage(error));
+                return null;
+            }
+        }))).filter((ep) => ep !== null),
+    ])));
 }
 export async function pollOnce(ctx = defaultContext()) {
-    if (ctx.mesh)
-        await networkStep('wireguard key report failed', () => reportWgKey(ctx));
     const work = await api(ctx, '/agent/v1/work');
-    // The mesh comes up first so cross-node service endpoints are routable when containers start.
-    if (ctx.mesh && work.network) {
-        const network = work.network;
-        await networkStep('wireguard reconcile failed', () => reconcileMesh(ctx.host ?? createHostRunner(), ctx.stateDir, network));
-    }
     const projects = [];
     for (const desired of work.desiredStates || (work.desiredState ? [work.desiredState] : [])) {
         try {
-            projects.push(await reconcile(desired, ctx, work.serviceEndpoints?.[desired.project], work.ports, work.holds?.[desired.project]));
+            projects.push(await reconcile(desired, ctx, await endpointAddressesForAgent(work.serviceEndpoints?.[desired.project], new Set(Object.entries(desired.services)
+                .filter(([, spec]) => spec.placement?.nodes?.includes(ctx.nodeName))
+                .map(([name]) => name))), work.ports, work.holds?.[desired.project]));
         }
         catch (e) {
             projects.push({
@@ -1198,12 +1181,9 @@ async function enroll(joinKey) {
     const transport = createTransport(decoded.credential, CONTROL_URL_OVERRIDE || decoded.connectUrl);
     console.log(`enrolling with ${decoded.credential.url} as ${decoded.name}`);
     const csr = await agentIdentityCsr(STATE_DIR);
-    const wgPublicKey = MESH ? (await ensureWgKeys(STATE_DIR)).publicKey : undefined;
     const body = {
         csr,
         address: process.env.TUG_NODE_ADDRESS,
-        publicAddress: process.env.TUG_NODE_PUBLIC_ADDRESS,
-        wgPublicKey,
     };
     try {
         const res = await transport.http(`${transport.base}/agent/v1/enroll`, {
@@ -1214,8 +1194,6 @@ async function enroll(joinKey) {
         if (!res.ok)
             throw new Error(`enroll failed: ${res.status} ${await res.text()}`);
         const doc = await installIdentity(STATE_DIR, (await res.json()));
-        if (wgPublicKey)
-            await writeReportedKey(STATE_DIR, wgPublicKey);
         console.log(`enrolled as node ${doc.nodeName}`);
         return doc;
     }

package/dist/backup-check.d.ts

@@ -1,4 +1,4 @@
-/** The part of `GET /api/v1/backups` that `tug backup-list --check` judges. */
+/** The part of `GET /api/v1/backups` that `tug backup list --check` judges. */
 export interface BackupCheckInput {
     destinationConfigured: boolean;
     targets: {

package/dist/backup-config.js

@@ -184,18 +184,25 @@ export function resolveBackups(compose, project = '') {
             return null;
         return { schedule, retention, jitterSeconds, timeoutSec, memory };
     };
-    for (const [name, svc] of Object.entries(compose.services)) {
-        const config = svc['x-tug']?.backup;
-        if (config === undefined)
-            continue;
-        const label = `${name}: x-tug.backup`;
+    /** The object form of a backup config; null when it is off, or invalid (the error is reported). */
+    const specOf = (label, config) => {
         const spec = normalizeConfig(config);
         if (spec === null)
-            continue;
+            return null;
         if (typeof spec !== 'object' || Array.isArray(spec)) {
             errors.push(`${label} must be true, false, a cron string or an object`);
-            continue;
+            return null;
         }
+        return spec;
+    };
+    const serviceTarget = (name, svc) => {
+        const config = svc['x-tug']?.backup;
+        if (config === undefined)
+            return null;
+        const label = `${name}: x-tug.backup`;
+        const spec = specOf(label, config);
+        if (!spec)
+            return null;
         const before = errors.length;
         const base = common(label, 'service', name, spec);
         const argv = spec.command === undefined ? null : commandArgv(spec.command);
@@ -216,8 +223,8 @@ export function resolveBackups(compose, project = '') {
         if (global || replicas > 1 || nodes.length > 1)
             errors.push(`${label} needs a single instance; ${name} is global, replicated or placed on several nodes`);
         if (errors.length > before || !base || !argv)
-            continue;
-        targets.push({
+            return null;
+        return {
             target: `service:${name}`,
             kind: 'service',
             name,
@@ -229,20 +236,16 @@ export function resolveBackups(compose, project = '') {
             verify: spec.verify,
             writers: spec.writers,
             node: !dynamic && nodes.length === 1 ? nodes[0] : undefined,
-        });
-    }
-    for (const [name, def] of Object.entries(compose.volumes || {})) {
+        };
+    };
+    const volumeTarget = (name, def) => {
         const config = isObject(def) ? def['x-tug']?.backup : undefined;
         if (config === undefined)
-            continue;
+            return null;
         const label = `volumes.${name}: x-tug.backup`;
-        const spec = normalizeConfig(config);
-        if (spec === null)
-            continue;
-        if (typeof spec !== 'object' || Array.isArray(spec)) {
-            errors.push(`${label} must be true, false, a cron string or an object`);
-            continue;
-        }
+        const spec = specOf(label, config);
+        if (!spec)
+            return null;
         const before = errors.length;
         const base = common(label, 'volume', name, spec);
         if (spec.exclude !== undefined &&
@@ -260,8 +263,8 @@ export function resolveBackups(compose, project = '') {
         else if (!placed.length && dynamicPlacement)
             errors.push(`${label} needs x-tug.placement.nodes on volume ${name}, since its services are placed by label or global`);
         if (errors.length > before || !base)
-            continue;
-        targets.push({
+            return null;
+        return {
             target: `volume:${name}`,
             kind: 'volume',
             name,
@@ -270,7 +273,12 @@ export function resolveBackups(compose, project = '') {
             exclude: spec.exclude,
             writers: spec.writers,
             node: candidates.length === 1 ? candidates[0] : undefined,
-        });
-    }
+        };
+    };
+    const found = [
+        ...Object.entries(compose.services).map(([name, svc]) => serviceTarget(name, svc)),
+        ...Object.entries(compose.volumes || {}).map(([name, def]) => volumeTarget(name, def)),
+    ];
+    targets.push(...found.filter((target) => target !== null));
     return { targets, errors };
 }

package/dist/backup-destination.d.ts

@@ -15,7 +15,7 @@ export declare function projectRepository(destinationUrl: string, project: strin
 /**
  * The repository password of a project, derived from the control master key so nothing extra is stored and
  * a node only ever holds the passwords of projects it runs backups for. Losing the master key loses every
- * repository, which is why `tug backup-repository-password` exists for offline escrow.
+ * repository, which is why `tug backup password` exists for offline escrow.
  */
 export declare function repositoryPassword(masterKey: string, project: string): string;
 /** Removes every known secret value from text that is about to be stored, shown or put in an event. */

package/dist/backup-destination.js

@@ -54,7 +54,7 @@ export function projectRepository(destinationUrl, project) {
 /**
  * The repository password of a project, derived from the control master key so nothing extra is stored and
  * a node only ever holds the passwords of projects it runs backups for. Losing the master key loses every
- * repository, which is why `tug backup-repository-password` exists for offline escrow.
+ * repository, which is why `tug backup password` exists for offline escrow.
  */
 export function repositoryPassword(masterKey, project) {
     return createHmac('sha256', masterKey).update(`tug-restic:${project}`).digest('hex');

package/dist/backup-job.js

@@ -1,6 +1,6 @@
 import { Transform } from 'node:stream';
 import { redactSecrets } from "./backup-destination.js";
-import { LABEL_JOB, LABEL_PROJECT, LABEL_SERVICE } from "./container-spec.js";
+import { LABEL_JOB, LABEL_PROJECT, LABEL_SERVICE, assertVolumeOwnership, volumeName, } from "./container-spec.js";
 /** Docker volume holding restic's cache across jobs, so repeated runs do not re-read the repository index. */
 export const RESTIC_CACHE_VOLUME = 'tug_restic_cache';
 const TAIL_BYTES = 4096;
@@ -148,14 +148,26 @@ export async function runBackupJob(payload, commandId, ctx, kind = 'backup.run')
             if (scratch) {
                 if (!dockerVolume)
                     throw new JobError('a scratch restore needs a scratch volume name');
-                if (!(await docker.inspectVolume(dockerVolume)))
+                let existing = await docker.inspectVolume(dockerVolume);
+                if (!existing) {
                     await docker.createVolume(dockerVolume, {
                         labels: { 'tug.project': payload.project, 'tug.restore': payload.runId },
                     });
+                    existing = await docker.inspectVolume(dockerVolume);
+                }
+                if (existing?.Labels?.[LABEL_PROJECT] !== payload.project ||
+                    existing.Labels?.['tug.restore'] !== payload.runId)
+                    throw new JobError(`scratch volume ${dockerVolume} is not owned by this restore`);
+            }
+            else {
+                if (dockerVolume !== volumeName(payload.project, payload.volume.name))
+                    throw new JobError('volume name does not match the project and target');
+                const existing = await docker.inspectVolume(dockerVolume);
+                if (!existing)
+                    // Docker would silently create an empty volume for a missing name and the backup would succeed with nothing in it.
+                    throw new JobError(`volume ${dockerVolume} does not exist on this node; a volume is created when a service that uses it is placed here`);
+                assertVolumeOwnership(existing, payload.project, payload.volume.name);
             }
-            else if (!(await docker.inspectVolume(dockerVolume)))
-                // Docker would silently create an empty volume for a missing name and the backup would succeed with nothing in it.
-                throw new JobError(`volume ${dockerVolume} does not exist on this node; a volume is created when a service that uses it is placed here`);
             binds.push(`${dockerVolume}:/data${kind === 'backup.restore' ? '' : ':ro'}`);
         }
         const restic = await helper('restic', {