@wpelevator/tug 0.4.0 → 0.6.0

package/README.md

@@ -6,15 +6,15 @@ Deploy Docker Compose projects to one or more ordinary Linux VPSes through a sma
 
 ## Features
 
-- Control API with bearer-token auth, scoped `read`/`deploy`/`admin` API tokens for CI and teammates, an audit trail naming who acted, and SQLite metadata.
-- Nodes join over SSH (`node-install`) or enroll themselves with a single `docker run` (`node-enroll`), with per-node agent and registry tokens. Agents only dial out, so nodes behind NAT work.
-- Agents pull desired state from control, apply a generated per-node Compose file, run TCP/HTTP/Docker health checks and report status. An outbound WebSocket wakes them as soon as control queues work and carries live `exec`, followed logs, `ps` and `inspect` sessions, with long-poll and periodic-poll fallbacks for ordinary work.
+- Control API with bearer-token auth, scoped `read`/`deploy`/`admin` API tokens for CI and teammates, an audit trail naming who acted, and SQLite metadata through Node's built-in driver.
+- Nodes join over SSH (`node-install`) or enroll themselves with a single `docker run` (`node-enroll`), with one agent token per node. Agents only dial out, so nodes behind NAT work.
+- Agents pull desired state from control, reconcile containers through the Docker Engine API, run TCP/HTTP/Docker health checks and report status. An outbound WebSocket wakes them as soon as control queues work and carries live `exec`, followed logs, `ps` and `inspect` sessions, with long-poll and periodic-poll fallbacks for ordinary work.
 - `x-tug` placement of services and volumes on nodes, with immutable desired-state records per deployment.
-- Build application images on control from uploaded source, or locally with `deploy --build-local`. Agents download saved images directly from control by default; a bring-your-own registry is optional.
+- Build application images on control from uploaded source, or locally with `deploy --build-local`. Agents download saved images directly from control. Published images are pinned to registry digests at deployment creation.
 - Secrets encrypted at rest, released only to the nodes whose services reference them, with targeted service recreation when only a secret changes.
 - Deployment convergence, per-project operation locks, cancellation, rollback, per-service rollout events and log fetching through the agents.
 - Opt-in zero-downtime rolling rollouts for ingress-fronted services (`x-tug.rollout`) and automatic rollback of failed deployments (`x-tug.rollback: auto`). See [Compose extensions](docs/compose.md#rolling-rollouts).
-- Per-node registry pull tokens enforced by an htpasswd-backed registry you provide, with rotation and revocation.
+- Private published images use encrypted project credentials for digest resolution and Engine API pulls.
 - Optional WireGuard mesh between nodes, health-aware cross-node service names and a managed Caddy ingress with automatic HTTPS, each opt-in. See [Networking](docs/networking.md).
 - Private-by-default published ports: generated Compose binds them to the overlay IP, loopback or the node address, never every interface unless you write `0.0.0.0`. See [Networking](docs/networking.md#published-ports-private-by-default).
 - Control-plane backup and restore, and `tug doctor` diagnostics.
@@ -31,7 +31,6 @@ Deploy Docker Compose projects to one or more ordinary Linux VPSes through a sma
 |   Control API (role-checked)        |
 |   SQLite: desired states, secrets   |
 |   image builds, saved images        |
-|   registry (optional)               |
 +-------------------------------------+
        ^                       ^
        |  agents dial out: work, status,
@@ -62,25 +61,25 @@ npm install -g @wpelevator/tug
 The CLI installs the matching `ghcr.io/wpelevator/tug:<version>` image on your hosts.
 
 1. Install the control plane: `tug control-install root@control.example.com --install-docker`. The flag installs missing Docker Engine and Compose on supported Debian, Ubuntu, Fedora and RHEL-compatible hosts; omit it when Docker is ready. Put HTTPS in front of the control API as described in [Control plane](docs/control-plane.md).
-2. Point the CLI at it: `tug init --control-url https://control.example.com --token "$TUG_TOKEN"`.
+2. Point the CLI at it: `tug init --control-url https://control.example.com --token "$TUG_TOKEN"`. It writes `tug.yml` (commit it, and list your servers in it) and keeps the token in the git-ignored `tug.local.yml`; see [CLI reference](docs/cli.md#config-file).
 3. Set the secrets your services need: `tug secret-set DB_PASSWORD`.
 4. Add nodes with `tug node-install root@app1.example.com --name app1 --install-docker`, or `tug node-enroll --name app1` and run the printed Docker command. For a bare cloud host, `tug node-enroll --name app1 --install-docker` prints cloud-init user-data.
 5. Deploy: `tug deploy -f docker-compose.yml`. It waits until every node reports the new version healthy.
-6. Operate with `tug status`, `events`, `exec`, `logs`, `ps`, `inspect`, `restart`, `rollback` and `doctor`.
+6. Operate with `tug status`, `events`, `exec`, `run`, `logs`, `ps`, `inspect`, `restart`, `rollback`, `backup-run`, `restore` and `doctor`.
 
 ## Documentation
 
 - [Architecture](docs/architecture.md): components, reconciliation, workflow diagrams, secrets and runtime paths.
 - [Compose extensions](docs/compose.md): `x-tug` placement, health checks, volumes and limitations.
-- [Control plane](docs/control-plane.md): install, TLS, registry credentials, backup and restore.
-- [Backups](docs/backups.md): interim WordPress backup sidecar with restic, per-node and NFS variants, and a restore drill until `x-tug.backup` (M11).
+- [Control plane](docs/control-plane.md): install, TLS, image delivery, backup and restore.
+- [Backups](docs/backups.md): interim WordPress backup sidecar with restic, per-node variants, and a restore drill until `x-tug.backup` (M11).
 - [Access control](docs/access-control.md): scoped API tokens, roles, audit trail and a GitHub Actions deploy example.
 - [Networking](docs/networking.md): WireGuard mesh, service DNS, ingress and TLS, firewall notes.
 - [Nodes](docs/nodes.md): SSH install, self-service enrollment, registration and status.
 - [CLI reference](docs/cli.md): every `tug` command and option.
 - [Configuration](docs/configuration.md): CLI, control and agent environment variables.
 - [Development](docs/development.md): local setup, CI, Compose stack, host mode and the integration test.
-- [Testing](docs/testing.md): test layers, the numbers rollouts are held to, and the backlog of mocked tests.
+- [Testing](docs/testing.md): test layers, rollout metrics, mocked test backlog, and the [Caddy/PHP and static-file benchmark](docs/testing.md#caddy-proxy-overhead-benchmark).
 - [Observability](docs/observability.md): what can be observed today, and the proposed logs, metrics and alerting.
 - [Releases](docs/releases.md): changesets, npm and Docker image publishing.
 - [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution.

package/dist/agent-jobs.d.ts

@@ -0,0 +1,41 @@
+import type { DockerApi } from './docker-api.ts';
+import type { BackupCredentialsResponse, Command, CommandResult, JobStatus } from './protocol.ts';
+export interface JobHost {
+    docker: DockerApi;
+    stateDir: string;
+    fetchCredentials(commandId: string): Promise<BackupCredentialsResponse>;
+    /** Parallel jobs on this node; extra jobs wait queued. */
+    concurrency?: number;
+}
+/**
+ * The agent's background jobs. Reconciliation must not wait for an hour-long backup, so commands of
+ * these types are queued here, report a heartbeat through `statuses()`, and deliver their result
+ * through `drain()` on a later poll, even if the agent restarted in between.
+ */
+export declare class JobRunner {
+    private readonly host;
+    private readonly active;
+    private readonly queue;
+    private results;
+    private ready;
+    private settled;
+    constructor(host: JobHost);
+    /** Called whenever a job finishes, so the agent can report its result without waiting a full interval. */
+    onSettled(callback: () => void): void;
+    private file;
+    private save;
+    /** First use: remove helpers of a previous run and fail the jobs it never finished. */
+    private init;
+    /** Starts cleanup of a previous process's leftovers; safe to call more than once. */
+    warm(): Promise<void>;
+    /** Queues a job; the command is acknowledged immediately and its result arrives later. */
+    start(command: Command): Promise<void>;
+    private pump;
+    private execute;
+    /** Jobs that are running or waiting, for the status report's heartbeat. */
+    statuses(): JobStatus[];
+    /** Finished results to put in the next status report. They are removed only after `acknowledge`. */
+    pending(): CommandResult[];
+    /** Forgets results control has stored. */
+    acknowledge(delivered: CommandResult[]): Promise<void>;
+}

package/dist/agent-jobs.js

@@ -0,0 +1,161 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { runBackupJob, removeStaleJobContainers } from "./backup-job.js";
+import { redactSecrets } from "./backup-destination.js";
+import { ensureDir, errMessage, now } from "./util.js";
+const RESULTS_FILE = 'job-results.json';
+/**
+ * The agent's background jobs. Reconciliation must not wait for an hour-long backup, so commands of
+ * these types are queued here, report a heartbeat through `statuses()`, and deliver their result
+ * through `drain()` on a later poll, even if the agent restarted in between.
+ */
+export class JobRunner {
+    host;
+    active = new Map();
+    queue = [];
+    results = [];
+    ready;
+    settled;
+    constructor(host) {
+        this.host = host;
+    }
+    /** Called whenever a job finishes, so the agent can report its result without waiting a full interval. */
+    onSettled(callback) {
+        this.settled = callback;
+    }
+    file() {
+        return path.join(this.host.stateDir, RESULTS_FILE);
+    }
+    async save() {
+        const data = {
+            results: this.results,
+            running: [...this.active.keys(), ...this.queue.map((job) => job.command.id)],
+        };
+        await ensureDir(this.host.stateDir);
+        await fs.writeFile(this.file(), JSON.stringify(data), { mode: 0o600 });
+    }
+    /** First use: remove helpers of a previous run and fail the jobs it never finished. */
+    init() {
+        this.ready ??= (async () => {
+            await removeStaleJobContainers(this.host.docker).catch(() => 0);
+            try {
+                const saved = JSON.parse(await fs.readFile(this.file(), 'utf8'));
+                this.results = saved.results ?? [];
+                for (const commandId of saved.running ?? [])
+                    this.results.push({
+                        commandId,
+                        state: 'failed',
+                        result: { error: 'the agent restarted while the job was running' },
+                    });
+                if (saved.running?.length)
+                    await this.save();
+            }
+            catch {
+                /* no saved state */
+            }
+        })();
+        return this.ready;
+    }
+    /** Starts cleanup of a previous process's leftovers; safe to call more than once. */
+    warm() {
+        return this.init();
+    }
+    /** Queues a job; the command is acknowledged immediately and its result arrives later. */
+    async start(command) {
+        await this.init();
+        let payload;
+        try {
+            payload = JSON.parse(command.payload_json ?? '{}');
+            if (!payload.runId || !payload.repository || !payload.resticImage)
+                throw new Error('the job payload is incomplete');
+        }
+        catch (error) {
+            this.results.push({
+                commandId: command.id,
+                state: 'failed',
+                result: { error: errMessage(error) },
+            });
+            await this.save();
+            return;
+        }
+        this.queue.push({ command, payload });
+        await this.save();
+        this.pump();
+    }
+    pump() {
+        const limit = this.host.concurrency ?? 1;
+        while (this.active.size < limit && this.queue.length) {
+            const job = this.queue.shift();
+            void this.execute(job.command, job.payload);
+        }
+    }
+    async execute(command, payload) {
+        const controller = new AbortController();
+        const status = {
+            commandId: command.id,
+            runId: payload.runId,
+            phase: 'pulling',
+            startedAt: now(),
+            controller,
+        };
+        this.active.set(command.id, status);
+        const timeout = setTimeout(() => controller.abort(new Error(`the job exceeded its ${payload.limits.timeoutSec}s limit`)), payload.limits.timeoutSec * 1000);
+        timeout.unref();
+        let credentials;
+        try {
+            credentials = await this.host.fetchCredentials(command.id);
+            const result = await runBackupJob(payload, command.id, {
+                docker: this.host.docker,
+                credentials,
+                signal: controller.signal,
+                onPhase: (phase, bytes) => {
+                    status.phase = phase;
+                    if (bytes !== undefined)
+                        status.bytes = bytes;
+                },
+            }, command.type === 'backup.check'
+                ? 'backup.check'
+                : command.type === 'backup.restore'
+                    ? 'backup.restore'
+                    : 'backup.run');
+            this.results.push({ commandId: command.id, state: 'succeeded', result });
+        }
+        catch (error) {
+            const values = credentials ? Object.values(credentials.env) : [];
+            this.results.push({
+                commandId: command.id,
+                state: 'failed',
+                result: { error: redactSecrets(errMessage(controller.signal.reason ?? error), values) },
+            });
+        }
+        finally {
+            clearTimeout(timeout);
+            this.active.delete(command.id);
+            await this.save().catch(() => { });
+            this.settled?.();
+            this.pump();
+        }
+    }
+    /** Jobs that are running or waiting, for the status report's heartbeat. */
+    statuses() {
+        return [
+            ...[...this.active.values()].map(({ controller: _controller, ...status }) => status),
+            ...this.queue.map(({ command, payload }) => ({
+                commandId: command.id,
+                runId: payload.runId,
+                phase: 'queued',
+                startedAt: now(),
+            })),
+        ];
+    }
+    /** Finished results to put in the next status report. They are removed only after `acknowledge`. */
+    pending() {
+        return [...this.results];
+    }
+    /** Forgets results control has stored. */
+    async acknowledge(delivered) {
+        const ids = new Set(delivered.map((result) => result.commandId));
+        this.results = this.results.filter((result) => !ids.has(result.commandId));
+        await this.save().catch(() => { });
+    }
+}

package/dist/agent.d.ts

@@ -85,7 +85,7 @@ export declare function pruneManagedImages(ctx: AgentContext, retained: string[]
 export declare function resolveBindAddress(address: string | null, lookup?: (host: string) => Promise<{
     address: string;
 }>): Promise<string>;
-export declare function reconcile(desiredState: DesiredState, ctx?: AgentContext, endpoints?: Record<string, ServiceEndpoint[]>, ports?: PortBinding): Promise<ProjectStatus>;
+export declare function reconcile(desiredState: DesiredState, ctx?: AgentContext, endpoints?: Record<string, ServiceEndpoint[]>, ports?: PortBinding, held?: string[]): Promise<ProjectStatus>;
 /** Reports this node's WireGuard public key when control has not seen the current one yet. */
 export declare function reportWgKey(ctx: AgentContext): Promise<void>;
 export declare function pollOnce(ctx?: AgentContext): Promise<ProjectStatus[]>;
@@ -93,8 +93,6 @@ interface Creds {
     agentToken: string;
     nodeName?: string;
     controlUrl?: string;
-    registry?: string | null;
-    registryToken?: string | null;
 }
 export declare function loadOrEnroll(): Promise<Creds>;
 export declare function main(): Promise<void>;

package/dist/agent.js

@@ -1,18 +1,20 @@
 #!/usr/bin/env node
 import fs from 'node:fs/promises';
+import { imageLocation } from "./registry.js";
 import { createReadStream, createWriteStream } from 'node:fs';
 import { Readable } from 'node:stream';
 import { pipeline } from 'node:stream/promises';
 import { validateImageArchive, verifyDownload } from "./artifacts.js";
 import path from 'node:path';
-import { spawn, spawnSync } from 'node:child_process';
+import { spawnSync } from 'node:child_process';
 import net from 'node:net';
 import dns from 'node:dns/promises';
 import { WebSocket } from 'ws';
 import { createHash } from 'node:crypto';
 import { addressBoundServices, isRolling, serviceSecretVersions } from "./compose.js";
-import { containerSpecs, networkName, volumeName } from "./container-spec.js";
+import { LABEL_PROJECT, LABEL_SERVICE, containerSpecs, networkName, volumeName, } from "./container-spec.js";
 import { API_VERSION, DockerEngine, isNotFound } from "./docker-api.js";
+import { JobRunner } from "./agent-jobs.js";
 import { LIVE_MAX_FRAME_BYTES, LIVE_START_TIMEOUT_MS } from "./protocol.js";
 import { LiveOperationEngine } from "./live.js";
 import { activeRollingStates, clearRolloutState, ensureProjectResources, loadRolloutState, projectContainers, reconcileRecreate, reconcileRolling, sweepOrphans, } from "./reconciler.js";
@@ -45,6 +47,23 @@ export function defaultContext() {
         rolloutPorts: process.env.TUG_ROLLOUT_PORTS,
     };
 }
+const jobRunners = new Map();
+/** One job runner per state directory, so a background backup outlives the polls that started it. */
+function jobRunnerFor(ctx) {
+    let runner = jobRunners.get(ctx.stateDir);
+    if (!runner) {
+        runner = new JobRunner({
+            docker: ctx.docker,
+            stateDir: ctx.stateDir,
+            concurrency: Number(process.env.TUG_BACKUP_CONCURRENCY) || 1,
+            fetchCredentials: (commandId) => api(ctx, `/agent/v1/backup/credentials?command=${encodeURIComponent(commandId)}`),
+        });
+        jobRunners.set(ctx.stateDir, runner);
+        // Clear helper containers a previous agent process left behind, before any live session can start one.
+        void runner.warm();
+    }
+    return runner;
+}
 const dbsByStateDir = new Map();
 async function agentDb(stateDir) {
     let db = dbsByStateDir.get(stateDir);
@@ -425,6 +444,7 @@ export class AgentLiveOperations {
                     container,
                     command: request.command,
                     interactive: request.interactive,
+                    oneOff: request.oneOff,
                     tty: request.tty,
                     columns: request.columns,
                     rows: request.rows,
@@ -801,7 +821,15 @@ export async function resolveBindAddress(address, lookup = (host) => dns.lookup(
 }
 // Last public-port warning per project, so the event is sent when it changes rather than on every poll.
 const publicPortWarnings = new Map();
-export async function reconcile(desiredState, ctx = defaultContext(), endpoints = {}, ports = { address: null, loopback: {} }) {
+/** Stops every running container of a service, including the revisions of a rolling service. */
+async function stopService(docker, project, service) {
+    const running = await docker.listContainers({
+        label: [`${LABEL_PROJECT}=${project}`, `${LABEL_SERVICE}=${service}`],
+    });
+    for (const container of running)
+        await docker.stopContainer(container.Id);
+}
+export async function reconcile(desiredState, ctx = defaultContext(), endpoints = {}, ports = { address: null, loopback: {} }, held = []) {
     const dir = path.join(ctx.stateDir, 'projects', desiredState.project);
     await ensureDir(dir);
     const serviceNames = Object.keys(containerSpecs(desiredState, ctx.nodeName).specs);
@@ -837,10 +865,14 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
     const sec = await api(ctx, `/agent/v1/secrets?project=${encodeURIComponent(desiredState.project)}&services=${encodeURIComponent(serviceNames.join(','))}`);
     const currentSecretVersions = serviceSecretVersions(desiredState.compose, serviceNames, sec.versions || {});
     const db = await agentDb(ctx.stateDir);
-    const previousRow = await db.get('SELECT version, revision FROM applied_state WHERE project = ?', desiredState.project);
-    const rolling = serviceNames.filter((service) => isRolling(desiredState.compose.services[service]));
+    const previousRow = db.get('SELECT version, revision FROM applied_state WHERE project = ?', desiredState.project);
+    // A held service is stopped and left alone (a restore is rewriting its data) until control drops the hold.
+    const heldSet = new Set(held.filter((service) => serviceNames.includes(service)));
+    for (const service of heldSet)
+        serviceStates[service].rolloutState = 'held';
+    const rolling = serviceNames.filter((service) => !heldSet.has(service) && isRolling(desiredState.compose.services[service]));
     const rollingSet = new Set(rolling);
-    const mainServices = serviceNames.filter((service) => !rollingSet.has(service));
+    const mainServices = serviceNames.filter((service) => !rollingSet.has(service) && !heldSet.has(service));
     // Published images are pulled only for a new desired revision. Pulling on every poll would let a tag that moved
     // upstream (`mariadb:11`) recreate the container without a deploy.
     const sameDesiredVersion = previousRow
@@ -882,7 +914,7 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
         .digest('hex')
         .slice(0, 16);
     const knownEvents = mainServices.length
-        ? await db.get('SELECT fingerprint, phase FROM event_state WHERE project = ?', desiredState.project)
+        ? db.get('SELECT fingerprint, phase FROM event_state WHERE project = ?', desiredState.project)
         : undefined;
     const announce = knownEvents?.fingerprint !== eventFingerprint;
     const wasFailed = !announce && knownEvents?.phase === 'failed';
@@ -894,7 +926,7 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
             if (artifact)
                 await ensureArtifact(ctx, artifact);
         },
-        pullImage: (service) => ctx.docker.pullImage(plan.specs[service].body.Image),
+        pullImage: (service) => ctx.docker.pullImage(plan.specs[service].body.Image, sec.registryAuth?.[imageLocation(plan.specs[service].body.Image).host]),
         emit,
     };
     let failing;
@@ -902,6 +934,8 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
         try {
             await ensureProjectResources(ctx.docker, desiredState.project, plan);
             await sweepOrphans(ctx.docker, desiredState.project, new Set(serviceNames));
+            for (const service of heldSet)
+                await stopService(ctx.docker, desiredState.project, service);
             if (mainServices.length)
                 await reconcileRecreate(ctx.docker, {
                     desired: desiredState,
@@ -938,13 +972,13 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
                     },
                 });
             if (mainServices.length && (announce || wasFailed))
-                await markEvents('serving');
+                markEvents('serving');
         }
         catch (e) {
             if (mainServices.length && !wasFailed) {
                 // Announce a failure once; retries of the same failure stay quiet until it recovers or the inputs change.
                 await emit(failing ?? mainServices.join(', '), 'failed', errMessage(e), 'error');
-                await markEvents('failed');
+                markEvents('failed');
             }
             // Rolling services are untouched by a failure here, so keep reporting their current revision as serving.
             if (!rolling.length)
@@ -974,7 +1008,7 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
         for (const service of serviceNames)
             serviceStates[service].rolloutState = 'serving';
     }
-    await db.run(`INSERT INTO applied_state(project, version, revision, applied_at, service_states_json, secret_versions_json)
+    db.run(`INSERT INTO applied_state(project, version, revision, applied_at, service_states_json, secret_versions_json)
      VALUES(?,?,?,?,?,?)
      ON CONFLICT(project) DO UPDATE SET version=excluded.version, revision=excluded.revision, applied_at=excluded.applied_at,
        service_states_json=excluded.service_states_json, secret_versions_json=excluded.secret_versions_json`, desiredState.project, desiredState.version, desiredState.revision || null, now(), JSON.stringify(serviceStates), JSON.stringify(currentSecretVersions));
@@ -1050,7 +1084,7 @@ export async function pollOnce(ctx = defaultContext()) {
     const projects = [];
     for (const desired of work.desiredStates || (work.desiredState ? [work.desiredState] : [])) {
         try {
-            projects.push(await reconcile(desired, ctx, work.serviceEndpoints?.[desired.project], work.ports));
+            projects.push(await reconcile(desired, ctx, work.serviceEndpoints?.[desired.project], work.ports, work.holds?.[desired.project]));
         }
         catch (e) {
             projects.push({
@@ -1071,7 +1105,13 @@ export async function pollOnce(ctx = defaultContext()) {
             await removeIngress(ingressCtx).catch(() => false);
     }
     const commandResults = [];
+    const jobs = jobRunnerFor(ctx);
     for (const command of work.commands || []) {
+        // Backups can run for an hour, so they run in the background and report through later polls.
+        if (command.type.startsWith('backup.')) {
+            await jobs.start(command);
+            continue;
+        }
         try {
             commandResults.push(await executeCommand(command, ctx));
         }
@@ -1083,7 +1123,12 @@ export async function pollOnce(ctx = defaultContext()) {
             });
         }
     }
+    const jobResults = jobs.pending();
+    commandResults.push(...jobResults);
     const report = { node: ctx.nodeName, projects, commandResults };
+    const running = jobs.statuses();
+    if (running.length)
+        report.jobs = running;
     if (ingress)
         report.ingress = ingress;
     if (work.retainedImages && projects.every((project) => project.health !== 'failed'))
@@ -1093,6 +1138,7 @@ export async function pollOnce(ctx = defaultContext()) {
         headers: { 'content-type': 'application/json' },
         body: JSON.stringify(report),
     });
+    await jobs.acknowledge(jobResults);
     return projects;
 }
 const CREDS_FILE = () => path.join(STATE_DIR, 'agent.json');
@@ -1122,19 +1168,6 @@ async function readJoinKey() {
     }
     return null;
 }
-function dockerLogin(registry, user, password) {
-    return new Promise((resolve, reject) => {
-        const p = spawn('docker', ['login', registry, '-u', user, '--password-stdin'], {
-            stdio: ['pipe', 'ignore', 'pipe'],
-        });
-        let err = '';
-        p.stderr.on('data', (d) => (err += d));
-        p.on('error', reject);
-        p.on('close', (code) => code === 0 ? resolve() : reject(new Error(err.trim() || `exit ${code}`)));
-        p.stdin.write(password);
-        p.stdin.end();
-    });
-}
 async function enroll(joinKey) {
     let decoded;
     try {
@@ -1184,8 +1217,6 @@ async function enroll(joinKey) {
         agentToken: out.agentToken,
         nodeName: out.node?.name || NODE_NAME,
         controlUrl,
-        registry: out.registry || null,
-        registryToken: out.registryToken || null,
     };
     await saveCreds(creds);
     if (wgPublicKey)
@@ -1209,14 +1240,6 @@ export async function loadOrEnroll() {
     NODE_NAME = creds.nodeName || NODE_NAME;
     if (!CONTROL_URL_FROM_ENV && creds.controlUrl)
         CONTROL_URL = creds.controlUrl;
-    if (APPLY && creds.registry && creds.registryToken) {
-        try {
-            await dockerLogin(creds.registry, NODE_NAME, creds.registryToken);
-        }
-        catch (e) {
-            console.error(`registry login failed (pulls from ${creds.registry} may fail): ${errMessage(e)}`);
-        }
-    }
     return creds;
 }
 export async function main() {

package/dist/backup-config.d.ts

@@ -0,0 +1,47 @@
+import type { BackupRetention, BackupType, ComposeFile } from './protocol.ts';
+/** One backup target after defaults and placement are resolved. */
+export interface ResolvedBackup {
+    /** `service:db` or `volume:uploads`, unique within a project. */
+    target: string;
+    kind: 'service' | 'volume';
+    name: string;
+    /** `command` for a service, `volume` for a volume. */
+    type: BackupType;
+    schedule: string;
+    retention: BackupRetention;
+    jitterSeconds: number;
+    timeoutSec: number;
+    /** Service targets: the command run in the service's container, and the snapshot file it is stored as. */
+    command?: string[];
+    /** Run in the service container to restore; the stored file arrives on its standard input. */
+    restore?: string[];
+    filename?: string;
+    verify?: string;
+    /** Volume targets. */
+    exclude?: string[];
+    /** Services held during a restore; unset means "discover from the Compose file". */
+    writers?: string[];
+    /** The one node the backup runs on, when it is known from literal placement. Label placement resolves at deploy time. */
+    node?: string;
+}
+export declare const DEFAULT_RETENTION: BackupRetention;
+export declare const DEFAULT_BACKUP_TIMEOUT_SEC: number;
+/** `30m`, `2h`, `90s` or a number of seconds. */
+export declare function parseBackupTimeout(value: unknown): number | null;
+/** Deterministic delay in `[0, 900)` seconds so many sites on one node do not start together. */
+export declare function defaultJitterSeconds(project: string, target: string): number;
+/** Validates the project-level defaults in `x-tug.backup`. */
+export declare function validateBackupDefaults(value: unknown, errors: string[]): void;
+/** The argv of a `command`: a list is used as is, a string runs through `sh -c` like a shell one-liner. */
+export declare function commandArgv(command: unknown): string[] | null;
+interface Resolved {
+    targets: ResolvedBackup[];
+    errors: string[];
+}
+/**
+ * Resolves and validates every `x-tug.backup` in a Compose file. Tug knows nothing about what a service
+ * is: a service backup is a command whose output is stored, a volume backup is its files. `project`
+ * seeds the default jitter. Errors name the key, so they can be shown to the operator as they are.
+ */
+export declare function resolveBackups(compose: ComposeFile, project?: string): Resolved;
+export {};