@wpelevator/tug 0.6.0 → 0.8.0

package/README.md

@@ -2,12 +2,12 @@
 
 The name: tugboats pull ships into place, like the pull-based agents here; `tug` is the CLI.
 
-Deploy Docker Compose projects to one or more ordinary Linux VPSes through a small self-hosted control plane and a pull-based agent on each node, without Kubernetes or a hosted PaaS. What it is for, the deployment shapes it targets and the planned milestones are in the [Roadmap](docs/roadmap.md).
+Deploy Docker Compose applications to one or more ordinary Linux VPSes through a small self-hosted control plane and a pull-based agent on each node, without Kubernetes or a hosted PaaS. Tug runs an agent on each node; the agents drive the node's Docker Engine API directly, and Compose is only used on your machine to read the file. What it is for, the deployment shapes it targets and the planned milestones are in the [Roadmap](docs/roadmap.md).
 
 ## Features
 
-- Control API with bearer-token auth, scoped `read`/`deploy`/`admin` API tokens for CI and teammates, an audit trail naming who acted, and SQLite metadata through Node's built-in driver.
-- Nodes join over SSH (`node-install`) or enroll themselves with a single `docker run` (`node-enroll`), with one agent token per node. Agents only dial out, so nodes behind NAT work.
+- One mutual-TLS port: every operator, CI job and node authenticates with a certificate from the control plane's private CA, and control authorizes it by public key. Scoped `read`/`deploy`/`admin` operator credentials for CI and teammates, an audit trail naming who acted, and SQLite metadata through Node's built-in driver.
+- Nodes join over SSH (`node-install`) or enroll themselves with a single `docker run` after `node-prepare-enrollment`, each with its own certificate and an immutable node ID. Agents only dial out, so nodes behind NAT work.
 - Agents pull desired state from control, reconcile containers through the Docker Engine API, run TCP/HTTP/Docker health checks and report status. An outbound WebSocket wakes them as soon as control queues work and carries live `exec`, followed logs, `ps` and `inspect` sessions, with long-poll and periodic-poll fallbacks for ordinary work.
 - `x-tug` placement of services and volumes on nodes, with immutable desired-state records per deployment.
 - Build application images on control from uploaded source, or locally with `deploy --build-local`. Agents download saved images directly from control. Published images are pinned to registry digests at deployment creation.
@@ -15,22 +15,22 @@ Deploy Docker Compose projects to one or more ordinary Linux VPSes through a sma
 - Deployment convergence, per-project operation locks, cancellation, rollback, per-service rollout events and log fetching through the agents.
 - Opt-in zero-downtime rolling rollouts for ingress-fronted services (`x-tug.rollout`) and automatic rollback of failed deployments (`x-tug.rollback: auto`). See [Compose extensions](docs/compose.md#rolling-rollouts).
 - Private published images use encrypted project credentials for digest resolution and Engine API pulls.
-- Optional WireGuard mesh between nodes, health-aware cross-node service names and a managed Caddy ingress with automatic HTTPS, each opt-in. See [Networking](docs/networking.md).
+- Optional WireGuard mesh between nodes, health-aware cross-node service names (hosts-file entries injected into containers) and a managed Caddy ingress with automatic HTTPS, each opt-in. See [Networking](docs/networking.md).
 - Private-by-default published ports: generated Compose binds them to the overlay IP, loopback or the node address, never every interface unless you write `0.0.0.0`. See [Networking](docs/networking.md#published-ports-private-by-default).
-- Control-plane backup and restore, and `tug doctor` diagnostics.
+- Workload backups of service commands and volumes with restic, a separate control-plane backup and restore, and `tug doctor` diagnostics for certificate expiry, node connectivity and clock skew.
 
 ## How it fits together
 
 ```text
-  tug CLI / CI (API token)
+  tug CLI / CI (operator credential)
        |
-       |  HTTPS: deploy, status, logs, exec
+       |  mTLS: deploy, status, logs, exec
        v
 +-------------------------------------+
-| Control plane                       |
+| Control plane (one mTLS port)       |
 |   Control API (role-checked)        |
 |   SQLite: desired states, secrets   |
-|   image builds, saved images        |
+|   private CA, image builds          |
 +-------------------------------------+
        ^                       ^
        |  agents dial out: work, status,
@@ -39,7 +39,7 @@ Deploy Docker Compose projects to one or more ordinary Linux VPSes through a sma
 +------+------------+   +------+------------------+
 | Node app1         |   | Node app2 (ingress node)|
 |   Agent           |<->|   Agent                 |
-|   Docker Compose  |   |   Docker Compose        |
+|   Docker Engine   |   |   Docker Engine         |
 |   services        |   |   services              |
 |                   |   |   managed Caddy ingress |
 +-------------------+   +-------------------------+
@@ -48,7 +48,7 @@ Deploy Docker Compose projects to one or more ordinary Linux VPSes through a sma
   SSH is used only by the CLI to install the control plane and nodes.
 ```
 
-Operators and CI talk to the control API. The control plane stores desired state and builds images; each agent dials out to fetch the work for its node, applies it with Docker Compose and reports health back, so nodes behind NAT need no inbound ports. SSH is used only to install control and agents. The optional WireGuard mesh and Caddy ingress are described in [Networking](docs/networking.md), and every flow in detail in [Architecture](docs/architecture.md).
+Operators and CI talk to the control API. The control plane stores desired state and builds images; each agent dials out to fetch the work for its node, applies it through the Docker Engine API and reports health back, so nodes behind NAT need no inbound ports. SSH is used only to install the control plane and its nodes. The optional WireGuard mesh and Caddy ingress are described in [Networking](docs/networking.md), and every flow in detail in [Architecture](docs/architecture.md).
 
 ## Quick start
 
@@ -60,29 +60,31 @@ npm install -g @wpelevator/tug
 
 The CLI installs the matching `ghcr.io/wpelevator/tug:<version>` image on your hosts.
 
-1. Install the control plane: `tug control-install root@control.example.com --install-docker`. The flag installs missing Docker Engine and Compose on supported Debian, Ubuntu, Fedora and RHEL-compatible hosts; omit it when Docker is ready. Put HTTPS in front of the control API as described in [Control plane](docs/control-plane.md).
-2. Point the CLI at it: `tug init --control-url https://control.example.com --token "$TUG_TOKEN"`. It writes `tug.yml` (commit it, and list your servers in it) and keeps the token in the git-ignored `tug.local.yml`; see [CLI reference](docs/cli.md#config-file).
+1. Install the control plane: `tug control-install root@control.example.com --public-url https://control.example.com:8443 --install-docker`. The flag installs missing Docker Engine and the Compose plugin that runs the control container on supported Debian, Ubuntu, Fedora and RHEL-compatible hosts; omit it when Docker is ready. Control serves mutual TLS directly on its published port (`8443`); do not terminate TLS in front of it ([Control plane](docs/control-plane.md#authentication-and-pki)).
+2. Point the CLI at it: `tug init tug-root.credential.json` (the file `control-install` wrote). It writes `tug.yml` (commit it, and list your servers in it), copies the credential to a `0600` file in `~/.config/tug/credentials/` and points the git-ignored `tug.local.yml` at it; see [CLI reference](docs/cli.md#config-file).
 3. Set the secrets your services need: `tug secret-set DB_PASSWORD`.
-4. Add nodes with `tug node-install root@app1.example.com --name app1 --install-docker`, or `tug node-enroll --name app1` and run the printed Docker command. For a bare cloud host, `tug node-enroll --name app1 --install-docker` prints cloud-init user-data.
-5. Deploy: `tug deploy -f docker-compose.yml`. It waits until every node reports the new version healthy.
-6. Operate with `tug status`, `events`, `exec`, `run`, `logs`, `ps`, `inspect`, `restart`, `rollback`, `backup-run`, `restore` and `doctor`.
+4. Add nodes with `tug node-install root@app1.example.com --name app1 --install-docker`, or run `tug node-prepare-enrollment --name app1` and then the printed Docker command on the host. Nodes need Docker Engine only. For a bare cloud host, `tug node-prepare-enrollment --name app1 --install-docker` prints cloud-init user-data.
+5. Deploy: `tug deploy -f docker-compose.yml`. It waits until every node reports the new desired-state revision healthy.
+6. Operate with `tug status`, `events`, `exec`, `run`, `logs`, `ps`, `inspect`, `restart`, `rollback`, `backup-run`, `restore` and `doctor`. Create more operator credentials for CI and teammates with `tug credential-create`.
 
 ## Documentation
 
+- [Terminology](docs/terminology.md): core concepts, the naming review (which recommendations are implemented), industry comparisons and the remaining proposed interface improvements.
 - [Architecture](docs/architecture.md): components, reconciliation, workflow diagrams, secrets and runtime paths.
 - [Compose extensions](docs/compose.md): `x-tug` placement, health checks, volumes and limitations.
-- [Control plane](docs/control-plane.md): install, TLS, image delivery, backup and restore.
-- [Backups](docs/backups.md): interim WordPress backup sidecar with restic, per-node variants, and a restore drill until `x-tug.backup` (M11).
-- [Access control](docs/access-control.md): scoped API tokens, roles, audit trail and a GitHub Actions deploy example.
-- [Networking](docs/networking.md): WireGuard mesh, service DNS, ingress and TLS, firewall notes.
-- [Nodes](docs/nodes.md): SSH install, self-service enrollment, registration and status.
+- [Control plane](docs/control-plane.md): install, mutual TLS and the private CA, image delivery, control-plane backup and restore.
+- [Backups](docs/backups.md): scheduled workload backups with `x-tug.backup` (restic), restore, recipes for common databases, and a restore drill. See also the [WordPress example](examples/wordpress/README.md).
+- [Access control](docs/access-control.md): certificate identity classes, scoped operator credentials, roles, audit trail and a GitHub Actions deploy example.
+- [Security](docs/security.md): deployment trust boundaries, security audit findings and validation limits.
+- [Networking](docs/networking.md): WireGuard mesh, cross-node service names, ingress and TLS, firewall notes.
+- [Nodes](docs/nodes.md): node names and IDs, SSH install, self-service enrollment, certificate renewal and recovery, registration and status.
 - [CLI reference](docs/cli.md): every `tug` command and option.
 - [Configuration](docs/configuration.md): CLI, control and agent environment variables.
 - [Development](docs/development.md): local setup, CI, Compose stack, host mode and the integration test.
 - [Testing](docs/testing.md): test layers, rollout metrics, mocked test backlog, and the [Caddy/PHP and static-file benchmark](docs/testing.md#caddy-proxy-overhead-benchmark).
 - [Observability](docs/observability.md): what can be observed today, and the proposed logs, metrics and alerting.
 - [Releases](docs/releases.md): changesets, npm and Docker image publishing.
-- [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution.
+- [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution, with the mTLS dependency findings.
 - [Roadmap](docs/roadmap.md): goals, principles, the reference WordPress topology and planned milestones.
 
 ## Development

package/dist/agent-credential.d.ts

@@ -0,0 +1,39 @@
+import { Agent } from 'undici';
+import type { Credential, PersistHooks } from './pki.ts';
+import type { IdentityResponse } from './protocol.ts';
+/** The agent's persisted identity: the common credential plus the node it names. */
+export interface AgentDocument extends Credential {
+    nodeId: string;
+    nodeName: string;
+}
+export declare const agentFile: (stateDir: string) => string;
+export declare const pendingFile: (stateDir: string) => string;
+export declare function loadAgentDocument(stateDir: string): Promise<AgentDocument | null>;
+/** `agent identity`: reuse or generate the pending key and return its CSR. Safe to repeat. */
+export declare function agentIdentityCsr(stateDir: string): Promise<string>;
+/** Validates an identity response against the pending key and durably installs it as the agent document. */
+export declare function installIdentity(stateDir: string, response: IdentityResponse, hooks?: PersistHooks): Promise<AgentDocument>;
+/** `agent install-cert`: reads an identity response from stdin text. */
+export declare function installCertFromText(stateDir: string, text: string): Promise<AgentDocument>;
+export declare function describeMissingIdentity(): string;
+export type HttpClient = (url: string, init?: RequestInit) => Promise<Response>;
+/**
+ * A client pinned to the credential's CA that presents its certificate. `connectUrl` only changes where the TCP
+ * connection goes; the server name is still verified against the credential's public host. Redirects are refused.
+ */
+export declare function createTransport(credential: Credential, connectUrl?: string): {
+    http: HttpClient;
+    base: string;
+    dispatcher: Agent;
+    close: () => Promise<void>;
+};
+export declare function certificateWindow(certPem: string): {
+    notBefore: number;
+    notAfter: number;
+};
+/** When to renew: two thirds of the lifetime plus up to ten percent random jitter, so a fleet does not sign together. */
+export declare function renewalTime(certPem: string, random?: number): number;
+/** Sends a CSR for the existing key, validates the result, saves it durably and returns the new document. */
+export declare function renewCertificate(stateDir: string, doc: AgentDocument, post: (csr: string) => Promise<{
+    certificate: string;
+}>, hooks?: PersistHooks): Promise<AgentDocument>;

package/dist/agent-credential.js

@@ -0,0 +1,135 @@
+import crypto from 'node:crypto';
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { Agent, fetch as undiciFetch } from 'undici';
+import { clientTlsOptions, createCsr, generateKeyPair, identityUri, parseCredential, validateInstalledCertificate, writeFileDurable, } from "./pki.js";
+export const agentFile = (stateDir) => path.join(stateDir, 'agent.json');
+export const pendingFile = (stateDir) => path.join(stateDir, 'agent-pending.json');
+const NO_AGENT_IDENTITY = 'this node has no identity: run `tug node-install`, `tug node-prepare-enrollment` or `tug node-repair-identity` from an operator machine';
+async function readJson(file) {
+    let text;
+    try {
+        text = await fs.readFile(file, 'utf8');
+    }
+    catch (e) {
+        if (e.code === 'ENOENT')
+            return null;
+        throw e;
+    }
+    try {
+        return JSON.parse(text);
+    }
+    catch (e) {
+        throw new Error(`${file} is corrupt; run \`tug node-repair-identity\` to replace this node's identity`, {
+            cause: e,
+        });
+    }
+}
+export async function loadAgentDocument(stateDir) {
+    const raw = await readJson(agentFile(stateDir));
+    if (raw === null)
+        return null;
+    try {
+        const credential = parseCredential(raw);
+        const { nodeId, nodeName } = raw;
+        if (typeof nodeId !== 'string' || typeof nodeName !== 'string')
+            throw new Error('missing node identity');
+        return { ...credential, nodeId, nodeName };
+    }
+    catch (e) {
+        throw new Error(`${agentFile(stateDir)} is invalid (${e instanceof Error ? e.message : String(e)}); run \`tug node-repair-identity\``, { cause: e });
+    }
+}
+async function pendingKey(stateDir) {
+    const existing = (await readJson(pendingFile(stateDir)));
+    if (existing?.key)
+        return existing.key;
+    const keys = await generateKeyPair();
+    await writeFileDurable(pendingFile(stateDir), JSON.stringify({ v: 1, key: keys.privateKeyPem }));
+    return keys.privateKeyPem;
+}
+/** `agent identity`: reuse or generate the pending key and return its CSR. Safe to repeat. */
+export async function agentIdentityCsr(stateDir) {
+    return createCsr(await pendingKey(stateDir));
+}
+/** Validates an identity response against the pending key and durably installs it as the agent document. */
+export async function installIdentity(stateDir, response, hooks = {}) {
+    const pending = (await readJson(pendingFile(stateDir)));
+    if (!pending?.key)
+        throw new Error('no pending key: run `agent identity` first');
+    validateInstalledCertificate({
+        cert: response.certificate,
+        ca: response.ca,
+        keyPem: pending.key,
+        uri: identityUri('node', response.node.id),
+    });
+    const doc = {
+        ...parseCredential({
+            v: 1,
+            url: response.url,
+            ca: response.ca,
+            cert: response.certificate,
+            key: pending.key,
+        }),
+        nodeId: response.node.id,
+        nodeName: response.node.name,
+    };
+    await writeFileDurable(agentFile(stateDir), JSON.stringify(doc), 0o600, hooks);
+    await fs.rm(pendingFile(stateDir), { force: true });
+    return doc;
+}
+/** `agent install-cert`: reads an identity response from stdin text. */
+export async function installCertFromText(stateDir, text) {
+    return installIdentity(stateDir, JSON.parse(text));
+}
+export function describeMissingIdentity() {
+    return NO_AGENT_IDENTITY;
+}
+/**
+ * A client pinned to the credential's CA that presents its certificate. `connectUrl` only changes where the TCP
+ * connection goes; the server name is still verified against the credential's public host. Redirects are refused.
+ */
+export function createTransport(credential, connectUrl) {
+    const dispatcher = new Agent({ connect: clientTlsOptions(credential) });
+    const base = new URL(connectUrl ?? credential.url).origin;
+    const http = (url, init) => {
+        let request = init;
+        // The runtime's FormData is not the undici package's FormData, which would stringify it. Serialize it with the
+        // runtime's own Response and stream the result.
+        if (typeof FormData !== 'undefined' && init?.body instanceof FormData) {
+            const encoded = new Response(init.body);
+            const headers = new Headers(init.headers);
+            headers.set('content-type', encoded.headers.get('content-type') ?? 'multipart/form-data');
+            request = { ...init, headers, body: encoded.body, duplex: 'half' };
+        }
+        return undiciFetch(url, {
+            ...request,
+            redirect: 'error',
+            dispatcher,
+        });
+    };
+    return { http, base, dispatcher, close: () => dispatcher.close() };
+}
+export function certificateWindow(certPem) {
+    const x = new crypto.X509Certificate(certPem);
+    return { notBefore: Date.parse(x.validFrom), notAfter: Date.parse(x.validTo) };
+}
+/** When to renew: two thirds of the lifetime plus up to ten percent random jitter, so a fleet does not sign together. */
+export function renewalTime(certPem, random = Math.random()) {
+    const { notBefore, notAfter } = certificateWindow(certPem);
+    const life = notAfter - notBefore;
+    return notBefore + (life * 2) / 3 + random * life * 0.1;
+}
+/** Sends a CSR for the existing key, validates the result, saves it durably and returns the new document. */
+export async function renewCertificate(stateDir, doc, post, hooks = {}) {
+    const { certificate } = await post(await createCsr(doc.key));
+    validateInstalledCertificate({
+        cert: certificate,
+        ca: doc.ca,
+        keyPem: doc.key,
+        uri: identityUri('node', doc.nodeId),
+    });
+    const next = { ...doc, cert: certificate };
+    await writeFileDurable(agentFile(stateDir), JSON.stringify(next), 0o600, hooks);
+    return next;
+}

package/dist/agent.d.ts

@@ -1,5 +1,7 @@
 #!/usr/bin/env node
 import { WebSocket } from 'ws';
+import type { AgentDocument } from './agent-credential.ts';
+import type { Credential } from './pki.ts';
 import type { ContainerSummary, DockerApi } from './docker-api.ts';
 import type { AgentLiveCommand, AgentLiveFrame } from './protocol.ts';
 import type { LiveEngineOptions } from './live.ts';
@@ -13,7 +15,8 @@ export interface AgentContext {
     stateDir: string;
     apply: boolean;
     controlUrl: string;
-    token: string | undefined;
+    /** The node credential presented to control; absent only in tests with a fake `http`. */
+    credential?: Credential;
     /** Runs `wg` / `ip` for the mesh. */
     host?: HostRunner;
     /** Manage the WireGuard mesh (generate a key, report it, reconcile `wg0`). */
@@ -22,14 +25,14 @@ export interface AgentContext {
     /** Host port range (`LOW-HIGH`) for rolling-rollout revisions; default 20000-20999. */
     rolloutPorts?: string;
 }
-export declare function defaultContext(): AgentContext;
+export declare function defaultContext(doc?: AgentDocument | undefined): AgentContext;
 export type AgentLiveControlFrame = AgentLiveCommand;
 export type AgentLiveOutputFrame = AgentLiveFrame;
 interface LiveChannelHandlers {
     onLive?: (frame: AgentLiveControlFrame) => void | Promise<void>;
     onDisconnect?: () => void;
 }
-type WebSocketFactory = (url: string, token: string) => WebSocket;
+type WebSocketFactory = (url: string, credential: Credential | undefined) => WebSocket;
 /** Keeps one outbound stream open and falls back to a bounded HTTP wait when an intermediary rejects WebSockets. */
 export declare class AgentWakeChannel {
     private readonly ctx;
@@ -66,6 +69,13 @@ export declare class AgentLiveOperations {
 }
 /** The container's address on the project network, which health checks dial. */
 export declare function containerIp(ctx: AgentContext, project: string, container: string): Promise<string>;
+/** True when this process runs inside a container: Docker, containerd, Podman or Kubernetes. */
+export declare function runsInContainer(): boolean;
+/**
+ * Health checks dial the container's address on the project network. An agent that runs as a plain host process
+ * cannot reach it on a machine where Docker lives in a VM (macOS, Windows), and the bare timeout does not say why.
+ */
+export declare function outsideContainerHint(health: HealthCheck, inContainer?: boolean): string;
 export declare function waitForHealth(ctx: AgentContext, project: string, container: string, service: string, health: HealthCheck | null | undefined): Promise<void>;
 export { parseDuration };
 export interface PublicListener {
@@ -89,10 +99,6 @@ export declare function reconcile(desiredState: DesiredState, ctx?: AgentContext
 /** Reports this node's WireGuard public key when control has not seen the current one yet. */
 export declare function reportWgKey(ctx: AgentContext): Promise<void>;
 export declare function pollOnce(ctx?: AgentContext): Promise<ProjectStatus[]>;
-interface Creds {
-    agentToken: string;
-    nodeName?: string;
-    controlUrl?: string;
-}
-export declare function loadOrEnroll(): Promise<Creds>;
+/** Loads the persisted identity, or enrolls with a join key. Without either the agent refuses to start. */
+export declare function loadOrEnroll(): Promise<AgentDocument>;
 export declare function main(): Promise<void>;