@celilo/cli 0.21.0 → 0.23.0

package/CELILO_CORE_MODULES.md

@@ -14,19 +14,20 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 > the `@celilo/cli` npm package). Re-grep manifests if an entry looks stale.
 
 > How the graph closes: a public app (`requires: public_web`) is served by **caddy**, which
-> `requires: dns_registrar` (**namecheap**) to publish its name and `firewall` (**greenwave** /
-> **iptables**) to open the port. Internal name resolution comes from a **dns_internal** provider
+> `requires: dns_registrar` (**namecheap**) to publish its name and `firewall` (**axon** /
+> **greenwave** / **iptables**) to open the port. Internal name resolution comes from a **dns_internal** provider
 > (**knot-unbound-internal** / **technitium**). Identity comes from **authentik** (`idp`). That's
 > the whole edge: DNS + firewall + ingress + identity, each a swappable provider module.
 
 ## Network fabric (DNS / firewall / DHCP)
 
-- **greenwave** — GreenWave C4000XG ISP router driver; port-forwarding + public-IP discovery via REST. **provides:** `firewall`, `dhcp_server`.
+- **axon** — Axon Networks Q1000K ISP router driver (Brightspeed-branded); port-forwarding + public-IP discovery + DHCP DNS via the TR-181 CGI API. **provides:** `firewall`, `dhcp_server`. Fork of **greenwave** — identical protocol, differing only in the vendor extension prefix (`X_AXON_` vs `X_GWS_`/`X_LANTIQ_COM_`). Pick by device: Q1000K → **axon**, C4000XG → **greenwave**.
+- **greenwave** — GreenWave C4000XG ISP router driver; port-forwarding + public-IP discovery via REST. **provides:** `firewall`, `dhcp_server`. Legacy device; new deployments on Axon hardware want **axon**.
 - **iptables** — iptables firewall + NAT; cross-VLAN port exposure with recursive upstream delegation. Converge model: `exposeService` registers into the shared-core port-forward registry, then renders the complete ruleset and applies it atomically via `iptables-restore` (default-DROP FORWARD + coarse zone-tier matrix; SSH-free). **provides:** `firewall`.
 - **knot-unbound-internal** — split-horizon internal DNS via Knot (authoritative) + Unbound (recursive); lightweight, plain apt, no .NET. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/knot-unbound-internal/base-module-aspect/`).
 - **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
 - **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). A caller supplies a NAME and nothing else: the address is the source IP of celilo's own update, re-derived on every assert. Registering `<domain>` also claims `www.<domain>` and vice versa (best effort, reported back as `outputs.companion_fqdn` so the framework's `public_dns` check watches it — Namecheap answers `ErrCount 0` for `www` updates it does not apply). DDNS passwords are keyed by the **registrable domain**, never the FQDN. **provides:** `dns_registrar`.
-- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. Records `network.control-plane-vpn.subnet`, which the internal resolver's split-horizon view also consumes. Adopts a running tunnel in place (existing key and peers retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does, `greenwave` does not).
+- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. Records `network.control-plane-vpn.subnet`, which the internal resolver's split-horizon view also consumes. Adopts a running tunnel in place (existing key and peers retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
 
 ## Public edge (ingress / identity)
 

package/CELILO_SUBSYSTEMS.md

@@ -170,8 +170,26 @@ runner seam (`execRunner` real / `createMockRunner` for tests) lives in
 - **DNS provider backfill** (re-emit registrations when a provider deploys) — `apps/celilo/src/services/dns-provider-backfill.ts` — `isDnsInternalProvider`, `backfillProviderDns`.
 - **Base-module aspects (fan-out across the fleet)** — `apps/celilo/src/services/aspect-runner.ts` — `planAspectFanOut`, `runAspectFanOut`, `maybeRunAspectForTrigger`. Aspect content lives in `modules/<m>/base-module-aspect/` (e.g. knot-unbound-internal, technitium).
 - **In-flight operation lock** — `apps/celilo/src/services/module-operations.ts` — `startOperation`/`completeOperation`/`failOperation` record deploy/uninstall/backup/restore in `module_operations`; `refuseIfInFlight`/`checkInFlight` are what backup and restore consult. Deploy and uninstall REGISTER but never check: it is a one-way guard protecting backup/restore consistency, not a general mutex (`openspec/specs/management-server-backup/spec.md` "In-flight operation refusal"). A row stops holding the lock once it is GONE, STOPPED/zombie (`isPidRunnable`, `ps -o state=` — `kill(pid,0)` calls a Ctrl-Z'd process alive), or older than `OPERATION_TTL_MS` (2h). The TTL is not redundancy: a pid is a recycled number, and once the pid space wraps an old row names an unrelated healthy process. Operator surface: `celilo module operations [list|clear] [--abandoned] [--all]` (`apps/celilo/src/cli/commands/module-operations.ts`); `list` shows only what holds the lock, abandoned rows are summarised unless `--abandoned`. Abandoned rows are reclaimed hourly by the `celilo-operations-sweep` bus subscriber (`timer.tick.1h` → `celilo module operations clear`, armed by `ensureOperationsSweepSubscriber` from module registration and `celilo system migrate`). `clear` MARKS rows failed rather than deleting them, and that is load-bearing: the `abandoned_operations` audit reads exactly those released rows to notice one module's operation dying over and over.
+- **Module pause / unpause (control-plane quiescence)** — `apps/celilo/src/services/module-pause.ts` — pure `planPause`/`planUnpause` producing an ordered plan, `executePause`/`executeUnpause` performing it, plus `listPausedModules`/`pausedAmong`/`formatPausedDuration`/`describePausedModule` (the ONE place an age is formatted). CLI: `apps/celilo/src/cli/commands/module-pause.ts` (`celilo module pause|unpause <id> [--cascade] [--stop-infra] [--reason] [--dry-run] [--yes]`). Pausing takes a module out of the CONTROL plane — no dispatched events, no timer hooks, no health checks, alerts suppressed — while leaving the DATA plane running, because capability consumption is deploy-time: every consumer calls `firewall`/`dhcp_server` from `on_install` and nothing calls it while serving. Config, secrets, IPAM/VMID and placement are preserved; `on_uninstall` does NOT run. Quiescence is enforced in two places: pausing drops the module's bus subscriptions (`unregisterModuleSubscriptions`), and `run-named-hook.ts` refuses any non-lifecycle hook for a PAUSED module (`skippedPaused`), which catches the paths that skip the bus — `events resync-subscriptions`, a restore that starts events.db empty, aspect fan-out, public-web republish. `on_install`/`on_uninstall` are exempt by hook NAME (not a caller flag): unpause redeploys through `on_install`, and removing a paused provider needs `on_uninstall`. Unpause always REDEPLOYS (`deployModule`) — that is what rebinds a consumer to a replacement provider and recreates provider-local state from the consumers that own it — and re-registers subscriptions, which a plain deploy does not do. A failed unpause restores `PAUSED` rather than leaving the module live and mis-bound. Cascade order reuses `services/update/dep-graph.ts` unchanged (pause = consumers first, unpause = providers first) and is computed from the GRAPH, never from which modules are currently paused, so a cascade walks THROUGH already-done members and is resumable. See `openspec/changes/module-pause-lifecycle/`.
+- **Provider-removal guard** — `apps/celilo/src/services/remove-guard.ts` — `findRemovalBlockers`/`describeRemovalRefusal`, called from `apps/celilo/src/cli/commands/module-remove.ts`. A PAUSED module is not a dependent (unpause cannot return it to service without a redeploy, and a redeploy re-resolves capabilities), which is what makes a provider swap possible at all. A dependent is one declaring the capability under `requires` **or** `optional` — the same relation `dep-graph.ts` uses, so the guard and the cascade agree on the set; the guard previously read `requires` alone, which let a removal silently orphan `technitium`'s `optional` `dhcp_server`. Refusals name each blocker AND which declaration makes it one. It deliberately does NOT exempt a dependent because another provider of the same capability exists (celilo#683).
 - **Backup artifact encryption** — `apps/celilo/src/services/backup-cipher.ts` — `encryptFileToFile`/`decryptFileToFile`, file-in/file-out and streamed, used by every backup writer (`backup-create.ts`) and reader (`backup-restore.ts`, `restore-from-file.ts`). Do NOT route artifacts through `secrets/encryption.ts`: that API is string-in/string-out for short DB values, and feeding it a tar cost base64 (1.33x) then hex (2x) then `JSON.stringify` — ~9x the artifact in memory, which OOM-killed forgejo's 774 MB backup, and a hard ~805 MB ceiling from the max string length that no amount of RAM raises. On-disk format is `magic "CELILOBK" (8) | version (1) | iv (16) | ciphertext | GCM tag (16)`; the tag is a trailer because it does not exist until the last byte is encrypted. `decryptFileToFile` still reads the pre-2026-07 JSON-envelope artifacts, discriminating on the magic bytes — the envelope's own `schemaVersion` cannot serve, as it lives inside the encrypted tar.
 
+### Paused modules are conspicuous
+
+A pause switches OFF the alerting that would otherwise report the module as
+down, so the paused-ness itself is the signal. Four surfaces, none optional:
+`celilo module list` / `celilo status` render `PAUSED (3d)` with the reason;
+`checkPausedModules` in `apps/celilo/src/services/fleet-checks.ts` makes ANY
+paused module a `system doctor` FAILURE (no threshold — a pause is a degraded
+state, and a threshold is just something to tune until the detector stops
+firing); `findSuppressor` in `apps/celilo/src/services/alerting/suppression.ts`
+attributes the module's suppressed alerts to the pause rather than dropping them
+anonymously; and `fleetWarnings()` in `apps/celilo/src/api/serve.ts` stamps a
+warning naming every paused module and its age onto EVERY management-API result
+(`ResultMessage.warnings`), including on commands unrelated to those modules.
+That last one is the important half: it does not depend on the operator choosing
+to look, which is how a forgotten pause actually gets found.
+
 ## Generation & templating
 
 - **Generator** — `apps/celilo/src/templates/generator.ts` — `generateTemplates` (orchestration), plus Terraform/Ansible file handling.
@@ -238,6 +256,20 @@ Creation, scheduling and freshness. A module declares an `on_backup` hook and a
 - **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`). **Exactly one dispatcher per bus**: `runDispatcher` refuses to start while another dispatcher's process is alive (`assertSoleDispatcher` in `dispatcher.ts`, liveness via `bus.liveDispatchers()` — `kill(pid,0)`, not heartbeat age, since a tick blocks for as long as its slowest handler). The exclusion is here rather than in the systemd unit because a stranded dispatcher can sit outside the unit's cgroup where `KillMode` cannot reach it (#580). `bus.health()` reports `dispatcherCount`/`dispatchers` and a `duplicate_dispatcher` status; `checkDispatcher` (`services/fleet-checks.ts`) fails on more than one. **Supervision** (`services/events-daemon.ts`): the unit is named `celilo-events.service` in BOTH the user and system scope, so the two are indistinguishable in every operator-facing string — `install-daemon` (which defaults to **user** scope) therefore refuses when the other scope's unit exists, and `checkDispatcher` compares the live pid against each installed unit's `MainPID` (`unitMainPid`) rather than merely testing that a unit file exists, since a file nobody is running is not supervision (#610).
 - **Dispatcher supervision (install / restart)** — `apps/celilo/src/services/events-daemon.ts` — `installDaemon`/`planDaemonInstall`/`uninstallDaemon`/`readInstalledUnit` write the systemd unit or launchd plist without touching supervisor state, and `restartDaemon` (+ pure `orphanDispatcherPids`, `resolveRestartScope`, `supervisorCommands`) is the one verb that DOES cycle it. CLI: `celilo events install-daemon|uninstall-daemon|show-daemon|restart-daemon [--system]`. `restart-daemon` exists because the dispatcher runs the code it LOADED: celilo-mgr sat 9 days on event-bus v0.1.8 after apt installed v0.2.0, running the very bug the release fixed (celilo#604). Two things make it non-trivial and both are load-bearing: (1) it stops any live dispatcher the supervisor does not own first — an ORPHAN (PPID 1) is invisible to `systemctl restart`, and `assertSoleDispatcher` then crash-loops the unit while the old code keeps serving; (2) it verifies on the BUS that a NEW pid is live reporting `BUS_VERSION`, never off systemctl's exit code, which returns 0 into exactly that crash loop. System scope shells `sudo systemctl` (the unit is root-owned; celilo runs unprivileged), covered by the scoped `/etc/sudoers.d/celilo-events-restart` conffile that `celilo-bootstrap` ships — a unit test asserts the argv and the grant cannot drift apart.
 
+## Terminal UI (`@celilo/cli-display`)
+
+Every CLI-facing primitive celilo renders with. Lives in `packages/cli-display/src/` so module code running in-process (capability functions, hook scripts) can reach it via `@celilo/capabilities`' re-export, and so `@celilo/core` can use it without depending on `@celilo/cli`. `@clack/prompts` was removed tree-wide in celilo#699; nothing here wraps a third-party prompt library.
+
+**The stream contract:** stdout carries a command's RESULT and nothing else — no glyph, no box-drawing prefix, no ANSI. Everything else (prompts, progress, banners, log lines, diagnostics) goes to **stderr**. This is what lets `celilo module list | grep '^caddy '` and `celilo events tail | jq` work without a repair step; the previous renderer put chrome on stdout, which cost a full e2e run when a `│  ` prefix read as a missing module (celilo#695) and made ten JSON commands unparseable (celilo#698).
+
+- **Prompts** — `packages/cli-display/src/prompt.ts` — `text`, `password`, `confirm`, `select`, `multiselect`, plus `CANCEL`/`isCancel` and `isInteractive`. Raw-mode keypress handling with cursor movement, submit-time validation, and per-keystroke tokenization (one stdin chunk can carry a whole paste). **Refuses a non-TTY stdin** with `PromptUnavailableError` instead of hanging on it or taking the next newline as a considered answer — both real past failures. New interactive decisions must NOT be added here; they belong on the event-bus interview, which a headless responder can answer.
+- **Message chrome** — `packages/cli-display/src/messages.ts` — `intro`, `outro`, `note`, `cancel`, and `log.{success,error,warn,info,message,step}`. All to stderr; routed into the active `ProgressDisplay` when one is set, so the two renderers never fight over the cursor.
+- **Colour** — `packages/cli-display/src/colors.ts` — `colorEnabled()`, `paint()`, and the `colors` token object. Gated per use on `NO_COLOR` → `FORCE_COLOR` → `stderr.isTTY`, so colour never reaches a pipe, a file, or protocol mode.
+- **Progress display** — `packages/cli-display/src/progress-display.ts` — `ProgressDisplay`: a flat append-only stream with a time gutter and one status-glyph vocabulary in `render` mode, or structured `[progress:start|done|fail|sub]` markers in `protocol` mode (what the Remote API server emits and the client re-renders locally).
+- **Active-display singleton** — `packages/cli-display/src/active-display.ts` — `getActiveDisplay`/`setActiveDisplay`; how a long-running command lets everything down its call chain route output through one display.
+- **celilo-side wrapper** — `apps/celilo/src/cli/prompts.ts` — `celiloIntro`/`celiloOutro`, `promptText`/`promptPassword`/`promptConfirm`, `showNote`, `log`. Roughly 160 `log.*` call sites route through this one module, which is why swapping the underlying implementation was a one-file change.
+- **FuelGauge** — `apps/celilo/src/cli/fuel-gauge.ts` — the Cylon-style indicator for operations over ~3s (ESC to background, ^C to cancel); delegates to the active `ProgressDisplay` when one is set.
+
 ## Remote API (drive the CLI over the wire)
 
 Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <host> celilo …`. Typed, streamed, per-operation authz, mid-run interviews. Design: `openspec/changes/replace-ssh-cli-api/proposal.md`.
@@ -246,7 +278,7 @@ Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <h
 - **Wire protocol** — `packages/core/src/protocol.ts` (`@celilo/core`) — versioned NDJSON tagged union (`command`/`progress`/`log`/`result`/`error`/`interview`/`answer`) + `translateOutputLine`.
 - **Registry serialization** — `apps/celilo/src/cli/commands/commands-json.ts` — `celilo commands --json` prints the full `COMMANDS` tree as JSON; the live source of truth the MCP fetches to generate its tool surface (so it mirrors whatever celilo version the server runs). `service list --json` similarly exposes configured providers for MCP auto-detect. `module list --json` prints the module roster (id/version/state) as stable JSON — the backbone the MCP composite troubleshooting tools correlate `audit --json` findings against.
 - **Server** — `apps/celilo/src/api/serve.ts` (`apiServeMode`); the `celilo api-serve --principal=<id>` sshd forced-command entry point (dispatched in `apps/celilo/src/cli/index.ts`). Authorizes per principal, runs the command as a protocol-mode child, streams output, audits to stderr.
-- **Client** — `packages/core/src/remote-client.ts` (`@celilo/core`) — `resolveRemote` (`--remote <dest>` / `CELILO_REMOTE`), `runRemoteClient` (`ssh -T`, renders progress via the local ProgressDisplay, answers interviews via clack).
+- **Client** — `packages/core/src/remote-client.ts` (`@celilo/core`) — `resolveRemote` (`--remote <dest>` / `CELILO_REMOTE`), `runRemoteClient` (`ssh -T`, renders progress via the local ProgressDisplay, answers interviews via the `@celilo/cli-display` prompts). Refuses to prompt on a non-TTY stdin, replying `unanswerable` rather than submitting a default as if a human had chosen it.
 - **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`).
 - **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire.
 - **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop.
@@ -259,6 +291,6 @@ Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh <h
 - **cele2e harness** — `packages/e2e/src/` — `runner.ts`, `container-manager.ts` (`startNetwork`, `reconnectNetwork`), `network-builder.ts` (`NetworkBuilder`).
 - **Run wrapper** — `infra/scripts/cele2e-run.sh` lives in the separate `infra/` clone, **not** in this repo. See the cele2e section of the repo-root `CLAUDE.md` for the operator workflow.
 - **Signal simulators** — three containers, three jobs. `docker/Dockerfile.signal-cli` runs the REAL unlinked daemon (`network().withSignalCli()`, reachable at `signal-cli.lab`) so `e2e/tests/signal-contract.test.ts` can re-check celilo's understanding of the JSON-RPC surface against the actual binary — the pebble pattern. `docker/Dockerfile.signal-sim` runs `simulators/signal-cli/server.ts` (`withSignalSim()`, `signal-sim.lab`), the drivable stand-in with a control surface (`/_control/inbound`, `/_control/sent`, `/_control/unlink`). `/_control/inbound` from the LINKED account's own number emits a `syncMessage.sentMessage` transcript rather than a `dataMessage`, because that is the only shape the real daemon delivers a note-to-self in — and a note-to-self is what every reply is in the default single-operator setup (#460). `docker/Dockerfile.signal-release` (`withSignalRelease()`, `signal-release.lab`) serves the signal-cli release tarball so the module's deploy-time download resolves inside the sealed network — the download is served, never skipped. The libsignal aarch64 native is compiled at `build-infra` time from the module's own recipe (`packages/e2e/scripts/stage-libsignal.ts` → `modules/signal/build/`) and staged into the apt-repo pool, so the recipe is exercised for real on every rebuild while the deploy stays fast and the network stays sealed.
-- **Public-boundary NAT model** — `packages/e2e/config/routing/` — exactly ONE NAT sits between the fleet and the simulated internet: the customer firewall (`fw-main` in `direct-internet`, `fw-isp` in the two-layer default), which MASQUERADEs to `100.100.0.100`. The ISP edge `fw-ext` ROUTES the customer's `100.100.0.0/24` and must never re-NAT it (`-s 100.100.0.0/24 -j RETURN` ahead of its MASQUERADE) — an ISP does not NAT a subscriber that already holds a public address. This is load-bearing, not cosmetic: Namecheap-style DDNS registers the SOURCE address when the caller omits `ip=`, which is how celilo registers public names since #464/#466, so a second NAT here publishes the simulator's own address for every public hostname and quietly breaks ACME, inbound reach, and seeded apex records. Constants: `externalWanIp()` / `externalWanSubnet()` in `src/types.ts`. The corollary: every simulator on `internet-external` must default-route via fw-ext (`100.64.0.1`) so it can reply to the customer's public address — Docker's bridge gateway has no path across networks. `config/routing/public-sim-entrypoint.sh` is the shared two-liner; the DNS hierarchy, pebble, isitup and celilo-website-sim already carried it, and npm-registry / registry / apt-repo / minio / cpanel-host only appeared to work because the double NAT put fw-ext's on-link address in the source field.
+- **Public-boundary NAT model** — `packages/e2e/config/routing/` — exactly ONE NAT sits between the fleet and the simulated internet: the customer firewall (`fw-main` in `direct-internet`, `fw-isp` in the two-layer default), which MASQUERADEs to `203.0.113.100`. The ISP edge `fw-ext` ROUTES the customer's `203.0.113.0/24` and must never re-NAT it (`-s 203.0.113.0/24 -j RETURN` ahead of its MASQUERADE) — an ISP does not NAT a subscriber that already holds a public address. This is load-bearing, not cosmetic: Namecheap-style DDNS registers the SOURCE address when the caller omits `ip=`, which is how celilo registers public names since #464/#466, so a second NAT here publishes the simulator's own address for every public hostname and quietly breaks ACME, inbound reach, and seeded apex records. Constants: `externalWanIp()` / `externalWanSubnet()` in `src/types.ts`. The corollary: every simulator on `internet-external` must default-route via fw-ext (`100.64.0.1`) so it can reply to the customer's public address — Docker's bridge gateway has no path across networks. `config/routing/public-sim-entrypoint.sh` is the shared two-liner; the DNS hierarchy, pebble, isitup and celilo-website-sim already carried it, and npm-registry / registry / apt-repo / minio / cpanel-host only appeared to work because the double NAT put fw-ext's on-link address in the source field.
 - **Simulated address plan** — `packages/e2e/src/types.ts` — `SIM_PRIVATE_SUPERNET` (`10.226.0.0/16`), `zoneIp(zone, host)`, `ZONE_SUBNETS`, `ZONE_GATEWAYS`. Every simulated PRIVATE zone is derived from this one table; the compose generator, `zone-classifier.ts` and the harness's `system init` all read it, so renumbering the whole sim is a one-line change. The sim deliberately does NOT reuse a real fleet's zone /24s (#539): the previous plan was byte-identical to production's, so a stack leaked on celilo's own forgejo-builder — which lives in the real dmz — claimed the builder's own subnet and blackholed every containerized CI job's route to the forge for ~135s. Teardown cannot prevent that (a SIGKILL runs no handler), so the addresses moved instead; the second, better reason is that a suite which only passes on production's exact octets is asserting one site's address plan rather than celilo's behaviour. `src/address-plan.test.ts` is the recurrence gate — it fails if a zone leaves the supernet, or if any retired fleet prefix reappears anywhere in `packages/e2e`, `e2e/tests` or `modules/*/e2e`.
 - **MCP server (agent-driven e2e)** — `packages/mcp-server/src/index.ts` — stdio MCP server exposing `start_run`/`run_status`/`run_result`/`stop_run` (detached cele2e runs read off the event bus, no ANSI scraping) + `env_check` (docker VM, run-lock, shared-infra, mgmt image CLI version, netapps). Dev/ops tool, `private`, not shipped to consumers.

package/drizzle/0024_module_pause.sql

@@ -0,0 +1,20 @@
+-- Module pause/unpause (openspec/changes/module-pause-lifecycle).
+--
+-- Purely additive: `state` gains a new legal value 'PAUSED' (no constraint to
+-- alter — the column is free text typed only in TypeScript), plus two nullable
+-- columns. Every existing module is simply not paused, so there is nothing to
+-- backfill.
+--
+-- `paused_at` is not decoration. A pause suppresses the alerting that would
+-- otherwise report the module as down, so the paused-ness itself has to be
+-- loud; the DURATION is what distinguishes a maintenance window from an outage
+-- someone forgot about, and `state` alone cannot answer it (design D7).
+--
+-- Note for rollback: an older CLI reading state='PAUSED' hits a value outside
+-- its union and fails to recognise it, rather than reading the module as live.
+-- Loud beats silent — but it still has no `unpause`, so unpause everything
+-- before downgrading.
+
+ALTER TABLE `modules` ADD `paused_at` integer;--> statement-breakpoint
+ALTER TABLE `modules` ADD `pause_reason` text;--> statement-breakpoint
+CREATE INDEX `modules_state_idx` ON `modules` (`state`);

package/drizzle/meta/_journal.json

@@ -169,6 +169,13 @@
       "when": 1783900000000,
       "tag": "0023_public_dns_evidence",
       "breakpoints": true
+    },
+    {
+      "idx": 24,
+      "version": "6",
+      "when": 1784000000000,
+      "tag": "0024_module_pause",
+      "breakpoints": true
     }
   ]
-}
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "0.21.0",
+  "version": "0.23.0",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -57,11 +57,10 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1024.0",
-    "@celilo/capabilities": "^1.0.0",
-    "@celilo/cli-display": "^0.1.10",
-    "@celilo/core": "^0.5.0",
-    "@celilo/event-bus": "^0.3.0",
-    "@clack/prompts": "^1.1.0",
+    "@celilo/capabilities": "^1.1.0",
+    "@celilo/cli-display": "^0.2.0",
+    "@celilo/core": "^0.7.0",
+    "@celilo/event-bus": "^0.4.0",
     "ajv": "^8.18.0",
     "drizzle-orm": "^0.36.4",
     "ink": "^7.0.1",

package/src/__integration__/container-services-cli.integration.test.ts

@@ -111,7 +111,11 @@ describe('Container Services and Machine Pool CLI Integration', () => {
       // Remove service with --force flag to skip confirmation
       const output = runCli(ctx.cli, 'service remove service-to-remove --force');
 
-      expect(output).toContain('removed');
+      // The command's RESULT, which is what stdout carries. This used to assert
+      // on the lowercase "removed" of the `celiloOutro` banner, which clack put
+      // on stdout while putting the result on stderr; celilo#699 swapped those
+      // so stdout is the data channel and the banner is chrome on stderr.
+      expect(output).toContain('Removed service: service-to-remove');
 
       // Verify deletion
       const services = await db.select().from(containerServices).all();
@@ -236,7 +240,9 @@ describe('Container Services and Machine Pool CLI Integration', () => {
       // Remove machine with --force flag to skip confirmation
       const output = runCli(ctx.cli, 'machine remove remove-me --force');
 
-      expect(output).toContain('removed');
+      // See the service-remove case above: stdout now carries the result, not
+      // the outro banner (celilo#699).
+      expect(output).toContain('Removed machine: remove-me');
 
       // Verify deletion
       const machinesList = await db.select().from(machines).all();

package/src/api/remote-client.test.ts

@@ -93,11 +93,12 @@ test('renders a forwarded interview and sends the answer back', async () => {
 /**
  * Regression for the fabricated "operator declined".
  *
- * The default renderer prompts with clack, which reads keypresses off stdin
- * whether or not stdin is a terminal. Driven over the MCP (stdin = the JSON-RPC
- * stream) the next newline submitted the prompt at its `initialValue` — the
- * question's `defaultValue` — so a breaking update nobody saw came back as a
- * considered "no". With no terminal the client must say it cannot answer.
+ * The default renderer prompts on the terminal. The renderer it replaced read
+ * keypresses off stdin whether or not stdin was a terminal, so driven over the
+ * MCP (stdin = the JSON-RPC stream) the next newline submitted the prompt at
+ * its `initialValue` — the question's `defaultValue` — and a breaking update
+ * nobody saw came back as a considered "no". With no terminal the client must
+ * say it cannot answer.
  *
  * And it must say so as `unanswerable`, NOT as an `answer` of any shape: an
  * `answer` is what consumes the query and destroys a question nobody decided

package/src/api/serve.ts

@@ -27,8 +27,10 @@ import {
 } from '@celilo/core';
 import { parseArguments } from '../cli/parser';
 import { getEventBusPath } from '../config/paths';
+import { getDb } from '../db/client';
 import { isAuthorized } from '../services/api-access';
 import { EVENT_TYPES } from '../services/bus-interview';
+import { describePausedModule, listPausedModules } from '../services/module-pause';
 import { type WireInterview, startRemoteResponder } from '../services/remote-responder';
 import {
   SessionWriter,
@@ -50,6 +52,38 @@ function send(msg: ServerMessage): void {
   process.stdout.write(`${JSON.stringify(msg)}\n`);
 }
 
+/**
+ * The fleet-level warnings stamped onto every terminal result (design D7).
+ *
+ * Deliberately unconditional and deliberately on UNRELATED commands: a pause
+ * suppresses its module's alerting, so the paused-ness itself is the only
+ * remaining signal, and a forgotten pause is found incidentally rather than by
+ * someone choosing to look. One indexed query (`modules_state_idx`), so this
+ * stays cheap enough to run on every call.
+ *
+ * Never throws: a broken or missing DB must not turn a working command into a
+ * failure over a warning.
+ */
+function fleetWarnings(): string[] {
+  try {
+    const paused = listPausedModules(getDb());
+    if (paused.length === 0) return [];
+    return [
+      `${paused.length} module(s) PAUSED: ${paused.map((m) => describePausedModule(m)).join(', ')}. A paused module is quiesced and its alerts are suppressed. Unpause with "celilo module unpause <id>".`,
+    ];
+  } catch {
+    return [];
+  }
+}
+
+/** Terminal result, with the fleet warnings attached. */
+function resultMessage(success: boolean, exitCode: number): ServerMessage {
+  const warnings = fleetWarnings();
+  return warnings.length > 0
+    ? { type: 'result', success, exitCode, warnings }
+    : { type: 'result', success, exitCode };
+}
+
 function audit(principal: string, op: string, decision: string, exitCode?: number): void {
   const suffix = exitCode === undefined ? '' : ` exit=${exitCode}`;
   process.stderr.write(
@@ -107,7 +141,7 @@ async function runCommand(argv: string[], forward: (msg: ServerMessage) => void)
 
   await Promise.all([pumpLines(child.stdout, forward), pumpLines(child.stderr, forward)]);
   const exitCode = await child.exited;
-  forward({ type: 'result', success: exitCode === 0, exitCode });
+  forward(resultMessage(exitCode === 0, exitCode));
   return exitCode;
 }
 
@@ -119,7 +153,7 @@ async function handleAttach(principal: string, sessionId: string): Promise<void>
   const record = readSession(sessionId);
   if (!record) {
     send({ type: 'error', error: `no such session: ${sessionId}` });
-    send({ type: 'result', success: false, exitCode: 1 });
+    send(resultMessage(false, 1));
     return;
   }
   if (record.principal !== principal) {
@@ -127,7 +161,7 @@ async function handleAttach(principal: string, sessionId: string): Promise<void>
       type: 'error',
       error: `permission denied: session ${sessionId} belongs to another principal`,
     });
-    send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
+    send(resultMessage(false, EXIT_PERMISSION_DENIED));
     audit(principal, `attach:${sessionId}`, 'deny');
     return;
   }
@@ -167,7 +201,7 @@ function handleCancel(principal: string, sessionId: string): void {
   const record = readSession(sessionId);
   if (!record) {
     send({ type: 'error', error: `no such session: ${sessionId}` });
-    send({ type: 'result', success: false, exitCode: 1 });
+    send(resultMessage(false, 1));
     return;
   }
   if (record.principal !== principal) {
@@ -175,7 +209,7 @@ function handleCancel(principal: string, sessionId: string): void {
       type: 'error',
       error: `permission denied: session ${sessionId} belongs to another principal`,
     });
-    send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
+    send(resultMessage(false, EXIT_PERMISSION_DENIED));
     audit(principal, `cancel:${sessionId}`, 'deny');
     return;
   }
@@ -185,7 +219,7 @@ function handleCancel(principal: string, sessionId: string): void {
     emittedBy: `api:${principal}`,
   });
   audit(principal, `cancel:${sessionId}`, 'allow');
-  send({ type: 'result', success: true, exitCode: 0 });
+  send(resultMessage(true, 0));
 }
 
 export async function apiServeMode(principal: string): Promise<void> {
@@ -251,7 +285,7 @@ export async function apiServeMode(principal: string): Promise<void> {
 
     if (!(await isAuthorized(principal, command, subcommand))) {
       send({ type: 'error', error: `permission denied: "${principal}" is not granted "${label}"` });
-      send({ type: 'result', success: false, exitCode: EXIT_PERMISSION_DENIED });
+      send(resultMessage(false, EXIT_PERMISSION_DENIED));
       audit(principal, label, 'deny');
       return;
     }

package/src/api-clients/proxmox.ts

@@ -636,6 +636,40 @@ export class ProxmoxClient {
     if (!result.success) return result;
     return { success: true, data: findNodeForVmid(result.data, vmid) };
   }
+
+  /**
+   * Power a guest down or up. Used by `module pause --stop-infra` (design D2),
+   * where stopping the box is opt-in rather than what a pause means.
+   *
+   * `shutdown` rather than `stop` on the way down: it asks the guest to go
+   * quietly instead of pulling its power, and a pause is a planned operation,
+   * not a fault. Proxmox returns a task id; callers that need to know it
+   * finished poll with `pollTaskUntilDone`.
+   */
+  async setGuestPower(
+    vmid: number,
+    kind: 'lxc' | 'qemu',
+    power: 'shutdown' | 'start',
+  ): Promise<ProxmoxResult<string>> {
+    const node = await this.nodeForVmid(vmid);
+    if (!node.success) return node;
+    if (!node.data) {
+      return { success: false, message: `No Proxmox node hosts vmid ${vmid}` };
+    }
+    return makeProxmoxPost<string>(
+      this.credentials,
+      `/nodes/${node.data}/${kind}/${vmid}/status/${power}`,
+      {},
+    );
+  }
+
+  /** Current run state of a guest (`running`, `stopped`, …). */
+  async guestStatus(vmid: number): Promise<ProxmoxResult<string | null>> {
+    const result = await this.clusterResources();
+    if (!result.success) return result;
+    const guest = result.data.find((r) => r.vmid === vmid);
+    return { success: true, data: guest?.status ?? null };
+  }
 }
 
 /**

package/src/cli/commands/alerts-sweep.ts

@@ -21,6 +21,7 @@ import type { SuppressionTopology } from '../../services/alerting/suppression';
 import { runSweep } from '../../services/alerting/sweep-runner';
 import { getModuleSystems } from '../../services/deployed-systems';
 import { runModuleHealthCheck } from '../../services/health-runner';
+import { listPausedModules } from '../../services/module-pause';
 import type { CommandResult } from '../types';
 
 /** Grace window before a newly-fired alert may notify. */
@@ -65,6 +66,7 @@ export async function handleAlertsSweep(): Promise<CommandResult> {
     },
     loadTopology: () => loadTopology(db),
     loadDeployWindowModules: () => modulesInDeployWindow(db),
+    loadPausedModules: () => new Set(listPausedModules(db).map((m) => m.id)),
     isSuppressible: (alert) => suppressibleByMonitor.get(alert.monitorId) ?? true,
     notifyDepsFor: (alert) => buildNotifyDeps(db, alert, new Date()),
     now: () => new Date(),

package/src/cli/commands/events.test.ts

@@ -4,10 +4,12 @@ import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { defineEvents, openBus } from '@celilo/event-bus';
 import {
+  type FailedDeliveryReport,
   handleEventsAck,
   handleEventsDrain,
   handleEventsEmit,
   handleEventsFail,
+  handleEventsListFailed,
   handleEventsListPending,
   handleEventsListSubscribers,
   handleEventsRepair,
@@ -262,3 +264,67 @@ describe('celilo events command handlers', () => {
     expect(replies[0].payload).toEqual({ value: 'a.net' });
   });
 });
+
+describe('celilo events list-failed', () => {
+  let dir: string;
+  let dbPath: string;
+
+  beforeEach(() => {
+    dir = mkdtempSync(join(tmpdir(), 'events-failed-test-'));
+    dbPath = join(dir, 'events.db');
+    process.env.EVENT_BUS_DB = dbPath;
+    const bus = openBus({ dbPath, events: defineEvents({}) });
+    const sub = bus.subscribe({ name: 'namecheap.ddns', pattern: 'ddns.*', handler: 'echo' });
+    for (let i = 0; i < 60; i++) {
+      const event = bus.emitRaw('ddns.refresh', { n: i });
+      bus.markFailed({ eventId: event.id, subscriberId: sub.id }, new Error('boom'), {
+        abandoned: true,
+      });
+    }
+    bus.close();
+  });
+  afterEach(() => {
+    process.env.EVENT_BUS_DB = undefined;
+    try {
+      rmSync(dir, { recursive: true, force: true });
+    } catch {
+      /* ignore */
+    }
+  });
+
+  // The whole point of the command: `total` is a COUNT, `shown` is the limit.
+  it('separates the true total from the capped sample', async () => {
+    const result = await handleEventsListFailed([], { limit: '5' });
+    expect(result.success).toBe(true);
+    if (!result.success) throw new Error('expected success');
+    const report = result.data as FailedDeliveryReport;
+    expect(report.total).toBe(60);
+    expect(report.shown).toBe(5);
+    expect(report.deliveries).toHaveLength(5);
+    expect(report.bySubscriber).toEqual([
+      { subscriber: 'namecheap.ddns', count: 60, latestFinishedAt: expect.any(Number) },
+    ]);
+  });
+
+  it('names the subscriber and event type on every row', async () => {
+    const result = await handleEventsListFailed([], { limit: '1' });
+    if (!result.success) throw new Error('expected success');
+    const row = (result.data as FailedDeliveryReport).deliveries[0];
+    expect(row.subscriber).toBe('namecheap.ddns');
+    expect(row.eventType).toBe('ddns.refresh');
+    expect(row.status).toBe('abandoned');
+    expect(row.error).toBe('boom');
+    expect(row.finishedAt).toBeGreaterThan(0);
+  });
+
+  it('--subscriber scopes both the rows and the total', async () => {
+    const mine = await handleEventsListFailed([], { subscriber: 'namecheap.ddns', limit: '3' });
+    if (!mine.success) throw new Error('expected success');
+    expect((mine.data as FailedDeliveryReport).total).toBe(60);
+
+    const other = await handleEventsListFailed([], { subscriber: 'nobody' });
+    if (!other.success) throw new Error('expected success');
+    expect((other.data as FailedDeliveryReport).total).toBe(0);
+    expect((other.data as FailedDeliveryReport).deliveries).toHaveLength(0);
+  });
+});