@celilo/cli 1.6.0 → 1.8.0

package/CELILO_CORE_MODULES.md

@@ -37,6 +37,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 - **axon** — Axon Networks Q1000K ISP router driver (Brightspeed-branded); port-forwarding + public-IP discovery + DHCP DNS + DHCP address-pool bounds via the TR-181 CGI API. **provides:** `firewall`, `dhcp_server`. Fork of **greenwave** — identical protocol, differing only in the vendor extension prefix (`X_AXON_` vs `X_GWS_`/`X_LANTIQ_COM_`). Pick by device: Q1000K → **axon**, C4000XG → **greenwave**. Two device behaviours shape this driver and are not shared by **greenwave** (celilo#739): the router **regenerates** `Pool.1.DNSServers` from its own upstream resolver list every ~15 min and on any config commit, so `setDhcpDns` also writes `Device.DNS.X_AXON_CustomServer` — the *input* to that computation, which survives — and **technitium re-asserts** the pool value on `timer.tick.1m`. Optional `dhcp_pool_start`/`dhcp_pool_end` bound the leased range; set them below the addresses IPAM allocates, because **IPAM has no knowledge of the router's DHCP pool** and nothing otherwise stops the router leasing an address celilo already assigned.
 - **greenwave** — GreenWave C4000XG ISP router driver; port-forwarding + public-IP discovery via REST. **provides:** `firewall`, `dhcp_server`. Legacy device; new deployments on Axon hardware want **axon**.
 - **iptables** — iptables firewall + NAT; cross-VLAN port exposure with recursive upstream delegation. Converge model: `exposeService` registers into the shared-core port-forward registry, then renders the complete ruleset and applies it atomically via `iptables-restore` (default-DROP FORWARD + coarse zone-tier matrix; SSH-free). **REQUIRES the networks it has legs on and defines none of them** (`requires.networks: [{from: $self:zones}]`) — the required set is resolved from the same `zones` value that says which NICs the box has, so the legs and the declared networks are ONE list. They used to be two: `zones` and a `provided_networks` array this module wrote into system config from its install hook, with nothing forcing them to agree. A leg whose network has no declared subnet classifies `alien`, and on a firewall with a recorded baseline an alien interface is DISABLED — so under-declaring was never cosmetic. `firewall-interface-classification` §3/D8 reached that conclusion for the test harness; `networks-are-declared-not-written` applies it in production and deletes `provided_networks` outright (breaking: iptables 3.0.0). celilo supplies each range, asking the operator when it holds none. **provides:** `firewall`.
+- **dnsmasq-dhcp** — a DHCP-only dnsmasq in **`internal`**, providing the zone-scoped `dhcp_server` capability for a network with no upstream router to serve it. **provides:** `dhcp_server` (zone `internal`). The other two `dhcp_server` providers, **axon** and **greenwave**, do not run a DHCP server at all — they drive an ISP router's built-in one over its API, which works only while the home network shares a broadcast domain with that router. Put `internal` behind the celilo firewall and DHCP discovery, being a broadcast, no longer reaches it: celilo could describe that topology and not make it work. Importing this module at all depends on well-known-provider uniqueness being scoped PER ZONE (celilo#1028), since axon/greenwave already claim `dhcp_server` fleet-wide. **`port=0` is load-bearing**: dnsmasq ships a resolver as well as a DHCP server, and `dns_internal` is a different capability with its own providers — two resolvers answering on one network is a split-brain that is very hard to see from outside. celilo's transactional artifacts go to `/var/lib/celilo/config-state/` via `applyRenderedConfig`'s `statePath`, NOT beside the managed fragment, because dnsmasq parses every file in `/etc/dnsmasq.d` and a `.celilo-bak` there is configuration the daemon reads. It declares a `requires.system` and is placed by celilo like any other module; an earlier shape declared none, which meant no host in the inventory, an `apt-get` shelled over SSH from a hook, and an operator-typed address — and which also evaded `validateMachineRoleForModule`, the check that refuses to place a non-firewall module on a multi-interface router.
 - **knot-unbound-internal** — split-horizon internal DNS via Knot (authoritative) + Unbound (recursive); lightweight, plain apt, no .NET. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/knot-unbound-internal/base-module-aspect/`).
 - **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
 - **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). A caller supplies a NAME and nothing else: the address is the source IP of celilo's own update, re-derived on every assert. Registering `<domain>` also claims `www.<domain>` and vice versa (best effort, reported back as `outputs.companion_fqdn` so the framework's `public_dns` check watches it — Namecheap answers `ErrCount 0` for `www` updates it does not apply). DDNS passwords are keyed by the **registrable domain**, never the FQDN. **provides:** `dns_registrar`.
@@ -46,7 +47,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 ## Public edge (ingress / identity)
 
 - **caddy** — reverse proxy with automatic HTTPS (HTTP-01 ACME); the standard HTTPS ingress. Emits a 301 redirect block for each served name's `www`/apex companion (kept out of the served-hostname set, so a companion never blocks the ACME wait). Its `health_check`'s DNS item is `internal_dns_resolution` — it digs from the management host behind the split-horizon resolver and is evidence about the in-fleet view only; public reachability is the framework's `public_dns` check. **provides:** `public_web`. **requires:** `dns_registrar`, `firewall`.
-- **caddy-internal** — a SECOND Caddy that serves the fleet and nothing else, and is a sibling of **caddy** rather than a mode of it. It never exposes a port on the firewall's EXTERNAL interface, never publishes a public record, and takes its certificates from Caddy's own local CA (`tls internal`) — those absences ARE the capability, and `public_web` cannot express them because it treats an unreachable route as a deploy failure and publishes a public A record to prevent one (design D10). Lives in the **`dmz`** zone, in front of the things it fronts, mirroring the public `caddy` (celilo#879). LAN devices reach it through an IPAM-allocated `internal`-subnet `ingress_ip` that `on_install` passes to `firewall.exposeService({ ingressIp })` — one DNAT on the firewall's INTERNAL side and nothing external, the same mechanism the dmz-resident `dns_internal` resolver has used for `:53` since ISS-0156. Systems already inside `dmz`/`app`/`secure` use its dmz address instead (carried as `zoneRoutableValue` on the internal record), and VPN clients arrive as a registered trusted source. It previously lived in `internal` on the claim that a dmz ingress could not be reached from a LAN without a public port-forward; that conflated a public forward with an internal-side ingress IP, and cost it the ability to serve a browser inside a segmented zone at all. **`getCaCertificate()` is why the contract has a method `public_web` does not** — an internally-issued cert means clients must trust a CA celilo runs, and here that is nearly free because the people who must trust it are the ones who just downloaded a bundle from the service behind it, so the anchor ships with it. **The route table is this module's OWN config (`routes`), never celilo's `web_routes`** (celilo#846): caddy derives its served hostnames from every row of that table, so a private route stored there would be picked up and served PUBLICLY — storage is the privacy boundary, not policy. Route changes reconcile synchronously in the capability call (there is no `routes_changed` event for private routes, and inventing one would only add delay). **Every proxied route strips a client-supplied `X-Forwarded-User`** (`header_up -X-Forwarded-User`): this ingress routes and does not authenticate, so a backend believing that header would believe whatever the client sent — which `wireguard-manager` shipped doing. That module verifies a signed token now, so the strip protects the NEXT backend written against the same assumption. It is defence in depth, NOT an authenticating proxy: adding `forward_auth` to `private_web` is a capability change and deliberately not bundled with it. A consumer cannot register a route on a hostname the ingress is not already configured for: a capability factory gets no capabilities of its own, so it cannot add the internal DNS record a new name would need, and it refuses rather than serving an unresolvable site block. **provides:** `private_web`. **requires:** `dns_internal` — declared under `optional.capabilities` ONLY to route around celilo#854 (the import-time secret gate refuses a consumer over a secret it never reads); `on_install` throws without it.
+- **caddy-internal** — a SECOND Caddy that serves the fleet and nothing else, and is a sibling of **caddy** rather than a mode of it. It never exposes a port on the firewall's EXTERNAL interface, never publishes a public record, and takes its certificates from Caddy's own local CA (`tls internal`) — those absences ARE the capability, and `public_web` cannot express them because it treats an unreachable route as a deploy failure and publishes a public A record to prevent one (design D10). Lives in the **`dmz`** zone, in front of the things it fronts, mirroring the public `caddy` (celilo#879). LAN devices reach it through an IPAM-allocated `internal`-subnet `ingress_ip` that `on_install` passes to `firewall.exposeService({ ingressIp })` — one DNAT on the firewall's INTERNAL side and nothing external, the same mechanism the dmz-resident `dns_internal` resolver has used for `:53` since ISS-0156. Systems already inside `dmz`/`app`/`secure` use its dmz address instead (carried as `zoneRoutableValue` on the internal record), and VPN clients arrive as a registered trusted source. It previously lived in `internal` on the claim that a dmz ingress could not be reached from a LAN without a public port-forward; that conflated a public forward with an internal-side ingress IP, and cost it the ability to serve a browser inside a segmented zone at all. **`getCaCertificate()` is why the contract has a method `public_web` does not** — an internally-issued cert means clients must trust a CA celilo runs, and here that is nearly free because the people who must trust it are the ones who just downloaded a bundle from the service behind it, so the anchor ships with it. **The route table is this module's OWN config (`routes`), never celilo's `web_routes`** (celilo#846): caddy derives its served hostnames from every row of that table, so a private route stored there would be picked up and served PUBLICLY — storage is the privacy boundary, not policy. Route changes reconcile synchronously in the capability call (there is no `routes_changed` event for private routes, and inventing one would only add delay). **Every proxied route strips a client-supplied `X-Forwarded-User`** (`header_up -X-Forwarded-User`): this ingress routes and does not authenticate, so a backend believing that header would believe whatever the client sent — which `wireguard-manager` shipped doing. That module verifies a signed token now, so the strip protects the NEXT backend written against the same assumption. It is defence in depth, NOT an authenticating proxy: adding `forward_auth` to `private_web` is a capability change and deliberately not bundled with it. A consumer cannot register a route on a hostname the ingress is not already configured for: a capability factory gets no capabilities of its own, so it cannot add the internal DNS record a new name would need, and it refuses rather than serving an unresolvable site block. **provides:** `private_web`. **requires:** `dns_internal` — a hard requirement, since `on_install` throws without it. It sat under `optional.capabilities` until celilo#854 was fixed, because the import-time secret gate refused a consumer over a secret it never reads.
 - **generic-cpanel-hosting-provider** — publishes static sites into a subfolder of a domain on a cPanel/SSH web host celilo does **not** govern (the host owns the domain, DNS, TLS and web server). Systemless like **namecheap** — no `requires.system`, no zone, no IPAM; it holds credentials for an external party. One provider serves many accounts, resolved by hostname; onboarding takes the account password once, `ssh-copy-id`s celilo's key, then discards it. **provides:** `external_web`.
 - **authentik** — Authentik identity provider with OIDC (Docker Compose: server, worker, Postgres, Redis). **provides:** `idp`. **requires:** `public_web`, `dns_registrar`, `firewall`.
 
@@ -57,6 +58,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 - **celilo-apt-repo** — Debian apt repository (reprepro + Bun HTTP server) serving the celilo `.deb` at apt.celilo.computer. **provides:** `apt_publish`. **requires:** `public_web`, `dns_registrar`.
 - **signal** — bidirectional Signal transport for alerts and deploy-interview questions; runs signal-cli in daemon mode with its JSON-RPC socket bound to the host's own address (never public) and **`--receive-mode=manual`**, which is load-bearing: signal-cli's default (`on-start`) leaves the daemon permanently receiving, so it drains every reply into an SSE stream nothing is attached to and REFUSES celilo's `receive` call — replies arrive and are unreadable, while `send` and every health check keep passing. Enrolled as a SECONDARY DEVICE of an existing Signal account rather than registering its own number — Signal blocks most VOIP ranges and bans bot-ish registrations. Recipient addresses live on celilo routes, not in module config, so adding a person never requires a redeploy. Runs on x86_64 and aarch64. `libsignal-client` ships no linux-aarch64 native, so celilo builds one (`modules/signal/build/`) and installs it as a `libsignal-jni` .deb on ARM hosts; x86_64 uses the JAR's bundled native. **provides:** `notification` (`send`, `receive`). **requires:** no capabilities — a transport that depended on the proxy, registrar or firewall could not tell you those were broken — and a system in the **`secure-mgmt`** zone: it holds a linked Signal account (the operator's own messaging identity and keys), and its job is to observe every tier while depending on none, which is what the control-plane zone is for. See `openspec/changes/add-alerting/`.
 - **celilo-website** — public docs site (static Astro) served via Caddy on celilo.computer. **requires:** `public_web`, `dns_registrar`.
+- **celilo-canary** — a deliberately minimal nginx serving one static page in **`dmz`**, permanently deployed, whose health check IS the assertion that the deploy path still works. Nothing about it is interesting except that it is *always there*: modules already deployed keep running when the pipeline breaks, so without a canary a regression in IPAM allocation, Terraform provisioning, Ansible convergence or capability wiring stays invisible until the next real deploy — which is exactly when nobody wants to discover it. One deploy exercises all four, plus a live cross-module capability call. **Fleet-only, and the absence is the design**: it registers one route through `private_web` and requires nothing that reaches the perimeter, so there is no public record, no ACME certificate and no port forward. Its `health_check` probes nginx over systemd and its `/healthz` endpoint with **`probeHttp`, from the management server** rather than by SSHing in and running `curl` — the production `ubuntu-22.04-standard` LXC ships no curl, so the old form could not tell a missing binary from a dead service, and reaching the canary at its own address additionally catches a service bound only to loopback. **requires:** `private_web`.
 
 ## Git forge & CI pipeline
 

package/CELILO_SUBSYSTEMS.md

@@ -32,7 +32,11 @@ see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MOD
 - **Network requirement + ensure (celilo owns the namespace)** — `apps/celilo/src/services/network-ensure.ts` (`ensureRequiredNetworks`), `NetworkRequirementSchema` / `getRequiredNetworkNames` in `apps/celilo/src/manifest/schema.ts`. A module declares `requires.networks: [{name}]` — a NAME, never a value; the schema is `.strict()` so a `subnet:` on the requirement is rejected with a message saying why. The deploy calls `ensureRequiredNetworks` in its interview phase, BEFORE generation and before any hook, and asks over the generic bus interview (`askText`, so it is answerable headless) for anything undefined. Which attributes a network has is celilo's answer, taken from `apps/celilo/schemas/system_config.json`: that file declares `network.<n>.gateway` for the routed segments and omits it for `control-plane-vpn`, so a gateway is never asked for a network that has none. Well-known names carry a `suggested` range there — deliberately NOT `default`, which `getDefaultConfiguration()` would seed at `system init`. Spec: `openspec/specs/network-declaration/spec.md`.
 - **The network write path (closed) + celilo's own discovery** — `celilo system apply-config` (`apps/celilo/src/cli/commands/system-apply-config.ts`) REFUSES the whole `network.` namespace, `network.bridge` excepted (a Proxmox bridge name is not addressing, and it is the one network key with a schema default). That is the automation surface a module hook shells out to, so closing it there is what makes "networks are celilo's" an authority rather than a convention every module has to remember. The refusal names the alternative — declare the network, read it with `$system:` — because a bare rejection sends a module author hunting for a typo. The one write that legitimately needed the surface moved INTO celilo: `celilo system discover-network` (`apps/celilo/src/services/network-discovery.ts`, `cli/commands/system-discover-network.ts`) parses the box's own `ip route` and records `network.internal.*` — or `network.secure-mgmt.*` when the box is off the internal LAN (#300). `celilo-mgmt` calls it and decides nothing; it used to parse and write this itself. Idempotent and never overwrites addressing already set. Recurrence gate: `test-integration/module/no-module-writes-networks.test.ts`.
 - **Firewall interface audit** — `apps/celilo/src/services/audit/interface-classification.ts` — reports per-firewall classification in `celilo audit`: alien interfaces by name and address (drift), and the blocking findings a converge refuses on — a carrier-grade NAT leg, an ambiguous external edge, a default route on the wrong leg.
-- **Zone taxonomy (canonical list)** — `apps/celilo/src/db/schema.ts` — `NETWORK_ZONES` is the single array; `NetworkZone` is DERIVED from it. Never hand-maintain a second copy: a duplicate that dropped a member made zone validation return null and silently fall back to a wrong-but-valid zone.
+- **Zone taxonomy (canonical list)** — `apps/celilo/src/db/schema.ts` — `NETWORK_ZONES` is the single array; `NetworkZone` is DERIVED from it. Never hand-maintain a second copy: a duplicate that dropped a member made zone validation return null and silently fall back to a wrong-but-valid zone. Two copies are unavoidable and BOTH are guarded: `modules/iptables`' `zones` picker (`test-integration/module/zone-options-cover-network-zones.test.ts` fails on any drift, and did when `isp-transit` was added to core alone) and the generated `schemas/module-manifest.schema.json` (`bun run check:schema`, run from `apps/celilo` — a FOURTH gate CI runs that the "Validation gates" list in CLAUDE.md does not name).
+  - **`isp-transit`** — the private segment between a celilo firewall and the router upstream of it. Non-placement and non-allocatable (it joins `external` and `control-plane-vpn` in the excluded set) because that router owns the addressing. It exists because on a DOWNSTREAM firewall the default route and the workload network share one interface, so `internal` meant both at once and every egress rule also reasoned about the workload network. A firewall with no separate transit leg declares `default_route_zone: internal` and behaves as it always did.
+- **Declared egress zone** — `modules/iptables` `default_route_zone`, consumed by `defaultRouteFinding` in `packages/capabilities/src/interface-classification.ts` — celilo already classifies whichever interface carries the default route; what it could not do is tell an INTENDED egress leg from a drifted one, so the comparison was hardcoded to "internal or external". Declaring the intent makes a wandered default route a finding. Defaults to `internal`.
+- **Capability-secret access is reference-driven** — `apps/celilo/src/capabilities/validation.ts` — `validateCapabilityAccess` fires on the `$capability:<name>.<secret>` paths the CONSUMER references, collected from its manifest AND from the templates `validateModuleTemplates` already parses at import. It used to iterate the secrets the PROVIDER declares, so requiring a capability that merely carries a secret put a module on the hook for that secret's `readable_by` list (celilo#854, celilo#1038). Reading a secret still goes through `checkCapabilitySecretAccess` at resolution time.
+- **Zone-first capability provider selection** — `apps/celilo/src/capabilities/lookup.ts` — one shared selection function used by both the direct lookup and the hook capability loader, which had drifted apart. Well-known-provider uniqueness is scoped per zone, which is what lets a second `dhcp_server` provider exist at all while `axon`/`greenwave` claim it fleet-wide.
 - **Control-plane network (`secure-mgmt`)** — `apps/celilo/src/hooks/capability-loader.ts` — `loadControlPlaneSubnet` returns the subnet of the zone `celilo-mgmt` is deployed in. `secure-mgmt` is a placement zone AND the control-plane tier, deliberately NOT in `ZONE_TIER_ORDER` (it is not part of the `dmz → app → secure` data-plane chain; it reaches every tier by trust). The firewall's `trustedSubnets` derives from this rather than assuming celilo-mgr sits on `internal`. Reported as an actionable gap by `checkControlPlaneNetwork` in `apps/celilo/src/services/fleet-checks.ts` when the management address matches no configured subnet.
 - **A deployed system's zone** — `apps/celilo/src/services/deployed-systems.ts` — for machine-pool deploys the zone recorded is the ZONE OF THE MACHINE, not `requires.system.zone` (which is only the minimum used to *select* a host, as with sizing). Three writers must agree: `recordDeployedSystemForModule`, `backfillModuleSystems`, and `apps/celilo/src/variables/context.ts` — the last runs latest and will overwrite the others.
 - **Host discovery ("which host serves module X?")** — `apps/celilo/src/cli/commands/module-where.ts` (`celilo module where <id> [--json]`, MCP `celilo_module_where`) — reads deployed hosts from `module_systems` via `getModuleSystems`, reconciles the live Proxmox node via `reconcilePlacement`, and adds a role-based reachability hint per zone. CI/build infra (builder VM, Forgejo runners) is out of scope (not in `module_systems`).
@@ -185,6 +189,7 @@ runner seam (`execRunner` real / `createMockRunner` for tests) lives in
 - **Module pause / unpause (control-plane quiescence)** — `apps/celilo/src/services/module-pause.ts` — pure `planPause`/`planUnpause` producing an ordered plan, `executePause`/`executeUnpause` performing it, plus `listPausedModules`/`pausedAmong`/`formatPausedDuration`/`describePausedModule` (the ONE place an age is formatted). CLI: `apps/celilo/src/cli/commands/module-pause.ts` (`celilo module pause|unpause <id> [--cascade] [--stop-infra] [--reason] [--dry-run] [--yes]`). Pausing takes a module out of the CONTROL plane — no dispatched events, no timer hooks, no health checks, alerts suppressed — while leaving the DATA plane running, because capability consumption is deploy-time: every consumer calls `firewall`/`dhcp_server` from `on_install` and nothing calls it while serving. Config, secrets, IPAM/VMID and placement are preserved; `on_uninstall` does NOT run. Quiescence is enforced in two places: pausing drops the module's bus subscriptions (`unregisterModuleSubscriptions`), and `run-named-hook.ts` refuses any non-lifecycle hook for a PAUSED module (`skippedPaused`), which catches the paths that skip the bus — `events resync-subscriptions`, a restore that starts events.db empty, aspect fan-out, public-web republish. `on_install`/`on_uninstall` are exempt by hook NAME (not a caller flag): unpause redeploys through `on_install`, and removing a paused provider needs `on_uninstall`. Unpause always REDEPLOYS (`deployModule`) — that is what rebinds a consumer to a replacement provider and recreates provider-local state from the consumers that own it — and re-registers subscriptions, which a plain deploy does not do. A failed unpause restores `PAUSED` rather than leaving the module live and mis-bound. Cascade order reuses `services/update/dep-graph.ts` unchanged (pause = consumers first, unpause = providers first) and is computed from the GRAPH, never from which modules are currently paused, so a cascade walks THROUGH already-done members and is resumable. See `openspec/specs/module-pause/spec.md`.
 - **Provider-removal guard** — `apps/celilo/src/services/remove-guard.ts` — `findRemovalBlockers`/`describeRemovalRefusal`, called from `apps/celilo/src/cli/commands/module-remove.ts`. A PAUSED module is not a dependent (unpause cannot return it to service without a redeploy, and a redeploy re-resolves capabilities), which is what makes a provider swap possible at all. A dependent is one declaring the capability under `requires` **or** `optional` — the same relation `dep-graph.ts` uses, so the guard and the cascade agree on the set; the guard previously read `requires` alone, which let a removal silently orphan `technitium`'s `optional` `dhcp_server`. Refusals name each blocker AND which declaration makes it one. It deliberately does NOT exempt a dependent because another provider of the same capability exists (celilo#683).
 - **Consumer-removal cleanup (every provider is told)** — `apps/celilo/src/services/consumer-cleanup.ts` — pure `planConsumerCleanup` + `loadConsumerCleanupPlan` + `runConsumerCleanup`, called from `apps/celilo/src/cli/commands/module-remove.ts` after `on_uninstall` and before `terraform destroy`. A capability is two-sided: the consumer asks, the provider mints something in ITS world (a caddy site block, a DNAT rule, an OIDC client at authentik, a registered CI runner), and removal only ever touched one side — the FK cascade dropped celilo's row, so the provider's next converge had no way to learn the thing existed. This dispatches the `on_consumer_removed` hook to every provider of every capability the departing module declared under `requires` OR `optional` (the same relation `remove-guard.ts` counts as a dependency edge), **once per provider** rather than once per capability, sorted by provider id. The hook receives one input, `consumer`, and NOTHING else: a provider that cannot answer "what do I hold for this module" without being told has a different defect — the consumer's id was never recorded at mint time. It replaces `services/web-route-cleanup.ts`, which did the same job for exactly one capability, by name, from core. Semantics that are easy to get backwards: **a failed withdrawal never blocks the removal** — the consumer goes and the failing PROVIDER is marked `ERROR` with the departing consumer named in `error_message` (surfaced as a `blocked` finding by `services/audit/undeployed-modules.ts`), because the hook is a full converge and after a failure the provider's state is unknown rather than "one thing missed". **Dispatch continues past a failure**, so one broken provider cannot leave the others holding state. A PAUSED provider is skipped with a warning naming what it keeps (`run-named-hook.ts` refuses non-lifecycle hooks on a paused module, and `on_consumer_removed` must NOT join `LIFECYCLE_HOOKS`), and a never-deployed one is skipped silently. Providers implementing it: caddy, caddy-internal, iptables, greenwave, axon, authentik, forgejo, generic-cpanel-hosting-provider. See `openspec/changes/consumer-removal-cleanup/`.
+- **Provider-arrival backfill (every consumer is re-run)** — `apps/celilo/src/services/provider-arrival.ts` — pure `planProviderBackfill` + `loadProviderBackfillPlan` + `runProviderBackfill`, called from `module-deploy.ts` at the end of BOTH deploy paths (config-only and full). The mirror of consumer-removal cleanup, and it exists because celilo handled one side of a capability generically and the other by hand: a provider arriving had three hand-written pieces covering two capabilities, and `firewall` — provided by `axon`, `greenwave` AND `iptables` — had none, so its consumers inherited nothing (celilo#1011). When a module deploys, every already-installed module declaring one of its capabilities under `requires` OR `optional` has its `on_install` re-run. **PULL, never push** (design D7): the consumer re-registers through the path that worked the first time, rather than core replaying history into the provider by calling the provider's own hooks on a consumer's behalf — the push shape is a second implementation with its own bug surface, and `backfillWebRouteDns` had already drifted into one. Never fatal to the provider's own deploy: a failed consumer is named with `celilo module deploy <id>` to retry, and a PAUSED or never-deployed consumer is reported as skipped rather than silently passed over. Shares `PRE_DEPLOY_STATES` with `consumer-cleanup.ts` — one definition, also read by `module-remove.ts`. It replaced `services/public-web-republish.ts` (deleted) and deliberately did NOT replace `services/dns-provider-backfill.ts`, whose docblock records the two reasons: one half replays celilo's HOST inventory rather than a capability set, and the other covers FQDNs published through `public_web` by modules that declare `public_web` and never `dns_internal`. See `openspec/changes/capability-owned-tables/` stage 1.
 - **Firewall registry ownership** — `apps/celilo/src/services/port-forwards.ts` + `apps/celilo/src/services/trusted-sources.ts`. Both stores are bound to the CONSUMING module and stamp `registered_by` themselves; a caller cannot supply it, so a registration is never attributed to the wrong module. Both writes are **declarative**: `replace()` states a consumer's COMPLETE set for a target, so a port or subnet it previously registered and now omits is withdrawn (celilo#855 — before this, a module that exposed `:8080` and redeployed exposing `:9090` kept both, forever). The owner is IN both unique indexes, not merely beside them: two consumers wanting the same forward are two ROWS, so one leaving cannot delete a rule the other still needs; `modules/iptables/scripts/ruleset-renderer.ts` dedupes on the rule tuple so the pair renders once. Neither column is a FK, so `runConsumerCleanup` deletes these rows explicitly after every provider has converged without them.
 - **Backup artifact encryption** — `apps/celilo/src/services/backup-cipher.ts` — `encryptFileToFile`/`decryptFileToFile`, file-in/file-out and streamed, used by every backup writer (`backup-create.ts`) and reader (`backup-restore.ts`, `restore-from-file.ts`). Do NOT route artifacts through `secrets/encryption.ts`: that API is string-in/string-out for short DB values, and feeding it a tar cost base64 (1.33x) then hex (2x) then `JSON.stringify` — ~9x the artifact in memory, which OOM-killed forgejo's 774 MB backup, and a hard ~805 MB ceiling from the max string length that no amount of RAM raises. On-disk format is `magic "CELILOBK" (8) | version (1) | iv (16) | ciphertext | GCM tag (16)`; the tag is a trailer because it does not exist until the last byte is encrypted. `decryptFileToFile` still reads the pre-2026-07 JSON-envelope artifacts, discriminating on the magic bytes — the envelope's own `schemaVersion` cannot serve, as it lives inside the encrypted tar.
 
@@ -305,6 +310,7 @@ upgrade controls — as opposed to how the module configures itself.
 ## Persistence
 
 - **DB schema** — `apps/celilo/src/db/schema.ts`. Client: `apps/celilo/src/db/client.ts`. Migration runner: `apps/celilo/src/db/migrate.ts`. Migrations: `apps/celilo/drizzle/`.
+- **Frozen-watermark repair** — `runMigrationsOn` (`db/migrate.ts`) is what `createDbClient` calls on open, not drizzle's migrator directly. drizzle is watermark-only, so a DB from the imperative hand-list era — schema applied, `__drizzle_migrations` never told — makes it re-run migrations and die on `duplicate column name`. That throw is inside the OPEN, so every celilo command on that box fails, `celilo system migrate` included: it reaches its own repair through `getDb()`. The repair engages only after the stock migrator has failed AND only when `findSchemaDrift` reports the declared schema entirely present, and it corrects the LEDGER without running SQL — replaying is not an option, since `drizzle/0021_dns_registration_consumers.sql` rebuilds `dns_registrations` by dropping the original and renaming a copy over it. A PARTIALLY applied schema (celilo-mgr's own pre-remediation state) still fails, naming the hand remediation. Gate: `db/migrate.test.ts`. Both entrypoints run `celilo system migrate` — the `.deb` postinst and the `celilo-mgmt` Ansible role, before the dispatcher starts (celilo#169).
 
 ## Events
 

package/MODULE_PRIMITIVES.md

@@ -92,7 +92,7 @@ for liveness use `probe`.
 serviceCtl(sys, 'caddy', 'reload', run);
 ```
 
-### `applyRenderedConfig({ target, path, content, validate, apply, runner?, timeoutMs? }) → RunResult`
+### `applyRenderedConfig({ target, path, content, validate, apply, statePath?, runner?, timeoutMs? }) → RunResult`
 **The converge primitive.** For config whose *content* your hook computes at
 deploy-time from celilo's DB (caddy's Caddyfile from `web_routes`, knot's views
 from the DNS ledger, the firewall ruleset from the port-forward registry) —
@@ -100,6 +100,11 @@ content a static Ansible render can't know. One atomic on-box script, one SSH
 round-trip: back up → write (`content` rides **stdin**, newline-safe) → `validate`
 (rolled back + not activated if it fails) → `apply` (rolled back if it fails).
 `{path}` is substituted into `validate`/`apply`.
+
+By default, rollback and last-render state are retained beside `path` as
+`.celilo-bak` and `.celilo-prev`. Set `statePath` to a prefix outside daemon
+include directories (such as `/etc/dnsmasq.d`) when the daemon parses every
+neighboring file.
 ```ts
 applyRenderedConfig({
   target: sys, path: '/etc/caddy/Caddyfile', content: rendered,

package/drizzle/0027_dns_internal_records_consumer_cascade.sql

@@ -0,0 +1,43 @@
+-- `dns_internal_records` cascades on its PROVIDER as well as its consumer, so
+-- swapping the internal DNS provider deletes the fleet's entire internal DNS
+-- ledger — including the `zone_routable_ip` view overrides the resolver's
+-- split-horizon config is reconciled from (celilo#1010).
+--
+-- `web_routes` does not do this. Its rows cascade on the consumer only, which is
+-- why a `public_web` provider swap keeps every route. The two tables' docblocks
+-- claim to be siblings and nothing marks the difference as a decision, because it
+-- was not one.
+--
+-- The claim on a capability-owned table is the CONSUMER
+-- (openspec/changes/capability-owned-tables D3/D8), and `dns_internal`'s
+-- declaration cannot express anything else. This makes the table agree with it.
+--
+-- `provider_module_id` STAYS. It is how a record is attributed to the resolver
+-- serving it and it is half the uniqueness key; only its ON DELETE action is
+-- wrong. It becomes a plain column with no foreign key rather than a cascade,
+-- so removing a provider leaves the ledger intact for the next one to reconcile
+-- from — which is the behaviour stage 1's provider-arrival backfill assumes.
+--
+-- SQLite cannot alter a foreign key, so the table is rebuilt. Order matters:
+-- `foreign_keys` is ON during migration, so the copy is made BEFORE the drop and
+-- the index is recreated after the rename. Same shape as
+-- `0021_dns_registration_consumers`.
+
+CREATE TABLE `__new_dns_internal_records` (
+	`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
+	`provider_module_id` text NOT NULL,
+	`consumer_module_id` text NOT NULL,
+	`host` text NOT NULL,
+	`ip` text NOT NULL,
+	`zone_routable_ip` text,
+	`registered_at` integer DEFAULT (unixepoch()) NOT NULL,
+	FOREIGN KEY (`consumer_module_id`) REFERENCES `modules`(`id`) ON UPDATE no action ON DELETE cascade
+);
+--> statement-breakpoint
+INSERT INTO `__new_dns_internal_records` SELECT `id`, `provider_module_id`, `consumer_module_id`, `host`, `ip`, `zone_routable_ip`, `registered_at` FROM `dns_internal_records`;
+--> statement-breakpoint
+DROP TABLE `dns_internal_records`;
+--> statement-breakpoint
+ALTER TABLE `__new_dns_internal_records` RENAME TO `dns_internal_records`;
+--> statement-breakpoint
+CREATE UNIQUE INDEX `dns_internal_records_provider_host_idx` ON `dns_internal_records` (`provider_module_id`,`host`);

package/drizzle/meta/_journal.json

@@ -190,6 +190,13 @@
       "when": 1784200000000,
       "tag": "0026_module_integrity_version",
       "breakpoints": true
+    },
+    {
+      "idx": 27,
+      "version": "6",
+      "when": 1784300000000,
+      "tag": "0027_dns_internal_records_consumer_cascade",
+      "breakpoints": true
     }
   ]
-}
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "1.6.0",
+  "version": "1.8.0",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -58,9 +58,9 @@
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1109.0",
     "@aws-sdk/lib-storage": "^3.1101.0",
-    "@celilo/capabilities": "^2.3.0",
+    "@celilo/capabilities": "^2.6.0",
     "@celilo/cli-display": "^0.2.0",
-    "@celilo/core": "^0.9.0",
+    "@celilo/core": "^0.9.1",
     "@celilo/event-bus": "^0.6.0",
     "ajv": "^8.18.0",
     "drizzle-orm": "^0.36.4",

package/src/capabilities/lookup.ts

@@ -18,6 +18,43 @@ export interface CapabilityProviderInfo {
   zones: string[] | null;
 }
 
+type CapabilityProviderRow = {
+  id: number;
+  moduleId: string;
+  capabilityName: string;
+  version: string;
+  data: Record<string, unknown>;
+  zones: string[] | null;
+};
+
+/**
+ * Select a provider from an already-loaded candidate set.
+ *
+ * Keeping this policy separate from the database query lets runtime consumers
+ * that already loaded every provider (notably the hook capability loader) use
+ * exactly the same explicit-zone-first semantics as direct lookups.
+ */
+export function selectCapabilityProvider(
+  all: CapabilityProviderRow[],
+  zone?: string,
+): CapabilityProviderInfo | null {
+  if (all.length === 0) return null;
+
+  if (zone) {
+    const zoneMatch = all.find((candidate) => candidate.zones?.includes(zone));
+    if (zoneMatch) return toInfo(zoneMatch);
+
+    const agnostic = all.find(
+      (candidate) => candidate.zones === null || candidate.zones === undefined,
+    );
+    if (agnostic) return toInfo(agnostic);
+
+    return null;
+  }
+
+  return toInfo(all[0]);
+}
+
 /**
  * Find a capability provider, optionally filtered by zone.
  *
@@ -35,27 +72,7 @@ export function findCapabilityProvider(
   zone?: string,
 ): CapabilityProviderInfo | null {
   const all = db.select().from(capabilities).where(eq(capabilities.capabilityName, name)).all();
-
-  if (all.length === 0) return null;
-
-  if (zone) {
-    // First: try to find a provider that explicitly covers this zone
-    const zoneMatch = all.find((c) => {
-      const zones = c.zones as string[] | null;
-      return zones?.includes(zone);
-    });
-    if (zoneMatch) return toInfo(zoneMatch);
-
-    // Second: fall back to zone-agnostic provider (zones is null)
-    const agnostic = all.find((c) => c.zones === null || c.zones === undefined);
-    if (agnostic) return toInfo(agnostic);
-
-    // No match for this zone
-    return null;
-  }
-
-  // No zone specified: return first provider
-  return toInfo(all[0]);
+  return selectCapabilityProvider(all, zone);
 }
 
 /**
@@ -70,14 +87,7 @@ export function findAllCapabilityProviders(name: string, db: DbClient): Capabili
     .map(toInfo);
 }
 
-function toInfo(row: {
-  id: number;
-  moduleId: string;
-  capabilityName: string;
-  version: string;
-  data: Record<string, unknown>;
-  zones: string[] | null;
-}): CapabilityProviderInfo {
+function toInfo(row: CapabilityProviderRow): CapabilityProviderInfo {
   return {
     id: row.id,
     moduleId: row.moduleId,

package/src/capabilities/secret-ref.test.ts

@@ -10,6 +10,7 @@ import { join } from 'node:path';
 import { createDbClient } from '@/db/client';
 import { encryptSecret } from '@/secrets/encryption';
 import { getOrCreateMasterKey } from '@/secrets/master-key';
+import { validateCapabilitySecrets } from './secret-validation';
 import { getCapabilitySecret } from './secrets';
 
 let testDirs: string[] = [];
@@ -108,6 +109,11 @@ describe('Capability secret_ref resolution', () => {
       )
       .run(capabilityId.id, 'tsig', 'TSIG secret for DNS updates');
 
+    // A configured secret_ref satisfies generation validation without
+    // duplicating the secret into capability_secrets.
+    const validation = await validateCapabilitySecrets('dns-external', db.$client);
+    expect(validation).toEqual({ success: true });
+
     // Test: Resolve capability secret via secret_ref
     const result = await getCapabilitySecret('dns_external', 'tsig', db.$client);
 
@@ -172,6 +178,17 @@ describe('Capability secret_ref resolution', () => {
       )
       .run(capabilityId.id, 'tsig');
 
+    const validation = await validateCapabilitySecrets('dns-external', db.$client);
+    expect(validation.success).toBe(false);
+    expect(validation.missingSecrets).toEqual([
+      {
+        capabilityId: capabilityId.id,
+        capabilityName: 'dns_external',
+        secretName: 'tsig',
+        description: null,
+      },
+    ]);
+
     // Test: Should throw error about missing module secret
     await expect(getCapabilitySecret('dns_external', 'tsig', db.$client)).rejects.toThrow(
       "Module secret 'nonexistent_secret' not found",
@@ -241,6 +258,9 @@ describe('Capability secret_ref resolution', () => {
       )
       .run(capabilityId.id, 'api_key', encrypted.encryptedValue, encrypted.iv, encrypted.authTag);
 
+    const validation = await validateCapabilitySecrets('test-module', db.$client);
+    expect(validation).toEqual({ success: true });
+
     // Test: Should read from capability_secrets table
     const result = await getCapabilitySecret('test_capability', 'api_key', db.$client);
 
@@ -305,6 +325,10 @@ describe('Capability secret_ref resolution', () => {
       )
       .run(capabilityId.id, 'api_key');
 
+    const validation = await validateCapabilitySecrets('test-module', db.$client);
+    expect(validation.success).toBe(false);
+    expect(validation.missingSecrets?.map((secret) => secret.secretName)).toEqual(['api_key']);
+
     // Test: Should throw helpful error
     await expect(getCapabilitySecret('test_capability', 'api_key', db.$client)).rejects.toThrow(
       "Secret 'api_key' in capability 'test_capability' has not been set",

package/src/capabilities/secret-validation.ts

@@ -5,6 +5,7 @@
 
 import type { Database } from 'bun:sqlite';
 import { celiloIntro, promptPassword } from '../cli/prompts';
+import type { ModuleManifest } from '../manifest/schema';
 import { encryptSecret } from '../secrets/encryption';
 import { getOrCreateMasterKey } from '../secrets/master-key';
 
@@ -21,6 +22,46 @@ export interface SecretValidationResult {
   missingSecrets?: MissingCapabilitySecret[];
 }
 
+/**
+ * Check whether an unset capability-secret row is backed by a configured
+ * provider module secret through `secret_ref`.
+ *
+ * Capability registration deliberately stores metadata-only rows even when
+ * the manifest delegates storage to `$secret:<name>`. In that case the NULL
+ * capability value is expected and must not trigger a second secret prompt.
+ */
+function hasConfiguredSecretRef(
+  moduleId: string,
+  manifest: ModuleManifest | null,
+  capabilityName: string,
+  secretName: string,
+  db: Database,
+): boolean {
+  const capability = manifest?.provides?.capabilities?.find(
+    (candidate) => candidate.name === capabilityName,
+  );
+  const secret = capability?.secrets?.find((candidate) => candidate.name === secretName);
+  const match = secret?.secret_ref?.match(/^\$secret:(.+)$/);
+
+  if (!match) {
+    return false;
+  }
+
+  const referencedSecret = db
+    .prepare(
+      `SELECT 1
+       FROM secrets
+       WHERE module_id = ? AND name = ?
+         AND encrypted_value IS NOT NULL
+         AND iv IS NOT NULL
+         AND auth_tag IS NOT NULL
+       LIMIT 1`,
+    )
+    .get(moduleId, match[1]);
+
+  return referencedSecret !== null && referencedSecret !== undefined;
+}
+
 /**
  * Check if module has any missing capability secrets
  *
@@ -43,6 +84,11 @@ export async function validateCapabilitySecrets(
     return { success: true }; // No capabilities = no secrets needed
   }
 
+  const moduleResult = db.prepare('SELECT manifest_data FROM modules WHERE id = ?').get(moduleId) as
+    | { manifest_data: string }
+    | undefined;
+  const manifest = moduleResult ? (JSON.parse(moduleResult.manifest_data) as ModuleManifest) : null;
+
   // Check for secrets with NULL encrypted_value
   const missingSecrets: MissingCapabilitySecret[] = [];
 
@@ -56,6 +102,10 @@ export async function validateCapabilitySecrets(
       .all(capability.id) as Array<{ name: string; description: string | null }>;
 
     for (const secret of secrets) {
+      if (hasConfiguredSecretRef(moduleId, manifest, capability.capability_name, secret.name, db)) {
+        continue;
+      }
+
       missingSecrets.push({
         capabilityId: capability.id,
         capabilityName: capability.capability_name,