@celilo/cli 0.23.0 → 0.24.0

package/CELILO_CORE_MODULES.md

@@ -21,13 +21,13 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 
 ## Network fabric (DNS / firewall / DHCP)
 
-- **axon** — Axon Networks Q1000K ISP router driver (Brightspeed-branded); port-forwarding + public-IP discovery + DHCP DNS via the TR-181 CGI API. **provides:** `firewall`, `dhcp_server`. Fork of **greenwave** — identical protocol, differing only in the vendor extension prefix (`X_AXON_` vs `X_GWS_`/`X_LANTIQ_COM_`). Pick by device: Q1000K → **axon**, C4000XG → **greenwave**.
+- **axon** — Axon Networks Q1000K ISP router driver (Brightspeed-branded); port-forwarding + public-IP discovery + DHCP DNS + DHCP address-pool bounds via the TR-181 CGI API. **provides:** `firewall`, `dhcp_server`. Fork of **greenwave** — identical protocol, differing only in the vendor extension prefix (`X_AXON_` vs `X_GWS_`/`X_LANTIQ_COM_`). Pick by device: Q1000K → **axon**, C4000XG → **greenwave**. Two device behaviours shape this driver and are not shared by **greenwave** (celilo#739): the router **regenerates** `Pool.1.DNSServers` from its own upstream resolver list every ~15 min and on any config commit, so `setDhcpDns` also writes `Device.DNS.X_AXON_CustomServer` — the *input* to that computation, which survives — and **technitium re-asserts** the pool value on `timer.tick.1m`. Optional `dhcp_pool_start`/`dhcp_pool_end` bound the leased range; set them below the addresses IPAM allocates, because **IPAM has no knowledge of the router's DHCP pool** and nothing otherwise stops the router leasing an address celilo already assigned.
 - **greenwave** — GreenWave C4000XG ISP router driver; port-forwarding + public-IP discovery via REST. **provides:** `firewall`, `dhcp_server`. Legacy device; new deployments on Axon hardware want **axon**.
 - **iptables** — iptables firewall + NAT; cross-VLAN port exposure with recursive upstream delegation. Converge model: `exposeService` registers into the shared-core port-forward registry, then renders the complete ruleset and applies it atomically via `iptables-restore` (default-DROP FORWARD + coarse zone-tier matrix; SSH-free). **provides:** `firewall`.
 - **knot-unbound-internal** — split-horizon internal DNS via Knot (authoritative) + Unbound (recursive); lightweight, plain apt, no .NET. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/knot-unbound-internal/base-module-aspect/`).
 - **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
 - **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). A caller supplies a NAME and nothing else: the address is the source IP of celilo's own update, re-derived on every assert. Registering `<domain>` also claims `www.<domain>` and vice versa (best effort, reported back as `outputs.companion_fqdn` so the framework's `public_dns` check watches it — Namecheap answers `ErrCount 0` for `www` updates it does not apply). DDNS passwords are keyed by the **registrable domain**, never the FQDN. **provides:** `dns_registrar`.
-- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. Records `network.control-plane-vpn.subnet`, which the internal resolver's split-horizon view also consumes. Adopts a running tunnel in place (existing key and peers retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
+- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. **Declares `network.control-plane-vpn.subnet` BEFORE it creates the interface** — the first side effect of `on_install`, before any command touches the box. The ordering is load-bearing, not tidiness: the firewall classifies every interface on its host and one no declaration accounts for is `alien`, so creating `wg0` first would leave a window in which a converge — a timer, the registry-poll CD, any unrelated module deploy — could shut the admin VPN down as an intruder's interface. The module declares the NETWORK, never the device: only the firewall decides what an interface is for. `health_check` asserts the declaration still holds, because the other checks all pass while it is missing. That same key is what the internal resolver's split-horizon view consumes. Adopts a running tunnel in place (existing key and peers retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
 
 ## Public edge (ingress / identity)
 

package/CELILO_SUBSYSTEMS.md

@@ -25,7 +25,10 @@ see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MOD
 
 - **IPAM (IP/VMID allocation)** — `apps/celilo/src/ipam/allocator.ts` — `allocateIPFromSubnet`, `allocateVMID`, `reserveIP`/`unreserveIP`, `inferZoneFromIP`, `getAllocation`. Auto-wrapper: `apps/celilo/src/ipam/auto-allocator.ts` — `allocateForModule` / `deallocateForModule`.
 - **Infrastructure selection (container-service vs machine pool)** — `apps/celilo/src/services/machine-pool.ts` (`getMachineByHostname`, `addMachine`, `assignModuleToMachine`) and `apps/celilo/src/services/container-service.ts` (`getContainerServiceByName`, `addContainerService`, `verifyContainerService`). Provider API clients: `apps/celilo/src/api-clients/proxmox.ts`, `apps/celilo/src/api-clients/digitalocean.ts`.
-- **Zone detection / system config** — `apps/celilo/src/services/zone-detector.ts` — `detectZoneFromIp` reads `network.<zone>.subnet` from the `systemConfig` table.
+- **Zone detection / system config** — `apps/celilo/src/services/zone-detector.ts` — `detectZoneFromIp` reads `network.<zone>.subnet` from the `systemConfig` table and returns `NetworkZone | 'unknown'`. It answers CONTAINMENT ONLY. It used to return `'external'` on no-match, which conflated "no declared subnet contains this" with "the internet can route to this" — on a firewall with five RFC1918 legs that reported four of them as facing the internet. `'unknown'` is the honest answer; the caller resolves it (see `machine add`: publicly routable → `external`, otherwise fail asking for `--zone`). The subnet-backed zone list is derived from `NETWORK_ZONES` minus `external`, which has no subnet and must never be given one.
+- **Interface classification** — `packages/capabilities/src/interface-classification.ts` — THE shared classifier, used by the backend and every firewall provider module so the two cannot drift apart again. `isPubliclyRoutable(ip)` is a property of the address alone (false for RFC 1918, RFC 6598 carrier-grade NAT, loopback, link-local, multicast, reserved). `classifyInterfaces(interfaces, zones)` assigns each interface `zone → external → alien`, first match winning, where `external` is the RESIDUAL — routable and claimed by no declared zone — and is never subnet-matched. `externalEdge()` returns none/single/**ambiguous** rather than silently picking the first public address. `defaultRouteFinding()` enforces the invariant that the default route leaves through `internal` or `external`. **`subnetContains(cidr, ip)` lives here and is the ONLY implementation** — three existed and disagreed (the backend's mishandled `/0`); the other two are deleted, not aliased. Design: `openspec/changes/firewall-interface-classification/design.md`.
+- **Declared networks (classification input)** — `readDeclaredNetworks(db)` in `apps/celilo/src/hooks/capability-loader.ts` — every `network.<name>.subnet` in system config, which is what an interface is attributed against. Read from the CONFIG, not from `NETWORK_ZONES`: celilo declares networks that are not placement zones (`network.control-plane-vpn.subnet`, written by `wireguard` before it brings `wg0` up). Injected into the firewall capability as a LIVE reader (`declaredNetworks`), alongside the trusted-source store and for the same reason — a declaration written during a hook run must be visible to the converge that follows it in that same run, which a snapshot taken at capability-build time cannot be.
+- **Firewall interface audit** — `apps/celilo/src/services/audit/interface-classification.ts` — reports per-firewall classification in `celilo audit`: alien interfaces by name and address (drift), and the blocking findings a converge refuses on — a carrier-grade NAT leg, an ambiguous external edge, a default route on the wrong leg.
 - **Zone taxonomy (canonical list)** — `apps/celilo/src/db/schema.ts` — `NETWORK_ZONES` is the single array; `NetworkZone` is DERIVED from it. Never hand-maintain a second copy: a duplicate that dropped a member made zone validation return null and silently fall back to a wrong-but-valid zone.
 - **Control-plane network (`secure-mgmt`)** — `apps/celilo/src/hooks/capability-loader.ts` — `loadControlPlaneSubnet` returns the subnet of the zone `celilo-mgmt` is deployed in. `secure-mgmt` is a placement zone AND the control-plane tier, deliberately NOT in `ZONE_TIER_ORDER` (it is not part of the `dmz → app → secure` data-plane chain; it reaches every tier by trust). The firewall's `trustedSubnets` derives from this rather than assuming celilo-mgr sits on `internal`. Reported as an actionable gap by `checkControlPlaneNetwork` in `apps/celilo/src/services/fleet-checks.ts` when the management address matches no configured subnet.
 - **A deployed system's zone** — `apps/celilo/src/services/deployed-systems.ts` — for machine-pool deploys the zone recorded is the ZONE OF THE MACHINE, not `requires.system.zone` (which is only the minimum used to *select* a host, as with sizing). Three writers must agree: `recordDeployedSystemForModule`, `backfillModuleSystems`, and `apps/celilo/src/variables/context.ts` — the last runs latest and will overwrite the others.
@@ -217,7 +220,10 @@ is currently wrong, and routes carry the message to a person's phone. Design:
 - **Alert identity** — `apps/celilo/src/services/alerting/keys.ts` — the key grammar (`module:<id>[/check:<name>]`, `builtin:<check>[/<kind>:<target>]`) that makes "the same problem" the same alert across runs. `moduleAlertKey`, `moduleCheckAlertKey`, `builtinAlertKey`, `parseAlertKey`.
 - **Reconciliation** — `apps/celilo/src/services/alerting/reconcile.ts` (`reconcile`) — a successful run's failing-key set is authoritative and resolution is by SET DIFFERENCE (absent ⇒ resolved). A run whose outcome is `error` resolves NOTHING and fires a module-level alert instead: the false-all-clear guard.
 - **Monitor execution** — `apps/celilo/src/services/alerting/run-monitor.ts` (`runOneMonitor`) + `sweep.ts` (`selectDueMonitors`) + `builtin-monitors.ts` / `health-coverage.ts` (the built-in checks and the "module with no health check" coverage check).
-- **Scheduled audit categories (`builtin_check` monitors)** — `apps/celilo/src/services/alerting/builtin-source.ts` — `SCHEDULABLE_BUILTIN_CHECKS` is the list of `celilo system audit` categories cheap enough to run every sweep: `machines_reachable`, `backups`, `disk_space`, `abandoned_operations`, and `public_dns` (`apps/celilo/src/services/audit/abandoned-operations.ts` — ≥3 abandonments of the same (module, operation) in 7d, the fingerprint of an operation being killed mid-flight). Everything else in the audit needs the whole world injected (proxmox, terraform, registry) and is not schedulable. Enable one with `celilo monitor add backups --interval 1h`. Targets are tab-completable — `completion.ts` reads `SCHEDULABLE_BUILTIN_CHECKS` directly rather than a hand-copied list, so a newly-schedulable check is completable immediately.
+- **Health-check cadence (one accessor)** — `apps/celilo/src/services/alerting/health-cadence.ts` — `effectiveHealthCheckCadence(manifest, override)`, `isScheduled`, `loadModuleHealthCadences`, `reconcileModuleWatchState`. The manifest's `hooks.health_check.interval` SUGGESTS; the operator's `health_check_interval` decides; both resolve at read time. `null` means nobody named a cadence (a coverage gap); `'manual'` means the operator opted out (a decision — raises no coverage finding, and resolves the module's live alerts on the way down, since nothing will report on them again).
+- **⚠️ `monitors.intervalMinutes` and `monitors.enabled` are `builtin_check`-only.** A `module_hook` row carries severity, escalation policy and `lastRunAt`; its cadence and whether it is watched resolve through the accessor above. The columns' meaning depending on `kind` is a named smell (design.md D8) — the alternatives are a cached resolved value that rots, or splitting the table, which needs a synthetic monitor identity for `alerts.monitorId`. Gates: `sweep-runner.test.ts` asserts a module row's stored values are NOT consulted; `cadence-migration.test.ts` asserts the same through `loadModuleHealthCadences`.
+- **Carrying an existing fleet over** — `apps/celilo/src/services/alerting/cadence-migration.ts` (`migrateMonitorCadences`), run from `celilo system migrate` (the `.deb` postinst runs it on every apt upgrade). A monitor row whose cadence diverges from its manifest gets that cadence written as an override; a disabled one gets `manual`. Not bookkeeping: without it the upgrade that ships read-time resolution silently reverts every hand-set cadence to the author's suggestion and resumes watching modules an operator deliberately disabled. Idempotent — writes only where no override exists.
+- **Scheduled audit categories (`builtin_check` monitors)** — `apps/celilo/src/services/alerting/builtin-source.ts` — `SCHEDULABLE_BUILTIN_CHECKS` is the list of `celilo system audit` categories cheap enough to run every sweep: `machines_reachable`, `backups`, `disk_space`, `abandoned_operations`, and `public_dns` (`apps/celilo/src/services/audit/abandoned-operations.ts` — ≥3 abandonments of the same (module, operation) in 7d, the fingerprint of an operation being killed mid-flight). Everything else in the audit needs the whole world injected (proxmox, terraform, registry) and is not schedulable. Enable one with `celilo monitor add backups --interval 1h`, and re-cadence it later with `celilo monitor set-interval backups 6h` (in place, because the monitor id owns the alert history). `monitor set-interval`/`enable`/`disable` REFUSE a module target and name `celilo module config set <m> health_check_interval` — two ways to set one module's cadence would disagree about what `module status` shows. Targets are tab-completable — `completion.ts` reads `SCHEDULABLE_BUILTIN_CHECKS` directly rather than a hand-copied list, so a newly-schedulable check is completable immediately.
 - **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet.
 - **Public-DNS reachability check (the only check with an OFF-FLEET vantage)** — `apps/celilo/src/services/audit/public-dns.ts` (`auditPublicDns`, pure over an injected probe and the previous run's counters) + `apps/celilo/src/services/public-dns-probe.ts` (the probe) + `audit/public-dns-source.ts` (ledger names + the `public_dns_evidence` counters). Every other check in celilo looks from INSIDE, behind a split-horizon resolver that deliberately answers with an in-zone address — correct for its purpose, and why all of them reported healthy for the nine days of celilo#626. This one resolves every `dns_registrations` FQDN through an **off-fleet resolver** (`public_dns.resolver`, default `1.1.1.1`) and compares it against the address the fleet appears to come from per an independent **echo service** (`public_dns.echo_url`, default `https://api.ipify.org`). Three properties are load-bearing: `assertOffFleetResolver` REFUSES a resolver matching `dns.primary`/`dns.fallback` (a check that quietly used the fleet's resolver would pass forever — the original bug one layer up); the expectation never comes from the registrar's own response (self-agreement, and Namecheap returns `ErrCount 0` for `www` updates it does not apply); and a divergence is a finding only once it OUTLIVES the record's own TTL, measured from the last assert, or it would page on every ISP re-lease. Missing evidence is counted rather than read as success — one undetermined run is silent, N consecutive ones are their own finding (`public_dns_evidence`), which is the hole celilo-website's isitup.org probe demonstrated live. Codes: `public_dns_stale`, `public_dns_missing`, `public_dns_companion_unclaimed`, `public_dns_unverifiable`. Spec: `openspec/specs/public-dns-reachability/spec.md`.
 - **The sweep** — `apps/celilo/src/services/alerting/sweep-runner.ts` (`runSweep`) — the ordered pass that makes alerting run by itself: run due monitors → promote past-grace alerts → re-evaluate suppression → flush quiet-hours deferrals → notify. Driven by `celilo alerts sweep` on `timer.tick.5m`. Never throws for one bad monitor.
@@ -234,18 +240,32 @@ is currently wrong, and routes carry the message to a person's phone. Design:
 
 ## Backups
 
-Creation, scheduling and freshness. A module declares an `on_backup` hook and a
-`backup.schedule`; celilo runs it on that cadence and alerts when it stops.
+Creation, scheduling and freshness. A module declares an `on_backup` hook and
+SUGGESTS a `backup.schedule`; the operator's override decides; celilo runs it on
+the resolved cadence and alerts when it stops.
 
-- **Creation** — `apps/celilo/src/services/backup-create.ts` — `createModuleBackup` (invokes the module's `on_backup` hook into an encrypted envelope), `createSystemStateBackup` (celilo.db), `findBackupEligibleModules`, `isBackupDue`. Storage destinations: `backup-storage.ts`. Retention: `backup-retention.ts` (`pruneBackupsForModule`). Restore: `backup-restore.ts`.
-- **Cadence (one accessor)** — `apps/celilo/src/services/backup-schedule.ts` — `effectiveBackupSchedule(manifest)`. An absent `backup.schedule` means `daily`, NOT `manual`; opting out takes an explicit `manual`. Both the freshness audit and the backup sweep must read cadence through this one function, or a module can be alerted-on but never backed up.
+- **Creation** — `apps/celilo/src/services/backup-create.ts` — `createModuleBackup` (invokes the module's `on_backup` hook into an encrypted envelope), `createSystemStateBackup` (celilo.db), `findBackupEligibleModules` (returns each module's operator config alongside its manifest, because every caller has to resolve a policy out of the two together), `isBackupDue`. Storage destinations: `backup-storage.ts`. Restore: `backup-restore.ts`.
+- **Cadence (one accessor)** — `apps/celilo/src/services/backup-schedule.ts` — `effectiveBackupSchedule(manifest, override)`. The manifest SUGGESTS; the operator's `backup_schedule` row in `module_configs` decides; absent from both means `daily`, NOT `manual` (opting out takes an explicit `manual`). Resolution happens at READ time — nothing is materialised at install or deploy — so a corrected manifest reaches every install that has not overridden. The one-argument form was DELETED rather than kept as an overload (Rule 3.9): it would let an un-updated reader compile clean while silently ignoring overrides. Both the freshness audit and the backup sweep must read cadence through this one function, or a module can be alerted-on but never backed up.
+- **The cadence type** — `apps/celilo/src/services/cadence.ts` — `Cadence` (`{minutes}` | `'manual'`), `parseCadence` / `formatCadence` / `cadenceMs`, and `cadenceSchema({floorMinutes})`. One spelling set for every cadence in celilo: a named period (`hourly`/`daily`/`weekly`/`monthly`), a duration (`6h`, `90m`, `3d`), or `manual`. ⚠️ **The floors are DERIVED from the sweep ticks, never written down**: this file owns `BACKUP_SWEEP_PATTERN` / `ALERTING_SWEEP_PATTERN` and computes `BACKUP_CADENCE_FLOOR_MINUTES` / `MONITOR_INTERVAL_FLOOR_MINUTES` from them (the sweeps import their pattern from here), so changing a tick moves what it can serve in the same edit. A cadence finer than its sweep's tick is REFUSED, not coerced — accepting it leaves the operator believing they configured something that can silently never happen.
+- **Retention (one accessor)** — `apps/celilo/src/services/backup-retention.ts` — `effectiveBackupRetention(manifest, configs)`, `prunesNothing`, `identifyExpiredBackups`, `pruneBackupsForModule`. Two INDEPENDENT dimensions (copies, age), each resolved override → manifest → **unbounded**. ⚠️ **An unset dimension is unbounded, never a default bound.** `backup.retention` is an optional block, so a manifest omitting it prunes nothing at all and its inner `count: 7` / `max_age_days: 30` defaults never apply; if setting one dimension let the other fall back to those, an operator asking to keep 3 copies would silently arm a 30-day deletion on a module that had been keeping everything. Unbounded is `Infinity`, which `identifyExpiredBackups` needs no special case for. Gate: `services/backup-retention.test.ts`. The four sites that used to read `manifest.backup.retention` directly (`cli/commands/backup-sweep.ts`, `backup-create.ts`, `backup-prune.ts` twice) all go through the accessor — that duplication is what let the schedule readers drift.
 - **The sweep** — `apps/celilo/src/services/backup-sweep.ts` — `runBackupSweep` (the pass that makes backups run by themselves: reclaim orphaned staging → for each eligible module, is its declared cadence due → back it up → apply declared retention) + `ensureBackupSweepSubscriber`. Driven by `celilo backup sweep` on `timer.tick.1h` — the coarsest tick that can still serve an `hourly` cadence. Armed from BOTH `registerModuleSubscriptions` (any module declaring an `on_backup` hook, so it appears on install or `module update`) AND `celilo system migrate`, which the `.deb` postinst runs on every apt upgrade — registering only from module install/update meant a corrected budget never reached an existing fleet, since the row already exists and module updates can be weeks apart. A run refused by the in-flight operation lock is a skip retried next tick, never a failure.
 - **⚠️ The sweep's budget is stated, never inherited** — `BACKUP_SWEEP_TIMEOUT_MS` (4h) and `BACKUP_SWEEP_MAX_ATTEMPTS` (1) in `backup-sweep.ts`. The event bus defaults to `timeout_ms: 60000` / `max_attempts: 3`, and inheriting them made scheduled backups structurally impossible: one forgejo backup measured ~5.5 minutes (1.3 GB result, 3.9 GB peak staging), so the dispatcher SIGTERMed it at 60s — three times an hour, for days. The retry count is half the bug, not a detail: an impossible pass retried 3x strands 3x the staging (27 GB in 5.7 hours on celilo-mgr). The hourly tick IS the retry.
 - **Staging reclamation** — `apps/celilo/src/services/backup-staging.ts` — `reapOrphanedStaging`, `stagingDirFor` (the single source of truth for `/tmp/celilo-backup-<record.id>`, shared with `backup-create.ts` so writer and reaper cannot drift). `backup-create.ts` removes its staging in a `finally`, which is correct and NOT enough: a `finally` never runs when the process is killed by a signal — dispatcher timeout, OOM, Ctrl-C, reboot — and those strand the LARGEST directories. So reclamation is kill-mode agnostic by construction: it asks "is anyone still using this?", answered from the `backups.pid` column plus `isPidRunnable`, both of which outlive the process. A directory is removed only when its owner is provably gone (record absent, record terminal, pid dead, or past `STAGING_TTL_MS` = 6h — the TTL is the only check surviving pid reuse). A live backup is always kept; an unrecognised name is ignored, never deleted. Reclaiming a record that still claimed `in_progress` also marks it failed with `ABANDONED_BACKUP_MESSAGE`, so `celilo backup list` stops showing phantom in-flight backups and the `backups` drift check cannot read a dead attempt as a fresh backup.
-- **Freshness audit** — `apps/celilo/src/services/audit/backups.ts` (`auditBackups`) — `backup_missing` / `backup_stale` drift findings against the same declared cadence.
+- **Freshness audit** — `apps/celilo/src/services/audit/backups.ts` (`auditBackups`, `backupStaleThresholdMs`) — `backup_missing` / `backup_stale` drift findings against the same EFFECTIVE cadence, loaded (with each module's override) by `audit/backup-source.ts`. The stale threshold is a formula, `cadence + max(1h, cadence × 0.1)`, not a lookup table: a table cannot answer for `6h`. An effective cadence of `manual` short-circuits BEFORE the never-backed-up check — a module the operator opted out of used to be reported as missing a backup forever, remediated only by the thing they declined.
+- **⚠️ Both age measurements read `backups.completedAt`** — `loadBackupHistory` (`backup-metadata.ts`) and `latestSuccessfulBackupByModule` (`audit/backup-source.ts`). The run path used to measure from `startedAt` and the audit from `completedAt`, so the two disagreed by the duration of the backup itself. Gate: `services/backup-age-agreement.test.ts`.
 - **CLI** — `apps/celilo/src/cli/commands/` — `backup-sweep.ts`, `backup-create.ts` (also `celilo module backup`), `backup-list.ts`, `backup-restore.ts`, `backup-prune.ts`, `backup-delete.ts`, `backup-import.ts`, `backup-pull.ts`, `backup-name.ts`.
 - **Storage destinations CLI** — `storage-add-local.ts`, `storage-add-s3.ts`, `storage-list.ts`, `storage-verify.ts`, `storage-set-default.ts`, `storage-set-path.ts` (relocate a local destination, migrating existing archives unless `--no-migrate`), `storage-remove.ts`. Any credential change goes through `updateStorageCredentials` in `backup-storage.ts`, which clears the verification stamp — a `✓ Verified` must never describe a destination it was not measured against (#566).
 
+## Per-module operator policy
+
+How celilo TREATS a module — its backup cadence, its health-check cadence, its
+upgrade controls — as opposed to how the module configures itself.
+
+- **The keys** — `FRAMEWORK_CONFIG_KEYS` in `apps/celilo/src/cli/commands/module-config.ts` — keys EVERY module accepts whether or not its manifest declares them: `auto_upgrade`, `upgrade_policy`, `backup_schedule`, `health_check_interval`, `backup_retention_count`, `backup_retention_max_age_days`. Each carries `{schema, why}`: a Zod schema, and the reason a wrong value is REFUSED rather than coerced. That reason is per key on purpose — the substance is what a wrong value would silently do (a cadence typo falls back to the manifest's suggestion; an `upgrade_policy` typo falls back to `by-semver`, which skips the pre-deploy backup on a patch), and no validator knows that.
+- **Storage** — rows in `module_configs`, unique on `(module_id, key)`, cascading on module removal. Nothing deletes them on `module update`, and `backup-create.ts` archives them into backup envelopes while `backup-restore.ts` restores them — so an override survives update, upgrade and restore with no new table and no migration.
+- **Operator surface** — `celilo module config set|get|unset <module> <key> [value]`. `unset` (→ `deleteModuleConfig` in `services/module-config.ts`) is what returns a module to following its manifest; without it an operator who once set a value could never go back, and later manifest corrections would stop reaching them permanently. Unsetting an absent key reports that and SUCCEEDS — it states a desired end state. All three are MCP tools with no bespoke code, generated from `packages/core/src/command-registry.ts`.
+- **Display** — `formatCadencePolicy` in `apps/celilo/src/cli/commands/module-status.ts` shows the effective value AND its source, naming the manifest's suggestion when an override is in effect. The stored override alone does not say what it was changed from; the effective value alone does not say who chose it.
+
 ## Persistence
 
 - **DB schema** — `apps/celilo/src/db/schema.ts`. Client: `apps/celilo/src/db/client.ts`. Migration runner: `apps/celilo/src/db/migrate.ts`. Migrations: `apps/celilo/drizzle/`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "0.23.0",
+  "version": "0.24.0",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -56,11 +56,12 @@
     "check:schema": "bun run scripts/export-manifest-schema.ts --check"
   },
   "dependencies": {
-    "@aws-sdk/client-s3": "^3.1024.0",
-    "@celilo/capabilities": "^1.1.0",
+    "@aws-sdk/client-s3": "^3.1109.0",
+    "@aws-sdk/lib-storage": "^3.1101.0",
+    "@celilo/capabilities": "^1.2.0",
     "@celilo/cli-display": "^0.2.0",
-    "@celilo/core": "^0.7.0",
-    "@celilo/event-bus": "^0.4.0",
+    "@celilo/core": "^0.8.0",
+    "@celilo/event-bus": "^0.5.0",
     "ajv": "^8.18.0",
     "drizzle-orm": "^0.36.4",
     "ink": "^7.0.1",

package/src/cli/commands/alerts-act.ts

@@ -7,7 +7,6 @@
  */
 
 import { getDb } from '../../db/client';
-import { parseIntervalMinutes } from '../../manifest/schema';
 import {
   acknowledgeAlert,
   findLiveAlertByKey,
@@ -15,6 +14,7 @@ import {
   silenceAlert,
 } from '../../services/alerting/ack';
 import { findPerson, listPeople } from '../../services/alerting/people';
+import { parseIntervalMinutes } from '../../services/cadence';
 import type { CommandResult } from '../types';
 
 function resolveKey(key: string | undefined) {

package/src/cli/commands/backup-create.ts

@@ -10,7 +10,17 @@ import {
   isBackupDue,
 } from '../../services/backup-create';
 import { formatSize } from '../../services/backup-metadata';
-import { pruneBackupsForModule } from '../../services/backup-retention';
+import {
+  effectiveBackupRetention,
+  pruneBackupsForModule,
+  prunesNothing,
+} from '../../services/backup-retention';
+import {
+  BACKUP_SCHEDULE_CONFIG_KEY,
+  effectiveBackupSchedule,
+} from '../../services/backup-schedule';
+import { formatCadence } from '../../services/cadence';
+import { configOverride } from '../../services/module-config';
 import { celiloIntro, celiloOutro } from '../prompts';
 import type { CommandResult } from '../types';
 
@@ -56,12 +66,20 @@ export async function handleBackupCreate(
       if (eligible.length > 0) {
         console.log('\nChecking module backup schedules...');
 
-        for (const { module: mod, manifest } of eligible) {
-          const schedule = manifest.backup?.schedule ?? 'manual';
+        for (const { module: mod, manifest, configs } of eligible) {
+          // Through the shared accessor, like every other reader. This site
+          // used to read the manifest directly AND default an absent cadence
+          // to `manual` — the opposite of what the accessor has always said —
+          // so `celilo backup create` quietly skipped modules the hourly sweep
+          // was backing up daily.
+          const schedule = effectiveBackupSchedule(
+            manifest,
+            configOverride(configs, BACKUP_SCHEDULE_CONFIG_KEY),
+          );
           const due = force || isBackupDue(mod.id, schedule);
 
           if (!due) {
-            console.log(`  ${mod.id}: skipping (not due, schedule: ${schedule})`);
+            console.log(`  ${mod.id}: skipping (not due, schedule: ${formatCadence(schedule)})`);
             moduleSkipCount++;
             continue;
           }
@@ -77,13 +95,10 @@ export async function handleBackupCreate(
           console.log(`  → ${result.storagePath}`);
           moduleBackupCount++;
 
-          // Auto-prune per retention policy
-          const retention = manifest.backup?.retention;
-          if (retention) {
-            const pruneResult = await pruneBackupsForModule(mod.id, {
-              count: retention.count,
-              maxAgeDays: retention.max_age_days,
-            });
+          // Auto-prune per the effective retention policy
+          const policy = effectiveBackupRetention(manifest, configs);
+          if (!prunesNothing(policy)) {
+            const pruneResult = await pruneBackupsForModule(mod.id, policy);
             if (pruneResult.deleted > 0) {
               console.log(`  ✓ Pruned ${pruneResult.deleted} old backup(s)`);
             }

package/src/cli/commands/backup-list.test.ts

@@ -0,0 +1,83 @@
+/**
+ * The per-module recovery summary under `celilo backup list`.
+ *
+ * The listing itself is a log of attempts, and on celilo-mgr a column of
+ * `✗ 0 B` rows on a tidy hourly rhythm was true for a day while forgejo had no
+ * usable backup at all (celilo#685). These assert the two facts that column
+ * structurally cannot carry: `never`, and how much has failed since.
+ */
+
+import { describe, expect, test } from 'bun:test';
+import type { Backup } from '../../db/schema';
+import { backedUpModuleIds, lastSuccessSummaryLines } from './backup-list';
+
+const NOW = Date.UTC(2026, 7, 13, 12, 0, 0);
+const ago = (ms: number) => new Date(NOW - ms);
+const HOUR = 60 * 60 * 1000;
+
+function backup(partial: Partial<Backup>): Backup {
+  return { backupType: 'module_data', moduleId: 'forgejo', ...partial } as Backup;
+}
+
+describe('backedUpModuleIds', () => {
+  test('deduplicates and sorts the modules a listing names', () => {
+    expect(
+      backedUpModuleIds([
+        backup({ moduleId: 'forgejo' }),
+        backup({ moduleId: 'authentik' }),
+        backup({ moduleId: 'forgejo' }),
+      ]),
+    ).toEqual(['authentik', 'forgejo']);
+  });
+
+  test('ignores system backups, which have no module and no cadence', () => {
+    expect(
+      backedUpModuleIds([
+        backup({ backupType: 'system_state', moduleId: null }),
+        backup({ moduleId: 'forgejo' }),
+      ]),
+    ).toEqual(['forgejo']);
+  });
+});
+
+describe('lastSuccessSummaryLines', () => {
+  test('says never, out loud, when a module has never been captured', () => {
+    // `signal` on celilo-mgr. A listing of its attempts shows rows on a
+    // schedule; none of them is a backup.
+    const [line] = lastSuccessSummaryLines(
+      [{ moduleId: 'signal', lastSuccessAt: null, consecutiveFailures: 8 }],
+      NOW,
+    );
+    expect(line).toContain('signal');
+    expect(line).toContain('never');
+    expect(line).toContain('8 failed attempts since');
+  });
+
+  test('reports how stale a real success is, with the failures since it', () => {
+    const [line] = lastSuccessSummaryLines(
+      [{ moduleId: 'forgejo', lastSuccessAt: ago(7 * 24 * HOUR), consecutiveFailures: 20 }],
+      NOW,
+    );
+    // The existing relative formatter buckets 7 days as "last week".
+    expect(line).toContain('last week');
+    expect(line).toContain('20 failed attempts since');
+  });
+
+  test('a healthy module carries no failure clause at all', () => {
+    const [line] = lastSuccessSummaryLines(
+      [{ moduleId: 'authentik', lastSuccessAt: ago(3 * HOUR), consecutiveFailures: 0 }],
+      NOW,
+    );
+    expect(line).toContain('3h ago');
+    expect(line).not.toContain('failed');
+  });
+
+  test('counts one failure in the singular', () => {
+    const [line] = lastSuccessSummaryLines(
+      [{ moduleId: 'caddy', lastSuccessAt: ago(2 * HOUR), consecutiveFailures: 1 }],
+      NOW,
+    );
+    expect(line).toContain('1 failed attempt since');
+    expect(line).not.toContain('attempts');
+  });
+});

package/src/cli/commands/backup-list.ts

@@ -3,7 +3,13 @@
  * Lists available backups with metadata.
  */
 
-import { formatSize, getBackup, listBackups } from '../../services/backup-metadata';
+import type { Backup } from '../../db/schema';
+import {
+  formatSize,
+  getBackup,
+  listBackups,
+  loadBackupHistory,
+} from '../../services/backup-metadata';
 import { getBackupStorage } from '../../services/backup-storage';
 import { celiloIntro } from '../prompts';
 import type { CommandResult } from '../types';
@@ -12,8 +18,7 @@ import type { CommandResult } from '../types';
  * Format a timestamp as a human-friendly relative string.
  * Uses relative terms for recent backups and dates for older ones.
  */
-function formatRelativeDate(date: Date): string {
-  const now = Date.now();
+function formatRelativeDate(date: Date, now: number = Date.now()): string {
   const diffMs = now - date.getTime();
   const diffMins = Math.floor(diffMs / 60_000);
   const diffHours = Math.floor(diffMs / 3_600_000);
@@ -73,6 +78,64 @@ function showBackupDetail(backupId: string): CommandResult {
   return { success: true, message: `Backup detail: ${backupId}` };
 }
 
+/**
+ * Say, per module, when its data was last actually captured.
+ *
+ * The listing above is a log of ATTEMPTS, and an attempt log is the one thing
+ * that cannot answer the question an operator is really asking. On celilo-mgr
+ * the forgejo rows read:
+ *
+ *   ✗ 17cedba4  forgejo      0 B        1h ago
+ *   ✗ 99a9f819  forgejo      0 B        2h ago
+ *   ✗ fbb4d355  forgejo      0 B        3h ago
+ *
+ * — a tidy hourly cadence, every row on time, and not one byte of backup
+ * anywhere in it. Regular failure and regular success have the same silhouette
+ * at a glance; the `✗` is one character wide and the rhythm is what the eye
+ * reads. That listing was true for a day while production had no forgejo
+ * backup at all (celilo#685).
+ *
+ * So the recovery question gets answered directly rather than left to be
+ * inferred from a column of marks. `never` and a failure count are the two
+ * facts a column of attempts structurally cannot show.
+ *
+ * System backups are skipped: they are not per-module and have no cadence.
+ */
+function printLastSuccessSummary(backupList: Backup[]): void {
+  const moduleIds = backedUpModuleIds(backupList);
+  if (moduleIds.length === 0) return;
+
+  const lines = lastSuccessSummaryLines(
+    moduleIds.map((moduleId) => ({ moduleId, ...loadBackupHistory(moduleId) })),
+    Date.now(),
+  );
+  console.log('\nLast successful backup:');
+  for (const line of lines) console.log(line);
+}
+
+/** The modules named by a listing, deduplicated. System rows have no module. */
+export function backedUpModuleIds(backupList: Backup[]): string[] {
+  const ids = backupList
+    .filter((b) => b.backupType === 'module_data' && b.moduleId)
+    .map((b) => b.moduleId as string);
+  return [...new Set(ids)].sort();
+}
+
+/** One line per module: when it was last captured, and what has failed since. */
+export function lastSuccessSummaryLines(
+  entries: Array<{ moduleId: string; lastSuccessAt: Date | null; consecutiveFailures: number }>,
+  now: number,
+): string[] {
+  return entries.map(({ moduleId, lastSuccessAt, consecutiveFailures }) => {
+    const when = lastSuccessAt ? formatRelativeDate(lastSuccessAt, now) : 'never';
+    const since =
+      consecutiveFailures > 0
+        ? `  (${consecutiveFailures} failed attempt${consecutiveFailures === 1 ? '' : 's'} since)`
+        : '';
+    return `  ${moduleId.padEnd(20)} ${when}${since}`;
+  });
+}
+
 export async function handleBackupList(
   args: string[],
   flags: Record<string, boolean | string> = {},
@@ -120,6 +183,7 @@ export async function handleBackupList(
     }
 
     console.log(`\n${backupList.length} backup${backupList.length === 1 ? '' : 's'} shown.`);
+    printLastSuccessSummary(backupList);
     console.log('Run "celilo backup list <id>" for details.\n');
 
     return { success: true, message: `Found ${backupList.length} backup(s)` };

package/src/cli/commands/backup-prune.ts

@@ -8,7 +8,12 @@ import { getDb } from '../../db/client';
 import { modules } from '../../db/schema';
 import type { ModuleManifest } from '../../manifest/schema';
 import { findBackupEligibleModules } from '../../services/backup-create';
-import { pruneBackupsForModule } from '../../services/backup-retention';
+import {
+  effectiveBackupRetention,
+  pruneBackupsForModule,
+  prunesNothing,
+} from '../../services/backup-retention';
+import { loadModuleConfigs } from '../../services/module-config';
 import { celiloIntro, celiloOutro } from '../prompts';
 import type { CommandResult } from '../types';
 
@@ -37,17 +42,16 @@ export async function handleBackupPrune(
       }
 
       const manifest = mod.manifestData as unknown as ModuleManifest;
-      const retention = manifest.backup?.retention;
-      if (!retention) {
-        console.log(`Module '${specificModule}' has no retention policy defined.`);
+      const policy = effectiveBackupRetention(manifest, loadModuleConfigs(db, specificModule));
+      if (prunesNothing(policy)) {
+        console.log(
+          `Module '${specificModule}' has no retention policy — every backup is kept.\n` +
+            `Set one with: celilo module config set ${specificModule} backup_retention_count <n>`,
+        );
         return { success: true, message: 'No retention policy' };
       }
 
-      const result = await pruneBackupsForModule(
-        specificModule,
-        { count: retention.count, maxAgeDays: retention.max_age_days },
-        dryRun,
-      );
+      const result = await pruneBackupsForModule(specificModule, policy, dryRun);
 
       if (result.deleted > 0) {
         const verb = dryRun ? 'Would delete' : 'Deleted';
@@ -63,15 +67,11 @@ export async function handleBackupPrune(
       // Prune all modules with retention policies
       const eligible = findBackupEligibleModules();
 
-      for (const { module: mod, manifest } of eligible) {
-        const retention = manifest.backup?.retention;
-        if (!retention) continue;
+      for (const { module: mod, manifest, configs } of eligible) {
+        const policy = effectiveBackupRetention(manifest, configs);
+        if (prunesNothing(policy)) continue;
 
-        const result = await pruneBackupsForModule(
-          mod.id,
-          { count: retention.count, maxAgeDays: retention.max_age_days },
-          dryRun,
-        );
+        const result = await pruneBackupsForModule(mod.id, policy, dryRun);
 
         if (result.deleted > 0) {
           const verb = dryRun ? 'Would delete' : 'Deleted';

package/src/cli/commands/backup-sweep.ts

@@ -18,13 +18,19 @@ import {
   isBackupDue,
 } from '../../services/backup-create';
 import { failBackup, getBackup, listInProgressBackups } from '../../services/backup-metadata';
-import { pruneBackupsForModule } from '../../services/backup-retention';
+import {
+  effectiveBackupRetention,
+  pruneBackupsForModule,
+  prunesNothing,
+} from '../../services/backup-retention';
+import { BACKUP_SCHEDULE_CONFIG_KEY } from '../../services/backup-schedule';
 import {
   STAGING_PREFIX,
   reapOrphanedStaging,
   resolveAbandonedBackups,
 } from '../../services/backup-staging';
 import { type BackupSweepReport, runBackupSweep } from '../../services/backup-sweep';
+import { configOverride } from '../../services/module-config';
 import { isPidRunnable } from '../../services/module-operations';
 import type { CommandResult } from '../types';
 
@@ -79,20 +85,26 @@ function resolveAbandonedRecords() {
 }
 
 export async function handleBackupSweep(): Promise<CommandResult> {
+  // Read once, so the cadence the sweep schedules on and the retention it
+  // prunes with come from the same snapshot of the operator's config.
+  const eligible = findBackupEligibleModules();
+  const configsByModule = new Map(eligible.map(({ module, configs }) => [module.id, configs]));
+
   const report = await runBackupSweep({
     reapStaging,
     resolveAbandonedRecords,
     listEligible: () =>
-      findBackupEligibleModules().map(({ module, manifest }) => ({ id: module.id, manifest })),
+      eligible.map(({ module, manifest, configs }) => ({
+        id: module.id,
+        manifest,
+        scheduleOverride: configOverride(configs, BACKUP_SCHEDULE_CONFIG_KEY),
+      })),
     isDue: (moduleId, schedule) => isBackupDue(moduleId, schedule),
     backup: (moduleId) => createModuleBackup(moduleId),
     prune: async ({ id, manifest }) => {
-      const retention = manifest.backup?.retention;
-      if (!retention) return;
-      await pruneBackupsForModule(id, {
-        count: retention.count,
-        maxAgeDays: retention.max_age_days,
-      });
+      const policy = effectiveBackupRetention(manifest, configsByModule.get(id));
+      if (prunesNothing(policy)) return;
+      await pruneBackupsForModule(id, policy);
     },
   });
 

package/src/cli/commands/firewall-interface-list.test.ts

@@ -0,0 +1,85 @@
+/**
+ * §8.2 — the classification, on demand.
+ *
+ * The value of this command is entirely in what it SAYS, so that is what these
+ * assert: the role of every interface, and — for anything unaccounted for —
+ * what the next converge will actually do about it. "1 alien interface" with no
+ * further guidance would repeat the failure this whole change is about, which
+ * was never a lack of information but a lack of celilo saying it.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+let testDir: string;
+let logs: string[];
+const originalLog = console.log;
+
+/** Capture stdout — the command's output IS its contract. */
+function captureLogs() {
+  logs = [];
+  console.log = (...args: unknown[]) => {
+    logs.push(args.map(String).join(' '));
+  };
+}
+
+beforeEach(() => {
+  testDir = mkdtempSync(join(tmpdir(), 'celilo-fw-iface-'));
+  process.env.CELILO_DB_PATH = join(testDir, 'test.db');
+  process.env.CELILO_DATA_DIR = testDir;
+  captureLogs();
+});
+
+afterEach(() => {
+  console.log = originalLog;
+  rmSync(testDir, { recursive: true, force: true });
+  process.env.CELILO_DB_PATH = undefined;
+});
+
+describe('celilo firewall interface list', () => {
+  test('an empty pool says so, and says what makes a machine a firewall', async () => {
+    const { handleFirewallInterfaceList } = await import('./firewall-interface-list');
+    const result = await handleFirewallInterfaceList([], {});
+
+    expect(result.success).toBe(false);
+    if (result.success) throw new Error('unreachable — asserted above');
+    expect(result.error).toContain('No firewalls');
+    // Not a dead end: it explains the rule rather than leaving the operator to
+    // guess why their box is not listed.
+    expect(result.error).toContain('router');
+  });
+
+  test('a named machine that does not exist names it, and points at machine list', async () => {
+    const { handleFirewallInterfaceList } = await import('./firewall-interface-list');
+    const result = await handleFirewallInterfaceList(['no-such-box'], {});
+
+    expect(result.success).toBe(false);
+    if (result.success) throw new Error('unreachable — asserted above');
+    expect(result.error).toContain('no-such-box');
+    expect(result.error).toContain('celilo machine list');
+  });
+});
+
+/**
+ * The classification itself is `@celilo/capabilities`' job and is tested there
+ * against far more cases than a CLI test should duplicate.
+ *
+ * What is worth pinning here is STRUCTURAL. Asserting the wording of the output
+ * by grepping this command's own source would just be reading my strings back to
+ * me — it would pass no matter what they said. The property below is different:
+ * it is a claim about what the command may DO, and it can genuinely fail.
+ */
+describe('read-only by construction', () => {
+  test('never touches the box', async () => {
+    // Safe to run against a firewall whose converge is currently refusing —
+    // which is exactly when an operator reaches for it. If this ever grows an
+    // SSH call it is a different command and should be named like one.
+    const src = await Bun.file(join(import.meta.dir, 'firewall-interface-list.ts')).text();
+
+    expect(src).not.toContain('runAppCommand');
+    expect(src).not.toContain('execFileSync');
+    expect(src).not.toContain('ssh ');
+  });
+});