@celilo/cli 0.27.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CELILO_CORE_MODULES.md +16 -0
- package/CELILO_SUBSYSTEMS.md +5 -2
- package/drizzle/0025_port_forward_owner.sql +29 -0
- package/drizzle/meta/_journal.json +7 -0
- package/package.json +2 -2
- package/src/cli/commands/module-remove.ts +26 -23
- package/src/cli/commands/system-audit.ts +5 -1
- package/src/cli/commands/system-update.ts +10 -2
- package/src/db/schema.ts +24 -6
- package/src/hooks/capability-loader.ts +59 -13
- package/src/hooks/define-hook.test.ts +0 -6
- package/src/hooks/executor.ts +2 -1
- package/src/hooks/types.ts +9 -17
- package/src/manifest/contracts/index.ts +20 -0
- package/src/manifest/contracts/v1.ts +33 -1
- package/src/manifest/schema.ts +48 -58
- package/src/services/audit/undeployed-modules.ts +18 -1
- package/src/services/consumer-cleanup.test.ts +347 -0
- package/src/services/consumer-cleanup.ts +244 -0
- package/src/services/module-validator/index.test.ts +9 -0
- package/src/services/port-forwards.test.ts +93 -40
- package/src/services/port-forwards.ts +74 -48
- package/src/services/trusted-sources.test.ts +52 -13
- package/src/services/trusted-sources.ts +25 -15
- package/src/templates/generator.ts +46 -28
- package/src/templates/{dns-ingress-ip.test.ts → ingress-ip.test.ts} +38 -22
- package/src/test-utils/cli-context.ts +15 -2
- package/src/services/web-route-cleanup.test.ts +0 -250
- package/src/services/web-route-cleanup.ts +0 -144
package/CELILO_CORE_MODULES.md
CHANGED
|
@@ -19,6 +19,19 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
|
|
|
19
19
|
> (**knot-unbound-internal** / **technitium**). Identity comes from **authentik** (`idp`). That's
|
|
20
20
|
> the whole edge: DNS + firewall + ingress + identity, each a swappable provider module.
|
|
21
21
|
|
|
22
|
+
> How the graph OPENS again: when a consumer is removed, celilo dispatches
|
|
23
|
+
> `on_consumer_removed` to every provider of every capability that consumer
|
|
24
|
+
> declared, and each withdraws what it minted on its behalf
|
|
25
|
+
> (openspec/changes/consumer-removal-cleanup). Providers implementing it:
|
|
26
|
+
> **caddy** and **caddy-internal** (drop the routes, reclaim `/srv/www/<slug>`),
|
|
27
|
+
> **iptables** / **greenwave** / **axon** (drop the forwards and trusted
|
|
28
|
+
> sources), **authentik** (delete the OIDC application), **forgejo**
|
|
29
|
+
> (deregister the CI runner), **generic-cpanel-hosting-provider** (remove the
|
|
30
|
+
> published docroot). Consumers do NOT withdraw their own state any more — a
|
|
31
|
+
> module gives a port or a subnet back by narrowing its own declaration, and
|
|
32
|
+
> departing entirely is the provider's job. A provider that fails to withdraw
|
|
33
|
+
> is marked ERROR; the removal still completes.
|
|
34
|
+
|
|
22
35
|
## Network fabric (DNS / firewall / DHCP)
|
|
23
36
|
|
|
24
37
|
- **axon** — Axon Networks Q1000K ISP router driver (Brightspeed-branded); port-forwarding + public-IP discovery + DHCP DNS + DHCP address-pool bounds via the TR-181 CGI API. **provides:** `firewall`, `dhcp_server`. Fork of **greenwave** — identical protocol, differing only in the vendor extension prefix (`X_AXON_` vs `X_GWS_`/`X_LANTIQ_COM_`). Pick by device: Q1000K → **axon**, C4000XG → **greenwave**. Two device behaviours shape this driver and are not shared by **greenwave** (celilo#739): the router **regenerates** `Pool.1.DNSServers` from its own upstream resolver list every ~15 min and on any config commit, so `setDhcpDns` also writes `Device.DNS.X_AXON_CustomServer` — the *input* to that computation, which survives — and **technitium re-asserts** the pool value on `timer.tick.1m`. Optional `dhcp_pool_start`/`dhcp_pool_end` bound the leased range; set them below the addresses IPAM allocates, because **IPAM has no knowledge of the router's DHCP pool** and nothing otherwise stops the router leasing an address celilo already assigned.
|
|
@@ -28,10 +41,12 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
|
|
|
28
41
|
- **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
|
|
29
42
|
- **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). A caller supplies a NAME and nothing else: the address is the source IP of celilo's own update, re-derived on every assert. Registering `<domain>` also claims `www.<domain>` and vice versa (best effort, reported back as `outputs.companion_fqdn` so the framework's `public_dns` check watches it — Namecheap answers `ErrCount 0` for `www` updates it does not apply). DDNS passwords are keyed by the **registrable domain**, never the FQDN. **provides:** `dns_registrar`.
|
|
30
43
|
- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. **REQUIRES the `control-plane-vpn` network and READS its range** (`requires.networks`; `client_subnet` is a `source: system` derive of `network.control-plane-vpn.subnet`). It does not write that network and has no way to — celilo owns the namespace, and the deploy will not reach any hook until the network is defined, asking for a range if one is missing. So `wg0` is attributable the moment it exists. This replaces a declare-before-you-create ordering inside `on_install`, which could only narrow the window and not close it: a consumer that captured config before the hook started could not see a value the hook wrote, whatever order it wrote it in, and that is exactly what left `wg0` unattributable (celilo#759). `health_check` still asserts the declaration matches what the tunnel serves — what it catches now is divergence, celilo's network having changed since the module resolved its config. That same key is what the internal resolver's split-horizon view consumes. Adopts a running tunnel in place (existing key retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **Adoption is a one-time IMPORT and it ENDS.** The marker is `registered_peers` — celilo's own config key, separate from the operator's `peers` so a machine can never rewrite what an operator typed. While that key has never been written the tunnel is not yet celilo's; the deploy that writes it (even as `[]`) CLAIMS the tunnel, importing whatever `[Peer]` blocks are running, and from then on the render is closed-world (`peers` ∪ `registered_peers`) with a `[Peer]` on the box in neither reported as drift by `on_install` and by `health_check`'s `unknown_peers`. An ABSENT `peers` declaration and a declared-EMPTY one are deliberately different (no `default: []` on either variable): they used to be the same value, so the module read every empty list as "adopt what is running", the last peer could not be revoked, and a hand-added peer rode along on every deploy unreported (celilo#765). On a tunnel celilo has not yet claimed, a declared-empty `peers` is REFUSED rather than obeyed — celilo's own variable-default seeding wrote a real `peers = []` row for every install of the previous version that never set one, so on the installed base an empty list cannot be told from a stored default, and obeying it would `wg syncconf` the admin tunnel down to zero peers. **PROVIDES `control_plane_vpn`** so another module can enrol clients without an operator editing YAML: `registerClient` / `revokeClient` write to `registered_peers` — the same key the adoption claim uses — and `getEndpoint` reads the server's public key LIVE off the host rather than storing a copy that could be republished after the key changed. `client_pool` is the range a consumer may allocate from, a strict subset of the client subnet, and `validate_config` REFUSES any operator-declared peer inside it: the two allocators cannot see each other, so the range is divided rather than negotiated, and that check is the only place a bad division is caught. ⚠️ Granting this capability grants fleet-wide reach — the tunnel's client subnet is a registered trusted source, so every client enrolled through it reaches every managed zone. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
|
|
44
|
+
- **wireguard-manager** — self-service enrolment for the control-plane VPN, so adding an administrator's laptop is a web page rather than an operator editing `peers` by hand. A browser generates the WireGuard keypair with WebCrypto and **the private key never leaves it** — celilo never sees it and the app never stores it. The app allocates an address from `client_pool` (the range `wireguard` reserved for a consumer; celilo's IPAM deliberately does not cover VPN clients) and enrols the peer through **`control_plane_vpn`**, so it never edits wireguard's config and cannot reach a peer the operator declared by hand. Its own device list is the app's SQLite, reconciled onto the tunnel by the `reconcile_clients` hook on a `timer.tick.1m` subscription. Two guards make a failed poll harmless and both are structural rather than a check: `planReconcile` accepts only the `read` variant of a discriminated union, so an unreadable poll cannot reach it, and revocation is TOMBSTONE-driven — a device is revoked because the app says it was, never because it is missing from a list — so a short or empty read can only fail to add someone, never cut anyone off. A peer on the tunnel the app does not know and has not tombstoned is reported as an orphan and LEFT ALONE (usually an app restored from an older backup). **REQUIRES `private_web`, never `public_web`**: there is no public record, no ACME certificate and no port forward, and the internal CA's certificate is bundled with each downloaded config so a freshly enrolled device trusts the manager it just connected to. ⚠️ **Everyone enrolled becomes a fleet administrator** — the tunnel's client subnet is a registered trusted source, so the `admin_group` gate on the idp application is the entire access-control story; per-peer narrowing is not possible, because server-side `AllowedIPs` is crypto-key routing rather than access control. Lives in **`app`**, one tier behind `caddy-internal` in `dmz`, and is never reached directly (celilo#879). Enrolment works from INSIDE the fleet only (a device not yet on the VPN cannot reach a manager reachable only over it) — from the LAN through the ingress IP, from a segmented zone at the ingress's dmz address, or over an existing VPN session, and a revocation takes effect at the next reconcile — that interval IS the window in which a revoked device still has reach, which is why the UI shows it pending and why `celilo module run-hook wireguard-manager reconcile_clients` exists. **requires:** `control_plane_vpn` (wireguard), `idp` (authentik), `private_web` (caddy-internal).
|
|
31
45
|
|
|
32
46
|
## Public edge (ingress / identity)
|
|
33
47
|
|
|
34
48
|
- **caddy** — reverse proxy with automatic HTTPS (HTTP-01 ACME); the standard HTTPS ingress. Emits a 301 redirect block for each served name's `www`/apex companion (kept out of the served-hostname set, so a companion never blocks the ACME wait). Its `health_check`'s DNS item is `internal_dns_resolution` — it digs from the management host behind the split-horizon resolver and is evidence about the in-fleet view only; public reachability is the framework's `public_dns` check. **provides:** `public_web`. **requires:** `dns_registrar`, `firewall`.
|
|
49
|
+
- **caddy-internal** — a SECOND Caddy that serves the fleet and nothing else, and is a sibling of **caddy** rather than a mode of it. It never exposes a port on the firewall's EXTERNAL interface, never publishes a public record, and takes its certificates from Caddy's own local CA (`tls internal`) — those absences ARE the capability, and `public_web` cannot express them because it treats an unreachable route as a deploy failure and publishes a public A record to prevent one (design D10). Lives in the **`dmz`** zone, in front of the things it fronts, mirroring the public `caddy` (celilo#879). LAN devices reach it through an IPAM-allocated `internal`-subnet `ingress_ip` that `on_install` passes to `firewall.exposeService({ ingressIp })` — one DNAT on the firewall's INTERNAL side and nothing external, the same mechanism the dmz-resident `dns_internal` resolver has used for `:53` since ISS-0156. Systems already inside `dmz`/`app`/`secure` use its dmz address instead (carried as `zoneRoutableValue` on the internal record), and VPN clients arrive as a registered trusted source. It previously lived in `internal` on the claim that a dmz ingress could not be reached from a LAN without a public port-forward; that conflated a public forward with an internal-side ingress IP, and cost it the ability to serve a browser inside a segmented zone at all. **`getCaCertificate()` is why the contract has a method `public_web` does not** — an internally-issued cert means clients must trust a CA celilo runs, and here that is nearly free because the people who must trust it are the ones who just downloaded a bundle from the service behind it, so the anchor ships with it. **The route table is this module's OWN config (`routes`), never celilo's `web_routes`** (celilo#846): caddy derives its served hostnames from every row of that table, so a private route stored there would be picked up and served PUBLICLY — storage is the privacy boundary, not policy. Route changes reconcile synchronously in the capability call (there is no `routes_changed` event for private routes, and inventing one would only add delay). A consumer cannot register a route on a hostname the ingress is not already configured for: a capability factory gets no capabilities of its own, so it cannot add the internal DNS record a new name would need, and it refuses rather than serving an unresolvable site block. **provides:** `private_web`. **requires:** `dns_internal` — declared under `optional.capabilities` ONLY to route around celilo#854 (the import-time secret gate refuses a consumer over a secret it never reads); `on_install` throws without it.
|
|
35
50
|
- **generic-cpanel-hosting-provider** — publishes static sites into a subfolder of a domain on a cPanel/SSH web host celilo does **not** govern (the host owns the domain, DNS, TLS and web server). Systemless like **namecheap** — no `requires.system`, no zone, no IPAM; it holds credentials for an external party. One provider serves many accounts, resolved by hostname; onboarding takes the account password once, `ssh-copy-id`s celilo's key, then discards it. **provides:** `external_web`.
|
|
36
51
|
- **authentik** — Authentik identity provider with OIDC (Docker Compose: server, worker, Postgres, Redis). **provides:** `idp`. **requires:** `public_web`, `dns_registrar`, `firewall`.
|
|
37
52
|
|
|
@@ -57,6 +72,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
|
|
|
57
72
|
## E2E fixtures & probes (not production apps)
|
|
58
73
|
|
|
59
74
|
- **hello-foo** / **hello-bar** — hostless single-page sites (one hand-written `site/index.html`, no build step) published at a configurable `domain` + `path`. They exist so the caddy route-lifecycle e2e can put several independent routes on one caddy and watch them interact — celilo-website is the only other static-site module, and one site can't answer "does removing `/` break `/foo`?". **requires:** `public_web`, `dns_registrar`.
|
|
75
|
+
- **hello-private-foo** / **hello-private-bar** — the `private_web` twins of the pair above: hostless single-page sites published through the fleet-only ingress. Two of them for the same reason — one consumer cannot answer "does withdrawing `/foo` leave `/bar` alone?", and per-consumer scoping is exactly what `unregisterRoutes()` promises. Their hook is `private_web ?? public_web` with no branching after it, which is the spec's "same consumer code, either ingress" requirement made executable. **hello-private-foo** additionally calls `getCaCertificate()` and writes the PEM into its own site before publishing — the CA-bundling pattern a real consumer uses, and the only way to observe that the capability returned something usable. **requires:** `private_web`.
|
|
60
76
|
- **source-forge-probe** — hostless e2e probe that exercises the `source_forge` capability (issue/revoke runner token) against a deployed Forgejo. **requires:** `source_forge`.
|
|
61
77
|
- **test-ansible-output** — e2e fixture that runs 25×1s sleep tasks to assert Ansible output streams in real time. Not a real app.
|
|
62
78
|
- **vm-smoke** — throwaway module proving celilo's generic `type: vm` deploy path end-to-end (template clone → cloud-init → Ansible reach). No capabilities.
|
package/CELILO_SUBSYSTEMS.md
CHANGED
|
@@ -24,6 +24,7 @@ see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MOD
|
|
|
24
24
|
## Allocation & infrastructure selection
|
|
25
25
|
|
|
26
26
|
- **IPAM (IP/VMID allocation)** — `apps/celilo/src/ipam/allocator.ts` — `allocateIPFromSubnet`, `allocateVMID`, `reserveIP`/`unreserveIP`, `inferZoneFromIP`, `getAllocation`. Auto-wrapper: `apps/celilo/src/ipam/auto-allocator.ts` — `allocateForModule` / `deallocateForModule`.
|
|
27
|
+
- **Ingress IPs (how the LAN reaches a service in a segmented zone)** — `ensureIngressIps` in `apps/celilo/src/templates/generator.ts`, called from `generateTemplates`. A module opts in by declaring an infrastructure variable whose name ENDS IN `ingress_ip` (`dns_ingress_ip` on the resolver, `ingress_ip` on `caddy-internal`); generate allocates and RESERVES a free `internal`-subnet address once and reuses it forever after, and the module's `on_install` passes it to `firewall.exposeService({ ingressIp })`. That renders ONE DNAT on the firewall's internal side and nothing on the external interface — it is not a public port-forward, and conflating the two is what pinned `caddy-internal` into the `internal` zone until celilo#879. Idempotence is load-bearing: re-allocating on a later generate silently moves the address clients use, with every command still reporting success (`apps/celilo/src/templates/ingress-ip.test.ts`).
|
|
27
28
|
- **Infrastructure selection (container-service vs machine pool)** — `apps/celilo/src/services/machine-pool.ts` (`getMachineByHostname`, `addMachine`, `assignModuleToMachine`) and `apps/celilo/src/services/container-service.ts` (`getContainerServiceByName`, `addContainerService`, `verifyContainerService`). Provider API clients: `apps/celilo/src/api-clients/proxmox.ts`, `apps/celilo/src/api-clients/digitalocean.ts`.
|
|
28
29
|
- **Zone detection / system config** — `apps/celilo/src/services/zone-detector.ts` — `detectZoneFromIp` reads `network.<zone>.subnet` from the `systemConfig` table and returns `NetworkZone | 'unknown'`. It answers CONTAINMENT ONLY. It used to return `'external'` on no-match, which conflated "no declared subnet contains this" with "the internet can route to this" — on a firewall with five RFC1918 legs that reported four of them as facing the internet. `'unknown'` is the honest answer; the caller resolves it (see `machine add`: publicly routable → `external`, otherwise fail asking for `--zone`). The subnet-backed zone list is derived from `NETWORK_ZONES` minus `external`, which has no subnet and must never be given one.
|
|
29
30
|
- **Interface classification** — `packages/capabilities/src/interface-classification.ts` — THE shared classifier, used by the backend and every firewall provider module so the two cannot drift apart again. `isPubliclyRoutable(ip)` is a property of the address alone (false for RFC 1918, RFC 6598 carrier-grade NAT, loopback, link-local, multicast, reserved). `classifyInterfaces(interfaces, zones)` assigns each interface `zone → external → alien`, first match winning, where `external` is the RESIDUAL — routable and claimed by no declared zone — and is never subnet-matched. `externalEdge()` returns none/single/**ambiguous** rather than silently picking the first public address. `defaultRouteFinding()` enforces the invariant that the default route leaves through `internal` or `external`. **`subnetContains(cidr, ip)` lives here and is the ONLY implementation** — three existed and disagreed (the backend's mishandled `/0`); the other two are deleted, not aliased. Design: `openspec/changes/firewall-interface-classification/design.md`.
|
|
@@ -49,7 +50,7 @@ see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MOD
|
|
|
49
50
|
|
|
50
51
|
| capability | impl | provider module(s) |
|
|
51
52
|
|---|---|---|
|
|
52
|
-
| `public_web` | `packages/capabilities/src/public-web.ts` (`createPublicWeb`). **Route lifecycle** is framework-owned at both ends, not per-module: removing a consumer
|
|
53
|
+
| `public_web` | `packages/capabilities/src/public-web.ts` (`createPublicWeb`). **Route lifecycle** is framework-owned at both ends, not per-module: removing a consumer dispatches caddy's `on_consumer_removed` hook (re-render the Caddyfile without that module's routes → reclaim `/srv/www/<slug>`) via the generic **consumer-removal cleanup** below, whether or not the module has an `on_uninstall`; deploying a provider runs `apps/celilo/src/services/public-web-republish.ts` (re-run every static consumer's `on_install`) so a provider rebuild refills the web roots it destroyed. There is no consumer-facing `unregister_routes` — withdrawal is the provider's, by design (openspec/changes/consumer-removal-cleanup D11). | caddy |
|
|
53
54
|
| `idp` | `packages/capabilities/src/idp.ts` | authentik |
|
|
54
55
|
| `dns_registrar` | `packages/capabilities/src/dns-registrar.ts` (`registerHost({ fqdn })` — **there is no way to supply an address**; the provider re-derives it from its own update's source IP on every assert) | namecheap |
|
|
55
56
|
| `external_web` | `packages/capabilities/src/external-web.ts` (`publishStaticSite`). **OFF-FLEET static publishing** — the counterpart to `public_web`. See the in-fleet vs off-fleet note below. | generic-cpanel-hosting-provider |
|
|
@@ -61,7 +62,7 @@ see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MOD
|
|
|
61
62
|
| `notification` | `packages/capabilities/src/notification.ts` (`send`, optional `receive`) | signal (planned) — transports are ordinary modules, both self-hosted and credential-only |
|
|
62
63
|
| `cross_module_read` | `packages/capabilities/src/cross-module-read.ts` | framework (read other modules' capability data) |
|
|
63
64
|
| `control_plane_vpn` | `packages/capabilities/src/control-plane-vpn.ts` (`registerClient`, `revokeClient`, `listClients`, `getEndpoint`). **Its state is NOT in a celilo table** — registrations live in the PROVIDER's own module config under `registered_peers`, kept separate from the operator's `peers` so a machine can never rewrite what an operator typed, and so `revokeClient` structurally cannot reach a declared peer. Do not go looking for a `vpn_clients` table; there isn't one and that is deliberate. Addresses are allocated by the CONSUMER from `client_pool` — celilo's IPAM deliberately does not cover VPN clients. ⚠️ Every client registered here reaches every managed zone: the tunnel's client subnet is a registered trusted source. | wireguard |
|
|
64
|
-
| `private_web` | `packages/capabilities/src/private-web.ts` (`publishStaticSite`, `registerReverseProxy`, `unregisterRoutes`, `getCaCertificate`). **Fleet-only HTTP ingress** — internal DNS, an internally-issued certificate, and no public exposure. A SIBLING of `public_web` rather than a flag on it, because `public_web` treats an unreachable route as a deploy failure and publishes a public record to prevent one. Reuses `public_web`'s request types so a consumer can write `private_web ?? public_web` and call through the union without branching.
|
|
65
|
+
| `private_web` | `packages/capabilities/src/private-web.ts` (`publishStaticSite`, `registerReverseProxy`, `unregisterRoutes`, `getCaCertificate`). **Fleet-only HTTP ingress** — internal DNS, an internally-issued certificate, and no public exposure. A SIBLING of `public_web` rather than a flag on it, because `public_web` treats an unreachable route as a deploy failure and publishes a public record to prevent one. Reuses `public_web`'s request types so a consumer can write `private_web ?? public_web` and call through the union without branching. **MODULE-provided**, unlike `public_web`: the implementation is `modules/caddy-internal/scripts/private-web-functions.ts` (wired via `CAPABILITY_MODULE_MAP`), and the route table lives in that module's own config rather than celilo's `web_routes` — a private route in `web_routes` would be picked up and served by the PUBLIC caddy, which derives its served hostnames from every row (celilo#846). | caddy-internal |
|
|
65
66
|
|
|
66
67
|
### In-fleet (`public_web`) vs off-fleet (`external_web`)
|
|
67
68
|
|
|
@@ -179,6 +180,8 @@ runner seam (`execRunner` real / `createMockRunner` for tests) lives in
|
|
|
179
180
|
- **In-flight operation lock** — `apps/celilo/src/services/module-operations.ts` — `startOperation`/`completeOperation`/`failOperation` record deploy/uninstall/backup/restore in `module_operations`; `refuseIfInFlight`/`checkInFlight` are what backup and restore consult. Deploy and uninstall REGISTER but never check: it is a one-way guard protecting backup/restore consistency, not a general mutex (`openspec/specs/management-server-backup/spec.md` "In-flight operation refusal"). A row stops holding the lock once it is GONE, STOPPED/zombie (`isPidRunnable`, `ps -o state=` — `kill(pid,0)` calls a Ctrl-Z'd process alive), or older than `OPERATION_TTL_MS` (2h). The TTL is not redundancy: a pid is a recycled number, and once the pid space wraps an old row names an unrelated healthy process. Operator surface: `celilo module operations [list|clear] [--abandoned] [--all]` (`apps/celilo/src/cli/commands/module-operations.ts`); `list` shows only what holds the lock, abandoned rows are summarised unless `--abandoned`. Abandoned rows are reclaimed hourly by the `celilo-operations-sweep` bus subscriber (`timer.tick.1h` → `celilo module operations clear`, armed by `ensureOperationsSweepSubscriber` from module registration and `celilo system migrate`). `clear` MARKS rows failed rather than deleting them, and that is load-bearing: the `abandoned_operations` audit reads exactly those released rows to notice one module's operation dying over and over.
|
|
180
181
|
- **Module pause / unpause (control-plane quiescence)** — `apps/celilo/src/services/module-pause.ts` — pure `planPause`/`planUnpause` producing an ordered plan, `executePause`/`executeUnpause` performing it, plus `listPausedModules`/`pausedAmong`/`formatPausedDuration`/`describePausedModule` (the ONE place an age is formatted). CLI: `apps/celilo/src/cli/commands/module-pause.ts` (`celilo module pause|unpause <id> [--cascade] [--stop-infra] [--reason] [--dry-run] [--yes]`). Pausing takes a module out of the CONTROL plane — no dispatched events, no timer hooks, no health checks, alerts suppressed — while leaving the DATA plane running, because capability consumption is deploy-time: every consumer calls `firewall`/`dhcp_server` from `on_install` and nothing calls it while serving. Config, secrets, IPAM/VMID and placement are preserved; `on_uninstall` does NOT run. Quiescence is enforced in two places: pausing drops the module's bus subscriptions (`unregisterModuleSubscriptions`), and `run-named-hook.ts` refuses any non-lifecycle hook for a PAUSED module (`skippedPaused`), which catches the paths that skip the bus — `events resync-subscriptions`, a restore that starts events.db empty, aspect fan-out, public-web republish. `on_install`/`on_uninstall` are exempt by hook NAME (not a caller flag): unpause redeploys through `on_install`, and removing a paused provider needs `on_uninstall`. Unpause always REDEPLOYS (`deployModule`) — that is what rebinds a consumer to a replacement provider and recreates provider-local state from the consumers that own it — and re-registers subscriptions, which a plain deploy does not do. A failed unpause restores `PAUSED` rather than leaving the module live and mis-bound. Cascade order reuses `services/update/dep-graph.ts` unchanged (pause = consumers first, unpause = providers first) and is computed from the GRAPH, never from which modules are currently paused, so a cascade walks THROUGH already-done members and is resumable. See `openspec/changes/module-pause-lifecycle/`.
|
|
181
182
|
- **Provider-removal guard** — `apps/celilo/src/services/remove-guard.ts` — `findRemovalBlockers`/`describeRemovalRefusal`, called from `apps/celilo/src/cli/commands/module-remove.ts`. A PAUSED module is not a dependent (unpause cannot return it to service without a redeploy, and a redeploy re-resolves capabilities), which is what makes a provider swap possible at all. A dependent is one declaring the capability under `requires` **or** `optional` — the same relation `dep-graph.ts` uses, so the guard and the cascade agree on the set; the guard previously read `requires` alone, which let a removal silently orphan `technitium`'s `optional` `dhcp_server`. Refusals name each blocker AND which declaration makes it one. It deliberately does NOT exempt a dependent because another provider of the same capability exists (celilo#683).
|
|
183
|
+
- **Consumer-removal cleanup (every provider is told)** — `apps/celilo/src/services/consumer-cleanup.ts` — pure `planConsumerCleanup` + `loadConsumerCleanupPlan` + `runConsumerCleanup`, called from `apps/celilo/src/cli/commands/module-remove.ts` after `on_uninstall` and before `terraform destroy`. A capability is two-sided: the consumer asks, the provider mints something in ITS world (a caddy site block, a DNAT rule, an OIDC client at authentik, a registered CI runner), and removal only ever touched one side — the FK cascade dropped celilo's row, so the provider's next converge had no way to learn the thing existed. This dispatches the `on_consumer_removed` hook to every provider of every capability the departing module declared under `requires` OR `optional` (the same relation `remove-guard.ts` counts as a dependency edge), **once per provider** rather than once per capability, sorted by provider id. The hook receives one input, `consumer`, and NOTHING else: a provider that cannot answer "what do I hold for this module" without being told has a different defect — the consumer's id was never recorded at mint time. It replaces `services/web-route-cleanup.ts`, which did the same job for exactly one capability, by name, from core. Semantics that are easy to get backwards: **a failed withdrawal never blocks the removal** — the consumer goes and the failing PROVIDER is marked `ERROR` with the departing consumer named in `error_message` (surfaced as a `blocked` finding by `services/audit/undeployed-modules.ts`), because the hook is a full converge and after a failure the provider's state is unknown rather than "one thing missed". **Dispatch continues past a failure**, so one broken provider cannot leave the others holding state. A PAUSED provider is skipped with a warning naming what it keeps (`run-named-hook.ts` refuses non-lifecycle hooks on a paused module, and `on_consumer_removed` must NOT join `LIFECYCLE_HOOKS`), and a never-deployed one is skipped silently. Providers implementing it: caddy, caddy-internal, iptables, greenwave, axon, authentik, forgejo, generic-cpanel-hosting-provider. See `openspec/changes/consumer-removal-cleanup/`.
|
|
184
|
+
- **Firewall registry ownership** — `apps/celilo/src/services/port-forwards.ts` + `apps/celilo/src/services/trusted-sources.ts`. Both stores are bound to the CONSUMING module and stamp `registered_by` themselves; a caller cannot supply it, so a registration is never attributed to the wrong module. Both writes are **declarative**: `replace()` states a consumer's COMPLETE set for a target, so a port or subnet it previously registered and now omits is withdrawn (celilo#855 — before this, a module that exposed `:8080` and redeployed exposing `:9090` kept both, forever). The owner is IN both unique indexes, not merely beside them: two consumers wanting the same forward are two ROWS, so one leaving cannot delete a rule the other still needs; `modules/iptables/scripts/ruleset-renderer.ts` dedupes on the rule tuple so the pair renders once. Neither column is a FK, so `runConsumerCleanup` deletes these rows explicitly after every provider has converged without them.
|
|
182
185
|
- **Backup artifact encryption** — `apps/celilo/src/services/backup-cipher.ts` — `encryptFileToFile`/`decryptFileToFile`, file-in/file-out and streamed, used by every backup writer (`backup-create.ts`) and reader (`backup-restore.ts`, `restore-from-file.ts`). Do NOT route artifacts through `secrets/encryption.ts`: that API is string-in/string-out for short DB values, and feeding it a tar cost base64 (1.33x) then hex (2x) then `JSON.stringify` — ~9x the artifact in memory, which OOM-killed forgejo's 774 MB backup, and a hard ~805 MB ceiling from the max string length that no amount of RAM raises. On-disk format is `magic "CELILOBK" (8) | version (1) | iv (16) | ciphertext | GCM tag (16)`; the tag is a trailer because it does not exist until the last byte is encrypted. `decryptFileToFile` still reads the pre-2026-07 JSON-envelope artifacts, discriminating on the magic bytes — the envelope's own `schemaVersion` cannot serve, as it lives inside the encrypted tar.
|
|
183
186
|
|
|
184
187
|
### Paused modules are conspicuous
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
-- Ownership of a firewall registration is a SET, not a label
|
|
2
|
+
-- (openspec/changes/consumer-removal-cleanup, D5a).
|
|
3
|
+
--
|
|
4
|
+
-- `port_forwards` had no owner column at all, so nothing could tell which
|
|
5
|
+
-- consumer a DNAT rule was minted for — the reason removing a module left the
|
|
6
|
+
-- rule on the box with nothing in the registry to explain it.
|
|
7
|
+
--
|
|
8
|
+
-- Adding the column alone would introduce a WORSE bug than it fixes. Both
|
|
9
|
+
-- unique indexes keyed the row WITHOUT an owner, and both stores
|
|
10
|
+
-- delete-then-insert on that tuple, so consumer B registering what consumer A
|
|
11
|
+
-- already has silently replaces A's row. Put an owner beside such an index and
|
|
12
|
+
-- removing B deletes a rule A still needs. So the owner joins the index: two
|
|
13
|
+
-- owners of one forward are two rows, and the row dies exactly when the last
|
|
14
|
+
-- module that wants it goes away. `renderRuleset` dedupes on the rule tuple so
|
|
15
|
+
-- the pair still emits one DNAT rule.
|
|
16
|
+
--
|
|
17
|
+
-- NOT destructive, though Phase 0 would permit it. Existing rows keep serving
|
|
18
|
+
-- with `registered_by = ''` (unattributed): dropping them would empty the
|
|
19
|
+
-- desired-state registry while the boxes keep their applied rules, and the
|
|
20
|
+
-- first firewall converge after the upgrade would then render a ruleset missing
|
|
21
|
+
-- every forward except the one the deploying module just re-registered. The
|
|
22
|
+
-- unattributed rows are adopted the next time their owner re-declares the same
|
|
23
|
+
-- target (see `buildPortForwardStore.replace`).
|
|
24
|
+
|
|
25
|
+
ALTER TABLE `port_forwards` ADD `registered_by` text DEFAULT '' NOT NULL;--> statement-breakpoint
|
|
26
|
+
DROP INDEX `port_forwards_unique_idx`;--> statement-breakpoint
|
|
27
|
+
CREATE UNIQUE INDEX `port_forwards_unique_idx` ON `port_forwards` (`firewall_ip`,`internal_ip`,`port`,`protocol`,`ingress_ip`,`registered_by`);--> statement-breakpoint
|
|
28
|
+
DROP INDEX `trusted_sources_unique_idx`;--> statement-breakpoint
|
|
29
|
+
CREATE UNIQUE INDEX `trusted_sources_unique_idx` ON `trusted_sources` (`firewall_ip`,`subnet`,`registered_by`);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@celilo/cli",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "Celilo — home lab orchestration CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -58,7 +58,7 @@
|
|
|
58
58
|
"dependencies": {
|
|
59
59
|
"@aws-sdk/client-s3": "^3.1109.0",
|
|
60
60
|
"@aws-sdk/lib-storage": "^3.1101.0",
|
|
61
|
-
"@celilo/capabilities": "^
|
|
61
|
+
"@celilo/capabilities": "^2.0.0",
|
|
62
62
|
"@celilo/cli-display": "^0.2.0",
|
|
63
63
|
"@celilo/core": "^0.8.1",
|
|
64
64
|
"@celilo/event-bus": "^0.6.0",
|
|
@@ -14,7 +14,7 @@ import { capabilities as capabilitiesTable, moduleInfrastructure, modules } from
|
|
|
14
14
|
import { createGaugeLogger } from '../../hooks/logger';
|
|
15
15
|
import { runNamedHook } from '../../hooks/run-named-hook';
|
|
16
16
|
import { deallocateForModule } from '../../ipam/auto-allocator';
|
|
17
|
-
import { ModuleManifestSchema } from '../../manifest/schema';
|
|
17
|
+
import { type ModuleManifest, ModuleManifestSchema } from '../../manifest/schema';
|
|
18
18
|
import { executeBuildWithProgress } from '../../services/build-stream';
|
|
19
19
|
import { askConfirm, withInterviewSession } from '../../services/bus-interview';
|
|
20
20
|
import {
|
|
@@ -22,6 +22,7 @@ import {
|
|
|
22
22
|
emitUninstallFailed,
|
|
23
23
|
emitUninstallStarted,
|
|
24
24
|
} from '../../services/celilo-events';
|
|
25
|
+
import { loadConsumerCleanupPlan, runConsumerCleanup } from '../../services/consumer-cleanup';
|
|
25
26
|
import { getContainerService, getServiceCredentials } from '../../services/container-service';
|
|
26
27
|
import { completeOperation, failOperation, startOperation } from '../../services/module-operations';
|
|
27
28
|
import {
|
|
@@ -29,7 +30,6 @@ import {
|
|
|
29
30
|
describeRemovalRefusal,
|
|
30
31
|
findRemovalBlockers,
|
|
31
32
|
} from '../../services/remove-guard';
|
|
32
|
-
import { cleanupWebRoutesForModule } from '../../services/web-route-cleanup';
|
|
33
33
|
import { getArg, hasFlag, validateRequiredArgs } from '../parser';
|
|
34
34
|
import { log } from '../prompts';
|
|
35
35
|
import type { CommandResult } from '../types';
|
|
@@ -243,28 +243,31 @@ async function performModuleRemove(
|
|
|
243
243
|
}
|
|
244
244
|
}
|
|
245
245
|
|
|
246
|
-
//
|
|
247
|
-
//
|
|
248
|
-
//
|
|
246
|
+
// Tell every provider whose capability this module consumed that it is
|
|
247
|
+
// leaving, so each withdraws what it minted on its behalf
|
|
248
|
+
// (openspec/changes/consumer-removal-cleanup). BEFORE terraform destroy, so
|
|
249
|
+
// provider hosts are still reachable; AFTER on_uninstall, so a module that
|
|
250
|
+
// tears its own state down first still wins.
|
|
249
251
|
//
|
|
250
|
-
//
|
|
251
|
-
//
|
|
252
|
-
//
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
252
|
+
// This replaces the by-name `cleanupWebRoutesForModule` call, which did the
|
|
253
|
+
// same job for exactly one capability. A failed withdrawal never blocks the
|
|
254
|
+
// removal — the failing PROVIDER is marked ERROR instead (D6).
|
|
255
|
+
const cleanupLogger = {
|
|
256
|
+
info: (m: string) => log.info(m),
|
|
257
|
+
warn: (m: string) => log.warn(m),
|
|
258
|
+
error: (m: string) => log.warn(m),
|
|
259
|
+
success: (m: string) => log.info(m),
|
|
260
|
+
};
|
|
261
|
+
const cleanup = await runConsumerCleanup(
|
|
262
|
+
moduleId,
|
|
263
|
+
loadConsumerCleanupPlan(moduleId, module.manifestData as ModuleManifest, db),
|
|
264
|
+
db,
|
|
265
|
+
cleanupLogger,
|
|
266
|
+
);
|
|
267
|
+
if (cleanup.failures.length > 0) {
|
|
268
|
+
log.warn(
|
|
269
|
+
`${cleanup.failures.length} provider(s) could not withdraw state for '${moduleId}' and are now marked ERROR: ${cleanup.failures.map((f) => f.providerId).join(', ')}. Removal continues; run \`celilo system audit\` to see what each is holding.`,
|
|
270
|
+
);
|
|
268
271
|
}
|
|
269
272
|
|
|
270
273
|
// Check if module has infrastructure that needs to be destroyed
|
|
@@ -371,7 +371,11 @@ async function buildAuditDeps(onProgress?: (msg: string) => void) {
|
|
|
371
371
|
backups: { modules: installedBackupInfo },
|
|
372
372
|
abandonedOperations: { records: loadAbandonedOperations(db) },
|
|
373
373
|
undeployedModules: {
|
|
374
|
-
modules: installed.map((m) => ({
|
|
374
|
+
modules: installed.map((m) => ({
|
|
375
|
+
id: m.id,
|
|
376
|
+
state: m.state,
|
|
377
|
+
errorMessage: m.errorMessage,
|
|
378
|
+
})),
|
|
375
379
|
},
|
|
376
380
|
unconfiguredModules: {
|
|
377
381
|
modules: installed.map((m) => ({
|
|
@@ -573,7 +573,11 @@ export async function handleSystemUpdate(
|
|
|
573
573
|
},
|
|
574
574
|
abandonedOperations: { records: loadAbandonedOperations(db) },
|
|
575
575
|
undeployedModules: {
|
|
576
|
-
modules: installed.map((m) => ({
|
|
576
|
+
modules: installed.map((m) => ({
|
|
577
|
+
id: m.id,
|
|
578
|
+
state: m.state,
|
|
579
|
+
errorMessage: m.errorMessage,
|
|
580
|
+
})),
|
|
577
581
|
},
|
|
578
582
|
unconfiguredModules: {
|
|
579
583
|
modules: installed.map((m) => ({
|
|
@@ -801,7 +805,11 @@ export function rebuildAuditDepsForRerun(
|
|
|
801
805
|
})),
|
|
802
806
|
},
|
|
803
807
|
undeployedModules: {
|
|
804
|
-
modules: installed.map((m) => ({
|
|
808
|
+
modules: installed.map((m) => ({
|
|
809
|
+
id: m.id,
|
|
810
|
+
state: m.state,
|
|
811
|
+
errorMessage: m.errorMessage,
|
|
812
|
+
})),
|
|
805
813
|
},
|
|
806
814
|
unconfiguredModules: {
|
|
807
815
|
modules: installed.map((m) => ({
|
package/src/db/schema.ts
CHANGED
|
@@ -591,19 +591,34 @@ export const portForwards = sqliteTable(
|
|
|
591
591
|
/** Dedicated INTERNAL ingress IP (ISS-0156); NULL for the normal public path. */
|
|
592
592
|
ingressIp: text('ingress_ip'),
|
|
593
593
|
description: text('description').notNull().default(''),
|
|
594
|
+
/**
|
|
595
|
+
* The CONSUMER that registered it, stamped by the store from the calling
|
|
596
|
+
* module (openspec/changes/consumer-removal-cleanup, D5a). `''` for rows
|
|
597
|
+
* written before this column existed — unattributed, so no consumer removal
|
|
598
|
+
* withdraws them.
|
|
599
|
+
*/
|
|
600
|
+
registeredBy: text('registered_by').notNull().default(''),
|
|
594
601
|
createdAt: integer('created_at', { mode: 'timestamp' }).notNull().default(sql`(unixepoch())`),
|
|
595
602
|
},
|
|
596
603
|
(table) => ({
|
|
597
|
-
// One row per (firewall, backend, port, protocol, ingress). The
|
|
598
|
-
//
|
|
599
|
-
//
|
|
600
|
-
//
|
|
604
|
+
// One row per (OWNER, firewall, backend, port, protocol, ingress). The
|
|
605
|
+
// owner is IN the index, not merely alongside it, and that is the whole
|
|
606
|
+
// point: without it, consumer B exposing a forward A already has silently
|
|
607
|
+
// REPLACES A's row, and B leaving then deletes a rule A still needs. Two
|
|
608
|
+
// owners of one forward are two rows — the set, denormalised into the
|
|
609
|
+
// table that already exists rather than a second table. `renderRuleset`
|
|
610
|
+
// dedupes on the rule tuple so the duplicate renders once.
|
|
611
|
+
//
|
|
612
|
+
// This is the `dns_registration_consumers` lesson (see below): a single
|
|
613
|
+
// overwritten owner column is not a label, because what cascades on it
|
|
614
|
+
// decides when a LIVE record is forgotten.
|
|
601
615
|
forwardUnique: uniqueIndex('port_forwards_unique_idx').on(
|
|
602
616
|
table.firewallIp,
|
|
603
617
|
table.internalIp,
|
|
604
618
|
table.port,
|
|
605
619
|
table.protocol,
|
|
606
620
|
table.ingressIp,
|
|
621
|
+
table.registeredBy,
|
|
607
622
|
),
|
|
608
623
|
}),
|
|
609
624
|
);
|
|
@@ -632,11 +647,14 @@ export const trustedSources = sqliteTable(
|
|
|
632
647
|
createdAt: integer('created_at', { mode: 'timestamp' }).notNull().default(sql`(unixepoch())`),
|
|
633
648
|
},
|
|
634
649
|
(table) => ({
|
|
635
|
-
// One row per (firewall, subnet) — the
|
|
636
|
-
//
|
|
650
|
+
// One row per (OWNER, firewall, subnet) — same reason the owner is in
|
|
651
|
+
// `port_forwards_unique_idx`. Two modules trusting the same subnet are two
|
|
652
|
+
// rows, so one leaving does not revoke the other's reach. The renderer
|
|
653
|
+
// already dedupes trusted subnets (`new Set`), so the pair renders once.
|
|
637
654
|
trustedSourceUnique: uniqueIndex('trusted_sources_unique_idx').on(
|
|
638
655
|
table.firewallIp,
|
|
639
656
|
table.subnet,
|
|
657
|
+
table.registeredBy,
|
|
640
658
|
),
|
|
641
659
|
}),
|
|
642
660
|
);
|
|
@@ -113,6 +113,16 @@ export const CAPABILITY_MODULE_MAP: Record<string, { script: string; legacyFacto
|
|
|
113
113
|
script: 'scripts/control-plane-vpn-functions.ts',
|
|
114
114
|
legacyFactoryName: 'default',
|
|
115
115
|
},
|
|
116
|
+
// MODULE-provided, unlike its sibling public_web above, which the framework
|
|
117
|
+
// implements (createPublicWeb) against celilo's `web_routes` table. The
|
|
118
|
+
// asymmetry is deliberate: a private route stored in `web_routes` would be
|
|
119
|
+
// picked up by the PUBLIC caddy, which derives its served hostnames from
|
|
120
|
+
// every row of that table — so the provider keeps its own routes and celilo
|
|
121
|
+
// core holds no private-ingress business at all (celilo#846).
|
|
122
|
+
private_web: {
|
|
123
|
+
script: 'scripts/private-web-functions.ts',
|
|
124
|
+
legacyFactoryName: 'default',
|
|
125
|
+
},
|
|
116
126
|
};
|
|
117
127
|
|
|
118
128
|
/**
|
|
@@ -247,7 +257,7 @@ export async function loadCapabilityFunctions(
|
|
|
247
257
|
// buildFirewallChain handles the per-layer wrap with auto-logging
|
|
248
258
|
// internally, so we don't need to wrap the result here.
|
|
249
259
|
if (capName === 'firewall' && allProviders.length > 1) {
|
|
250
|
-
const
|
|
260
|
+
const { chain, self } = await buildFirewallChain(
|
|
251
261
|
allProviders,
|
|
252
262
|
moduleInfo,
|
|
253
263
|
masterKey,
|
|
@@ -256,8 +266,15 @@ export async function loadCapabilityFunctions(
|
|
|
256
266
|
debugLog,
|
|
257
267
|
consumingModuleId,
|
|
258
268
|
);
|
|
259
|
-
if (
|
|
260
|
-
result[capName] =
|
|
269
|
+
if (chain) {
|
|
270
|
+
result[capName] = chain;
|
|
271
|
+
}
|
|
272
|
+
// The chain hands a CONSUMER the innermost layer, which is not this
|
|
273
|
+
// provider's own layer when it sits further out. `on_consumer_removed`
|
|
274
|
+
// must converge THIS firewall, so it gets its own layer by name.
|
|
275
|
+
if (self) {
|
|
276
|
+
result.firewall_registry = self;
|
|
277
|
+
debugLog(`firewall_registry: provider view injected for ${consumingModuleId}`);
|
|
261
278
|
}
|
|
262
279
|
continue;
|
|
263
280
|
}
|
|
@@ -359,7 +376,7 @@ export async function loadCapabilityFunctions(
|
|
|
359
376
|
providerSecrets,
|
|
360
377
|
// Single firewall provider (no upstream chain) still needs the injected
|
|
361
378
|
// port-forward store — the chain path isn't taken when there's one provider.
|
|
362
|
-
capName === 'firewall' ? buildPortForwardStore(db) : undefined,
|
|
379
|
+
capName === 'firewall' ? buildPortForwardStore(db, consumingModuleId) : undefined,
|
|
363
380
|
capName === 'firewall' ? loadFirewallZones(db) : undefined,
|
|
364
381
|
// Bound to the CONSUMING module so a trusted-source registration is
|
|
365
382
|
// attributable — reach into every tier must never be anonymous.
|
|
@@ -372,6 +389,12 @@ export async function loadCapabilityFunctions(
|
|
|
372
389
|
wrapWithLogging(capabilityInterface as object, logger, capName),
|
|
373
390
|
);
|
|
374
391
|
debugLog(`${capName}: loaded via legacy factory`);
|
|
392
|
+
// Sole firewall provider running its OWN hook: the interface just built
|
|
393
|
+
// IS its layer, so hand it back under the provider-view name too.
|
|
394
|
+
if (capName === 'firewall' && consumingModuleId === capability.moduleId) {
|
|
395
|
+
result.firewall_registry = result[capName];
|
|
396
|
+
debugLog(`firewall_registry: provider view injected for ${consumingModuleId}`);
|
|
397
|
+
}
|
|
375
398
|
}
|
|
376
399
|
} catch (error) {
|
|
377
400
|
debugLog(
|
|
@@ -977,6 +1000,20 @@ function parseInterfaceBaseline(raw: unknown): string[] | undefined {
|
|
|
977
1000
|
return names.length > 0 ? names : undefined;
|
|
978
1001
|
}
|
|
979
1002
|
|
|
1003
|
+
/**
|
|
1004
|
+
* What a firewall build hands back.
|
|
1005
|
+
*
|
|
1006
|
+
* `chain` is the layer a CONSUMER talks to (the innermost). `self` is the layer
|
|
1007
|
+
* belonging to the module currently running a hook, and is null unless that
|
|
1008
|
+
* module is itself one of the providers — it is what `on_consumer_removed`
|
|
1009
|
+
* converges, and it differs from `chain` whenever the provider sits further out
|
|
1010
|
+
* than the innermost layer.
|
|
1011
|
+
*/
|
|
1012
|
+
interface FirewallChain {
|
|
1013
|
+
chain: unknown;
|
|
1014
|
+
self: unknown | null;
|
|
1015
|
+
}
|
|
1016
|
+
|
|
980
1017
|
async function buildFirewallChain(
|
|
981
1018
|
allProviders: Array<{
|
|
982
1019
|
id: number;
|
|
@@ -991,11 +1028,12 @@ async function buildFirewallChain(
|
|
|
991
1028
|
logger: HookLogger,
|
|
992
1029
|
debugLog: (msg: string) => void,
|
|
993
1030
|
consumingModuleId: string,
|
|
994
|
-
): Promise<
|
|
1031
|
+
): Promise<FirewallChain> {
|
|
995
1032
|
// The shared-core port-forward registry, injected into every firewall provider
|
|
996
1033
|
// in the chain so exposeService/converge reconcile against the one canonical
|
|
997
|
-
// store (openspec/changes/unified-management-no-ssh/proposal.md).
|
|
998
|
-
|
|
1034
|
+
// store (openspec/changes/unified-management-no-ssh/proposal.md). Bound to the
|
|
1035
|
+
// CONSUMING module so every forward it declares is attributable (D2).
|
|
1036
|
+
const store = buildPortForwardStore(db, consumingModuleId);
|
|
999
1037
|
// Trusted-source registry, bound to the CONSUMING module so a registration is
|
|
1000
1038
|
// attributable. Only the layers that render their own ruleset receive it.
|
|
1001
1039
|
const trustedSourceStore = buildTrustedSourceStore(db, consumingModuleId);
|
|
@@ -1011,17 +1049,17 @@ async function buildFirewallChain(
|
|
|
1011
1049
|
|
|
1012
1050
|
if (!hasExternal) {
|
|
1013
1051
|
debugLog('firewall chain: no provider with external interface found');
|
|
1014
|
-
return null;
|
|
1052
|
+
return { chain: null, self: null };
|
|
1015
1053
|
}
|
|
1016
1054
|
|
|
1017
1055
|
// Build the leaf (external) firewall first
|
|
1018
1056
|
const leafModule = db.select().from(modules).where(eq(modules.id, hasExternal.moduleId)).get();
|
|
1019
|
-
if (!leafModule) return null;
|
|
1057
|
+
if (!leafModule) return { chain: null, self: null };
|
|
1020
1058
|
|
|
1021
1059
|
const leafModulePath = join(leafModule.sourcePath, moduleInfo.script);
|
|
1022
1060
|
if (!existsSync(leafModulePath)) {
|
|
1023
1061
|
debugLog(`firewall chain: leaf module not found at ${leafModulePath}`);
|
|
1024
|
-
return null;
|
|
1062
|
+
return { chain: null, self: null };
|
|
1025
1063
|
}
|
|
1026
1064
|
|
|
1027
1065
|
const leafConfig = await loadModuleConfig(hasExternal.moduleId, db);
|
|
@@ -1030,7 +1068,7 @@ async function buildFirewallChain(
|
|
|
1030
1068
|
const leafExported =
|
|
1031
1069
|
typeof leafMod.default === 'function' ? leafMod.default : leafMod[moduleInfo.legacyFactoryName];
|
|
1032
1070
|
|
|
1033
|
-
if (typeof leafExported !== 'function') return null;
|
|
1071
|
+
if (typeof leafExported !== 'function') return { chain: null, self: null };
|
|
1034
1072
|
|
|
1035
1073
|
// Branded compiled factory (Phase 8 path): call with the canonical
|
|
1036
1074
|
// context. wrapWithLogging is applied internally so the leaf interface
|
|
@@ -1071,8 +1109,15 @@ async function buildFirewallChain(
|
|
|
1071
1109
|
// doesn't fit defineCapabilityFunction's single-context shape.
|
|
1072
1110
|
const downstream = allProviders.filter((p) => p.moduleId !== hasExternal.moduleId);
|
|
1073
1111
|
|
|
1112
|
+
// Each layer as its OWN provider sees it, so `on_consumer_removed` converges
|
|
1113
|
+
// the firewall that declares the hook rather than whichever layer a consumer
|
|
1114
|
+
// happens to talk to.
|
|
1115
|
+
const selfLayer = (id: string, iface: unknown): unknown | null =>
|
|
1116
|
+
id === consumingModuleId ? iface : null;
|
|
1117
|
+
let self = selfLayer(hasExternal.moduleId, leafFirewall);
|
|
1118
|
+
|
|
1074
1119
|
if (downstream.length === 0) {
|
|
1075
|
-
return leafFirewall;
|
|
1120
|
+
return { chain: leafFirewall, self };
|
|
1076
1121
|
}
|
|
1077
1122
|
|
|
1078
1123
|
let currentUpstream = leafFirewall;
|
|
@@ -1147,9 +1192,10 @@ async function buildFirewallChain(
|
|
|
1147
1192
|
wrapWithLogging(downstreamFirewall as object, logger, 'firewall'),
|
|
1148
1193
|
provider.moduleId,
|
|
1149
1194
|
);
|
|
1195
|
+
self = self ?? selfLayer(provider.moduleId, currentUpstream);
|
|
1150
1196
|
}
|
|
1151
1197
|
|
|
1152
|
-
return currentUpstream;
|
|
1198
|
+
return { chain: currentUpstream, self };
|
|
1153
1199
|
}
|
|
1154
1200
|
|
|
1155
1201
|
/**
|
|
@@ -70,9 +70,6 @@ const fakePublicWeb: PublicWebCapability = {
|
|
|
70
70
|
async upload_static_assets() {
|
|
71
71
|
return { success: true, filesUploaded: 0, contentHash: 'fake' };
|
|
72
72
|
},
|
|
73
|
-
async unregister_routes() {
|
|
74
|
-
return undefined;
|
|
75
|
-
},
|
|
76
73
|
async getServerIp() {
|
|
77
74
|
return '10.0.10.10';
|
|
78
75
|
},
|
|
@@ -298,9 +295,6 @@ describe('defineCapabilityFunction', () => {
|
|
|
298
295
|
async upload_static_assets() {
|
|
299
296
|
return { success: true, filesUploaded: 0, contentHash: 'x' };
|
|
300
297
|
},
|
|
301
|
-
async unregister_routes() {
|
|
302
|
-
return undefined;
|
|
303
|
-
},
|
|
304
298
|
async getServerIp() {
|
|
305
299
|
return '10.0.10.10';
|
|
306
300
|
},
|
package/src/hooks/executor.ts
CHANGED
|
@@ -28,6 +28,7 @@ import {
|
|
|
28
28
|
} from '@celilo/capabilities';
|
|
29
29
|
import {
|
|
30
30
|
type ContractHookSignature,
|
|
31
|
+
contractHookSignature,
|
|
31
32
|
resolveContract,
|
|
32
33
|
supportedContractVersions,
|
|
33
34
|
} from '../manifest/contracts';
|
|
@@ -435,7 +436,7 @@ export async function invokeHook(
|
|
|
435
436
|
};
|
|
436
437
|
}
|
|
437
438
|
|
|
438
|
-
const signature = contract.hooks
|
|
439
|
+
const signature = contractHookSignature(contract.hooks, hookName);
|
|
439
440
|
if (!signature) {
|
|
440
441
|
return {
|
|
441
442
|
success: false,
|
package/src/hooks/types.ts
CHANGED
|
@@ -104,25 +104,17 @@ export interface HookResult {
|
|
|
104
104
|
}
|
|
105
105
|
|
|
106
106
|
/**
|
|
107
|
-
* Supported lifecycle hook names
|
|
107
|
+
* Supported lifecycle hook names — re-exported from `@celilo/capabilities`,
|
|
108
|
+
* which owns the one list (celilo#821).
|
|
108
109
|
*
|
|
109
|
-
*
|
|
110
|
-
* `
|
|
111
|
-
*
|
|
110
|
+
* This was a second hand-maintained copy, and it had already drifted: it was
|
|
111
|
+
* missing `reconcile_routes`, so caddy's public_web reconcile hook was a hook
|
|
112
|
+
* the celilo side of the codebase did not believe in.
|
|
112
113
|
*/
|
|
113
|
-
export type HookName
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
| 'health_check'
|
|
118
|
-
| 'validate_config'
|
|
119
|
-
| 'on_backup'
|
|
120
|
-
| 'on_backup_analyze'
|
|
121
|
-
| 'on_restore'
|
|
122
|
-
| 'on_system_event'
|
|
123
|
-
| 'refresh_registrations'
|
|
124
|
-
| 'reassert_dhcp_dns'
|
|
125
|
-
| 'reconcile_clients';
|
|
114
|
+
export type { HookName } from '@celilo/capabilities';
|
|
115
|
+
export { HOOK_NAMES } from '@celilo/capabilities';
|
|
116
|
+
|
|
117
|
+
import type { HookName } from '@celilo/capabilities';
|
|
126
118
|
|
|
127
119
|
/**
|
|
128
120
|
* Hook manifest section - maps hook names to definitions
|
|
@@ -15,6 +15,26 @@ import type { ContractHookSignature, ContractHooks } from './v1';
|
|
|
15
15
|
|
|
16
16
|
export type { ContractHooks, ContractHookSignature };
|
|
17
17
|
|
|
18
|
+
/**
|
|
19
|
+
* Look up a hook signature by an UNTRUSTED name.
|
|
20
|
+
*
|
|
21
|
+
* `ContractHooks` is keyed by `HookName` (celilo#821), which is what makes a
|
|
22
|
+
* missing or unknown entry a compile error. The executor's `hookName` is a
|
|
23
|
+
* plain `string` on purpose — it comes from a module manifest, and deciding
|
|
24
|
+
* whether it names a real hook is precisely what the caller is asking. This
|
|
25
|
+
* helper is the one place that crossing is expressed, so every other use of
|
|
26
|
+
* `ContractHooks` stays exact.
|
|
27
|
+
*
|
|
28
|
+
* Returns `undefined` for a name the contract does not define; the caller
|
|
29
|
+
* turns that into "Hook 'x' is not part of celilo_contract 1.0".
|
|
30
|
+
*/
|
|
31
|
+
export function contractHookSignature(
|
|
32
|
+
hooks: ContractHooks,
|
|
33
|
+
name: string,
|
|
34
|
+
): ContractHookSignature | undefined {
|
|
35
|
+
return (hooks as Record<string, ContractHookSignature | undefined>)[name];
|
|
36
|
+
}
|
|
37
|
+
|
|
18
38
|
/**
|
|
19
39
|
* The shape of a registered contract.
|
|
20
40
|
*/
|