@celilo/cli 0.8.2 → 0.9.1

package/AGENTS.md

@@ -7,10 +7,6 @@ build and ship apps on celilo. **Read the local docs below before searching the
 web** — they are the source of truth; the hosted copies at
 `https://celilo.computer/docs` lag them.
 
-This file ships **inside the npm package**: the `./`-prefixed docs in the doc map
-are installed alongside it (grep them offline). The repo-relative entries point
-at the source repository / hosted docs, which the tarball does not carry.
-
 ## Mental model (learn these five things)
 
 - **Module** — a directory with a `manifest.yml` and `celilo/scripts/` hooks,
@@ -77,18 +73,14 @@ celilo module deploy <id> / update / health   # deploy + verify on the fleet
 
 ## Doc map (read in this order)
 
-Shipped **inside this package** (offline, grep-friendly):
-
-- `./CLI_USAGE.md` — full CLI reference + common workflows.
-- `./schemas/system_config.json` — the system-config contract.
-- `./src/manifest/schema.ts` — Zod source of the manifest schema (the published
-  `module-manifest.schema.json` is generated from it via `bun run export:schema`).
-
-In the **source repo / hosted docs** (not carried in the tarball):
-
-- `design/MODULE_DEVELOPMENT_GUIDE.md` — author a module (start here).
-- `v2/BUILD_BUS.md` — how a change reaches the fleet.
-- `v2/APP_CI_REFERENCE.md` — wire an app's CI/CD (the copy-me recipe).
-- `v2/MODULE_VERSIONING.md` — what `version` means; `version_source`.
-- `schemas/module-manifest.schema.json` — the manifest contract.
+- `./CELILO_SUBSYSTEMS.md` — map of the infrastructure primitives celilo already
+  implements (IPAM, capabilities, firewall/DNS, generator, …), with entry-point files.
+- `./CELILO_CORE_MODULES.md` — the production modules celilo ships in `modules/`
+  (what each provides/requires, its role in the fleet).
+- `../../design/MODULE_DEVELOPMENT_GUIDE.md` — author a module (start here).
+- `./CLI_USAGE.md` — full CLI reference + workflows.
+- `../../v2/BUILD_BUS.md` — how a change reaches the fleet.
+- `../../v2/APP_CI_REFERENCE.md` — wire an app's CI/CD (the copy-me recipe).
+- `../../v2/MODULE_VERSIONING.md` — what `version` means; `version_source`.
+- `../../schemas/module-manifest.schema.json` — the manifest contract.
 - `https://celilo.computer/docs` — hosted docs (LAN; the repo docs lead).

package/CELILO_CORE_MODULES.md

@@ -0,0 +1,61 @@
+# Celilo Core Modules
+
+**What this is:** the production modules celilo ships in `modules/` — the infrastructure celilo
+actually *deploys*. Companion to [CELILO_SUBSYSTEMS.md](./CELILO_SUBSYSTEMS.md) (which maps
+celilo's own engine). Use this to know what already exists before building a new module, and to
+understand how the fleet wires together via capabilities.
+
+Each entry: `module id` — what it is — **provides** / **requires** capabilities. Its source is
+`modules/<id>/manifest.yml`; capability contracts are listed in
+[CELILO_SUBSYSTEMS.md](./CELILO_SUBSYSTEMS.md).
+
+> **Paths below are relative to the celilo source-repo root** (e.g. `modules/caddy/manifest.yml`)
+> — references into the source tree, not links relative to this file (this doc also ships inside
+> the `@celilo/cli` npm package). Re-grep manifests if an entry looks stale.
+
+> How the graph closes: a public app (`requires: public_web`) is served by **caddy**, which
+> `requires: dns_registrar` (**namecheap**) to publish its name and `firewall` (**greenwave** /
+> **iptables**) to open the port. Internal name resolution comes from a **dns_internal** provider
+> (**knot-unbound-internal** / **technitium**). Identity comes from **authentik** (`idp`). That's
+> the whole edge: DNS + firewall + ingress + identity, each a swappable provider module.
+
+## Network fabric (DNS / firewall / DHCP)
+
+- **greenwave** — GreenWave C4000XG ISP router driver; port-forwarding + public-IP discovery via REST. **provides:** `firewall`, `dhcp_server`.
+- **iptables** — iptables firewall + NAT; cross-VLAN port exposure with recursive upstream delegation. **provides:** `firewall`.
+- **knot-unbound-internal** — split-horizon internal DNS via Knot (authoritative) + Unbound (recursive); lightweight, plain apt, no .NET. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/knot-unbound-internal/base-module-aspect/`).
+- **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
+- **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). **provides:** `dns_registrar`.
+
+## Public edge (ingress / identity)
+
+- **caddy** — reverse proxy with automatic HTTPS (HTTP-01 ACME); the standard HTTPS ingress. **provides:** `public_web`. **requires:** `dns_registrar`, `firewall`.
+- **authentik** — Authentik identity provider with OIDC (Docker Compose: server, worker, Postgres, Redis). **provides:** `idp`. **requires:** `public_web`, `dns_registrar`, `firewall`.
+
+## Celilo's own infrastructure (self-hosted)
+
+- **celilo-mgmt** — the celilo management server itself, deployed as a module (replaces install.sh + `system init`; ships daemon, runs migrations, self-registers). **provides:** `celilo_event_bus`, `celilo_module_deploy_worker`. **requires:** `cross_module_read`. See `v2/MANAGEMENT_AS_NETAPP.md`.
+- **celilo-registry** — module registry server (Cargo sparse protocol); stores `.netapp` files, serves index + search/download API. **provides:** `registry_publish`. **requires:** `public_web`, `dns_registrar`.
+- **celilo-apt-repo** — Debian apt repository (reprepro + Bun HTTP server) serving the celilo `.deb` at apt.celilo.computer. **provides:** `apt_publish`. **requires:** `public_web`, `dns_registrar`.
+- **celilo-website** — public docs site (static Astro) served via Caddy on celilo.computer. **requires:** `public_web`, `dns_registrar`.
+
+## Git forge & CI pipeline
+
+- **forgejo** — self-hosted Forgejo git forge (git-over-SSH, OIDC, public ingress). **provides:** `source_forge`. **requires:** `public_web`, `idp`, `firewall`, `dns_registrar`. See `v2/CI_PIPELINE.md`.
+- **forgejo-runner** — host-mode (LXC) Forgejo Actions runner for unit/lint jobs; outbound-only, lives in dmz. **requires:** `source_forge`. See `v2/FORGEJO_RUNNER.md`.
+- **forgejo-builder** — VM-based, Docker-capable Forgejo Actions runner (`requires.system.type: vm`) for hermetic release builds + the cele2e suite. **requires:** `source_forge`.
+- **npm-cache-node** — self-hosted npm registry as a pass-through disk cache (proxies upstream). **requires:** `public_web`, `dns_registrar`. See `v2/NPM_CACHE_NODE.md`.
+
+## Applications
+
+- **homebridge** — HomeKit bridge for smart-home devices (VeSync, Leviton, Lutron, TP-Link, Tuya). No capabilities (leaf app).
+
+## E2E fixtures & probes (not production apps)
+
+- **source-forge-probe** — hostless e2e probe that exercises the `source_forge` capability (issue/revoke runner token) against a deployed Forgejo. **requires:** `source_forge`.
+- **test-ansible-output** — e2e fixture that runs 25×1s sleep tasks to assert Ansible output streams in real time. Not a real app.
+- **vm-smoke** — throwaway module proving celilo's generic `type: vm` deploy path end-to-end (template clone → cloud-init → Ansible reach). No capabilities.
+
+## Archived / superseded
+
+`modules/archive/` holds retired modules — **dns-external** (VPS authoritative DNS + WireGuard, superseded by the `dns_internal`/`dns_registrar` split), **gmail** (email-reading capability), **namecheap-api** (registrar-config via Namecheap API, superseded by the **namecheap** DDNS module). Reference only; not deployed.

package/CELILO_SUBSYSTEMS.md

@@ -0,0 +1,83 @@
+# Celilo Subsystem Inventory
+
+**What this is:** a map of infrastructure primitives celilo **already implements**, with the
+entry-point file for each. It exists so agents stop reinventing features (the trigger case:
+IP/VMID allocation, which is already IPAM).
+
+**The rule:** Assume celilo already provides any infrastructure primitive (IP/VMID
+allocation, secrets, DNS, firewall, cross-module data, packaging, event scheduling).
+Before proposing to build one — or asking the operator how to do it — grep this
+inventory / the codebase and cite what you found. Only raise a question if the search
+is genuinely empty, or the real choice is between existing mechanisms.
+
+This is an *implemented-feature* map. For terminology see `GLOSSARY.md`; for architecture
+see `design/README.md`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MODULES.md)
+(the production modules celilo ships).
+
+> **All file paths below are relative to the celilo source-repo root** (e.g.
+> `apps/celilo/src/ipam/allocator.ts`). They are references into the source tree, not links
+> relative to this file — this doc also ships inside the `@celilo/cli` npm package, where the
+> wider repo isn't present. Paths drift; if an entry looks wrong, re-grep and fix it here.
+
+---
+
+## Allocation & infrastructure selection
+
+- **IPAM (IP/VMID allocation)** — `apps/celilo/src/ipam/allocator.ts` — `allocateIPFromSubnet`, `allocateVMID`, `reserveIP`/`unreserveIP`, `inferZoneFromIP`, `getAllocation`. Auto-wrapper: `apps/celilo/src/ipam/auto-allocator.ts` — `allocateForModule` / `deallocateForModule`.
+- **Infrastructure selection (container-service vs machine pool)** — `apps/celilo/src/services/machine-pool.ts` (`getMachineByHostname`, `addMachine`, `assignModuleToMachine`) and `apps/celilo/src/services/container-service.ts` (`getContainerServiceByName`, `addContainerService`, `verifyContainerService`). Provider API clients: `apps/celilo/src/api-clients/proxmox.ts`, `apps/celilo/src/api-clients/digitalocean.ts`.
+- **Zone detection / system config** — `apps/celilo/src/services/zone-detector.ts` — `detectZoneFromIp` reads `network.<zone>.subnet` from the `systemConfig` table.
+
+## Capability system (cross-module data & functions)
+
+- **Define a capability function** — `packages/capabilities/src/define-capability-function.ts` — `defineCapabilityFunction`.
+- **Canonical capability registry** — `packages/capabilities/src/capability-registry.ts` — `KNOWN_CAPABILITY_NAMES` (the authoritative list), `CapabilityRegistry` type. Public surface: `packages/capabilities/src/index.ts`.
+- **Loader (wires provider factories into hook contexts)** — `apps/celilo/src/hooks/capability-loader.ts` — `loadCapabilityFunctions`, `resolveFirewallNatIp`.
+- **Ledger wrappers (stateful capabilities)** — `apps/celilo/src/services/dns-registrations.ts` (`withDnsRegistrationLedger`), `apps/celilo/src/services/dns-internal-records.ts` (`withDnsInternalLedger`).
+
+### Known capabilities (impl → provider module)
+
+| capability | impl | provider module(s) |
+|---|---|---|
+| `public_web` | `packages/capabilities/src/public-web.ts` (`createPublicWeb`) | caddy |
+| `idp` | `packages/capabilities/src/idp.ts` | authentik |
+| `dns_registrar` | `packages/capabilities/src/dns-registrar.ts` (`registerHost`) | namecheap |
+| `firewall` | `packages/capabilities/src/firewall.ts` (`exposeService`, `unexposeService`, `listExposedServices`) | greenwave, iptables |
+| `dns_internal` | `packages/capabilities/src/dns-internal.ts` | knot-unbound-internal, technitium |
+| `dhcp_server` | `packages/capabilities/src/dhcp-server.ts` | greenwave |
+| `source_forge` | `packages/capabilities/src/source-forge.ts` | forgejo |
+| `registry_publish` | `packages/capabilities/src/registry-publish.ts` | celilo-registry |
+| `cross_module_read` | `packages/capabilities/src/cross-module-read.ts` | framework (read other modules' capability data) |
+
+## Hooks & deploy
+
+- **Hook executor / ABI** — `apps/celilo/src/hooks/executor.ts` (`invokeHook`, `executeHookScript`, `checkRequiredCapabilities`), types in `apps/celilo/src/hooks/types.ts` (`HookContext`, `HookDefinition`, `HookName`). Named-hook runner: `apps/celilo/src/hooks/run-named-hook.ts`. Manifest hook config: `apps/celilo/src/hooks/load-hook-config.ts`.
+- **Deploy pipeline** — `apps/celilo/src/services/module-deploy.ts`.
+- **DNS provider backfill** (re-emit registrations when a provider deploys) — `apps/celilo/src/services/dns-provider-backfill.ts` — `isDnsInternalProvider`, `backfillProviderDns`.
+- **Base-module aspects (fan-out across the fleet)** — `apps/celilo/src/services/aspect-runner.ts` — `planAspectFanOut`, `runAspectFanOut`, `maybeRunAspectForTrigger`. Aspect content lives in `modules/<m>/base-module-aspect/` (e.g. knot-unbound-internal, technitium).
+
+## Generation & templating
+
+- **Generator** — `apps/celilo/src/templates/generator.ts` — `generateTemplates` (orchestration), plus Terraform/Ansible file handling.
+- **Variable resolution** — `apps/celilo/src/variables/resolver.ts` (parser: `apps/celilo/src/variables/parser.ts`). Supported prefixes: `$self`, `$system`, `$secret`, `$system_secret`, `$capability`, `$infra`.
+
+## Secrets
+
+- **Vault / encryption** — `apps/celilo/src/secrets/vault.ts` (`deriveVaultPassword`, `getVaultPassword` — Ansible Vault) and `apps/celilo/src/secrets/encryption.ts` (`encryptSecret`/`decryptSecret`, AES-256-GCM).
+
+## Packaging, registry & publish
+
+- **Module packaging** — `apps/celilo/src/module/packaging/` — `build.ts` (`buildModule`), `extract.ts`, `checksum.ts`, `signature.ts` (`signChecksums`/`verifySignature`), `release-metadata.ts`, `audit.ts`.
+- **Publish driver** — `scripts/publish.ts` shims to `apps/celilo/src/cli/commands/publish/` (workspace npm packages via `bun publish` + module registry; preflight stale-version/stale-manifest gates).
+
+## Persistence
+
+- **DB schema** — `apps/celilo/src/db/schema.ts`. Client: `apps/celilo/src/db/client.ts`. Migration runner: `apps/celilo/src/db/migrate.ts`. Migrations: `apps/celilo/drizzle/`.
+
+## Events
+
+- **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`).
+
+## E2E simulation
+
+- **cele2e harness** — `packages/e2e/src/` — `runner.ts`, `container-manager.ts` (`startNetwork`, `reconnectNetwork`), `network-builder.ts` (`NetworkBuilder`).
+- **Run wrapper** — `infra/scripts/cele2e-run.sh` lives in the separate `infra/` clone, **not** in this repo. See the cele2e section of the repo-root `CLAUDE.md` for the operator workflow.