@celilo/cli 0.25.0 → 0.25.1

package/CELILO_CORE_MODULES.md

@@ -27,7 +27,7 @@ Each entry: `module id` — what it is — **provides** / **requires** capabilit
 - **knot-unbound-internal** — split-horizon internal DNS via Knot (authoritative) + Unbound (recursive); lightweight, plain apt, no .NET. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/knot-unbound-internal/base-module-aspect/`).
 - **technitium** — internal split-horizon DNS resolver + authoritative server (web UI + HTTP API); heavier alternative to knot-unbound. **provides:** `dns_internal`. Ships a base-module-aspect (`modules/technitium/base-module-aspect/`).
 - **namecheap** — public DNS A-record management via Namecheap Dynamic DNS API (HTTP, no browser automation). A caller supplies a NAME and nothing else: the address is the source IP of celilo's own update, re-derived on every assert. Registering `<domain>` also claims `www.<domain>` and vice versa (best effort, reported back as `outputs.companion_fqdn` so the framework's `public_dns` check watches it — Namecheap answers `ErrCount 0` for `www` updates it does not apply). DDNS passwords are keyed by the **registrable domain**, never the FQDN. **provides:** `dns_registrar`.
-- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. **REQUIRES the `control-plane-vpn` network and READS its range** (`requires.networks`; `client_subnet` is a `source: system` derive of `network.control-plane-vpn.subnet`). It does not write that network and has no way to — celilo owns the namespace, and the deploy will not reach any hook until the network is defined, asking for a range if one is missing. So `wg0` is attributable the moment it exists. This replaces a declare-before-you-create ordering inside `on_install`, which could only narrow the window and not close it: a consumer that captured config before the hook started could not see a value the hook wrote, whatever order it wrote it in, and that is exactly what left `wg0` unattributable (celilo#759). `health_check` still asserts the declaration matches what the tunnel serves — what it catches now is divergence, celilo's network having changed since the module resolved its config. That same key is what the internal resolver's split-horizon view consumes. Adopts a running tunnel in place (existing key and peers retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
+- **wireguard** — owns the admin WireGuard tunnel on the firewall host: interface, listen port, peers (as records), and client subnet are module config rather than hand-maintained state. Exposes the listen port and **registers the client subnet as a trusted source**, so VPN reach into the managed zones is in the firewall registry and every converge re-emits it. **REQUIRES the `control-plane-vpn` network and READS its range** (`requires.networks`; `client_subnet` is a `source: system` derive of `network.control-plane-vpn.subnet`). It does not write that network and has no way to — celilo owns the namespace, and the deploy will not reach any hook until the network is defined, asking for a range if one is missing. So `wg0` is attributable the moment it exists. This replaces a declare-before-you-create ordering inside `on_install`, which could only narrow the window and not close it: a consumer that captured config before the hook started could not see a value the hook wrote, whatever order it wrote it in, and that is exactly what left `wg0` unattributable (celilo#759). `health_check` still asserts the declaration matches what the tunnel serves — what it catches now is divergence, celilo's network having changed since the module resolved its config. That same key is what the internal resolver's split-horizon view consumes. Adopts a running tunnel in place (existing key retained; `wg syncconf`, never `wg-quick down`) because that tunnel is the operator's recovery path. **Adoption is a one-time IMPORT and it ENDS.** The marker is `registered_peers` — celilo's own config key, separate from the operator's `peers` so a machine can never rewrite what an operator typed. While that key has never been written the tunnel is not yet celilo's; the deploy that writes it (even as `[]`) CLAIMS the tunnel, importing whatever `[Peer]` blocks are running, and from then on the render is closed-world (`peers` ∪ `registered_peers`) with a `[Peer]` on the box in neither reported as drift by `on_install` and by `health_check`'s `unknown_peers`. An ABSENT `peers` declaration and a declared-EMPTY one are deliberately different (no `default: []` on either variable): they used to be the same value, so the module read every empty list as "adopt what is running", the last peer could not be revoked, and a hand-added peer rode along on every deploy unreported (celilo#765). On a tunnel celilo has not yet claimed, a declared-empty `peers` is REFUSED rather than obeyed — celilo's own variable-default seeding wrote a real `peers = []` row for every install of the previous version that never set one, so on the installed base an empty list cannot be told from a stored default, and obeying it would `wg syncconf` the admin tunnel down to zero peers. **requires:** `firewall` (and the provider must support trusted-source registration — `iptables` does; the ISP-router drivers `greenwave` and `axon` do not).
 
 ## Public edge (ingress / identity)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "0.25.0",
+  "version": "0.25.1",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -61,7 +61,7 @@
     "@celilo/capabilities": "^1.2.0",
     "@celilo/cli-display": "^0.2.0",
     "@celilo/core": "^0.8.0",
-    "@celilo/event-bus": "^0.5.0",
+    "@celilo/event-bus": "^0.6.0",
     "ajv": "^8.18.0",
     "drizzle-orm": "^0.36.4",
     "ink": "^7.0.1",

package/src/cli/stdout-is-undecorated.test.ts

@@ -54,7 +54,13 @@ describe('celilo#699 — stdout is undecorated', () => {
     // an empty roster would pass every assertion below without testing them.
     const imported = celilo('module import ../../modules/celilo-mgmt');
     expect(imported.status, `module import failed:\n${imported.stderr}`).toBe(0);
-  });
+    // The spawn above is budgeted 60s, but bun's DEFAULT hook timeout is 5s, so
+    // the hook killed the import at 5001ms and the failure surfaced as
+    // `status: null` with an empty stderr — which reads as "module import is
+    // broken" rather than "this hook is not allowed to take as long as the work
+    // inside it". `module import` runs ansible-galaxy, measured at ~7s here; CI
+    // is under 5s, so the suite is green there and red on a slower machine.
+  }, 90_000);
 
   afterAll(async () => {
     await ctx.cleanup();

package/src/db/client.test.ts

@@ -0,0 +1,63 @@
+import { Database } from 'bun:sqlite';
+import { afterEach, describe, expect, test } from 'bun:test';
+import { rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createDbClient } from './client';
+
+// #798: a command that opened the db while a deploy still held it died on
+// SQLITE_BUSY immediately, because the connection carried no busy timeout.
+describe('createDbClient concurrency', () => {
+  const paths: string[] = [];
+
+  const freshPath = () => {
+    const path = join(tmpdir(), `celilo-client-test-${Bun.nanoseconds()}.db`);
+    paths.push(path);
+    return path;
+  };
+
+  afterEach(() => {
+    for (const path of paths.splice(0)) {
+      for (const suffix of ['', '-wal', '-shm']) {
+        rmSync(`${path}${suffix}`, { force: true });
+      }
+    }
+  });
+
+  test('sets a non-zero busy timeout so a contended open waits instead of throwing', () => {
+    const { $client } = createDbClient({ path: freshPath() });
+
+    const timeout = $client.query<{ timeout: number }, []>('PRAGMA busy_timeout').get()?.timeout;
+
+    // Red before #798: bun:sqlite defaults to 0, i.e. fail on first contention.
+    expect(timeout).toBeGreaterThan(0);
+  });
+
+  test('opens in WAL mode', () => {
+    const { $client } = createDbClient({ path: freshPath() });
+
+    const mode = $client.query<{ journal_mode: string }, []>('PRAGMA journal_mode').get();
+
+    expect(mode?.journal_mode.toLowerCase()).toBe('wal');
+  });
+
+  test('a second open succeeds while another connection holds a write transaction', () => {
+    const path = freshPath();
+    createDbClient({ path });
+
+    // Hold the write lock the way a deploy in progress does.
+    const writer = new Database(path);
+    writer.run(`PRAGMA busy_timeout = ${0}`);
+    writer.run('BEGIN IMMEDIATE');
+
+    try {
+      // Reading is what `module show-config` does; it must not throw.
+      const { $client } = createDbClient({ path, readonly: true });
+      const mode = $client.query<{ journal_mode: string }, []>('PRAGMA journal_mode').get();
+      expect(mode?.journal_mode.toLowerCase()).toBe('wal');
+    } finally {
+      writer.run('ROLLBACK');
+      writer.close();
+    }
+  });
+});

package/src/db/client.ts

@@ -2,6 +2,7 @@ import { Database } from 'bun:sqlite';
 import { existsSync, mkdirSync } from 'node:fs';
 import { dirname, join } from 'node:path';
 import { fileURLToPath } from 'node:url';
+import { BUSY_TIMEOUT_MS, ensureWalMode } from '@celilo/event-bus/wal';
 import { drizzle } from 'drizzle-orm/bun-sqlite';
 import { migrate } from 'drizzle-orm/bun-sqlite/migrator';
 import { getDbPath } from '../config/paths';
@@ -60,12 +61,18 @@ export function createDbClient(config?: Partial<DatabaseConfig>) {
     create: true,
   });
 
+  // Set FIRST so it covers every statement below, including the migrations. A
+  // command that opens the db while a deploy still holds it used to die on
+  // SQLITE_BUSY immediately instead of waiting (#798).
+  sqlite.run(`PRAGMA busy_timeout = ${BUSY_TIMEOUT_MS}`);
+
   // Enable foreign keys
   sqlite.run('PRAGMA foreign_keys = ON');
 
-  // Enable WAL mode for better concurrency
+  // Enable WAL mode for better concurrency. busy_timeout does NOT cover this
+  // transition, so the switch needs its own retry — see ensureWalMode.
   if (!readonly) {
-    sqlite.run('PRAGMA journal_mode = WAL');
+    ensureWalMode(sqlite);
   }
 
   const db = drizzle(sqlite, { schema });

package/src/templates/dns-ingress-ip.test.ts

@@ -0,0 +1,123 @@
+/**
+ * The DNS-ingress allocate-and-reserve guard (ISS-0156).
+ *
+ * This invariant had NO test. Losing it is not a crash: `module generate`
+ * re-allocates a different address on every run, silently moving the address
+ * internal clients use to reach DNS, while every command still reports success.
+ * `module generate` runs repeatedly over a module's life, so "on the second
+ * run" is the normal case, not an edge one.
+ */
+
+import { beforeEach, describe, expect, test } from 'bun:test';
+import type { DbClient } from '../db/client';
+import type { ModuleManifest } from '../manifest/schema';
+import { getModuleConfigValue } from '../services/module-config';
+import { setupTestDatabase } from '../test-utils/database';
+import { ensureDnsIngressIp } from './generator';
+
+let db: DbClient;
+
+/** A manifest that opts in the way a `dns_internal` provider does. */
+const wantsIngress = {
+  variables: { owns: [{ name: 'dns_ingress_ip', source: 'infrastructure' }] },
+} as unknown as ModuleManifest;
+
+/** Same shape, but the variable is operator input rather than infrastructure. */
+const operatorSupplied = {
+  variables: { owns: [{ name: 'dns_ingress_ip', source: 'user_input' }] },
+} as unknown as ModuleManifest;
+
+/**
+ * Typed as `string | undefined` rather than `unknown`: every assertion here is
+ * about an address, and an untyped read pushes a cast onto each one.
+ */
+const storedIp = (moduleId: string): string | undefined => {
+  const value = getModuleConfigValue(moduleId, 'dns_ingress_ip', db)?.value;
+  return typeof value === 'string' ? value : undefined;
+};
+
+beforeEach(async () => {
+  db = await setupTestDatabase();
+  db.$client
+    .prepare('INSERT OR REPLACE INTO system_config (key, value) VALUES (?, ?)')
+    .run('network.internal.subnet', '10.226.1.0/24');
+});
+
+describe('ensureDnsIngressIp', () => {
+  test('allocates an address from the internal subnet on first generate', async () => {
+    const result = await ensureDnsIngressIp('technitium', wantsIngress, db);
+
+    expect(result.success).toBe(true);
+    expect(storedIp('technitium')).toMatch(/^10\.226\.1\.\d+$/);
+  });
+
+  test('REUSES the same address on a second generate', async () => {
+    // The guard itself. Re-allocating here moves the resolver's DNAT ingress
+    // every time the module is regenerated, and nothing reports a problem.
+    await ensureDnsIngressIp('technitium', wantsIngress, db);
+    const first = storedIp('technitium');
+
+    await ensureDnsIngressIp('technitium', wantsIngress, db);
+    const second = storedIp('technitium');
+
+    expect(second).toBe(first);
+  });
+
+  test('stays stable across many generates, not just two', async () => {
+    await ensureDnsIngressIp('technitium', wantsIngress, db);
+    const first = storedIp('technitium');
+
+    for (let i = 0; i < 5; i++) {
+      await ensureDnsIngressIp('technitium', wantsIngress, db);
+    }
+
+    expect(storedIp('technitium')).toBe(first);
+  });
+
+  test('RESERVES the address, so it is never handed out to something else', async () => {
+    // Allocation without reservation is the same bug one step later: a
+    // container gets the resolver's ingress address and DNS goes dark.
+    await ensureDnsIngressIp('technitium', wantsIngress, db);
+    const ip = storedIp('technitium');
+
+    const reserved = db.$client
+      .prepare('SELECT ip_start, reason FROM ip_reservations WHERE ip_start = ?')
+      .get(ip ?? '') as { ip_start: string; reason: string } | undefined;
+
+    expect(reserved?.ip_start).toBe(ip);
+    // The reason names the owner, so an operator reading the table can tell
+    // what an otherwise anonymous held address is for.
+    expect(reserved?.reason).toBe('dns-ingress:technitium');
+  });
+
+  test('two modules get two different addresses', async () => {
+    await ensureDnsIngressIp('technitium', wantsIngress, db);
+    await ensureDnsIngressIp('knot-unbound-internal', wantsIngress, db);
+
+    expect(storedIp('knot-unbound-internal')).not.toBe(storedIp('technitium'));
+  });
+
+  test('does nothing for a module that never asked for one', async () => {
+    const result = await ensureDnsIngressIp('caddy', {} as ModuleManifest, db);
+
+    expect(result.success).toBe(true);
+    expect(storedIp('caddy')).toBeUndefined();
+  });
+
+  test('only `source: infrastructure` opts in', async () => {
+    // A same-named variable the operator supplies is theirs to set; allocating
+    // over it would overwrite an operator's deliberate choice.
+    await ensureDnsIngressIp('technitium', operatorSupplied, db);
+
+    expect(storedIp('technitium')).toBeUndefined();
+  });
+
+  test('fails with an actionable message when the internal subnet is unset', async () => {
+    db.$client.prepare('DELETE FROM system_config WHERE key = ?').run('network.internal.subnet');
+
+    const result = await ensureDnsIngressIp('technitium', wantsIngress, db);
+
+    expect(result.success).toBe(false);
+    expect(result.success === false && result.error).toContain('network.internal.subnet');
+  });
+});

package/src/templates/generator.ts

@@ -653,6 +653,70 @@ celilo module import modules/${moduleId}
  * @param options - Generation options
  * @returns Generation result
  */
+/** Narrower than `GenerateResult`: this step produces no files, only an outcome. */
+export type DnsIngressResult = { success: true } | { success: false; error: string };
+
+/**
+ * Allocate-and-reserve the module's dedicated DNS-ingress IP, ONCE (ISS-0156).
+ *
+ * A `dns_internal` provider deploys into a PROTECTED zone (dmz) so it can see
+ * protected-zone query sources for split-horizon views. `internal` devices have
+ * no route into the 10-net, so they reach the resolver through a firewall DNAT
+ * on a dedicated `internal`-subnet address. A module opts in by declaring a
+ * `dns_ingress_ip` infrastructure variable.
+ *
+ * **Idempotence is the whole point, and it is load-bearing.** `module generate`
+ * runs repeatedly over a module's life. Re-allocating here on the second run
+ * would move the address internal clients use to reach DNS, every time — while
+ * every command still reports success. That is why the stored value is reused
+ * rather than re-derived, and why this is a named function instead of a branch
+ * buried in `generateTemplates`: an invariant nothing can call is an invariant
+ * nothing can test, and this one had no test at all.
+ */
+export async function ensureDnsIngressIp(
+  moduleId: string,
+  manifest: ModuleManifest,
+  db: DbClient,
+): Promise<DnsIngressResult> {
+  const wantsDnsIngress = manifest.variables?.owns?.some(
+    (v) => v.name === 'dns_ingress_ip' && v.source === 'infrastructure',
+  );
+  if (!wantsDnsIngress) return { success: true };
+
+  const existing = getModuleConfigValue(moduleId, 'dns_ingress_ip', db)?.value;
+  if (typeof existing === 'string' && existing.length > 0) {
+    log.success(`Using existing DNS-ingress IP ${existing} for ${moduleId}`);
+    return { success: true };
+  }
+
+  const subnetRow = db.$client
+    .prepare('SELECT value FROM system_config WHERE key = ?')
+    .get('network.internal.subnet') as { value: string } | undefined;
+  if (!subnetRow?.value) {
+    return {
+      success: false,
+      error:
+        'network.internal.subnet is not configured — required to allocate the ' +
+        'dns_internal DNS-ingress IP (ISS-0156). Ensure the internal network is set up first.',
+    };
+  }
+
+  const { allocateIPFromSubnet, reserveIP } = await import('../ipam/allocator');
+  const { stripCIDR } = await import('../ipam/subnet-parser');
+  try {
+    const ip = stripCIDR(await allocateIPFromSubnet(subnetRow.value, 'internal', db));
+    await reserveIP(ip, 'internal', `dns-ingress:${moduleId}`, null, db);
+    upsertModuleConfig(db, moduleId, 'dns_ingress_ip', ip);
+    log.success(`Allocated DNS-ingress IP ${ip} (internal subnet) for ${moduleId}`);
+    return { success: true };
+  } catch (error) {
+    return {
+      success: false,
+      error: `DNS-ingress IP allocation failed: ${error instanceof Error ? error.message : String(error)}`,
+    };
+  }
+}
+
 export async function generateTemplates(options: GenerateOptions): Promise<GenerateResult> {
   const { moduleId, modulePath, outputPath, db = getDb() } = options;
 
@@ -815,49 +879,8 @@ export async function generateTemplates(options: GenerateOptions): Promise<Gener
     }
   }
 
-  // Dedicated DNS-ingress IP (ISS-0156). A dns_internal provider now deploys into
-  // a PROTECTED zone (dmz) so it can see protected-zone query sources for
-  // split-horizon views (openspec/specs/internal-dns-zone-views/spec.md). `internal` devices have
-  // no route into the 10-net, so they reach the resolver through a firewall DNAT
-  // on a dedicated `internal`-subnet address. A module opts in by declaring a
-  // `dns_ingress_ip` infrastructure variable; we allocate a free IP from the
-  // `internal` subnet via IPAM and RESERVE it (so it's never re-handed-out),
-  // idempotently (reuse the stored value on re-generate). The resolver's
-  // on_install passes it to firewall.exposeService({ ingressIp }).
-  const wantsDnsIngress = manifest.variables?.owns?.some(
-    (v) => v.name === 'dns_ingress_ip' && v.source === 'infrastructure',
-  );
-  if (wantsDnsIngress) {
-    const existing = getModuleConfigValue(moduleId, 'dns_ingress_ip', db)?.value;
-    if (typeof existing === 'string' && existing.length > 0) {
-      log.success(`Using existing DNS-ingress IP ${existing} for ${moduleId}`);
-    } else {
-      const subnetRow = db.$client
-        .prepare('SELECT value FROM system_config WHERE key = ?')
-        .get('network.internal.subnet') as { value: string } | undefined;
-      if (!subnetRow?.value) {
-        return {
-          success: false,
-          error:
-            'network.internal.subnet is not configured — required to allocate the ' +
-            'dns_internal DNS-ingress IP (ISS-0156). Ensure the internal network is set up first.',
-        };
-      }
-      const { allocateIPFromSubnet, reserveIP } = await import('../ipam/allocator');
-      const { stripCIDR } = await import('../ipam/subnet-parser');
-      try {
-        const ip = stripCIDR(await allocateIPFromSubnet(subnetRow.value, 'internal', db));
-        await reserveIP(ip, 'internal', `dns-ingress:${moduleId}`, null, db);
-        upsertModuleConfig(db, moduleId, 'dns_ingress_ip', ip);
-        log.success(`Allocated DNS-ingress IP ${ip} (internal subnet) for ${moduleId}`);
-      } catch (error) {
-        return {
-          success: false,
-          error: `DNS-ingress IP allocation failed: ${error instanceof Error ? error.message : String(error)}`,
-        };
-      }
-    }
-  }
+  const dnsIngress = await ensureDnsIngressIp(moduleId, manifest, db);
+  if (!dnsIngress.success) return dnsIngress;
 
   // Infrastructure Properties Resolution (Proxmox provider config)
   // For Proxmox services, extract provider config and store as temporary values