@celilo/cli 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CELILO_SUBSYSTEMS.md +12 -0
- package/package.json +2 -2
- package/src/cli/commands/hook-run.ts +5 -8
- package/src/cli/commands/system-audit.ts +2 -0
- package/src/cli/commands/system-doctor.ts +30 -1
- package/src/cli/commands/system-update.ts +2 -0
- package/src/cli/tui/audit-state.ts +2 -0
- package/src/db/schema.ts +41 -1
- package/src/hooks/artifact-retention.test.ts +136 -0
- package/src/hooks/artifact-retention.ts +159 -0
- package/src/hooks/executor.test.ts +80 -0
- package/src/hooks/executor.ts +68 -23
- package/src/hooks/test-fixtures/artifact-writing-hook.ts +25 -0
- package/src/hooks/types.ts +20 -2
- package/src/policy/module-business-baseline.ts +404 -0
- package/src/policy/no-module-business-in-core.test.ts +504 -0
- package/src/services/alerting/keys.ts +21 -1
- package/src/services/alerting/run-monitor.ts +6 -1
- package/src/services/audit/browser-pin.test.ts +167 -0
- package/src/services/audit/browser-pin.ts +185 -0
- package/src/services/audit/index.test.ts +1 -0
- package/src/services/audit/index.ts +3 -0
- package/src/services/audit/types.ts +1 -0
- package/src/services/health-runner.ts +15 -1
- package/src/services/module-deploy.ts +4 -4
- package/src/services/update/orchestrator.test.ts +1 -0
- package/src/system/browser-provisioning.test.ts +67 -0
- package/src/system/prereqs.test.ts +73 -0
- package/src/system/prereqs.ts +89 -12
package/CELILO_SUBSYSTEMS.md
CHANGED
|
@@ -200,6 +200,18 @@ warning naming every paused module and its age onto EVERY management-API result
|
|
|
200
200
|
That last one is the important half: it does not depend on the operator choosing
|
|
201
201
|
to look, which is how a forgotten pause actually gets found.
|
|
202
202
|
|
|
203
|
+
## Management-host browser runtime
|
|
204
|
+
|
|
205
|
+
- **The browser celilo provisions** — installed by `modules/celilo-mgmt/ansible/roles/celilo-mgmt/tasks/debian.yml` into `/var/lib/celilo/browsers`, behind `install_browser` (a manifest boolean **defaulting to false**, exactly how `install_docker` / `install_terraform` gate third-party-repo tooling: ~170 MB from an external CDN). The pin is `playwright_version` in `modules/celilo-mgmt/manifest.yml` — the fleet's single browser-version declaration. What is UNCONDITIONAL is the *declaration*, so a host that needs a browser and lacks one says so. It provisions `chromium-headless-shell`, **not** full Chromium: both builds on celilo-mgr are headless shells and `chromium.launch()` has been driving one in production the whole time, so the full browser would change what works rather than preserve it. Accepted ceiling: no headed browsing. macOS is an explicit `fail`, not a silent skip.
|
|
206
|
+
- **The seam** — `packages/capabilities/src/browser.ts` — `resolveBrowser()` returns `{ browser, flavor, executablePath, browserVersion, playwrightVersion, revision }` read from the stable symlink `/var/lib/celilo/browsers/current/chrome` plus the descriptor `current/celilo-browser.json` the install writes. Consumers pass `executablePath` to their launcher, which bypasses Playwright's revision-keyed cache resolution — the revision is a property of the CLIENT version (1.55.1 → 1193, 1.60.0 → 1223), so without it a consumer's client bump silently moves the browser out from under it. **A plain exported function, deliberately NOT a capability and NOT a `HookContext` field**: an event-bus subscriber runs as its own subprocess with no hook context at all, and `lunacycle` — the one consumer — launches from both a hook and a subscriber. There is no `CapabilityRegistry` / `KNOWN_CAPABILITY_NAMES` entry and there should not be one.
|
|
207
|
+
- **Two failures, two severities** — `BrowserUnavailableError.reason` is `'not_provisioned' | 'unusable'`, and the discriminant is load-bearing. Installation is opt-in, so *absent* is the NORMAL state of a host; a consumer reports it non-fatally. A module throws on any failing check item and celilo maps any fail → unhealthy → the module stays INSTALLED, so a single severity would take every browser-using module permanently unverified on any host that had not flipped a flag it never knew about. *Unusable* — dangling symlink, not executable, unreadable descriptor — is a regression and fails. **Provisioning status is decided from the BINARY, never from a directory existing**: a build directory present with no executable inside it satisfies every path check and then fails at launch.
|
|
208
|
+
- **`system doctor` rows** — `apps/celilo/src/system/prereqs.ts` — `browser` and `fonts` joined `PREREQUISITES`. `PrerequisiteSpec.command` is the second check kind: an ABSOLUTE path switches presence from `command -v` to exists + executable + reports-a-version. ⚠️ **Never ask Playwright which executable it would use.** `chromium.executablePath()` reports the FULL browser while a headless launch opens the SHELL — measured disagreeing on one machine in one run — so on a shell-only host it validates a path that is not there while the binary that actually runs is fine. Because a not-provisioned browser is deliberately not a check failure, this row is the ONLY place an operator learns the host has none. `fonts` runs `fc-match sans-serif`, which resolves an actual face rather than merely proving fontconfig is installed, and displays the family — that is what decides whether a retained screenshot has legible glyphs.
|
|
209
|
+
- **Per-run artifact directory** — `moduleArtifactDir(modulePath, runKey)` in the same file is the one definition of the module store's artifact layout, consumed by `apps/celilo/src/hooks/executor.ts` when it creates `HookContext.screenshotDir` and reachable from a bus subscriber that has no context. Per-run because a consumer writing fixed filenames — a reasonable thing to do — would otherwise overwrite itself every run and leave retention nothing to retain. The executor discards the directory when the hook wrote nothing, so an idle module accrues none; note it is created only AFTER every early return, because the capability pre-flight returns outside the `try/finally` that reclaims it and an earlier `mkdir` leaked one empty directory per affected run, forever.
|
|
210
|
+
- **Artifacts reach the operator** — the executor collects EVERY file written to the run directory (`collectArtifacts`), carried as `HookResult.artifactPaths` (there is no `screenshotPath`; it was deleted, not deprecated). `apps/celilo/src/services/health-runner.ts` carries them onto `HealthCheckResult` — including the hook-FAILED branch, where they matter most because there are no named checks to explain anything — and `failingKeysFromHealthItems` (`services/alerting/keys.ts`) appends them to each failing item's `details`, which is the field that actually reaches an alert. Collecting them and stopping short of `details` would accomplish nothing. `apps/celilo/src/hooks/types.ts` exports `describeArtifacts`, the one renderer the three error-message sites share.
|
|
211
|
+
- **Retention is by AGE, never by run count** — `apps/celilo/src/hooks/artifact-retention.ts` — 24 h plus a per-module byte ceiling, pruned on write so it needs no scheduler. Count-based pruning is the trap: for a persistent failure the FIRST artifact set carries the original cause and later ones repeat it, so "keep the last 5" discards the useful one within about an hour at a 15-minute cadence. A run is aged by its NEWEST file, not the directory's mtime, so an in-flight run cannot be evicted underneath itself. Artifacts are excluded from module backups (`modules/celilo-mgmt/scripts/on_backup.ts` `EXCLUDE_DIRS`) and preserved across `module update`.
|
|
212
|
+
- **Pin-drift guardrail** — `apps/celilo/src/services/audit/browser-pin.ts` — the `browser_pin` drift category compares each module's BUNDLED `playwright-core` against the provisioned browser, and it compares **revisions, not version strings**: the revision is what has to match a build, and each client states its own in the `browsers.json` inside the package, so nobody maintains a version→revision table that would rot. It WARNS (`drift`) and never blocks — under D2 the consumer passes an explicit `executablePath`, so a mismatch is protocol skew rather than a failure. Silent when no browser is provisioned (the opt-in default) and when no module bundles a client (almost all of them); a check that fired in either case would put a permanent finding on every host in the fleet. Adding a `DriftCategory` does NOT create a monitor — only 3 of the 21 categories have one — so this is a `system audit` finding, not a page (D4).
|
|
213
|
+
- Design and the declined alternatives (no probe capability, no fourth `builtin_check`, no concurrency semaphore, no off-box prober — each with its trigger): `openspec/changes/managed-browser-runtime/`.
|
|
214
|
+
|
|
203
215
|
## Generation & templating
|
|
204
216
|
|
|
205
217
|
- **Generator** — `apps/celilo/src/templates/generator.ts` — `generateTemplates` (orchestration), plus Terraform/Ansible file handling.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@celilo/cli",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.2.0",
|
|
4
4
|
"description": "Celilo — home lab orchestration CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -58,7 +58,7 @@
|
|
|
58
58
|
"dependencies": {
|
|
59
59
|
"@aws-sdk/client-s3": "^3.1109.0",
|
|
60
60
|
"@aws-sdk/lib-storage": "^3.1101.0",
|
|
61
|
-
"@celilo/capabilities": "^2.
|
|
61
|
+
"@celilo/capabilities": "^2.1.0",
|
|
62
62
|
"@celilo/cli-display": "^0.2.0",
|
|
63
63
|
"@celilo/core": "^0.8.1",
|
|
64
64
|
"@celilo/event-bus": "^0.6.0",
|
|
@@ -13,6 +13,7 @@ import { getDb } from '../../db/client';
|
|
|
13
13
|
import { modules } from '../../db/schema';
|
|
14
14
|
import { createConsoleLogger, createGaugeLogger } from '../../hooks/logger';
|
|
15
15
|
import { runNamedHook } from '../../hooks/run-named-hook';
|
|
16
|
+
import { describeArtifacts } from '../../hooks/types';
|
|
16
17
|
import type { HookName } from '../../hooks/types';
|
|
17
18
|
import type { ModuleManifest } from '../../manifest/schema';
|
|
18
19
|
import { getArg, hasFlag, validateRequiredArgs } from '../parser';
|
|
@@ -83,10 +84,8 @@ export async function handleHookRun(
|
|
|
83
84
|
});
|
|
84
85
|
|
|
85
86
|
if (!result.success) {
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
errorMsg += `\n\nScreenshot saved: ${result.screenshotPath}`;
|
|
89
|
-
}
|
|
87
|
+
const errorMsg =
|
|
88
|
+
(result.error || 'Hook execution failed') + describeArtifacts(result.artifactPaths);
|
|
90
89
|
return { success: false, error: errorMsg };
|
|
91
90
|
}
|
|
92
91
|
|
|
@@ -109,10 +108,8 @@ export async function handleHookRun(
|
|
|
109
108
|
|
|
110
109
|
if (!result.success) {
|
|
111
110
|
gauge.stop(false);
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
errorMsg += `\n\nScreenshot saved: ${result.screenshotPath}`;
|
|
115
|
-
}
|
|
111
|
+
const errorMsg =
|
|
112
|
+
(result.error || 'Hook execution failed') + describeArtifacts(result.artifactPaths);
|
|
116
113
|
return { success: false, error: errorMsg };
|
|
117
114
|
}
|
|
118
115
|
|
|
@@ -40,6 +40,7 @@ import { runAudit } from '../../services/audit';
|
|
|
40
40
|
import type { DriftFinding, SystemAuditReport } from '../../services/audit';
|
|
41
41
|
import { loadAbandonedOperations } from '../../services/audit/abandoned-operations';
|
|
42
42
|
import { loadBackupAuditInfo } from '../../services/audit/backup-source';
|
|
43
|
+
import { collectBrowserPinDeps } from '../../services/audit/browser-pin';
|
|
43
44
|
import {
|
|
44
45
|
type LatestCliVersionFetcher,
|
|
45
46
|
fetchLatestCliVersion,
|
|
@@ -358,6 +359,7 @@ async function buildAuditDeps(onProgress?: (msg: string) => void) {
|
|
|
358
359
|
manifest: m.manifestData as ModuleManifest,
|
|
359
360
|
})),
|
|
360
361
|
},
|
|
362
|
+
browserPin: collectBrowserPinDeps(deployedModules),
|
|
361
363
|
terraformPlan: {
|
|
362
364
|
modules: terraformModules,
|
|
363
365
|
run: realTerraformPlan,
|
|
@@ -33,6 +33,7 @@ import { spawnSync } from 'node:child_process';
|
|
|
33
33
|
import { existsSync, readFileSync, statSync } from 'node:fs';
|
|
34
34
|
import { createRequire } from 'node:module';
|
|
35
35
|
import { dirname, join, resolve } from 'node:path';
|
|
36
|
+
import { resolveBrowser } from '@celilo/capabilities';
|
|
36
37
|
import { defineEvents, openBus } from '@celilo/event-bus';
|
|
37
38
|
import cliPkg from '../../../package.json' with { type: 'json' };
|
|
38
39
|
import { getDbPath, getEventBusPath } from '../../config/paths';
|
|
@@ -279,6 +280,7 @@ function applyFix(drifted: DriftedDep[], cliRoot: string): string[] {
|
|
|
279
280
|
function renderPrereqSection(): { lines: string[]; failingCount: number } {
|
|
280
281
|
const checks = checkAllPrerequisites();
|
|
281
282
|
const lines: string[] = [];
|
|
283
|
+
const browserFlavor = provisionedBrowserFlavor();
|
|
282
284
|
|
|
283
285
|
lines.push('System prerequisites');
|
|
284
286
|
// Right-pad column for alignment. Take the longest tool name +1
|
|
@@ -288,7 +290,16 @@ function renderPrereqSection(): { lines: string[]; failingCount: number } {
|
|
|
288
290
|
for (const c of checks) {
|
|
289
291
|
if (c.present && c.meetsMinimum) {
|
|
290
292
|
const version = c.version ? c.version : `${ANSI.dim}(version unknown)${ANSI.reset}`;
|
|
291
|
-
|
|
293
|
+
// Flavour rides on the browser row because the ceiling it names is
|
|
294
|
+
// real — a headless-only build cannot run headed — and an operator
|
|
295
|
+
// reading doctor is the other party who benefits from that being
|
|
296
|
+
// legible rather than discovered by a launch failure. It cannot come
|
|
297
|
+
// from the check itself: a headless shell reports
|
|
298
|
+
// "Google Chrome for Testing <v>", the same string a full build
|
|
299
|
+
// prints, so only the descriptor knows which one is installed.
|
|
300
|
+
const flavor =
|
|
301
|
+
c.name === 'browser' && browserFlavor ? ` ${ANSI.dim}(${browserFlavor})${ANSI.reset}` : '';
|
|
302
|
+
lines.push(` ${ANSI.green}✔${ANSI.reset} ${c.name.padEnd(nameCol)} ${version}${flavor}`);
|
|
292
303
|
} else if (c.present && !c.meetsMinimum) {
|
|
293
304
|
// Present but below minimum (or version-parse failed with a minimum).
|
|
294
305
|
const detail = c.version ? `${c.version} — below minimum required` : 'version unreadable';
|
|
@@ -306,6 +317,24 @@ function renderPrereqSection(): { lines: string[]; failingCount: number } {
|
|
|
306
317
|
return { lines, failingCount: failingPrerequisites(checks).length };
|
|
307
318
|
}
|
|
308
319
|
|
|
320
|
+
/**
|
|
321
|
+
* Which flavour of browser is provisioned, or null when none is (or the
|
|
322
|
+
* descriptor cannot be read). Never throws — a doctor row is not worth
|
|
323
|
+
* failing the whole report over.
|
|
324
|
+
*
|
|
325
|
+
* This reads the DESCRIPTOR while the prerequisite check RUNS the binary.
|
|
326
|
+
* Those answer different questions and both are wanted: executing it is
|
|
327
|
+
* what distinguishes a real install from a build directory with no binary
|
|
328
|
+
* in it, and only the descriptor records what was installed.
|
|
329
|
+
*/
|
|
330
|
+
function provisionedBrowserFlavor(): string | null {
|
|
331
|
+
try {
|
|
332
|
+
return resolveBrowser().flavor;
|
|
333
|
+
} catch {
|
|
334
|
+
return null;
|
|
335
|
+
}
|
|
336
|
+
}
|
|
337
|
+
|
|
309
338
|
/**
|
|
310
339
|
* mtime (ms) of the installed dispatcher code (`@celilo/event-bus`
|
|
311
340
|
* package.json). The fleet dispatcher check compares this against the
|
|
@@ -27,6 +27,7 @@ import type { ModuleManifest } from '../../manifest/schema';
|
|
|
27
27
|
import { RegistryClient } from '../../registry/client';
|
|
28
28
|
import { runAudit } from '../../services/audit';
|
|
29
29
|
import { loadAbandonedOperations } from '../../services/audit/abandoned-operations';
|
|
30
|
+
import { collectBrowserPinDeps } from '../../services/audit/browser-pin';
|
|
30
31
|
import { fetchLatestCliVersion } from '../../services/audit/cli-version';
|
|
31
32
|
import { unusedPublicDnsProbe } from '../../services/audit/public-dns';
|
|
32
33
|
import { makeJournalReader, readAppliedMigrations } from '../../services/audit/schema';
|
|
@@ -536,6 +537,7 @@ export async function handleSystemUpdate(
|
|
|
536
537
|
manifest: m.manifestData as ModuleManifest,
|
|
537
538
|
})),
|
|
538
539
|
},
|
|
540
|
+
browserPin: collectBrowserPinDeps(upgradableModules),
|
|
539
541
|
terraformPlan: {
|
|
540
542
|
modules: upgradableModules.map((m) => ({
|
|
541
543
|
id: m.id,
|
|
@@ -72,6 +72,7 @@ export const ALL_CATEGORIES: readonly DriftCategory[] = [
|
|
|
72
72
|
'cli_version',
|
|
73
73
|
'schema',
|
|
74
74
|
'capability_abi',
|
|
75
|
+
'browser_pin',
|
|
75
76
|
'terraform_plan',
|
|
76
77
|
'module_versions',
|
|
77
78
|
'module_configs',
|
|
@@ -96,6 +97,7 @@ export const CATEGORY_LABELS: Record<DriftCategory, string> = {
|
|
|
96
97
|
cli_version: 'CLI version',
|
|
97
98
|
schema: 'Schema migrations',
|
|
98
99
|
capability_abi: 'Capability ABI',
|
|
100
|
+
browser_pin: 'Browser pin',
|
|
99
101
|
terraform_plan: 'Terraform plans',
|
|
100
102
|
module_versions: 'Module versions',
|
|
101
103
|
module_configs: 'Module configs',
|
package/src/db/schema.ts
CHANGED
|
@@ -60,6 +60,7 @@ export const IN_FLIGHT_STATES = [
|
|
|
60
60
|
|
|
61
61
|
/**
|
|
62
62
|
* Modules table - stores module metadata and manifest data
|
|
63
|
+
* @owner celilo — the module registry itself
|
|
63
64
|
*/
|
|
64
65
|
export const modules = sqliteTable(
|
|
65
66
|
'modules',
|
|
@@ -112,6 +113,7 @@ export const modules = sqliteTable(
|
|
|
112
113
|
* `valueJson` only for arrays/objects. That was Defect 1 — TS types
|
|
113
114
|
* generated from the manifest claimed `number` while the runtime
|
|
114
115
|
* value was string. Now closed.
|
|
116
|
+
* @owner celilo — generic per-module KV; the migration destination (T9)
|
|
115
117
|
*/
|
|
116
118
|
export const moduleConfigs = sqliteTable(
|
|
117
119
|
'module_configs',
|
|
@@ -134,6 +136,7 @@ export const moduleConfigs = sqliteTable(
|
|
|
134
136
|
/**
|
|
135
137
|
* Capabilities table - stores registered capabilities provided by modules
|
|
136
138
|
* Example: namecheap module provides dns_registrar capability
|
|
139
|
+
* @owner celilo — the brokering table, who provides what (T8)
|
|
137
140
|
*/
|
|
138
141
|
export const capabilities = sqliteTable('capabilities', {
|
|
139
142
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -154,6 +157,7 @@ export const capabilities = sqliteTable('capabilities', {
|
|
|
154
157
|
* Capability secrets table - stores encrypted secrets owned by capabilities
|
|
155
158
|
* Values are encrypted with AES-256-GCM using master key
|
|
156
159
|
* Example: dns-external capability owns TSIG secret
|
|
160
|
+
* @owner celilo — custody of ciphertext keyed by capability id; holds no format knowledge
|
|
157
161
|
*/
|
|
158
162
|
export const capabilitySecrets = sqliteTable(
|
|
159
163
|
'capability_secrets',
|
|
@@ -178,6 +182,7 @@ export const capabilitySecrets = sqliteTable(
|
|
|
178
182
|
/**
|
|
179
183
|
* Secrets table - stores encrypted secrets per module
|
|
180
184
|
* Values are encrypted with AES-256-GCM using master key
|
|
185
|
+
* @owner celilo — custody of ciphertext keyed by module; holds no format knowledge
|
|
181
186
|
*/
|
|
182
187
|
export const secrets = sqliteTable('secrets', {
|
|
183
188
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -196,6 +201,7 @@ export const secrets = sqliteTable('secrets', {
|
|
|
196
201
|
* System configuration - system-wide settings
|
|
197
202
|
* Used for $system: variables in templates
|
|
198
203
|
* Examples: DNS servers, network settings, domain names
|
|
204
|
+
* @owner celilo — generic operator KV
|
|
199
205
|
*/
|
|
200
206
|
export const systemConfig = sqliteTable('system_config', {
|
|
201
207
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -210,6 +216,7 @@ export const systemConfig = sqliteTable('system_config', {
|
|
|
210
216
|
* System secrets table - stores encrypted system-level secrets
|
|
211
217
|
* Values are encrypted with AES-256-GCM using master key
|
|
212
218
|
* Examples: Proxmox root password, API tokens, SSH keys
|
|
219
|
+
* @owner celilo — generic operator KV, encrypted
|
|
213
220
|
*/
|
|
214
221
|
export const systemSecrets = sqliteTable('system_secrets', {
|
|
215
222
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -225,6 +232,7 @@ export const systemSecrets = sqliteTable('system_secrets', {
|
|
|
225
232
|
/**
|
|
226
233
|
* Module integrity table - stores checksums and signature for package verification
|
|
227
234
|
* Used for runtime auditing to detect tampering, missing files, or extra files
|
|
235
|
+
* @owner celilo — package checksums + signature, tamper detection
|
|
228
236
|
*/
|
|
229
237
|
export const moduleIntegrity = sqliteTable('module_integrity', {
|
|
230
238
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -242,6 +250,7 @@ export const moduleIntegrity = sqliteTable('module_integrity', {
|
|
|
242
250
|
* IPAM (IP Address Management) allocations table
|
|
243
251
|
* Tracks VMID and IP address assignments per module
|
|
244
252
|
* Prevents conflicts and enables automatic allocation from zone subnets
|
|
253
|
+
* @owner celilo — IPAM is a core primitive
|
|
245
254
|
*/
|
|
246
255
|
export const ipAllocations = sqliteTable('ip_allocations', {
|
|
247
256
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -258,6 +267,7 @@ export const ipAllocations = sqliteTable('ip_allocations', {
|
|
|
258
267
|
* IP reservations table
|
|
259
268
|
* Allows users to reserve IPs for infrastructure or external services
|
|
260
269
|
* IPAM allocator skips reserved IPs
|
|
270
|
+
* @owner celilo — IPAM is a core primitive
|
|
261
271
|
*/
|
|
262
272
|
export const ipReservations = sqliteTable('ip_reservations', {
|
|
263
273
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -272,6 +282,7 @@ export const ipReservations = sqliteTable('ip_reservations', {
|
|
|
272
282
|
* VMID reservations table
|
|
273
283
|
* Allows users to reserve VMIDs for existing VMs or external systems
|
|
274
284
|
* IPAM allocator skips reserved VMIDs
|
|
285
|
+
* @owner celilo — IPAM is a core primitive; infra-provider columns; out of scope per S17, not blessed
|
|
275
286
|
*/
|
|
276
287
|
export const vmidReservations = sqliteTable('vmid_reservations', {
|
|
277
288
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -284,6 +295,7 @@ export const vmidReservations = sqliteTable('vmid_reservations', {
|
|
|
284
295
|
* Module builds table
|
|
285
296
|
* Tracks build metadata for modules with custom compilation requirements
|
|
286
297
|
* Example: Caddy with RFC2136 DNS provider, custom Go binaries
|
|
298
|
+
* @owner celilo — build metadata per module version
|
|
287
299
|
*/
|
|
288
300
|
export type BuildStatus = 'success' | 'failed' | 'in_progress';
|
|
289
301
|
|
|
@@ -381,6 +393,7 @@ export function isAllocatableZone(zone: NetworkZone): zone is AllocatableZone {
|
|
|
381
393
|
* Container services table
|
|
382
394
|
* Stores container service providers (Proxmox, Digital Ocean, etc.)
|
|
383
395
|
* that can provision new containers/VMs on demand
|
|
396
|
+
* @owner celilo — infra-provider registry; infra-provider columns; out of scope per S17, not blessed
|
|
384
397
|
*/
|
|
385
398
|
export const containerServices = sqliteTable('container_services', {
|
|
386
399
|
id: text('id').primaryKey(), // UUID
|
|
@@ -405,6 +418,7 @@ export const containerServices = sqliteTable('container_services', {
|
|
|
405
418
|
* Machines table
|
|
406
419
|
* Stores pre-existing machines (Raspberry Pi, VPS, bare metal)
|
|
407
420
|
* that users have added to the pool for hosting modules
|
|
421
|
+
* @owner celilo — the machine pool
|
|
408
422
|
*/
|
|
409
423
|
export const machines = sqliteTable('machines', {
|
|
410
424
|
id: text('id').primaryKey(), // UUID
|
|
@@ -450,6 +464,7 @@ export const machines = sqliteTable('machines', {
|
|
|
450
464
|
/**
|
|
451
465
|
* Module infrastructure table
|
|
452
466
|
* Tracks which infrastructure (machine or container service) is used for each module
|
|
467
|
+
* @owner celilo — which infra hosts which module; infra-provider columns; out of scope per S17, not blessed
|
|
453
468
|
*/
|
|
454
469
|
export const moduleInfrastructure = sqliteTable('module_infrastructure', {
|
|
455
470
|
id: text('id').primaryKey(), // UUID
|
|
@@ -491,6 +506,7 @@ export const moduleInfrastructure = sqliteTable('module_infrastructure', {
|
|
|
491
506
|
* `$infra:<name>.…`. `hostname` is the runtime DNS hostname (often == name, but
|
|
492
507
|
* user/well-known-assignable), used by DNS and events. See
|
|
493
508
|
* openspec/specs/module-systems-addressing/spec.md.
|
|
509
|
+
* @owner celilo — deployment state per addressed host; infra-provider columns; out of scope per S17, not blessed
|
|
494
510
|
*/
|
|
495
511
|
export const moduleSystems = sqliteTable(
|
|
496
512
|
'module_systems',
|
|
@@ -538,6 +554,7 @@ export const moduleSystems = sqliteTable(
|
|
|
538
554
|
* Tracks routes registered by modules via public_web capability functions.
|
|
539
555
|
* Routes are registered during on_install hooks and removed during on_uninstall.
|
|
540
556
|
* The public_web provider (Caddy) uses these to generate its configuration.
|
|
557
|
+
* @owner capability:public_web — reverse-proxy configuration; migrates to the provider (T1)
|
|
541
558
|
*/
|
|
542
559
|
export const webRoutes = sqliteTable(
|
|
543
560
|
'web_routes',
|
|
@@ -576,6 +593,7 @@ export const webRoutes = sqliteTable(
|
|
|
576
593
|
* Keyed by `firewall_ip` (the converge target) so multiple firewalls each render
|
|
577
594
|
* their own set. Shared-core (not a per-module JSON file) so any firewall
|
|
578
595
|
* provider reconciles against the one canonical store.
|
|
596
|
+
* @owner capability:firewall — one row is one DNAT rule; migrates to the provider (T2)
|
|
579
597
|
*/
|
|
580
598
|
export const portForwards = sqliteTable(
|
|
581
599
|
'port_forwards',
|
|
@@ -632,6 +650,7 @@ export const portForwards = sqliteTable(
|
|
|
632
650
|
*
|
|
633
651
|
* Distinct from a port forward: that publishes one backend on specific ports;
|
|
634
652
|
* this is a whole origin subnet permitted to initiate into the segmented tiers.
|
|
653
|
+
* @owner capability:firewall — the firewall ruleset; migrates to the provider (T3)
|
|
635
654
|
*/
|
|
636
655
|
export const trustedSources = sqliteTable(
|
|
637
656
|
'trusted_sources',
|
|
@@ -678,6 +697,7 @@ export const trustedSources = sqliteTable(
|
|
|
678
697
|
* until its LAST consumer is removed. The remote DNS record itself stays
|
|
679
698
|
* (Namecheap DDNS has no delete API).
|
|
680
699
|
* See designs/DISPATCHER_DAEMON_AND_TIMER_EVENTS.md (B2).
|
|
700
|
+
* @owner celilo — claim ledger, deliberately stores no address (T4)
|
|
681
701
|
*/
|
|
682
702
|
export const dnsRegistrations = sqliteTable(
|
|
683
703
|
'dns_registrations',
|
|
@@ -721,6 +741,7 @@ export const dnsRegistrations = sqliteTable(
|
|
|
721
741
|
* last module that wants the name goes away (design.md D5).
|
|
722
742
|
*
|
|
723
743
|
* The introducing module is the earliest row by `id`.
|
|
744
|
+
* @owner celilo — the consumer SET, dies with the last consumer (T5)
|
|
724
745
|
*/
|
|
725
746
|
export const dnsRegistrationConsumers = sqliteTable(
|
|
726
747
|
'dns_registration_consumers',
|
|
@@ -755,6 +776,7 @@ export const dnsRegistrationConsumers = sqliteTable(
|
|
|
755
776
|
* modules at once — so absences are counted rather than reported, and become
|
|
756
777
|
* their own finding only once they persist. A subject that answers has its row
|
|
757
778
|
* dropped, which is what makes the count consecutive.
|
|
779
|
+
* @owner celilo — the audit subsystem's state about its own ability to observe (T7)
|
|
758
780
|
*/
|
|
759
781
|
export const publicDnsEvidence = sqliteTable('public_dns_evidence', {
|
|
760
782
|
subject: text('subject').primaryKey(),
|
|
@@ -775,6 +797,7 @@ export const publicDnsEvidence = sqliteTable('public_dns_evidence', {
|
|
|
775
797
|
* `celilo system doctor` reads this to assert service hostnames resolve to
|
|
776
798
|
* the firewall natIp (LAN-reachable) and not a zone-side container IP that
|
|
777
799
|
* a LAN device can't route to. Rows die with either module via FK cascade.
|
|
800
|
+
* @owner capability:dns_internal — resolver configuration; migrates to the provider (T6)
|
|
778
801
|
*/
|
|
779
802
|
export const dnsInternalRecords = sqliteTable(
|
|
780
803
|
'dns_internal_records',
|
|
@@ -814,6 +837,7 @@ export const dnsInternalRecords = sqliteTable(
|
|
|
814
837
|
/**
|
|
815
838
|
* Backup storage providers - destinations for backup archives
|
|
816
839
|
* Supports local filesystem and S3-compatible storage (AWS S3, MinIO, Backblaze B2, Wasabi)
|
|
840
|
+
* @owner celilo — backup destinations; infra-provider columns; out of scope per S17, not blessed
|
|
817
841
|
*/
|
|
818
842
|
export type BackupStorageProvider = 'local' | 's3';
|
|
819
843
|
|
|
@@ -837,6 +861,7 @@ export const backupStorages = sqliteTable('backup_storages', {
|
|
|
837
861
|
/**
|
|
838
862
|
* Backup records - metadata for each backup taken
|
|
839
863
|
* Tracks both system state backups and module data backups
|
|
864
|
+
* @owner celilo — backup metadata
|
|
840
865
|
*/
|
|
841
866
|
export type BackupType = 'module_data' | 'system_state';
|
|
842
867
|
export type BackupStatus = 'in_progress' | 'completed' | 'failed';
|
|
@@ -878,6 +903,7 @@ export const backups = sqliteTable('backups', {
|
|
|
878
903
|
* The `pid` column carries the process that started the operation; a row whose
|
|
879
904
|
* pid is no longer alive is treated as abandoned (the process crashed before
|
|
880
905
|
* the completion update landed) and ignored by in-flight checks.
|
|
906
|
+
* @owner celilo — in-flight operation tracking
|
|
881
907
|
*/
|
|
882
908
|
export type ModuleOperationKind =
|
|
883
909
|
| 'deploy'
|
|
@@ -918,6 +944,7 @@ export const moduleOperations = sqliteTable('module_operations', {
|
|
|
918
944
|
* approvals don't linger.
|
|
919
945
|
*
|
|
920
946
|
* See openspec/specs/base-module-aspects/spec.md D2 + D7.
|
|
947
|
+
* @owner celilo — operator consent, per module version + scope hash
|
|
921
948
|
*/
|
|
922
949
|
export const aspectApprovals = sqliteTable(
|
|
923
950
|
'aspect_approvals',
|
|
@@ -961,6 +988,7 @@ export const aspectApprovals = sqliteTable(
|
|
|
961
988
|
* ponytail: one key per principal (a person wanting a second device makes a
|
|
962
989
|
* second principal, e.g. `alice-laptop`). If multiple keys per identity is ever
|
|
963
990
|
* needed, split into an `api_keys` child table — not worth it yet.
|
|
991
|
+
* @owner celilo — remote-API identity + grants
|
|
964
992
|
*/
|
|
965
993
|
export const apiPrincipals = sqliteTable('api_principals', {
|
|
966
994
|
id: text('id').primaryKey(), // UUID
|
|
@@ -1008,6 +1036,7 @@ export type AlertState = 'pending' | 'firing' | 'acked' | 'suppressed' | 'resolv
|
|
|
1008
1036
|
/**
|
|
1009
1037
|
* People celilo can reach. Deliberately independent of any transport or
|
|
1010
1038
|
* module — a person exists before any notification module is deployed.
|
|
1039
|
+
* @owner celilo — deliberately independent of any transport or module
|
|
1011
1040
|
*/
|
|
1012
1041
|
export const people = sqliteTable('people', {
|
|
1013
1042
|
id: text('id').primaryKey(), // UUID
|
|
@@ -1029,6 +1058,7 @@ export const people = sqliteTable('people', {
|
|
|
1029
1058
|
* adding a recipient would require redeploying the module. signal-cli holds
|
|
1030
1059
|
* one credential (its own registration); recipients are addresses, not
|
|
1031
1060
|
* credentials. See design D8.
|
|
1061
|
+
* @owner celilo — who celilo pages and how; the calibration example in section 5
|
|
1032
1062
|
*/
|
|
1033
1063
|
export const routes = sqliteTable(
|
|
1034
1064
|
'routes',
|
|
@@ -1064,6 +1094,7 @@ export const routes = sqliteTable(
|
|
|
1064
1094
|
/**
|
|
1065
1095
|
* Named, reusable escalation policy. Steps reference ROUTES rather than
|
|
1066
1096
|
* people — "page Peter" is ambiguous about which transport to use.
|
|
1097
|
+
* @owner celilo — named escalation policy
|
|
1067
1098
|
*/
|
|
1068
1099
|
export const escalationPolicies = sqliteTable('escalation_policies', {
|
|
1069
1100
|
id: text('id').primaryKey(), // UUID
|
|
@@ -1079,7 +1110,11 @@ export const escalationPolicies = sqliteTable('escalation_policies', {
|
|
|
1079
1110
|
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull().default(sql`(unixepoch())`),
|
|
1080
1111
|
});
|
|
1081
1112
|
|
|
1082
|
-
/**
|
|
1113
|
+
/**
|
|
1114
|
+
* One ordered step of an escalation policy. Delay is from escalation start.
|
|
1115
|
+
*
|
|
1116
|
+
* @owner celilo — the ordered steps of one
|
|
1117
|
+
*/
|
|
1083
1118
|
export const escalationSteps = sqliteTable(
|
|
1084
1119
|
'escalation_steps',
|
|
1085
1120
|
{
|
|
@@ -1105,6 +1140,7 @@ export const escalationSteps = sqliteTable(
|
|
|
1105
1140
|
* The module manifest's `hooks.health_check.interval` is only a SUGGESTION;
|
|
1106
1141
|
* this row is the effective schedule and an operator edit survives module
|
|
1107
1142
|
* upgrades. See design D3.
|
|
1143
|
+
* @owner celilo — what runs, how often, how failures route
|
|
1108
1144
|
*/
|
|
1109
1145
|
export const monitors = sqliteTable(
|
|
1110
1146
|
'monitors',
|
|
@@ -1139,6 +1175,7 @@ export const monitors = sqliteTable(
|
|
|
1139
1175
|
* whether the last run actually ran (design D5) — without a persisted
|
|
1140
1176
|
* outcome there is nothing to distinguish "found nothing wrong" from
|
|
1141
1177
|
* "couldn't look".
|
|
1178
|
+
* @owner celilo — did the run execute at all
|
|
1142
1179
|
*/
|
|
1143
1180
|
export const monitorRuns = sqliteTable('monitor_runs', {
|
|
1144
1181
|
id: integer('id').primaryKey({ autoIncrement: true }),
|
|
@@ -1155,6 +1192,7 @@ export const monitorRuns = sqliteTable('monitor_runs', {
|
|
|
1155
1192
|
* A deliberate, time-boxed suppression source. Today only deploys create
|
|
1156
1193
|
* these — a deploy is the same suppression mechanism as a machine-down alert,
|
|
1157
1194
|
* with a window as the source instead of an ancestor alert (design D7).
|
|
1195
|
+
* @owner celilo — time-boxed deliberate suppression
|
|
1158
1196
|
*/
|
|
1159
1197
|
export const suppressionWindows = sqliteTable('suppression_windows', {
|
|
1160
1198
|
id: text('id').primaryKey(), // UUID
|
|
@@ -1174,6 +1212,7 @@ export const suppressionWindows = sqliteTable('suppression_windows', {
|
|
|
1174
1212
|
* `acked`, `suppressed`, and `silenced` are deliberately THREE separate
|
|
1175
1213
|
* concerns (design S5) — collapsing any two is how alerting systems become
|
|
1176
1214
|
* untrustworthy.
|
|
1215
|
+
* @owner celilo — the alert lifecycle
|
|
1177
1216
|
*/
|
|
1178
1217
|
export const alerts = sqliteTable(
|
|
1179
1218
|
'alerts',
|
|
@@ -1265,6 +1304,7 @@ export const alerts = sqliteTable(
|
|
|
1265
1304
|
* Per DELIVERY, not per alert: the token identifies WHO replied, which is
|
|
1266
1305
|
* what "an ack from the secondary is broadcast to everyone paged" needs, and
|
|
1267
1306
|
* doubles as the audit trail (design D10).
|
|
1307
|
+
* @owner celilo — one outbound message + its reply token
|
|
1268
1308
|
*/
|
|
1269
1309
|
export const notificationDeliveries = sqliteTable('notification_deliveries', {
|
|
1270
1310
|
id: text('id').primaryKey(), // UUID
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Retention is by AGE, and the test that matters is the one asserting the
|
|
3
|
+
* FIRST failing run survives a long streak. A count-based rule passes every
|
|
4
|
+
* other test here and fails that one — which is exactly the bug this
|
|
5
|
+
* policy exists to avoid, since the first artifact set carries the original
|
|
6
|
+
* cause and later ones repeat it.
|
|
7
|
+
*
|
|
8
|
+
* Uses a real temp directory rather than an injected filesystem: mtime
|
|
9
|
+
* ordering and directory sizing are the behaviour under test, and a fake
|
|
10
|
+
* would be asserting my own model of them.
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
|
14
|
+
import { mkdirSync, mkdtempSync, readdirSync, rmSync, utimesSync, writeFileSync } from 'node:fs';
|
|
15
|
+
import { tmpdir } from 'node:os';
|
|
16
|
+
import { join } from 'node:path';
|
|
17
|
+
import { ARTIFACT_RETENTION_MS, pruneModuleArtifacts } from './artifact-retention';
|
|
18
|
+
|
|
19
|
+
const HOUR = 60 * 60 * 1000;
|
|
20
|
+
const NOW = Date.UTC(2026, 7, 18, 12, 0, 0);
|
|
21
|
+
|
|
22
|
+
let root: string;
|
|
23
|
+
|
|
24
|
+
beforeEach(() => {
|
|
25
|
+
root = mkdtempSync(join(tmpdir(), 'celilo-artifacts-'));
|
|
26
|
+
});
|
|
27
|
+
|
|
28
|
+
afterEach(() => {
|
|
29
|
+
rmSync(root, { recursive: true, force: true });
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
/** Create a run directory whose files are `ageMs` old, with `bytes` of content. */
|
|
33
|
+
function makeRun(name: string, ageMs: number, bytes = 16): string {
|
|
34
|
+
const dir = join(root, name);
|
|
35
|
+
mkdirSync(dir, { recursive: true });
|
|
36
|
+
const file = join(dir, 'spa-failure.png');
|
|
37
|
+
writeFileSync(file, Buffer.alloc(bytes));
|
|
38
|
+
const when = new Date(NOW - ageMs);
|
|
39
|
+
utimesSync(file, when, when);
|
|
40
|
+
utimesSync(dir, when, when);
|
|
41
|
+
return dir;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
function survivors(): string[] {
|
|
45
|
+
return readdirSync(root).sort();
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
describe('pruneModuleArtifacts', () => {
|
|
49
|
+
test('keeps everything inside the retention window', () => {
|
|
50
|
+
makeRun('run-a', 1 * HOUR);
|
|
51
|
+
makeRun('run-b', 23 * HOUR);
|
|
52
|
+
|
|
53
|
+
const outcome = pruneModuleArtifacts(root, { now: NOW });
|
|
54
|
+
|
|
55
|
+
expect(outcome.removed).toEqual([]);
|
|
56
|
+
expect(survivors()).toEqual(['run-a', 'run-b']);
|
|
57
|
+
});
|
|
58
|
+
|
|
59
|
+
test('prunes what is older than the retention window', () => {
|
|
60
|
+
makeRun('stale', 25 * HOUR);
|
|
61
|
+
makeRun('fresh', 1 * HOUR);
|
|
62
|
+
|
|
63
|
+
pruneModuleArtifacts(root, { now: NOW });
|
|
64
|
+
|
|
65
|
+
expect(survivors()).toEqual(['fresh']);
|
|
66
|
+
});
|
|
67
|
+
|
|
68
|
+
test('the boundary is exclusive: exactly at the window, it stays', () => {
|
|
69
|
+
// The off-by-one is the whole risk in a rule expressed as an inequality.
|
|
70
|
+
makeRun('exactly-24h', ARTIFACT_RETENTION_MS);
|
|
71
|
+
makeRun('a-ms-older', ARTIFACT_RETENTION_MS + 1);
|
|
72
|
+
|
|
73
|
+
pruneModuleArtifacts(root, { now: NOW });
|
|
74
|
+
|
|
75
|
+
expect(survivors()).toEqual(['exactly-24h']);
|
|
76
|
+
});
|
|
77
|
+
|
|
78
|
+
test('THE POINT: the first failure of a long streak survives', () => {
|
|
79
|
+
// A 12-hour streak at the 15-minute health cadence — every run failing,
|
|
80
|
+
// every one writing artifacts. The first set carries the original cause;
|
|
81
|
+
// "keep the last N" would have discarded it hours ago.
|
|
82
|
+
for (let quarterHour = 0; quarterHour <= 48; quarterHour++) {
|
|
83
|
+
makeRun(`run-${String(quarterHour).padStart(3, '0')}`, quarterHour * 15 * 60 * 1000);
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
pruneModuleArtifacts(root, { now: NOW });
|
|
87
|
+
|
|
88
|
+
const kept = survivors();
|
|
89
|
+
expect(kept).toContain('run-048'); // the oldest — the first failure
|
|
90
|
+
expect(kept).toContain('run-000'); // the most recent
|
|
91
|
+
expect(kept.length).toBe(49);
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
test('the size ceiling evicts oldest-first, and only down to the ceiling', () => {
|
|
95
|
+
makeRun('oldest', 3 * HOUR, 1000);
|
|
96
|
+
makeRun('middle', 2 * HOUR, 1000);
|
|
97
|
+
makeRun('newest', 1 * HOUR, 1000);
|
|
98
|
+
|
|
99
|
+
const outcome = pruneModuleArtifacts(root, { now: NOW, sizeCeilingBytes: 2500 });
|
|
100
|
+
|
|
101
|
+
// 3000 bytes exceeds 2500; dropping the oldest brings it to 2000.
|
|
102
|
+
expect(survivors()).toEqual(['middle', 'newest']);
|
|
103
|
+
expect(outcome.removed).toEqual([join(root, 'oldest')]);
|
|
104
|
+
expect(outcome.retainedBytes).toBe(2000);
|
|
105
|
+
});
|
|
106
|
+
|
|
107
|
+
test('a run still being written is aged by its newest file, not the directory', () => {
|
|
108
|
+
// The directory's own mtime can lag a file rewritten in place, which
|
|
109
|
+
// would let an in-flight run read as old enough to evict.
|
|
110
|
+
const dir = join(root, 'in-flight');
|
|
111
|
+
mkdirSync(dir, { recursive: true });
|
|
112
|
+
const old = new Date(NOW - 30 * HOUR);
|
|
113
|
+
utimesSync(dir, old, old);
|
|
114
|
+
const stale = join(dir, 'old.txt');
|
|
115
|
+
writeFileSync(stale, 'x');
|
|
116
|
+
utimesSync(stale, old, old);
|
|
117
|
+
writeFileSync(join(dir, 'just-written.png'), 'y'); // now
|
|
118
|
+
|
|
119
|
+
pruneModuleArtifacts(root, { now: NOW });
|
|
120
|
+
|
|
121
|
+
expect(survivors()).toEqual(['in-flight']);
|
|
122
|
+
});
|
|
123
|
+
|
|
124
|
+
test('a missing artifact root is not an error', () => {
|
|
125
|
+
expect(() => pruneModuleArtifacts(join(root, 'never-created'), { now: NOW })).not.toThrow();
|
|
126
|
+
});
|
|
127
|
+
|
|
128
|
+
test('loose files beside the run directories are left alone', () => {
|
|
129
|
+
writeFileSync(join(root, 'cookies.json'), '{}');
|
|
130
|
+
makeRun('stale', 30 * HOUR);
|
|
131
|
+
|
|
132
|
+
pruneModuleArtifacts(root, { now: NOW });
|
|
133
|
+
|
|
134
|
+
expect(survivors()).toEqual(['cookies.json']);
|
|
135
|
+
});
|
|
136
|
+
});
|