@celilo/cli 1.10.0 → 1.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -267,7 +267,8 @@ is currently wrong, and routes carry the message to a person's phone. Design:
267
267
  - **⚠️ `monitors.intervalMinutes` and `monitors.enabled` are `builtin_check`-only.** A `module_hook` row carries severity, escalation policy and `lastRunAt`; its cadence and whether it is watched resolve through the accessor above. The columns' meaning depending on `kind` is a named smell (design.md D8) — the alternatives are a cached resolved value that rots, or splitting the table, which needs a synthetic monitor identity for `alerts.monitorId`. Gates: `sweep-runner.test.ts` asserts a module row's stored values are NOT consulted; `cadence-migration.test.ts` asserts the same through `loadModuleHealthCadences`.
268
268
  - **Carrying an existing fleet over** — `apps/celilo/src/services/alerting/cadence-migration.ts` (`migrateMonitorCadences`), run from `celilo system migrate` (the `.deb` postinst runs it on every apt upgrade). A monitor row whose cadence diverges from its manifest gets that cadence written as an override; a disabled one gets `manual`. Not bookkeeping: without it the upgrade that ships read-time resolution silently reverts every hand-set cadence to the author's suggestion and resumes watching modules an operator deliberately disabled. Idempotent — writes only where no override exists.
269
269
  - **Scheduled audit categories (`builtin_check` monitors)** — `apps/celilo/src/services/alerting/builtin-source.ts` — `SCHEDULABLE_BUILTIN_CHECKS` is the list of `celilo system audit` categories cheap enough to run every sweep: `machines_reachable`, `backups`, `disk_space`, `abandoned_operations`, and `public_dns` (`apps/celilo/src/services/audit/abandoned-operations.ts` — ≥3 abandonments of the same (module, operation) in 7d, the fingerprint of an operation being killed mid-flight). Everything else in the audit needs the whole world injected (proxmox, terraform, registry) and is not schedulable. Enable one with `celilo monitor add backups --interval 1h`, and re-cadence it later with `celilo monitor set-interval backups 6h` (in place, because the monitor id owns the alert history). `monitor set-interval`/`enable`/`disable` REFUSE a module target and name `celilo module config set <m> health_check_interval` — two ways to set one module's cadence would disagree about what `module status` shows. Targets are tab-completable — `completion.ts` reads `SCHEDULABLE_BUILTIN_CHECKS` directly rather than a hand-copied list, so a newly-schedulable check is completable immediately.
270
- - **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet.
270
+ - **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`, target set from `diskProbeTargets`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. ⚠️ **It probes SYSTEMS, not just MACHINES.** `machine` is celilo's narrow term for an operator-pre-provisioned box in the machine pool; it excludes every LXC/VM celilo provisioned through a container_service — which is most of the fleet. The probe walked `listMachines()` alone until celilo#1133, so the registry, the forge and the firewall were never measured — and **two of them filled on the same night, wearing different disguises**: celilo-registry (vmid 204) hit 100% with 4 KB free and publishes began returning `Internal Server Error` while every celilo surface stayed green (reads work on a full disk; only writes fail), and git.celilo.computer (vmid 206) hit 100% and its runner logged `database or disk is full` every 2s, claiming no work for ~13 hours — presenting as a DEAD CI RUNNER when the runner was healthy and polling. Scheduling the monitor would not have caught either: the check would have reported all-clear about filesystems it never looked at. `diskProbeTargets` now unions the machine pool with `getProvisionedSystems`, deduplicated by address (co-hosted modules share one filesystem and so one finding), containers probed as `root`. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet.
271
+ - **Instance sizing (`celilo proxmox vm|ct list|resize`)** — `apps/celilo/src/cli/commands/proxmox-instance-resize.ts` + `proxmox-resize-guards.ts` (`validateResize`/`computeFloor`, pure and unit-tested without Proxmox) + `proxmox-instance-list.ts` (DESIRED vs ACTUAL c/m/d, drift). A celilo-provisioned instance's size is **canonical infrastructure state in `module_systems`**, never module config; `requires.system` is only the floor used to select a host. Sizing flows `module_systems` → `$self:{cores,memory,disk}` (seeded from `requires.system` on first provision, `apps/celilo/src/variables/context.ts`) → the instance Terraform. ⚠️ **Two apply paths.** cpu/memory reconcile declaratively — record the size, redeploy the owning module, and the provider stop/starts the guest, so that path costs a reboot and is gated on the operator approving one. Disk growth goes DIRECT to Proxmox's resize API (`ProxmoxClient.resizeGuestDisk`, `pct resize`/`qm resize`): online, additive, no power change and no redeploy. That is not a shortcut — a Terraform reconcile means a full Ansible pass, and the box being grown is the one that has run out of room to run one (celilo#1133). Guards: floor and disk-shrink are hard, node capacity yields to `--force`. lxc grows its filesystem too; qemu grows only the block device and the guest must extend its own partition, which the command says out loud.
271
272
  - **Public-DNS reachability check (the only check with an OFF-FLEET vantage)** — `apps/celilo/src/services/audit/public-dns.ts` (`auditPublicDns`, pure over an injected probe and the previous run's counters) + `apps/celilo/src/services/public-dns-probe.ts` (the probe) + `audit/public-dns-source.ts` (ledger names + the `public_dns_evidence` counters). Every other check in celilo looks from INSIDE, behind a split-horizon resolver that deliberately answers with an in-zone address — correct for its purpose, and why all of them reported healthy for the nine days of celilo#626. This one resolves every `dns_registrations` FQDN through an **off-fleet resolver** (`public_dns.resolver`, default `1.1.1.1`) and compares it against the address the fleet appears to come from per an independent **echo service** (`public_dns.echo_url`, default `https://api.ipify.org`). Three properties are load-bearing: `assertOffFleetResolver` REFUSES a resolver matching `dns.primary`/`dns.fallback` (a check that quietly used the fleet's resolver would pass forever — the original bug one layer up); the expectation never comes from the registrar's own response (self-agreement, and Namecheap returns `ErrCount 0` for `www` updates it does not apply); and a divergence is a finding only once it OUTLIVES the record's own TTL, measured from the last assert, or it would page on every ISP re-lease. Missing evidence is counted rather than read as success — one undetermined run is silent, N consecutive ones are their own finding (`public_dns_evidence`), which is the hole celilo-website's isitup.org probe demonstrated live. Codes: `public_dns_stale`, `public_dns_missing`, `public_dns_companion_unclaimed`, `public_dns_unverifiable`. Spec: `openspec/specs/public-dns-reachability/spec.md`.
272
273
  - **The sweep** — `apps/celilo/src/services/alerting/sweep-runner.ts` (`runSweep`) — the ordered pass that makes alerting run by itself: run due monitors → promote past-grace alerts → re-evaluate suppression → flush quiet-hours deferrals → notify. Driven by `celilo alerts sweep` on `timer.tick.5m`. Never throws for one bad monitor.
273
274
  - **Suppression (topology-derived, never configured)** — `apps/celilo/src/services/alerting/suppression.ts` — `ancestorKeysFor`/`findSuppressor`/`machineAlertKey`. A firing machine explains its modules' failures; a firing capability provider explains its zone's consumers. Derived from `module_systems`, so it cannot drift from reality. Deploy windows: `deploy-hooks.ts` (`openDeployWindow`/`closeDeployWindows` — closed by module, so a crashed deploy self-heals).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@celilo/cli",
3
- "version": "1.10.0",
3
+ "version": "1.12.0",
4
4
  "description": "Celilo — home lab orchestration CLI",
5
5
  "type": "module",
6
6
  "bin": {
@@ -58,7 +58,7 @@
58
58
  "dependencies": {
59
59
  "@aws-sdk/client-s3": "^3.1109.0",
60
60
  "@aws-sdk/lib-storage": "^3.1101.0",
61
- "@celilo/capabilities": "^3.1.0",
61
+ "@celilo/capabilities": "^3.2.0",
62
62
  "@celilo/cli-display": "^0.2.0",
63
63
  "@celilo/core": "^0.9.1",
64
64
  "@celilo/event-bus": "^0.6.0",
@@ -663,6 +663,43 @@ export class ProxmoxClient {
663
663
  );
664
664
  }
665
665
 
666
+ /**
667
+ * Grow a guest's disk, online. This is `pct resize` / `qm resize` (celilo#1133).
668
+ *
669
+ * Deliberately NOT routed through Terraform like a cpu/memory resize is. Two
670
+ * reasons, and the first is the one that matters: a Terraform reconcile means
671
+ * a full module redeploy — Ansible, package installs, service restarts — and
672
+ * the box being resized is by definition the one that has run out of room to
673
+ * do any of that. The cure would need the disk space it exists to provide.
674
+ * Second, the resize API is additive and online, so celilo can promise the
675
+ * guest is not power-cycled; delegating that to a provider's update path
676
+ * cannot promise it.
677
+ *
678
+ * `disk` is the config key of the volume to grow — `rootfs` for lxc, `scsi0`
679
+ * for a cloud-init VM. `size` is absolute (Proxmox also accepts `+NG`, but
680
+ * canonical state is an absolute size, so we send one).
681
+ *
682
+ * ⚠️ lxc grows the filesystem too; qemu grows only the block device and the
683
+ * guest must extend its own partition. Callers say so rather than implying
684
+ * the space is usable.
685
+ */
686
+ async resizeGuestDisk(
687
+ vmid: number,
688
+ kind: 'lxc' | 'qemu',
689
+ disk: string,
690
+ sizeGb: number,
691
+ ): Promise<ProxmoxResult<string>> {
692
+ const node = await this.nodeForVmid(vmid);
693
+ if (!node.success) return node;
694
+ if (!node.data) {
695
+ return { success: false, message: `No Proxmox node hosts vmid ${vmid}` };
696
+ }
697
+ return makeProxmoxPut<string>(this.credentials, `/nodes/${node.data}/${kind}/${vmid}/resize`, {
698
+ disk,
699
+ size: `${sizeGb}G`,
700
+ });
701
+ }
702
+
666
703
  /** Current run state of a guest (`running`, `stopped`, …). */
667
704
  async guestStatus(vmid: number): Promise<ProxmoxResult<string | null>> {
668
705
  const result = await this.clusterResources();
@@ -785,6 +822,14 @@ async function makeProxmoxPost<T>(
785
822
  return makeProxmoxFormRequest(credentials, 'POST', path, params);
786
823
  }
787
824
 
825
+ async function makeProxmoxPut<T>(
826
+ credentials: ProxmoxCredentials,
827
+ path: string,
828
+ params: Record<string, string>,
829
+ ): Promise<ProxmoxResult<T>> {
830
+ return makeProxmoxFormRequest(credentials, 'PUT', path, params);
831
+ }
832
+
788
833
  /**
789
834
  * Entry from Proxmox's appliance catalog (`pveam available`). The `template`
790
835
  * field is the canonical filename (revision included) that should be passed to
@@ -61,6 +61,7 @@ interface ManifestForPublish extends ManifestForCapabilityCheck {
61
61
  id: string;
62
62
  version: string;
63
63
  description?: string;
64
+ icon?: string;
64
65
  version_source?: { kind?: string };
65
66
  }
66
67
 
@@ -307,6 +308,7 @@ export async function publishOneModule(
307
308
  netappPath: buildResult.packagePath,
308
309
  token: opts.token,
309
310
  description: manifest.description?.trim() || undefined,
311
+ icon: manifest.icon?.trim() || undefined,
310
312
  });
311
313
  } catch (err) {
312
314
  return {
@@ -5,7 +5,7 @@
5
5
  * desired≠actual row means a resize is pending a reconcile/redeploy.
6
6
  *
7
7
  * Read-only sibling of `proxmox node list`; the foundation the `resize` verb
8
- * (ISS-0150 P2) builds on. See apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md.
8
+ * (ISS-0150 P2) builds on. See openspec/changes/proxmox-capacity-lifecycle/.
9
9
  */
10
10
 
11
11
  import {
@@ -1,13 +1,30 @@
1
1
  /**
2
- * `celilo proxmox vm resize <name> --memory <MB> [--cpu <n>]` (and `ct resize`) —
3
- * the canonical resize for a celilo-provisioned instance (ISS-0150 P2). Updates
4
- * the SYSTEM's canonical size in module_systems, then reconciles declaratively by
5
- * redeploying the owning module (its Terraform now reads the system size).
2
+ * `celilo proxmox vm resize <name> [--memory <MB>] [--cpu <n>] [--disk <GB>]`
3
+ * (and `ct resize`) — the canonical resize for a celilo-provisioned instance
4
+ * (ISS-0150 P2). Updates the SYSTEM's canonical size in module_systems.
5
+ *
6
+ * Two apply paths, because the two kinds of change are not alike:
7
+ *
8
+ * cpu / memory → declarative. Record the size, redeploy the owning module,
9
+ * and its Terraform writes the new size to the guest config.
10
+ * The provider stop/starts the guest to apply it, so this
11
+ * path costs a reboot and is gated on the operator saying so.
12
+ * disk (growth) → direct. Proxmox's resize API grows the volume online, so
13
+ * celilo calls it and skips the redeploy entirely. No reboot,
14
+ * no Ansible. This is not a shortcut: a full module redeploy
15
+ * needs disk space to run, and the box being grown is the one
16
+ * that has run out of it (celilo#1133).
17
+ *
18
+ * Both paths leave module_systems canonical, so `proxmox ct list` shows DESIRED
19
+ * matching ACTUAL and drift clears either way. The instance Terraform reads
20
+ * `$self:disk` (not `requires.system.disk`, which is only the floor), so the
21
+ * next deploy agrees with a resize instead of trying to undo it.
6
22
  *
7
23
  * Guardrails (proxmox-resize-guards.ts): floor = max(requires.system) across
8
- * co-hosted modules (hard); node capacity (overridable with --force). Reboot +
9
- * pre-resize backup are confirmed via the event-bus interview (CLAUDE.md: no
10
- * direct CLI prompts) — flags win: --allow-reboot / --skip-backup / --yes.
24
+ * co-hosted modules (hard); disk shrink (hard); node capacity (overridable with
25
+ * --force). Reboot + pre-resize backup are confirmed via the event-bus
26
+ * interview (CLAUDE.md: no direct CLI prompts) — flags win: --allow-reboot /
27
+ * --skip-backup / --yes.
11
28
  */
12
29
 
13
30
  import { and, eq } from 'drizzle-orm';
@@ -26,7 +43,9 @@ import type { InstanceKind } from './proxmox-instance-list';
26
43
  import { computeFloor, validateResize } from './proxmox-resize-guards';
27
44
  import { resolveProxmoxService } from './proxmox-service';
28
45
 
29
- const PROXMOX_TYPE: Record<InstanceKind, string> = { vm: 'qemu', ct: 'lxc' };
46
+ const PROXMOX_TYPE: Record<InstanceKind, 'qemu' | 'lxc'> = { vm: 'qemu', ct: 'lxc' };
47
+
48
+ const BYTES_PER_GB = 1024 * 1024 * 1024;
30
49
 
31
50
  function numFlag(v: string | boolean | undefined): number | undefined {
32
51
  if (typeof v !== 'string') return undefined;
@@ -43,20 +62,16 @@ export async function handleProxmoxInstanceResize(
43
62
 
44
63
  const name = args[0];
45
64
  if (!name) {
46
- const e = `Instance name required: celilo proxmox ${kind} resize <name> --memory <MB> [--cpu <n>]`;
65
+ const e = `Instance name required: celilo proxmox ${kind} resize <name> [--memory <MB>] [--cpu <n>] [--disk <GB>]`;
47
66
  console.log(`✗ ${e}`);
48
67
  return { success: false, error: e };
49
68
  }
50
69
 
51
- if (numFlag(flags.disk) != null) {
52
- const e = 'Disk resize is deferred (ISS-0150 D4). Use --memory / --cpu for now.';
53
- console.log(`✗ ${e}`);
54
- return { success: false, error: e };
55
- }
56
70
  const reqCpu = numFlag(flags.cpu);
57
71
  const reqMemory = numFlag(flags.memory);
58
- if (reqCpu == null && reqMemory == null) {
59
- const e = 'Specify at least one of --memory <MB> or --cpu <n>.';
72
+ const reqDisk = numFlag(flags.disk);
73
+ if (reqCpu == null && reqMemory == null && reqDisk == null) {
74
+ const e = 'Specify at least one of --memory <MB>, --cpu <n> or --disk <GB>.';
60
75
  console.log(`✗ ${e}`);
61
76
  return { success: false, error: e };
62
77
  }
@@ -116,13 +131,24 @@ export async function handleProxmoxInstanceResize(
116
131
  const nodeCaps = await client.nodeCapacities();
117
132
  const nodeCap = nodeCaps.success ? nodeCaps.data.find((n) => n.node === node) : undefined;
118
133
 
134
+ // Disk compares against the LIVE volume, not celilo's record of it. "Can
135
+ // Proxmox do this" is a question about the real disk: canonical state is
136
+ // seeded from the manifest floor and goes stale the moment anyone runs `pct
137
+ // resize` by hand — which, until this command existed, was the only way to
138
+ // grow one. Comparing a grow request against a stale-low record would wave
139
+ // through a shrink that Proxmox then refuses. cpu/memory keep using canonical
140
+ // state, which the reconcile path owns end to end.
141
+ const liveDiskGb =
142
+ guest?.maxdisk != null ? Math.round(guest.maxdisk / BYTES_PER_GB) : target.disk;
143
+
119
144
  const decision = validateResize(
120
- { cpu: reqCpu, memory: reqMemory },
145
+ { cpu: reqCpu, memory: reqMemory, disk: reqDisk },
121
146
  {
122
- current: { cpu: target.cpu, memory: target.memory, disk: target.disk },
147
+ current: { cpu: target.cpu, memory: target.memory, disk: liveDiskGb },
123
148
  floor,
124
149
  nodeFreeMemMb: nodeCap?.memFreeMb ?? Number.POSITIVE_INFINITY,
125
150
  nodeTotalCores: nodeCap?.cpuCores ?? Number.POSITIVE_INFINITY,
151
+ nodeFreeDiskGb: nodeCap?.diskFreeGb ?? Number.POSITIVE_INFINITY,
126
152
  },
127
153
  { force: flags.force === true },
128
154
  );
@@ -137,10 +163,24 @@ export async function handleProxmoxInstanceResize(
137
163
  const yes = flags.yes === true;
138
164
  const coHostedIds = coHosted.map((s) => s.moduleId);
139
165
 
166
+ // One decision, made in the guard layer where it is unit-tested, consumed
167
+ // twice below. `needsReboot` is false exactly when nothing but disk changed:
168
+ // disk growth applies online through the resize API, so it neither
169
+ // power-cycles the guest nor needs Terraform to run.
170
+ //
171
+ // Both skips that follow hang on it. Growing a disk is purely additive —
172
+ // nothing is destroyed, nothing is rewritten — so there is no state a backup
173
+ // would protect, and staging one would be actively harmful: the operator
174
+ // reaching for --disk has a box that has run out of room, and a backup needs
175
+ // the very resource it lacks. A cpu/memory change still backs up and still
176
+ // redeploys, because that one reboots.
177
+ const diskOnly = !decision.needsReboot;
178
+
140
179
  // Pre-resize backup — ON by default; --skip-backup opts out; otherwise the
141
180
  // skip decision is an event-bus interview question (never a direct CLI prompt).
142
181
  let doBackup: boolean;
143
- if (flags['skip-backup'] === true) doBackup = false;
182
+ if (diskOnly) doBackup = false;
183
+ else if (flags['skip-backup'] === true) doBackup = false;
144
184
  else if (yes) doBackup = true;
145
185
  else
146
186
  doBackup = await withInterviewSession(() =>
@@ -190,11 +230,39 @@ export async function handleProxmoxInstanceResize(
190
230
  }
191
231
  }
192
232
 
233
+ // Disk goes straight to Proxmox's resize API — online, additive, no power
234
+ // change. Do it BEFORE recording canonical state so a failure leaves celilo's
235
+ // record matching reality rather than claiming a size the guest doesn't have.
236
+ if (reqDisk != null) {
237
+ // rootfs for a container, scsi0 for a cloud-init VM — the volumes celilo's
238
+ // own Terraform declares (modules/*/terraform/main.tf.tpl).
239
+ const volume = kind === 'ct' ? 'rootfs' : 'scsi0';
240
+ console.log(`▸ Growing ${volume} to ${reqDisk}GB (online — no restart)…`);
241
+ const resized = await client.resizeGuestDisk(vmid, PROXMOX_TYPE[kind], volume, reqDisk);
242
+ if (!resized.success) {
243
+ const e = `Disk resize failed: ${resized.message}`;
244
+ console.log(`✗ ${e}`);
245
+ return { success: false, error: e };
246
+ }
247
+ console.log(` ✓ ${volume} is now ${reqDisk}GB`);
248
+ if (kind === 'vm') {
249
+ // qemu grows the block device only. Saying "resized" without this would
250
+ // send the operator back to a box whose df output has not moved.
251
+ console.log(
252
+ ` ⚠ The VM's block device grew, but the guest must extend its own partition and filesystem
253
+ (e.g. growpart /dev/sda 1 && resize2fs /dev/sda1). A container does this for itself; a VM does not.`,
254
+ );
255
+ }
256
+ }
257
+
193
258
  // Update the canonical size on every module_systems row for this instance.
194
259
  for (const s of coHosted) {
195
- const set: { cpu?: number; memory?: number; updatedAt: Date } = { updatedAt: new Date() };
260
+ const set: { cpu?: number; memory?: number; disk?: number; updatedAt: Date } = {
261
+ updatedAt: new Date(),
262
+ };
196
263
  if (reqCpu != null) set.cpu = reqCpu;
197
264
  if (reqMemory != null) set.memory = reqMemory;
265
+ if (reqDisk != null) set.disk = reqDisk;
198
266
  db.update(moduleSystems)
199
267
  .set(set)
200
268
  .where(and(eq(moduleSystems.moduleId, s.moduleId), eq(moduleSystems.name, s.name)))
@@ -202,7 +270,19 @@ export async function handleProxmoxInstanceResize(
202
270
  }
203
271
  const effCpu = reqCpu ?? target.cpu;
204
272
  const effMem = reqMemory ?? target.memory;
205
- console.log(`✓ Canonical size updated: ${name} → ${effCpu ?? '?'}c / ${effMem ?? '?'}MB`);
273
+ const effDisk = reqDisk ?? liveDiskGb;
274
+ console.log(
275
+ `✓ Canonical size updated: ${name} → ${effCpu ?? '?'}c / ${effMem ?? '?'}MB / ${effDisk ?? '?'}GB`,
276
+ );
277
+
278
+ // Disk-only: the guest already carries the new size and the canonical record
279
+ // now agrees, so Terraform has nothing to reconcile. Redeploying would run a
280
+ // full Ansible pass for no change — on a box that just filled up, that is the
281
+ // expensive, risky half of a cure whose cheap half already landed.
282
+ if (diskOnly) {
283
+ console.log(`✓ Resized ${name} to ${reqDisk}GB. No restart, no redeploy.`);
284
+ return { success: true, message: `resized ${name}` };
285
+ }
206
286
 
207
287
  // Reconcile declaratively: redeploy the owning module — its Terraform now reads
208
288
  // the updated system size and writes it to the Proxmox VM config. The proxmox
@@ -6,6 +6,7 @@ const ctx = (over: Partial<ResizeContext> = {}): ResizeContext => ({
6
6
  floor: { cpu: 4, memory: 8192, disk: 80 },
7
7
  nodeFreeMemMb: 20000,
8
8
  nodeTotalCores: 6,
9
+ nodeFreeDiskGb: 500,
9
10
  ...over,
10
11
  });
11
12
 
@@ -39,6 +40,56 @@ describe('validateResize (ISS-0150)', () => {
39
40
  });
40
41
  });
41
42
 
43
+ describe('validateResize — disk (celilo#1133)', () => {
44
+ // The whole point of the verb: the registry filled up and celilo had no cure.
45
+ test('growing the disk passes and needs NO reboot', () => {
46
+ const d = validateResize({ disk: 160 }, ctx());
47
+ expect(d.errors).toEqual([]);
48
+ expect(d.effective.disk).toBe(160);
49
+ // Load-bearing. The instances that fill up are the registry, the forge and
50
+ // the firewall; a resize that needlessly restarts one of those is a worse
51
+ // cure than the disease, and it would be discovered at the worst moment.
52
+ expect(d.needsReboot).toBe(false);
53
+ });
54
+
55
+ test('a cpu or memory change alongside a disk change still reboots', () => {
56
+ expect(validateResize({ disk: 160, memory: 16384 }, ctx()).needsReboot).toBe(true);
57
+ expect(validateResize({ disk: 160, cpu: 6 }, ctx()).needsReboot).toBe(true);
58
+ });
59
+
60
+ test('shrink is refused even well above the floor, and even with --force', () => {
61
+ // 60GB clears the 40GB floor, so this is NOT a floor violation — it is a
62
+ // shrink, and Proxmox cannot do one safely. The two need different errors
63
+ // because the operator's next move differs.
64
+ const roomy = ctx({ floor: { cpu: 4, memory: 8192, disk: 40 } });
65
+ const d = validateResize({ disk: 60 }, roomy, { force: true });
66
+ expect(d.errors.length).toBe(1);
67
+ expect(d.errors[0]).toContain('cannot shrink');
68
+ expect(d.capacityOnly).toBe(false);
69
+ });
70
+
71
+ test('shrinking below the floor reports the floor too', () => {
72
+ const d = validateResize({ disk: 10 }, ctx(), { force: true });
73
+ expect(d.errors.length).toBe(2);
74
+ expect(d.errors.some((e) => e.includes('below the module minimum'))).toBe(true);
75
+ expect(d.errors.some((e) => e.includes('cannot shrink'))).toBe(true);
76
+ });
77
+
78
+ test('growth beyond node free storage is a capacity error, overridable with --force', () => {
79
+ const tight = ctx({ nodeFreeDiskGb: 5 }); // 5GB free; +20GB requested
80
+ const blocked = validateResize({ disk: 100 }, tight, { force: false });
81
+ expect(blocked.errors.length).toBe(1);
82
+ expect(blocked.errors[0]).toContain('exceeds node free storage');
83
+ expect(blocked.capacityOnly).toBe(true);
84
+ expect(validateResize({ disk: 100 }, tight, { force: true }).errors).toEqual([]);
85
+ });
86
+
87
+ test('an equal-size disk request is neither shrink nor growth', () => {
88
+ const d = validateResize({ disk: 80 }, ctx({ nodeFreeDiskGb: 0 }));
89
+ expect(d.errors).toEqual([]);
90
+ });
91
+ });
92
+
42
93
  describe('computeFloor', () => {
43
94
  test('takes the max of each field across co-hosted modules', () => {
44
95
  expect(
@@ -1,7 +1,13 @@
1
1
  /**
2
- * Pure guardrails for `celilo proxmox … resize` (ISS-0150 D4). Kept separate from
3
- * the command so the floor + capacity logic is unit-testable without Proxmox or a
4
- * DB. See apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md.
2
+ * Pure guardrails for `celilo proxmox … resize` (ISS-0150 D4, disk landed in
3
+ * celilo#1133). Kept separate from the command so the floor + capacity logic is
4
+ * unit-testable without Proxmox or a DB which is what let the disk paths be
5
+ * proven with the fleet unreachable.
6
+ *
7
+ * See openspec/changes/proxmox-capacity-lifecycle/. (The header used to point at
8
+ * apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md, which does not exist and cannot:
9
+ * that directory is gitignored, so a doc written there is never committed and
10
+ * leaves with the worktree. CLAUDE.md flags the path as a trap.)
5
11
  */
6
12
 
7
13
  export interface ResizeRequest {
@@ -22,6 +28,8 @@ export interface ResizeContext {
22
28
  nodeFreeMemMb: number;
23
29
  /** Physical cores on the target node — a VM's vCPU can't exceed this (absolute). */
24
30
  nodeTotalCores: number;
31
+ /** Free storage on the target node — disk growth must fit (delta check). */
32
+ nodeFreeDiskGb: number;
25
33
  }
26
34
 
27
35
  export interface ResizeDecision {
@@ -36,10 +44,12 @@ export interface ResizeDecision {
36
44
  }
37
45
 
38
46
  /**
39
- * Validate a resize request against the floor and node capacity. Disk shrink is
40
- * rejected (qemu can't shrink safely); disk resize is otherwise deferred
41
- * (D4) callers should reject a disk change until that lands, but the floor/grow
42
- * checks are here for when it does.
47
+ * Validate a resize request against the floor and node capacity.
48
+ *
49
+ * Disk shrink is rejected: Proxmox grows a disk online but has no safe shrink
50
+ * for either lxc or qemu the guest filesystem would have to be shrunk first,
51
+ * from inside, and celilo does not do that. Disk GROWTH is additive and needs
52
+ * no stop/start, which is why `needsReboot` keys on cpu/memory alone.
43
53
  */
44
54
  export function validateResize(
45
55
  req: ResizeRequest,
@@ -52,15 +62,26 @@ export function validateResize(
52
62
  disk: req.disk ?? ctx.current.disk,
53
63
  };
54
64
 
55
- const floorErrors: string[] = [];
65
+ // Hard errors never overridable, not even with --force.
66
+ const hardErrors: string[] = [];
56
67
  if (req.cpu != null && req.cpu < ctx.floor.cpu) {
57
- floorErrors.push(`cpu ${req.cpu} is below the module minimum (${ctx.floor.cpu})`);
68
+ hardErrors.push(`cpu ${req.cpu} is below the module minimum (${ctx.floor.cpu})`);
58
69
  }
59
70
  if (req.memory != null && req.memory < ctx.floor.memory) {
60
- floorErrors.push(`memory ${req.memory}MB is below the module minimum (${ctx.floor.memory}MB)`);
71
+ hardErrors.push(`memory ${req.memory}MB is below the module minimum (${ctx.floor.memory}MB)`);
61
72
  }
62
73
  if (req.disk != null && req.disk < ctx.floor.disk) {
63
- floorErrors.push(`disk ${req.disk}GB is below the module minimum (${ctx.floor.disk}GB)`);
74
+ hardErrors.push(`disk ${req.disk}GB is below the module minimum (${ctx.floor.disk}GB)`);
75
+ }
76
+ // Shrink is refused on its own terms, not as a floor violation: a disk well
77
+ // above every tenant's minimum still cannot be shrunk. Say WHY rather than
78
+ // rejecting bare, because the operator's next move differs — a floor error
79
+ // means "pick a bigger number", a shrink error means "you cannot do this at
80
+ // all, migrate instead".
81
+ if (req.disk != null && ctx.current.disk != null && req.disk < ctx.current.disk) {
82
+ hardErrors.push(
83
+ `disk ${req.disk}GB is smaller than the current ${ctx.current.disk}GB — Proxmox cannot shrink a disk safely. Provision a smaller instance and migrate the data instead.`,
84
+ );
64
85
  }
65
86
 
66
87
  const capErrors: string[] = [];
@@ -76,10 +97,20 @@ export function validateResize(
76
97
  }
77
98
  }
78
99
 
79
- const capacityOnly = floorErrors.length === 0 && capErrors.length > 0;
80
- // Floor is a hard floor (never overridable). Capacity is overridable with --force.
81
- const errors = [...floorErrors, ...(opts.force ? [] : capErrors)];
100
+ // disk: only the GROWTH beyond the current size draws on the node's storage.
101
+ if (req.disk != null && ctx.current.disk != null) {
102
+ const addedGb = req.disk - ctx.current.disk;
103
+ if (addedGb > ctx.nodeFreeDiskGb) {
104
+ capErrors.push(`+${addedGb}GB exceeds node free storage (${ctx.nodeFreeDiskGb}GB)`);
105
+ }
106
+ }
107
+
108
+ const capacityOnly = hardErrors.length === 0 && capErrors.length > 0;
109
+ // Floor and shrink are hard (never overridable). Capacity yields to --force.
110
+ const errors = [...hardErrors, ...(opts.force ? [] : capErrors)];
82
111
 
112
+ // Disk growth is applied online by Proxmox's resize API — no stop/start. Only
113
+ // a cpu/memory change needs the guest power-cycled.
83
114
  const needsReboot = req.cpu != null || req.memory != null;
84
115
 
85
116
  return { errors, effective, needsReboot, capacityOnly };
@@ -0,0 +1,48 @@
1
+ /**
2
+ * The `icon` field's refinement (openspec/changes/module-icons, D3).
3
+ *
4
+ * This is the trust boundary: `manifest.yml` is hand-edited and the value ends
5
+ * up drawn in a coloured row, so the check that it is monochrome-capable lives
6
+ * here rather than in any consumer.
7
+ */
8
+
9
+ import { describe, expect, test } from 'bun:test';
10
+ import { ModuleManifestSchema } from './schema';
11
+
12
+ function manifestWith(icon?: string): Record<string, unknown> {
13
+ return {
14
+ celilo_contract: '1.0',
15
+ id: 'icon-fixture',
16
+ name: 'Icon Fixture',
17
+ version: '1.0.0',
18
+ ...(icon === undefined ? {} : { icon }),
19
+ };
20
+ }
21
+
22
+ describe('manifest icon', () => {
23
+ test('accepts a BMP non-emoji scalar', () => {
24
+ const parsed = ModuleManifestSchema.parse(manifestWith('⛨'));
25
+ expect(parsed.icon).toBe('⛨');
26
+ });
27
+
28
+ test('accepts absence', () => {
29
+ const parsed = ModuleManifestSchema.parse(manifestWith());
30
+ expect(parsed.icon).toBeUndefined();
31
+ });
32
+
33
+ test('rejects U+1F512, naming the monochrome reason rather than the range', () => {
34
+ const result = ModuleManifestSchema.safeParse(manifestWith('🔒'));
35
+ expect(result.success).toBe(false);
36
+ if (result.success) throw new Error('expected the padlock to be rejected');
37
+ const message = result.error.issues[0]?.message ?? '';
38
+ expect(message).toContain('monochrome');
39
+ expect(message).toContain('U+1F512');
40
+ });
41
+
42
+ test('rejects a two-character string', () => {
43
+ const result = ModuleManifestSchema.safeParse(manifestWith('⛨⛨'));
44
+ expect(result.success).toBe(false);
45
+ if (result.success) throw new Error('expected two characters to be rejected');
46
+ expect(result.error.issues[0]?.message ?? '').toContain('exactly one character');
47
+ });
48
+ });
@@ -563,6 +563,14 @@ export type ModuleSubscription = z.infer<typeof ModuleSubscriptionSchema>;
563
563
  * with via `celilo_contract`. The contract version determines the
564
564
  * canonical inputs/outputs of every lifecycle hook (see `./contracts/v1.ts`).
565
565
  */
566
+ /**
567
+ * Codepoints carrying Unicode's `Emoji` property, which the `icon` field
568
+ * rejects. Note this property is broader than "looks like an emoji": ASCII
569
+ * digits, `#` and `*` carry it too, because they form keycap sequences. That
570
+ * over-rejection costs nothing — none of them is a plausible module glyph.
571
+ */
572
+ const EMOJI_CODEPOINT = /\p{Emoji}/u;
573
+
566
574
  export const ModuleManifestSchema = z
567
575
  .object({
568
576
  /**
@@ -584,6 +592,55 @@ export const ModuleManifestSchema = z
584
592
  version: z.string().regex(/^\d+\.\d+\.\d+$/, 'Version must be semantic version (e.g., 1.0.0)'),
585
593
  description: z.string().optional(),
586
594
 
595
+ /**
596
+ * One glyph identifying this module wherever celilo draws it — the console
597
+ * roster, the topology boxes, the registry browse page. Optional: a module
598
+ * declaring none falls back to the consumer's built-in table, then to a
599
+ * placeholder (openspec/changes/module-icons, D3/D5).
600
+ *
601
+ * Exactly one Unicode scalar, inside the BMP, without Unicode's `Emoji`
602
+ * property. These glyphs inherit the colour of the row they are drawn in,
603
+ * so a firing module's icon goes red with the rest of the row. An emoji
604
+ * codepoint paints its own colours and would stay cheerful while its
605
+ * module's state said otherwise.
606
+ *
607
+ * BMP-and-not-Emoji is a PROXY for "renders monochrome", not a proof. Some
608
+ * BMP codepoints outside the Emoji property still get an emoji font on some
609
+ * platforms. What it does catch is the whole SMP emoji range, which is
610
+ * where an author reaching for a padlock or a shield actually lands, and
611
+ * that is the case worth catching.
612
+ *
613
+ * `ModuleManifestSchema` is strict, so a celilo predating this field
614
+ * rejects a manifest declaring it. The CLI release accepting `icon` ships
615
+ * before any module published to the registry declares one (D2).
616
+ */
617
+ icon: z
618
+ .string()
619
+ .superRefine((value, ctx) => {
620
+ const scalars = [...value];
621
+ if (scalars.length !== 1) {
622
+ ctx.addIssue({
623
+ code: z.ZodIssueCode.custom,
624
+ message: `icon must be exactly one character, got ${scalars.length}`,
625
+ });
626
+ return;
627
+ }
628
+ const codePoint = value.codePointAt(0) ?? 0;
629
+ if (codePoint > 0xffff || EMOJI_CODEPOINT.test(value)) {
630
+ const hex = codePoint.toString(16).toUpperCase().padStart(4, '0');
631
+ ctx.addIssue({
632
+ code: z.ZodIssueCode.custom,
633
+ message: [
634
+ `icon '${value}' (U+${hex}) must be a monochrome glyph: it inherits the colour`,
635
+ 'of the row it is drawn in, and an emoji codepoint paints its own colours, so it',
636
+ "would stay cheerful while the module went red. Use a BMP symbol outside Unicode's",
637
+ "Emoji property (a key '\u26bf', not a padlock '\u{1f512}').",
638
+ ].join(' '),
639
+ });
640
+ }
641
+ })
642
+ .optional(),
643
+
587
644
  /**
588
645
  * How `manifest.yml#version` (the PAYLOAD version) is determined — see
589
646
  * openspec/changes/module-version-semantics/proposal.md / ISS-0151. The capability *contract* version
@@ -28,6 +28,8 @@ export interface SearchResult {
28
28
  name: string;
29
29
  max_version: string;
30
30
  description: string;
31
+ /** The module's declared glyph, absent when it declared none. */
32
+ icon?: string;
31
33
  }
32
34
 
33
35
  export interface SearchResponse {
@@ -130,6 +132,12 @@ export class RegistryClient {
130
132
  * apps/celilo/designs/REGISTRY_BROWSE_UI.md (Phase 2 step 0).
131
133
  */
132
134
  description?: string;
135
+ /**
136
+ * The module's `manifest.yml#icon`, when it declares one. Optional and
137
+ * server-tolerated in its absence, exactly like `description`
138
+ * (openspec/changes/module-icons, D4).
139
+ */
140
+ icon?: string;
133
141
  }): Promise<{ ok: boolean; name: string; vers: string }> {
134
142
  const fileData = await readFile(opts.netappPath);
135
143
  const cksum = `sha256:${createHash('sha256').update(fileData).digest('hex')}`;
@@ -140,6 +148,7 @@ export class RegistryClient {
140
148
  deps: [],
141
149
  cksum,
142
150
  ...(opts.description ? { description: opts.description } : {}),
151
+ ...(opts.icon ? { icon: opts.icon } : {}),
143
152
  });
144
153
  const metaBuf = Buffer.from(meta, 'utf-8');
145
154
 
@@ -24,6 +24,11 @@ import { type FailingKey, builtinAlertKey } from './keys';
24
24
  */
25
25
  const TARGET_KIND_BY_CATEGORY: Partial<Record<DriftCategory, string>> = {
26
26
  machines_reachable: 'machine',
27
+ // Subject is a hostname, which since celilo#1133 may be a machine OR a
28
+ // celilo-provisioned container. A container's hostname finds no machine
29
+ // ancestor, so its alert is never suppressed by one — which is the safe
30
+ // direction: a full disk that pages when it did not have to beats one that
31
+ // stays silent because an unrelated ancestor happened to be firing.
27
32
  disk_space: 'machine',
28
33
  // Subject is an FQDN, which is neither a machine nor a module: a served
29
34
  // name can outlive any single module that asked for it.
@@ -50,10 +50,12 @@ export async function runBuiltinCheckForMonitor(
50
50
  return auditMachinesReachable({ results: await probeMachines() });
51
51
  }
52
52
 
53
- // Same shape as machines_reachable — one SSH round trip per machine, bounded
54
- // but it MEASURES the local box rather than exempting it. See disk-probe.ts.
53
+ // Same shape as machines_reachable — one bounded SSH round trip per target
54
+ // but it differs on both ends: it MEASURES the local box rather than exempting
55
+ // it, and it covers celilo-provisioned instances as well as the machine pool.
56
+ // See disk-probe.ts for why the second one is load-bearing.
55
57
  if (category === 'disk_space') {
56
- return auditDiskSpace({ results: await probeDiskUsage() });
58
+ return auditDiskSpace({ results: await probeDiskUsage(db) });
57
59
  }
58
60
 
59
61
  // Local DB reads only — cheap enough to run on every sweep, which is
@@ -1,5 +1,5 @@
1
1
  import { describe, expect, test } from 'bun:test';
2
- import { parseDfOutput, percentUsed } from './disk-probe';
2
+ import { diskProbeTargets, parseDfOutput, percentUsed } from './disk-probe';
3
3
 
4
4
  describe('percentUsed', () => {
5
5
  // Checked against what celilo-mgr's df actually printed, because an operator
@@ -72,3 +72,63 @@ describe('parseDfOutput', () => {
72
72
  expect(parseDfOutput(out)?.usedPercent).toBe(56);
73
73
  });
74
74
  });
75
+
76
+ describe('diskProbeTargets (celilo#1133)', () => {
77
+ // The two boxes that actually failed, on one night, in one fleet.
78
+ //
79
+ // vmid 204 celilo-registry 20G, 100% full, 4 KB free. Publishes returned
80
+ // `Internal Server Error`; five release attempts died. Reads kept
81
+ // working, so health, the sparse index and `module search` all
82
+ // stayed green.
83
+ // vmid 206 git.celilo.computer 40G, 100% full. The builder's runner logged
84
+ // `failed to fetch task ... database or disk is full` every two
85
+ // seconds and claimed nothing for ~13 hours. 33 GB of it was 785
86
+ // unpruned Actions bundles in repo-archive. It presented as a DEAD
87
+ // RUNNER; the runner was healthy, registered and polling.
88
+ //
89
+ // Both are container_service instances. Both were invisible to a probe that
90
+ // walked `listMachines()`, because a `machine` is an operator-pre-provisioned
91
+ // box and neither of these is one. Two failures, two disguises, one blind spot.
92
+ const registry = { hostname: 'celilo-registry', ipv4Address: '203.0.113.4', vmid: 204 };
93
+ const forge = { hostname: 'forgejo', ipv4Address: '203.0.113.6', vmid: 206 };
94
+ const machine = { hostname: 'iot', ipAddress: '203.0.113.5', sshUser: 'root' };
95
+
96
+ // THE regression, asserted on the two vmids that failed rather than on
97
+ // invented ones. Scheduling the monitor would NOT have caught either: the
98
+ // check would have gone on reporting all-clear about filesystems it never
99
+ // looked at, which is worse than not having the check at all.
100
+ test('includes the container_service instances that filled, not just the machine pool', () => {
101
+ const targets = diskProbeTargets([machine], [registry, forge]);
102
+ expect(targets.map((t) => t.hostname).sort()).toEqual(['celilo-registry', 'forgejo', 'iot']);
103
+ });
104
+
105
+ test('a container is probed as root, the user celilo already deploys to it as', () => {
106
+ const [target] = diskProbeTargets([], [registry]);
107
+ expect(target).toEqual({
108
+ hostname: 'celilo-registry',
109
+ ipAddress: '203.0.113.4',
110
+ sshUser: 'root',
111
+ });
112
+ });
113
+
114
+ test('a machine-pool placement contributes nothing — no vmid, already covered', () => {
115
+ const onMachine = { hostname: 'iot', ipv4Address: '203.0.113.5', vmid: null };
116
+ expect(diskProbeTargets([machine], [onMachine])).toEqual([machine]);
117
+ });
118
+
119
+ test('co-hosted modules share one filesystem and so produce one target', () => {
120
+ const caddy = { hostname: 'shared', ipv4Address: '203.0.113.9', vmid: 210 };
121
+ const authentik = { hostname: 'shared', ipv4Address: '203.0.113.9', vmid: 210 };
122
+ expect(diskProbeTargets([], [caddy, authentik]).length).toBe(1);
123
+ });
124
+
125
+ test('the machine pool wins on a shared address, keeping its configured ssh user', () => {
126
+ const asMachine = { hostname: 'vps', ipAddress: '203.0.113.4', sshUser: 'admin' };
127
+ const [target] = diskProbeTargets([asMachine], [registry]);
128
+ expect(target.sshUser).toBe('admin');
129
+ });
130
+
131
+ test('a system with no address is skipped rather than probed blindly', () => {
132
+ expect(diskProbeTargets([], [{ hostname: 'x', ipv4Address: '', vmid: 7 }])).toEqual([]);
133
+ });
134
+ });
@@ -1,10 +1,39 @@
1
1
  /**
2
- * Disk-usage probe for the machine pool.
2
+ * Disk-usage probe for every system celilo can reach.
3
3
  *
4
- * Structurally a sibling of `machine-probe.ts` — same pool, same SSH bounding —
5
- * with one deliberate difference that is the whole reason this file has its own
4
+ * Structurally a sibling of `machine-probe.ts` — same SSH bounding — with two
5
+ * deliberate differences, and they are the whole reason this file has its own
6
6
  * comment.
7
7
  *
8
+ * ⚠️ IT PROBES SYSTEMS, NOT JUST MACHINES.
9
+ *
10
+ * `machine` has celilo's narrow meaning: an operator-pre-provisioned box in the
11
+ * machine pool. It excludes every LXC and VM celilo provisioned itself through a
12
+ * container_service — which is most of the fleet, and all of the interesting
13
+ * parts of it. This probe walked `listMachines()` alone, so the registry, the
14
+ * forge and the firewall were never measured at all.
15
+ *
16
+ * That is not a coverage nicety. Two instances filled on the same night, and
17
+ * neither raised anything (celilo#1133):
18
+ *
19
+ * vmid 204, celilo-registry — 20G, 100%, four kilobytes free. Every celilo
20
+ * surface stayed green, because reads work on a full disk and only writes
21
+ * fail. The health check passed, the sparse index served, `module search`
22
+ * returned all 42 modules. The first symptom was a release dying with
23
+ * `Internal Server Error`, and finding out why took an ssh and a `df`.
24
+ * vmid 206, git.celilo.computer — 40G, 100%. The builder's runner logged
25
+ * `failed to fetch task … database or disk is full` every two seconds and
26
+ * claimed no work for ~13 hours. It presented as a DEAD CI RUNNER. The
27
+ * runner was healthy, registered and polling the whole time.
28
+ *
29
+ * Both are container_service instances, so both were invisible here for the same
30
+ * reason, and each wore a different disguise. Scheduling the `disk_space` monitor
31
+ * would not have caught either: the check would have gone on reporting all-clear
32
+ * about filesystems it never looked at, which is worse than not having it.
33
+ *
34
+ * So the unit here is the SYSTEM: the machine pool plus every provisioned
35
+ * instance with an address.
36
+ *
8
37
  * ⚠️ THE LOCAL BOX IS MEASURED, NOT EXEMPTED.
9
38
  *
10
39
  * `probeMachines()` reports the management server reachable WITHOUT probing it,
@@ -27,7 +56,9 @@
27
56
  import { execFile } from 'node:child_process';
28
57
  import { statfs } from 'node:fs/promises';
29
58
  import { promisify } from 'node:util';
59
+ import type { DbClient } from '../db/client';
30
60
  import type { DiskUsageResult } from './audit/disk-space';
61
+ import { getProvisionedSystems } from './deployed-systems';
31
62
  import { listMachines } from './machine-pool';
32
63
  import { LOCAL_MACHINE_IP } from './ssh-key-manager';
33
64
 
@@ -133,13 +164,44 @@ async function probeRemote(
133
164
  }
134
165
  }
135
166
 
136
- export async function probeDiskUsage(): Promise<DiskUsageResult[]> {
137
- const machines = await listMachines();
167
+ /**
168
+ * Every system worth measuring, deduplicated by address.
169
+ *
170
+ * A module with no `requires.system` (an API-only module like namecheap) has no
171
+ * instance and contributes nothing. Two modules co-hosted on one container
172
+ * contribute one entry, not two — the filesystem is shared and so is its
173
+ * finding.
174
+ *
175
+ * celilo provisions containers with its own key and deploys to them as root
176
+ * (the `ansible_user` default), which is the same access `df` needs here.
177
+ */
178
+ export function diskProbeTargets(
179
+ machines: Array<{ hostname: string; ipAddress: string; sshUser: string }>,
180
+ systems: Array<{ hostname: string; ipv4Address: string; vmid: number | null }>,
181
+ ): Array<{ hostname: string; ipAddress: string; sshUser: string }> {
182
+ const byAddress = new Map<string, { hostname: string; ipAddress: string; sshUser: string }>();
183
+ for (const m of machines) byAddress.set(m.ipAddress, m);
184
+ for (const s of systems) {
185
+ // No vmid means no celilo-provisioned instance behind this row (a
186
+ // machine-pool placement), and the machine pool above already covers it.
187
+ if (s.vmid == null || !s.ipv4Address) continue;
188
+ if (byAddress.has(s.ipv4Address)) continue;
189
+ byAddress.set(s.ipv4Address, {
190
+ hostname: s.hostname,
191
+ ipAddress: s.ipv4Address,
192
+ sshUser: 'root',
193
+ });
194
+ }
195
+ return [...byAddress.values()];
196
+ }
197
+
198
+ export async function probeDiskUsage(db: DbClient): Promise<DiskUsageResult[]> {
199
+ const targets = diskProbeTargets(await listMachines(), getProvisionedSystems(db));
138
200
  return Promise.all(
139
- machines.map((m) =>
140
- m.ipAddress === LOCAL_MACHINE_IP
141
- ? probeLocal(m.hostname, m.ipAddress)
142
- : probeRemote(m.hostname, m.ipAddress, m.sshUser),
201
+ targets.map((t) =>
202
+ t.ipAddress === LOCAL_MACHINE_IP
203
+ ? probeLocal(t.hostname, t.ipAddress)
204
+ : probeRemote(t.hostname, t.ipAddress, t.sshUser),
143
205
  ),
144
206
  );
145
207
  }