@celilo/cli 1.9.1 → 1.11.0

package/CELILO_SUBSYSTEMS.md

@@ -267,7 +267,8 @@ is currently wrong, and routes carry the message to a person's phone. Design:
 - **⚠️ `monitors.intervalMinutes` and `monitors.enabled` are `builtin_check`-only.** A `module_hook` row carries severity, escalation policy and `lastRunAt`; its cadence and whether it is watched resolve through the accessor above. The columns' meaning depending on `kind` is a named smell (design.md D8) — the alternatives are a cached resolved value that rots, or splitting the table, which needs a synthetic monitor identity for `alerts.monitorId`. Gates: `sweep-runner.test.ts` asserts a module row's stored values are NOT consulted; `cadence-migration.test.ts` asserts the same through `loadModuleHealthCadences`.
 - **Carrying an existing fleet over** — `apps/celilo/src/services/alerting/cadence-migration.ts` (`migrateMonitorCadences`), run from `celilo system migrate` (the `.deb` postinst runs it on every apt upgrade). A monitor row whose cadence diverges from its manifest gets that cadence written as an override; a disabled one gets `manual`. Not bookkeeping: without it the upgrade that ships read-time resolution silently reverts every hand-set cadence to the author's suggestion and resumes watching modules an operator deliberately disabled. Idempotent — writes only where no override exists.
 - **Scheduled audit categories (`builtin_check` monitors)** — `apps/celilo/src/services/alerting/builtin-source.ts` — `SCHEDULABLE_BUILTIN_CHECKS` is the list of `celilo system audit` categories cheap enough to run every sweep: `machines_reachable`, `backups`, `disk_space`, `abandoned_operations`, and `public_dns` (`apps/celilo/src/services/audit/abandoned-operations.ts` — ≥3 abandonments of the same (module, operation) in 7d, the fingerprint of an operation being killed mid-flight). Everything else in the audit needs the whole world injected (proxmox, terraform, registry) and is not schedulable. Enable one with `celilo monitor add backups --interval 1h`, and re-cadence it later with `celilo monitor set-interval backups 6h` (in place, because the monitor id owns the alert history). `monitor set-interval`/`enable`/`disable` REFUSE a module target and name `celilo module config set <m> health_check_interval` — two ways to set one module's cadence would disagree about what `module status` shows. Targets are tab-completable — `completion.ts` reads `SCHEDULABLE_BUILTIN_CHECKS` directly rather than a hand-copied list, so a newly-schedulable check is completable immediately.
-- **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet.
+- **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`, target set from `diskProbeTargets`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. ⚠️ **It probes SYSTEMS, not just MACHINES.** `machine` is celilo's narrow term for an operator-pre-provisioned box in the machine pool; it excludes every LXC/VM celilo provisioned through a container_service — which is most of the fleet. The probe walked `listMachines()` alone until celilo#1133, so the registry, the forge and the firewall were never measured — and **two of them filled on the same night, wearing different disguises**: celilo-registry (vmid 204) hit 100% with 4 KB free and publishes began returning `Internal Server Error` while every celilo surface stayed green (reads work on a full disk; only writes fail), and git.celilo.computer (vmid 206) hit 100% and its runner logged `database or disk is full` every 2s, claiming no work for ~13 hours — presenting as a DEAD CI RUNNER when the runner was healthy and polling. Scheduling the monitor would not have caught either: the check would have reported all-clear about filesystems it never looked at. `diskProbeTargets` now unions the machine pool with `getProvisionedSystems`, deduplicated by address (co-hosted modules share one filesystem and so one finding), containers probed as `root`. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet.
+- **Instance sizing (`celilo proxmox vm|ct list|resize`)** — `apps/celilo/src/cli/commands/proxmox-instance-resize.ts` + `proxmox-resize-guards.ts` (`validateResize`/`computeFloor`, pure and unit-tested without Proxmox) + `proxmox-instance-list.ts` (DESIRED vs ACTUAL c/m/d, drift). A celilo-provisioned instance's size is **canonical infrastructure state in `module_systems`**, never module config; `requires.system` is only the floor used to select a host. Sizing flows `module_systems` → `$self:{cores,memory,disk}` (seeded from `requires.system` on first provision, `apps/celilo/src/variables/context.ts`) → the instance Terraform. ⚠️ **Two apply paths.** cpu/memory reconcile declaratively — record the size, redeploy the owning module, and the provider stop/starts the guest, so that path costs a reboot and is gated on the operator approving one. Disk growth goes DIRECT to Proxmox's resize API (`ProxmoxClient.resizeGuestDisk`, `pct resize`/`qm resize`): online, additive, no power change and no redeploy. That is not a shortcut — a Terraform reconcile means a full Ansible pass, and the box being grown is the one that has run out of room to run one (celilo#1133). Guards: floor and disk-shrink are hard, node capacity yields to `--force`. lxc grows its filesystem too; qemu grows only the block device and the guest must extend its own partition, which the command says out loud.
 - **Public-DNS reachability check (the only check with an OFF-FLEET vantage)** — `apps/celilo/src/services/audit/public-dns.ts` (`auditPublicDns`, pure over an injected probe and the previous run's counters) + `apps/celilo/src/services/public-dns-probe.ts` (the probe) + `audit/public-dns-source.ts` (ledger names + the `public_dns_evidence` counters). Every other check in celilo looks from INSIDE, behind a split-horizon resolver that deliberately answers with an in-zone address — correct for its purpose, and why all of them reported healthy for the nine days of celilo#626. This one resolves every `dns_registrations` FQDN through an **off-fleet resolver** (`public_dns.resolver`, default `1.1.1.1`) and compares it against the address the fleet appears to come from per an independent **echo service** (`public_dns.echo_url`, default `https://api.ipify.org`). Three properties are load-bearing: `assertOffFleetResolver` REFUSES a resolver matching `dns.primary`/`dns.fallback` (a check that quietly used the fleet's resolver would pass forever — the original bug one layer up); the expectation never comes from the registrar's own response (self-agreement, and Namecheap returns `ErrCount 0` for `www` updates it does not apply); and a divergence is a finding only once it OUTLIVES the record's own TTL, measured from the last assert, or it would page on every ISP re-lease. Missing evidence is counted rather than read as success — one undetermined run is silent, N consecutive ones are their own finding (`public_dns_evidence`), which is the hole celilo-website's isitup.org probe demonstrated live. Codes: `public_dns_stale`, `public_dns_missing`, `public_dns_companion_unclaimed`, `public_dns_unverifiable`. Spec: `openspec/specs/public-dns-reachability/spec.md`.
 - **The sweep** — `apps/celilo/src/services/alerting/sweep-runner.ts` (`runSweep`) — the ordered pass that makes alerting run by itself: run due monitors → promote past-grace alerts → re-evaluate suppression → flush quiet-hours deferrals → notify. Driven by `celilo alerts sweep` on `timer.tick.5m`. Never throws for one bad monitor.
 - **Suppression (topology-derived, never configured)** — `apps/celilo/src/services/alerting/suppression.ts` — `ancestorKeysFor`/`findSuppressor`/`machineAlertKey`. A firing machine explains its modules' failures; a firing capability provider explains its zone's consumers. Derived from `module_systems`, so it cannot drift from reality. Deploy windows: `deploy-hooks.ts` (`openDeployWindow`/`closeDeployWindows` — closed by module, so a crashed deploy self-heals).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@celilo/cli",
-  "version": "1.9.1",
+  "version": "1.11.0",
   "description": "Celilo — home lab orchestration CLI",
   "type": "module",
   "bin": {
@@ -58,7 +58,7 @@
   "dependencies": {
     "@aws-sdk/client-s3": "^3.1109.0",
     "@aws-sdk/lib-storage": "^3.1101.0",
-    "@celilo/capabilities": "^3.1.0",
+    "@celilo/capabilities": "^3.2.0",
     "@celilo/cli-display": "^0.2.0",
     "@celilo/core": "^0.9.1",
     "@celilo/event-bus": "^0.6.0",

package/src/api-clients/proxmox.ts

@@ -663,6 +663,43 @@ export class ProxmoxClient {
     );
   }
 
+  /**
+   * Grow a guest's disk, online. This is `pct resize` / `qm resize` (celilo#1133).
+   *
+   * Deliberately NOT routed through Terraform like a cpu/memory resize is. Two
+   * reasons, and the first is the one that matters: a Terraform reconcile means
+   * a full module redeploy — Ansible, package installs, service restarts — and
+   * the box being resized is by definition the one that has run out of room to
+   * do any of that. The cure would need the disk space it exists to provide.
+   * Second, the resize API is additive and online, so celilo can promise the
+   * guest is not power-cycled; delegating that to a provider's update path
+   * cannot promise it.
+   *
+   * `disk` is the config key of the volume to grow — `rootfs` for lxc, `scsi0`
+   * for a cloud-init VM. `size` is absolute (Proxmox also accepts `+NG`, but
+   * canonical state is an absolute size, so we send one).
+   *
+   * ⚠️ lxc grows the filesystem too; qemu grows only the block device and the
+   * guest must extend its own partition. Callers say so rather than implying
+   * the space is usable.
+   */
+  async resizeGuestDisk(
+    vmid: number,
+    kind: 'lxc' | 'qemu',
+    disk: string,
+    sizeGb: number,
+  ): Promise<ProxmoxResult<string>> {
+    const node = await this.nodeForVmid(vmid);
+    if (!node.success) return node;
+    if (!node.data) {
+      return { success: false, message: `No Proxmox node hosts vmid ${vmid}` };
+    }
+    return makeProxmoxPut<string>(this.credentials, `/nodes/${node.data}/${kind}/${vmid}/resize`, {
+      disk,
+      size: `${sizeGb}G`,
+    });
+  }
+
   /** Current run state of a guest (`running`, `stopped`, …). */
   async guestStatus(vmid: number): Promise<ProxmoxResult<string | null>> {
     const result = await this.clusterResources();
@@ -785,6 +822,14 @@ async function makeProxmoxPost<T>(
   return makeProxmoxFormRequest(credentials, 'POST', path, params);
 }
 
+async function makeProxmoxPut<T>(
+  credentials: ProxmoxCredentials,
+  path: string,
+  params: Record<string, string>,
+): Promise<ProxmoxResult<T>> {
+  return makeProxmoxFormRequest(credentials, 'PUT', path, params);
+}
+
 /**
  * Entry from Proxmox's appliance catalog (`pveam available`). The `template`
  * field is the canonical filename (revision included) that should be passed to

package/src/cli/commands/proxmox-instance-list.ts

@@ -5,7 +5,7 @@
  * desired≠actual row means a resize is pending a reconcile/redeploy.
  *
  * Read-only sibling of `proxmox node list`; the foundation the `resize` verb
- * (ISS-0150 P2) builds on. See apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md.
+ * (ISS-0150 P2) builds on. See openspec/changes/proxmox-capacity-lifecycle/.
  */
 
 import {

package/src/cli/commands/proxmox-instance-resize.ts

@@ -1,13 +1,30 @@
 /**
- * `celilo proxmox vm resize <name> --memory <MB> [--cpu <n>]` (and `ct resize`) —
- * the canonical resize for a celilo-provisioned instance (ISS-0150 P2). Updates
- * the SYSTEM's canonical size in module_systems, then reconciles declaratively by
- * redeploying the owning module (its Terraform now reads the system size).
+ * `celilo proxmox vm resize <name> [--memory <MB>] [--cpu <n>] [--disk <GB>]`
+ * (and `ct resize`) — the canonical resize for a celilo-provisioned instance
+ * (ISS-0150 P2). Updates the SYSTEM's canonical size in module_systems.
+ *
+ * Two apply paths, because the two kinds of change are not alike:
+ *
+ *   cpu / memory  → declarative. Record the size, redeploy the owning module,
+ *                   and its Terraform writes the new size to the guest config.
+ *                   The provider stop/starts the guest to apply it, so this
+ *                   path costs a reboot and is gated on the operator saying so.
+ *   disk (growth) → direct. Proxmox's resize API grows the volume online, so
+ *                   celilo calls it and skips the redeploy entirely. No reboot,
+ *                   no Ansible. This is not a shortcut: a full module redeploy
+ *                   needs disk space to run, and the box being grown is the one
+ *                   that has run out of it (celilo#1133).
+ *
+ * Both paths leave module_systems canonical, so `proxmox ct list` shows DESIRED
+ * matching ACTUAL and drift clears either way. The instance Terraform reads
+ * `$self:disk` (not `requires.system.disk`, which is only the floor), so the
+ * next deploy agrees with a resize instead of trying to undo it.
  *
  * Guardrails (proxmox-resize-guards.ts): floor = max(requires.system) across
- * co-hosted modules (hard); node capacity (overridable with --force). Reboot +
- * pre-resize backup are confirmed via the event-bus interview (CLAUDE.md: no
- * direct CLI prompts) — flags win: --allow-reboot / --skip-backup / --yes.
+ * co-hosted modules (hard); disk shrink (hard); node capacity (overridable with
+ * --force). Reboot + pre-resize backup are confirmed via the event-bus
+ * interview (CLAUDE.md: no direct CLI prompts) — flags win: --allow-reboot /
+ * --skip-backup / --yes.
  */
 
 import { and, eq } from 'drizzle-orm';
@@ -26,7 +43,9 @@ import type { InstanceKind } from './proxmox-instance-list';
 import { computeFloor, validateResize } from './proxmox-resize-guards';
 import { resolveProxmoxService } from './proxmox-service';
 
-const PROXMOX_TYPE: Record<InstanceKind, string> = { vm: 'qemu', ct: 'lxc' };
+const PROXMOX_TYPE: Record<InstanceKind, 'qemu' | 'lxc'> = { vm: 'qemu', ct: 'lxc' };
+
+const BYTES_PER_GB = 1024 * 1024 * 1024;
 
 function numFlag(v: string | boolean | undefined): number | undefined {
   if (typeof v !== 'string') return undefined;
@@ -43,20 +62,16 @@ export async function handleProxmoxInstanceResize(
 
   const name = args[0];
   if (!name) {
-    const e = `Instance name required: celilo proxmox ${kind} resize <name> --memory <MB> [--cpu <n>]`;
+    const e = `Instance name required: celilo proxmox ${kind} resize <name> [--memory <MB>] [--cpu <n>] [--disk <GB>]`;
     console.log(`✗ ${e}`);
     return { success: false, error: e };
   }
 
-  if (numFlag(flags.disk) != null) {
-    const e = 'Disk resize is deferred (ISS-0150 D4). Use --memory / --cpu for now.';
-    console.log(`✗ ${e}`);
-    return { success: false, error: e };
-  }
   const reqCpu = numFlag(flags.cpu);
   const reqMemory = numFlag(flags.memory);
-  if (reqCpu == null && reqMemory == null) {
-    const e = 'Specify at least one of --memory <MB> or --cpu <n>.';
+  const reqDisk = numFlag(flags.disk);
+  if (reqCpu == null && reqMemory == null && reqDisk == null) {
+    const e = 'Specify at least one of --memory <MB>, --cpu <n> or --disk <GB>.';
     console.log(`✗ ${e}`);
     return { success: false, error: e };
   }
@@ -116,13 +131,24 @@ export async function handleProxmoxInstanceResize(
   const nodeCaps = await client.nodeCapacities();
   const nodeCap = nodeCaps.success ? nodeCaps.data.find((n) => n.node === node) : undefined;
 
+  // Disk compares against the LIVE volume, not celilo's record of it. "Can
+  // Proxmox do this" is a question about the real disk: canonical state is
+  // seeded from the manifest floor and goes stale the moment anyone runs `pct
+  // resize` by hand — which, until this command existed, was the only way to
+  // grow one. Comparing a grow request against a stale-low record would wave
+  // through a shrink that Proxmox then refuses. cpu/memory keep using canonical
+  // state, which the reconcile path owns end to end.
+  const liveDiskGb =
+    guest?.maxdisk != null ? Math.round(guest.maxdisk / BYTES_PER_GB) : target.disk;
+
   const decision = validateResize(
-    { cpu: reqCpu, memory: reqMemory },
+    { cpu: reqCpu, memory: reqMemory, disk: reqDisk },
     {
-      current: { cpu: target.cpu, memory: target.memory, disk: target.disk },
+      current: { cpu: target.cpu, memory: target.memory, disk: liveDiskGb },
       floor,
       nodeFreeMemMb: nodeCap?.memFreeMb ?? Number.POSITIVE_INFINITY,
       nodeTotalCores: nodeCap?.cpuCores ?? Number.POSITIVE_INFINITY,
+      nodeFreeDiskGb: nodeCap?.diskFreeGb ?? Number.POSITIVE_INFINITY,
     },
     { force: flags.force === true },
   );
@@ -137,10 +163,24 @@ export async function handleProxmoxInstanceResize(
   const yes = flags.yes === true;
   const coHostedIds = coHosted.map((s) => s.moduleId);
 
+  // One decision, made in the guard layer where it is unit-tested, consumed
+  // twice below. `needsReboot` is false exactly when nothing but disk changed:
+  // disk growth applies online through the resize API, so it neither
+  // power-cycles the guest nor needs Terraform to run.
+  //
+  // Both skips that follow hang on it. Growing a disk is purely additive —
+  // nothing is destroyed, nothing is rewritten — so there is no state a backup
+  // would protect, and staging one would be actively harmful: the operator
+  // reaching for --disk has a box that has run out of room, and a backup needs
+  // the very resource it lacks. A cpu/memory change still backs up and still
+  // redeploys, because that one reboots.
+  const diskOnly = !decision.needsReboot;
+
   // Pre-resize backup — ON by default; --skip-backup opts out; otherwise the
   // skip decision is an event-bus interview question (never a direct CLI prompt).
   let doBackup: boolean;
-  if (flags['skip-backup'] === true) doBackup = false;
+  if (diskOnly) doBackup = false;
+  else if (flags['skip-backup'] === true) doBackup = false;
   else if (yes) doBackup = true;
   else
     doBackup = await withInterviewSession(() =>
@@ -190,11 +230,39 @@ export async function handleProxmoxInstanceResize(
     }
   }
 
+  // Disk goes straight to Proxmox's resize API — online, additive, no power
+  // change. Do it BEFORE recording canonical state so a failure leaves celilo's
+  // record matching reality rather than claiming a size the guest doesn't have.
+  if (reqDisk != null) {
+    // rootfs for a container, scsi0 for a cloud-init VM — the volumes celilo's
+    // own Terraform declares (modules/*/terraform/main.tf.tpl).
+    const volume = kind === 'ct' ? 'rootfs' : 'scsi0';
+    console.log(`▸ Growing ${volume} to ${reqDisk}GB (online — no restart)…`);
+    const resized = await client.resizeGuestDisk(vmid, PROXMOX_TYPE[kind], volume, reqDisk);
+    if (!resized.success) {
+      const e = `Disk resize failed: ${resized.message}`;
+      console.log(`✗ ${e}`);
+      return { success: false, error: e };
+    }
+    console.log(`  ✓ ${volume} is now ${reqDisk}GB`);
+    if (kind === 'vm') {
+      // qemu grows the block device only. Saying "resized" without this would
+      // send the operator back to a box whose df output has not moved.
+      console.log(
+        `  ⚠ The VM's block device grew, but the guest must extend its own partition and filesystem
+    (e.g. growpart /dev/sda 1 && resize2fs /dev/sda1). A container does this for itself; a VM does not.`,
+      );
+    }
+  }
+
   // Update the canonical size on every module_systems row for this instance.
   for (const s of coHosted) {
-    const set: { cpu?: number; memory?: number; updatedAt: Date } = { updatedAt: new Date() };
+    const set: { cpu?: number; memory?: number; disk?: number; updatedAt: Date } = {
+      updatedAt: new Date(),
+    };
     if (reqCpu != null) set.cpu = reqCpu;
     if (reqMemory != null) set.memory = reqMemory;
+    if (reqDisk != null) set.disk = reqDisk;
     db.update(moduleSystems)
       .set(set)
       .where(and(eq(moduleSystems.moduleId, s.moduleId), eq(moduleSystems.name, s.name)))
@@ -202,7 +270,19 @@ export async function handleProxmoxInstanceResize(
   }
   const effCpu = reqCpu ?? target.cpu;
   const effMem = reqMemory ?? target.memory;
-  console.log(`✓ Canonical size updated: ${name} → ${effCpu ?? '?'}c / ${effMem ?? '?'}MB`);
+  const effDisk = reqDisk ?? liveDiskGb;
+  console.log(
+    `✓ Canonical size updated: ${name} → ${effCpu ?? '?'}c / ${effMem ?? '?'}MB / ${effDisk ?? '?'}GB`,
+  );
+
+  // Disk-only: the guest already carries the new size and the canonical record
+  // now agrees, so Terraform has nothing to reconcile. Redeploying would run a
+  // full Ansible pass for no change — on a box that just filled up, that is the
+  // expensive, risky half of a cure whose cheap half already landed.
+  if (diskOnly) {
+    console.log(`✓ Resized ${name} to ${reqDisk}GB. No restart, no redeploy.`);
+    return { success: true, message: `resized ${name}` };
+  }
 
   // Reconcile declaratively: redeploy the owning module — its Terraform now reads
   // the updated system size and writes it to the Proxmox VM config. The proxmox

package/src/cli/commands/proxmox-resize-guards.test.ts

@@ -6,6 +6,7 @@ const ctx = (over: Partial<ResizeContext> = {}): ResizeContext => ({
   floor: { cpu: 4, memory: 8192, disk: 80 },
   nodeFreeMemMb: 20000,
   nodeTotalCores: 6,
+  nodeFreeDiskGb: 500,
   ...over,
 });
 
@@ -39,6 +40,56 @@ describe('validateResize (ISS-0150)', () => {
   });
 });
 
+describe('validateResize — disk (celilo#1133)', () => {
+  // The whole point of the verb: the registry filled up and celilo had no cure.
+  test('growing the disk passes and needs NO reboot', () => {
+    const d = validateResize({ disk: 160 }, ctx());
+    expect(d.errors).toEqual([]);
+    expect(d.effective.disk).toBe(160);
+    // Load-bearing. The instances that fill up are the registry, the forge and
+    // the firewall; a resize that needlessly restarts one of those is a worse
+    // cure than the disease, and it would be discovered at the worst moment.
+    expect(d.needsReboot).toBe(false);
+  });
+
+  test('a cpu or memory change alongside a disk change still reboots', () => {
+    expect(validateResize({ disk: 160, memory: 16384 }, ctx()).needsReboot).toBe(true);
+    expect(validateResize({ disk: 160, cpu: 6 }, ctx()).needsReboot).toBe(true);
+  });
+
+  test('shrink is refused even well above the floor, and even with --force', () => {
+    // 60GB clears the 40GB floor, so this is NOT a floor violation — it is a
+    // shrink, and Proxmox cannot do one safely. The two need different errors
+    // because the operator's next move differs.
+    const roomy = ctx({ floor: { cpu: 4, memory: 8192, disk: 40 } });
+    const d = validateResize({ disk: 60 }, roomy, { force: true });
+    expect(d.errors.length).toBe(1);
+    expect(d.errors[0]).toContain('cannot shrink');
+    expect(d.capacityOnly).toBe(false);
+  });
+
+  test('shrinking below the floor reports the floor too', () => {
+    const d = validateResize({ disk: 10 }, ctx(), { force: true });
+    expect(d.errors.length).toBe(2);
+    expect(d.errors.some((e) => e.includes('below the module minimum'))).toBe(true);
+    expect(d.errors.some((e) => e.includes('cannot shrink'))).toBe(true);
+  });
+
+  test('growth beyond node free storage is a capacity error, overridable with --force', () => {
+    const tight = ctx({ nodeFreeDiskGb: 5 }); // 5GB free; +20GB requested
+    const blocked = validateResize({ disk: 100 }, tight, { force: false });
+    expect(blocked.errors.length).toBe(1);
+    expect(blocked.errors[0]).toContain('exceeds node free storage');
+    expect(blocked.capacityOnly).toBe(true);
+    expect(validateResize({ disk: 100 }, tight, { force: true }).errors).toEqual([]);
+  });
+
+  test('an equal-size disk request is neither shrink nor growth', () => {
+    const d = validateResize({ disk: 80 }, ctx({ nodeFreeDiskGb: 0 }));
+    expect(d.errors).toEqual([]);
+  });
+});
+
 describe('computeFloor', () => {
   test('takes the max of each field across co-hosted modules', () => {
     expect(

package/src/cli/commands/proxmox-resize-guards.ts

@@ -1,7 +1,13 @@
 /**
- * Pure guardrails for `celilo proxmox … resize` (ISS-0150 D4). Kept separate from
- * the command so the floor + capacity logic is unit-testable without Proxmox or a
- * DB. See apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md.
+ * Pure guardrails for `celilo proxmox … resize` (ISS-0150 D4, disk landed in
+ * celilo#1133). Kept separate from the command so the floor + capacity logic is
+ * unit-testable without Proxmox or a DB — which is what let the disk paths be
+ * proven with the fleet unreachable.
+ *
+ * See openspec/changes/proxmox-capacity-lifecycle/. (The header used to point at
+ * apps/celilo/designs/PROXMOX_INSTANCE_SIZING.md, which does not exist and cannot:
+ * that directory is gitignored, so a doc written there is never committed and
+ * leaves with the worktree. CLAUDE.md flags the path as a trap.)
  */
 
 export interface ResizeRequest {
@@ -22,6 +28,8 @@ export interface ResizeContext {
   nodeFreeMemMb: number;
   /** Physical cores on the target node — a VM's vCPU can't exceed this (absolute). */
   nodeTotalCores: number;
+  /** Free storage on the target node — disk growth must fit (delta check). */
+  nodeFreeDiskGb: number;
 }
 
 export interface ResizeDecision {
@@ -36,10 +44,12 @@ export interface ResizeDecision {
 }
 
 /**
- * Validate a resize request against the floor and node capacity. Disk shrink is
- * rejected (qemu can't shrink safely); disk resize is otherwise deferred
- * (D4) — callers should reject a disk change until that lands, but the floor/grow
- * checks are here for when it does.
+ * Validate a resize request against the floor and node capacity.
+ *
+ * Disk shrink is rejected: Proxmox grows a disk online but has no safe shrink
+ * for either lxc or qemu — the guest filesystem would have to be shrunk first,
+ * from inside, and celilo does not do that. Disk GROWTH is additive and needs
+ * no stop/start, which is why `needsReboot` keys on cpu/memory alone.
  */
 export function validateResize(
   req: ResizeRequest,
@@ -52,15 +62,26 @@ export function validateResize(
     disk: req.disk ?? ctx.current.disk,
   };
 
-  const floorErrors: string[] = [];
+  // Hard errors — never overridable, not even with --force.
+  const hardErrors: string[] = [];
   if (req.cpu != null && req.cpu < ctx.floor.cpu) {
-    floorErrors.push(`cpu ${req.cpu} is below the module minimum (${ctx.floor.cpu})`);
+    hardErrors.push(`cpu ${req.cpu} is below the module minimum (${ctx.floor.cpu})`);
   }
   if (req.memory != null && req.memory < ctx.floor.memory) {
-    floorErrors.push(`memory ${req.memory}MB is below the module minimum (${ctx.floor.memory}MB)`);
+    hardErrors.push(`memory ${req.memory}MB is below the module minimum (${ctx.floor.memory}MB)`);
   }
   if (req.disk != null && req.disk < ctx.floor.disk) {
-    floorErrors.push(`disk ${req.disk}GB is below the module minimum (${ctx.floor.disk}GB)`);
+    hardErrors.push(`disk ${req.disk}GB is below the module minimum (${ctx.floor.disk}GB)`);
+  }
+  // Shrink is refused on its own terms, not as a floor violation: a disk well
+  // above every tenant's minimum still cannot be shrunk. Say WHY rather than
+  // rejecting bare, because the operator's next move differs — a floor error
+  // means "pick a bigger number", a shrink error means "you cannot do this at
+  // all, migrate instead".
+  if (req.disk != null && ctx.current.disk != null && req.disk < ctx.current.disk) {
+    hardErrors.push(
+      `disk ${req.disk}GB is smaller than the current ${ctx.current.disk}GB — Proxmox cannot shrink a disk safely. Provision a smaller instance and migrate the data instead.`,
+    );
   }
 
   const capErrors: string[] = [];
@@ -76,10 +97,20 @@ export function validateResize(
     }
   }
 
-  const capacityOnly = floorErrors.length === 0 && capErrors.length > 0;
-  // Floor is a hard floor (never overridable). Capacity is overridable with --force.
-  const errors = [...floorErrors, ...(opts.force ? [] : capErrors)];
+  // disk: only the GROWTH beyond the current size draws on the node's storage.
+  if (req.disk != null && ctx.current.disk != null) {
+    const addedGb = req.disk - ctx.current.disk;
+    if (addedGb > ctx.nodeFreeDiskGb) {
+      capErrors.push(`+${addedGb}GB exceeds node free storage (${ctx.nodeFreeDiskGb}GB)`);
+    }
+  }
+
+  const capacityOnly = hardErrors.length === 0 && capErrors.length > 0;
+  // Floor and shrink are hard (never overridable). Capacity yields to --force.
+  const errors = [...hardErrors, ...(opts.force ? [] : capErrors)];
 
+  // Disk growth is applied online by Proxmox's resize API — no stop/start. Only
+  // a cpu/memory change needs the guest power-cycled.
   const needsReboot = req.cpu != null || req.memory != null;
 
   return { errors, effective, needsReboot, capacityOnly };

package/src/manifest/contracts/v1.ts

@@ -311,6 +311,30 @@ export const V1_HOOKS: ContractHooks = {
     inputs: {},
     outputs: {},
   },
+  /**
+   * Reclaim disk a module's own store has accumulated and has no way to
+   * release on its own.
+   *
+   * Distinct from `on_backup`'s retention, which prunes copies celilo itself
+   * made and records. This is for state the module's SERVICE owns and celilo
+   * never sees — a registry's package store, a cache, an artifact directory —
+   * where the only thing that knows what is superseded is the service.
+   *
+   * Deliberately not a converge hook: convergence re-asserts a desired state
+   * and is safe to run constantly, while this DELETES and its mistakes are not
+   * recoverable. So it declares its retention policy in config, supports a dry
+   * run, and is expected to be conservative — the celilo-registry sweep never
+   * removes the last revision of a release, which is what makes running it
+   * unattended defensible.
+   *
+   * Driven on demand (`celilo module run-hook <module> sweep_revisions`) and,
+   * where the store grows without an operator, by a `timer.tick.*`
+   * subscription.
+   */
+  sweep_revisions: {
+    inputs: {},
+    outputs: {},
+  },
   /**
    * Build-bus upstream publish hook. The executor passes the
    * PublishEvent fields as env vars (CELILO_EVENT_PAYLOAD,

package/src/manifest/schema.ts

@@ -366,6 +366,7 @@ const LIFECYCLE_HOOK_SCHEMAS = {
   reconcile_clients: LifecycleHookSchema.optional(),
   list_peers: LifecycleHookSchema.optional(),
   reconcile_peers: LifecycleHookSchema.optional(),
+  sweep_revisions: LifecycleHookSchema.optional(),
 } satisfies Record<HookName, z.ZodTypeAny>;
 
 /**

package/src/services/alerting/builtin-monitors.ts

@@ -24,6 +24,11 @@ import { type FailingKey, builtinAlertKey } from './keys';
  */
 const TARGET_KIND_BY_CATEGORY: Partial<Record<DriftCategory, string>> = {
   machines_reachable: 'machine',
+  // Subject is a hostname, which since celilo#1133 may be a machine OR a
+  // celilo-provisioned container. A container's hostname finds no machine
+  // ancestor, so its alert is never suppressed by one — which is the safe
+  // direction: a full disk that pages when it did not have to beats one that
+  // stays silent because an unrelated ancestor happened to be firing.
   disk_space: 'machine',
   // Subject is an FQDN, which is neither a machine nor a module: a served
   // name can outlive any single module that asked for it.

package/src/services/alerting/builtin-source.ts

@@ -50,10 +50,12 @@ export async function runBuiltinCheckForMonitor(
     return auditMachinesReachable({ results: await probeMachines() });
   }
 
-  // Same shape as machines_reachable — one SSH round trip per machine, bounded
-  // — but it MEASURES the local box rather than exempting it. See disk-probe.ts.
+  // Same shape as machines_reachable — one bounded SSH round trip per target —
+  // but it differs on both ends: it MEASURES the local box rather than exempting
+  // it, and it covers celilo-provisioned instances as well as the machine pool.
+  // See disk-probe.ts for why the second one is load-bearing.
   if (category === 'disk_space') {
-    return auditDiskSpace({ results: await probeDiskUsage() });
+    return auditDiskSpace({ results: await probeDiskUsage(db) });
   }
 
   // Local DB reads only — cheap enough to run on every sweep, which is

package/src/services/disk-probe.test.ts

@@ -1,5 +1,5 @@
 import { describe, expect, test } from 'bun:test';
-import { parseDfOutput, percentUsed } from './disk-probe';
+import { diskProbeTargets, parseDfOutput, percentUsed } from './disk-probe';
 
 describe('percentUsed', () => {
   // Checked against what celilo-mgr's df actually printed, because an operator
@@ -72,3 +72,63 @@ describe('parseDfOutput', () => {
     expect(parseDfOutput(out)?.usedPercent).toBe(56);
   });
 });
+
+describe('diskProbeTargets (celilo#1133)', () => {
+  // The two boxes that actually failed, on one night, in one fleet.
+  //
+  //   vmid 204  celilo-registry  20G, 100% full, 4 KB free. Publishes returned
+  //             `Internal Server Error`; five release attempts died. Reads kept
+  //             working, so health, the sparse index and `module search` all
+  //             stayed green.
+  //   vmid 206  git.celilo.computer  40G, 100% full. The builder's runner logged
+  //             `failed to fetch task ... database or disk is full` every two
+  //             seconds and claimed nothing for ~13 hours. 33 GB of it was 785
+  //             unpruned Actions bundles in repo-archive. It presented as a DEAD
+  //             RUNNER; the runner was healthy, registered and polling.
+  //
+  // Both are container_service instances. Both were invisible to a probe that
+  // walked `listMachines()`, because a `machine` is an operator-pre-provisioned
+  // box and neither of these is one. Two failures, two disguises, one blind spot.
+  const registry = { hostname: 'celilo-registry', ipv4Address: '203.0.113.4', vmid: 204 };
+  const forge = { hostname: 'forgejo', ipv4Address: '203.0.113.6', vmid: 206 };
+  const machine = { hostname: 'iot', ipAddress: '203.0.113.5', sshUser: 'root' };
+
+  // THE regression, asserted on the two vmids that failed rather than on
+  // invented ones. Scheduling the monitor would NOT have caught either: the
+  // check would have gone on reporting all-clear about filesystems it never
+  // looked at, which is worse than not having the check at all.
+  test('includes the container_service instances that filled, not just the machine pool', () => {
+    const targets = diskProbeTargets([machine], [registry, forge]);
+    expect(targets.map((t) => t.hostname).sort()).toEqual(['celilo-registry', 'forgejo', 'iot']);
+  });
+
+  test('a container is probed as root, the user celilo already deploys to it as', () => {
+    const [target] = diskProbeTargets([], [registry]);
+    expect(target).toEqual({
+      hostname: 'celilo-registry',
+      ipAddress: '203.0.113.4',
+      sshUser: 'root',
+    });
+  });
+
+  test('a machine-pool placement contributes nothing — no vmid, already covered', () => {
+    const onMachine = { hostname: 'iot', ipv4Address: '203.0.113.5', vmid: null };
+    expect(diskProbeTargets([machine], [onMachine])).toEqual([machine]);
+  });
+
+  test('co-hosted modules share one filesystem and so produce one target', () => {
+    const caddy = { hostname: 'shared', ipv4Address: '203.0.113.9', vmid: 210 };
+    const authentik = { hostname: 'shared', ipv4Address: '203.0.113.9', vmid: 210 };
+    expect(diskProbeTargets([], [caddy, authentik]).length).toBe(1);
+  });
+
+  test('the machine pool wins on a shared address, keeping its configured ssh user', () => {
+    const asMachine = { hostname: 'vps', ipAddress: '203.0.113.4', sshUser: 'admin' };
+    const [target] = diskProbeTargets([asMachine], [registry]);
+    expect(target.sshUser).toBe('admin');
+  });
+
+  test('a system with no address is skipped rather than probed blindly', () => {
+    expect(diskProbeTargets([], [{ hostname: 'x', ipv4Address: '', vmid: 7 }])).toEqual([]);
+  });
+});

package/src/services/disk-probe.ts

@@ -1,10 +1,39 @@
 /**
- * Disk-usage probe for the machine pool.
+ * Disk-usage probe for every system celilo can reach.
  *
- * Structurally a sibling of `machine-probe.ts` — same pool, same SSH bounding —
- * with one deliberate difference that is the whole reason this file has its own
+ * Structurally a sibling of `machine-probe.ts` — same SSH bounding — with two
+ * deliberate differences, and they are the whole reason this file has its own
  * comment.
  *
+ * ⚠️ IT PROBES SYSTEMS, NOT JUST MACHINES.
+ *
+ * `machine` has celilo's narrow meaning: an operator-pre-provisioned box in the
+ * machine pool. It excludes every LXC and VM celilo provisioned itself through a
+ * container_service — which is most of the fleet, and all of the interesting
+ * parts of it. This probe walked `listMachines()` alone, so the registry, the
+ * forge and the firewall were never measured at all.
+ *
+ * That is not a coverage nicety. Two instances filled on the same night, and
+ * neither raised anything (celilo#1133):
+ *
+ *   vmid 204, celilo-registry — 20G, 100%, four kilobytes free. Every celilo
+ *     surface stayed green, because reads work on a full disk and only writes
+ *     fail. The health check passed, the sparse index served, `module search`
+ *     returned all 42 modules. The first symptom was a release dying with
+ *     `Internal Server Error`, and finding out why took an ssh and a `df`.
+ *   vmid 206, git.celilo.computer — 40G, 100%. The builder's runner logged
+ *     `failed to fetch task … database or disk is full` every two seconds and
+ *     claimed no work for ~13 hours. It presented as a DEAD CI RUNNER. The
+ *     runner was healthy, registered and polling the whole time.
+ *
+ * Both are container_service instances, so both were invisible here for the same
+ * reason, and each wore a different disguise. Scheduling the `disk_space` monitor
+ * would not have caught either: the check would have gone on reporting all-clear
+ * about filesystems it never looked at, which is worse than not having it.
+ *
+ * So the unit here is the SYSTEM: the machine pool plus every provisioned
+ * instance with an address.
+ *
  * ⚠️ THE LOCAL BOX IS MEASURED, NOT EXEMPTED.
  *
  * `probeMachines()` reports the management server reachable WITHOUT probing it,
@@ -27,7 +56,9 @@
 import { execFile } from 'node:child_process';
 import { statfs } from 'node:fs/promises';
 import { promisify } from 'node:util';
+import type { DbClient } from '../db/client';
 import type { DiskUsageResult } from './audit/disk-space';
+import { getProvisionedSystems } from './deployed-systems';
 import { listMachines } from './machine-pool';
 import { LOCAL_MACHINE_IP } from './ssh-key-manager';
 
@@ -133,13 +164,44 @@ async function probeRemote(
   }
 }
 
-export async function probeDiskUsage(): Promise<DiskUsageResult[]> {
-  const machines = await listMachines();
+/**
+ * Every system worth measuring, deduplicated by address.
+ *
+ * A module with no `requires.system` (an API-only module like namecheap) has no
+ * instance and contributes nothing. Two modules co-hosted on one container
+ * contribute one entry, not two — the filesystem is shared and so is its
+ * finding.
+ *
+ * celilo provisions containers with its own key and deploys to them as root
+ * (the `ansible_user` default), which is the same access `df` needs here.
+ */
+export function diskProbeTargets(
+  machines: Array<{ hostname: string; ipAddress: string; sshUser: string }>,
+  systems: Array<{ hostname: string; ipv4Address: string; vmid: number | null }>,
+): Array<{ hostname: string; ipAddress: string; sshUser: string }> {
+  const byAddress = new Map<string, { hostname: string; ipAddress: string; sshUser: string }>();
+  for (const m of machines) byAddress.set(m.ipAddress, m);
+  for (const s of systems) {
+    // No vmid means no celilo-provisioned instance behind this row (a
+    // machine-pool placement), and the machine pool above already covers it.
+    if (s.vmid == null || !s.ipv4Address) continue;
+    if (byAddress.has(s.ipv4Address)) continue;
+    byAddress.set(s.ipv4Address, {
+      hostname: s.hostname,
+      ipAddress: s.ipv4Address,
+      sshUser: 'root',
+    });
+  }
+  return [...byAddress.values()];
+}
+
+export async function probeDiskUsage(db: DbClient): Promise<DiskUsageResult[]> {
+  const targets = diskProbeTargets(await listMachines(), getProvisionedSystems(db));
   return Promise.all(
-    machines.map((m) =>
-      m.ipAddress === LOCAL_MACHINE_IP
-        ? probeLocal(m.hostname, m.ipAddress)
-        : probeRemote(m.hostname, m.ipAddress, m.sshUser),
+    targets.map((t) =>
+      t.ipAddress === LOCAL_MACHINE_IP
+        ? probeLocal(t.hostname, t.ipAddress)
+        : probeRemote(t.hostname, t.ipAddress, t.sshUser),
     ),
   );
 }