@rubytech/create-maxy-code 0.1.477 → 0.1.478

package/payload/platform/docs/superpowers/specs/2026-07-20-task-1704-data-portal-standing-audit-design.md

@@ -0,0 +1,177 @@
+# Task 1704 — data portal standing audit
+
+**Date:** 2026-07-20 · **Lane:** cloudflare + storage-broker · data-portal observability
+**Depends on:** 1689 (portal, landed), 1694 (broker R2 object ops, landed), 1810 (loop registry, landed)
+
+## The problem
+
+The `[data-portal]` lifeline traces one upload end to end and ends in a re-read post-condition, so
+every failure that emits an event is already covered. Three conditions emit nothing:
+
+- **An orphan object** — an object in R2 with no manifest row. Stored, invisible to ingestion
+  forever, and the upload that produced it returned `200`.
+- **A phantom row** — a manifest row whose object is absent. Ingestion will pull a missing file,
+  and nothing says so until ingestion runs, which may be never.
+- **A stalled ingestion sweep** — an ingestion that never runs logs nothing, so a climbing backlog
+  and an empty one look identical.
+
+Only a standing reconcile reveals any of them. Absence of the audit's line is itself the signal that
+all three are unobservable.
+
+## Premise corrections carried from the task file
+
+The task brief predates three changes. All three point the same way; none change its intent.
+
+1. **The `[storage-audit]` idiom is now `registerLoop`.** Task 1810 moved every standing loop onto
+   `platform/ui/server/loop-registry.ts`, and `scripts/check-loops-registered.mjs` fails the build on
+   a bare `setInterval`. "Copy the `[storage-audit]` shape" therefore means registering a loop, not
+   hand-rolling two unref'd timers.
+2. **Every line number in the brief is stale**, though every cited thing still exists.
+   `runStorageAudit` is `storage-broker.ts:773`; the "silent when the config names no portal" idiom
+   is the `hasBookingSite` check in the availability loop; `r2ObjectList`'s drain-to-exhaustion
+   comment at `cf-exec.ts:443` names this audit as its own reason for existing.
+3. **`reconcilePages` (Task 1728) is a closer template than the brief knew.** It is already a pure
+   orphan/phantom set difference over two lists in `lib/storage-broker/src/audit.ts`, carrying the
+   concurrency contract and the refusal to print `orphan=<everything>` off a failed read. This audit
+   is that shape one level down: objects against rows instead of projects against records.
+
+## Components
+
+### 1. `reconcileDataPortal` — pure, in `lib/storage-broker/src/audit.ts`
+
+Third reconcile beside `reconcileStorage` and `reconcilePages`. It is D1 rows against R2 objects,
+which is that lib's domain, and the ui server imports the lib by source path so there is no dist
+rebuild in the loop.
+
+```ts
+interface DataPortalReconcile {
+  objects: number
+  rows: number
+  orphanObjects: string[]   // object keys with no manifest row
+  phantomRows: string[]     // manifest objectKeys with no object
+  oldestUningestedHrs: number | null   // null renders as `na`
+}
+
+function reconcileDataPortal(
+  objects: Array<{ key: string }>,
+  rows: Array<{ objectKey: string; ingested: number; uploadedAt: string }>,
+  nowMs: number,
+): DataPortalReconcile
+```
+
+`nowMs` is injected so the age arithmetic is deterministic under test rather than clock-dependent.
+
+`oldestUningestedHrs` is computed from rows with `ingested === 0` only, and is `null` when there are
+none. **An `uploadedAt` that will not parse is excluded from the minimum rather than allowed to
+produce `NaN`** — that is the `snapshotAgeMs=NaN` lesson from the availability loop, where a truthy
+but non-ISO value slipped past a `??` guard and logged `NaN`. If exclusion leaves no usable row, the
+result is `null`.
+
+### 2. `parseManifestRows` — pure, exported from the route module
+
+`cf.d1Query` shells `npx wrangler d1 execute --json` and is typed `unknown`. This function parses
+that envelope into the row shape above and **throws on anything it does not recognise**.
+
+Throwing rather than returning `[]` is the load-bearing decision. Zero rows against a populated
+bucket reads as "every object is an orphan" — a fabricated defect count off a failed read, which is
+precisely what `reconcilePages` refuses to print. A throw lands in the d1-degraded branch, where the
+line says the read failed instead of asserting a number it does not have.
+
+### 3. `runDataPortalAudit(root)` — the caller, in `ui/server/routes/storage-broker.ts`
+
+Sits with `runStorageAudit` and `runPagesAudit` at the foot of the same file.
+
+**Discovery.** Walks `data/accounts/*/data-portal.json`. A config that is absent, unparseable, or
+parses but names no `portalDbName` and `bucketName` is **silent** — the `hasBookingSite` idiom. An
+account with no portal is not a fault and must not produce a line.
+
+**It returns before minting a house token when no account has a portal.** Most installs have no data
+portal and no house Cloudflare credential; minting unconditionally would make this loop report an
+error forever on every one of them.
+
+**Reads.** One `makeHouseCfExec(root)` for the whole run, then per account:
+
+- rows — `d1Query(portalDbName, 'SELECT objectKey, ingested, uploadedAt FROM manifest')`
+- objects — `r2ObjectList(bucketName)`
+
+The two halves are caught **separately**, and the line is emitted unconditionally afterwards. This is
+`.tasks/backlog/1351` made structural: the log is not downstream of either read, so neither read can
+skip it. An audit that a data-source failure silences is the same blind spot one layer up, in a task
+that exists because failures emit nothing.
+
+**Degradation.** Each line carries exactly what is still measurable:
+
+| Failure | Line still carries |
+|---|---|
+| R2 read fails | `rows=N oldestUningestedHrs=N`, remainder `na`, `degraded=r2` |
+| D1 read fails | `objects=N`, remainder `na`, `degraded=d1` |
+| Both fail | all `na`, `degraded=d1+r2` |
+
+`oldestUningestedHrs` surviving an R2 failure is not incidental: the backlog age is computable from
+the manifest alone, and it is the one of the three conditions that a half-broken audit can still
+report.
+
+**Line format.** Counts only:
+
+```
+[data-portal-audit] account=<id> objects=N rows=N orphanObjects=N phantomRows=N oldestUningestedHrs=N
+```
+
+Healthy is `orphanObjects=0 phantomRows=0` with a bounded age. **Any non-zero orphan or phantom count
+is a defect, not noise.**
+
+`reconcilePages` reports orphan and phantom *names*, on the grounds that a count says something is
+wrong and a name says what to fix. This audit deliberately does not, and the pure function returning
+the keys does not change that: the keys here are `<ownerId>/<filename>`, real people's filenames,
+and the journal is not where they belong. The task's success criterion specifies counts, and the
+audit owning its own ownership discipline is called out in the brief as load-bearing precisely
+because the in-process path holds the account-wide credential and bypasses `objectGate` entirely.
+
+### 4. Arming, in `ui/server/index.ts`
+
+```ts
+registerLoop({
+  name: 'data-portal-audit',
+  intervalMs: 300_000,
+  firstRunDelayMs: 50_000,
+  run: runDataPortalAuditSafe,
+})
+```
+
+Same interval as the storage and pages audits. The first run is offset past storage (20s), pages
+(35s) and google-account (45s) so a boot does not fire four token mints in the same instant.
+
+**Degradation is not an error.** A degraded run returns a note (`portals=N degraded=M`) and the
+registry records `ok`, because the audit did the job it could and said so. A genuine throw — no
+house credential where a portal exists, an unreachable registry — is logged and **rethrown**, the
+`runStorageAuditSafe` shape, so /activity shows a fresh Last run against a stale Last OK.
+
+## Testing
+
+**Unit, `lib/storage-broker/src/__tests__/audit.test.ts`** — an object with no row counts one
+`orphanObjects`; a row with no object counts one `phantomRows`; a matched pair counts neither;
+`oldestUningestedHrs` comes from the oldest `ingested=0` row; `ingested=1` rows are excluded from it;
+`na` when there are no uningested rows; an unparseable `uploadedAt` does not yield `NaN`.
+
+**Envelope, same run** — `parseManifestRows` on a well-formed wrangler payload, and a throw on a
+malformed one. The throw case is what stops a parse failure reading as an empty manifest.
+
+**Route, new `ui/server/__tests__/data-portal-audit.test.ts`** — modelled on
+`storage-broker-object-routes.test.ts`: `// @vitest-environment node`, `vi.mock` the neo4j store,
+partially mock the lib via `importOriginal`, stub `makeHouseCfExec` with a fake carrying
+`r2ObjectList` and `d1Query`. Cases: two accounts, one with a portal, yields one line; a D1 throw
+still emits a line carrying `degraded=d1`; an R2 throw emits `degraded=r2` with `rows` still counted;
+a malformed D1 payload degrades rather than reporting zero rows; a config naming no portal is silent;
+no portal anywhere mints no token.
+
+**Mutation check** — reverting the reconcile to compare in one direction only must fail the opposite
+direction's test.
+
+## Scope boundaries
+
+**In scope:** the reconcile and its tests, the caller, the arming, the route-level tests, and
+`.docs/data-portal.md` § Observability, which currently states the audit is not built.
+
+**Out of scope:** the device-side ingestion sweep (unbuilt, agent-side); any change to the portal
+template or the gate (1689, landed); escalation beyond a log line — `.tasks/backlog/1351` records
+that a non-zero count escalating nowhere is its own problem, and that is lane-wide, not this task's.

package/payload/platform/docs/superpowers/specs/2026-07-20-task-1820-cpu-triage-admin-tools-design.md

@@ -0,0 +1,97 @@
+# Task 1820 — `cpu-triage` as a two-call admin diagnostic
+
+Design agreed 2026-07-20. Implements `.tasks/1820-expose-cpu-triage-to-the-admin-assistant-as-a-two-call-device-diagnostic.md`.
+
+## Problem restated
+
+`platform/scripts/cpu-triage.sh` answers "which cores are busy and what is on them" and is reachable only by an engineer with SSH. The measurement needs at least one full 120 s reconcile cycle to be trustworthy, and no admin tool anywhere exceeds a 12 s envelope. So the diagnostic has to be split across two calls: one that starts a run and returns immediately, one that reads a result.
+
+## Shape
+
+Two tools in the admin plugin, one new wrapper script, one JSON-contract fix in the measurement script.
+
+### `cpu-triage.sh` — JSON contract only
+
+The `--control` comparison is currently computed only inside the text-report branch (lines 275–285); the JSON branch ends at line 251 and emits no control field. Passing `--control` together with `--json` is therefore inert today, which would silently drop the single most valuable input to this measurement.
+
+Add to the JSON object:
+
+```json
+"control": { "brand": "<name>", "meanPct": <number>, "found": true }
+```
+
+`null` when `--control` was not passed. `{ "brand": "<name>", "found": false }` when the named brand is not among the running services. The measurement design, sampling loop, exit codes and text report are untouched.
+
+### `cpu-triage-run.sh` — new wrapper
+
+```
+cpu-triage-run.sh --run-id ID --out-dir DIR --log LOGFILE -- <cpu-triage args…>
+```
+
+Runs `cpu-triage.sh --json <args>`, streaming stdout to `DIR/cpu-triage-<ID>.json`. **After** that file is closed it writes `DIR/cpu-triage-<ID>.status`, then appends one completion line to `LOGFILE`.
+
+The ordering is load-bearing. The result file exists from the first byte of output, so its existence is not a readiness signal. `.status` is the only readiness signal, and it is written strictly after the result. A reader that keys on the `.json` existing reports a half-written run as complete — which is the exact failure the task's mutation check exists to catch.
+
+The wrapper exists rather than an inline `sh -c` string because the ordering guarantee should be reviewable in a file with exit-code gates, per the deterministic doctrine.
+
+### Per-run files, under `<configDir>/logs/`
+
+| File | Written by | Carries |
+|---|---|---|
+| `cpu-triage-<id>.meta.json` | `cpu-triage-start` | runId, startedAt, expectedCompleteAt, windowSec, args |
+| `cpu-triage-<id>.json` | wrapper, streaming | the report — **not** a readiness signal |
+| `cpu-triage-<id>.status` | wrapper, last | `exit=<n>` on line one, stderr on the lines below |
+
+Start-time metadata and end-time status cannot share one file, which is why the layout is three files rather than the task's "sibling status marker" phrasing implies.
+
+Run ids are timestamp-prefixed — `20260720T143000-a1b2` — so "most recent" resolves by filename sort rather than a stat walk.
+
+### `cpu-triage-start { windowSec?, intervalSec?, thresholdPct?, control? }`
+
+Returns `{ runId, startedAt, expectedCompleteAt, windowSec, resultPath }`. Spawns the wrapper detached with `stdio: "ignore"` and `unref()`, so the MCP process does not hold the run open. Logs `[cpu-triage] op=start runId=<id> windowSec=<n>`. Prunes to the 20 most recent runs.
+
+`windowSec` is validated at the tool boundary with a 120 s floor. The script already refuses a shorter window with exit 2, but detached that refusal arrives one full window late — the refusal has to be synchronous.
+
+A missing `cpu-triage.sh` returns a typed refusal naming the resolved path with `isError: true`, never an empty report.
+
+### `cpu-triage-read { runId? }`
+
+Defaults to the most recent run. Decides state from `.status` alone.
+
+| On disk | Returns |
+|---|---|
+| no runs at all | typed "no runs started" — not a failure |
+| `.meta.json` present, no `.status` | `running`, carrying `expectedCompleteAt` so an overdue run is visible |
+| `.status` exit 0 or 1 | `complete` plus the parsed report. Exit 1 is a finding, not a failure. |
+| `.status` any other code | `failed` plus exitCode and stderr |
+| `.status` exit 0 or 1, JSON unparseable | `failed`, reason naming the parse error |
+
+The exit-code contract survives to the agent: 0 clean, 1 sustained hot core, 2 precondition failure.
+
+## Registration
+
+`platform/plugins/admin/mcp/src/index.ts` (via `eagerTool`) and `platform/plugins/admin/PLUGIN.md` move in one commit. Both tools take `publicAllowlist: false`, `adminAllowlist: true` — the operator's "why is this box busy" question should not cost a permission round trip during an incident. `ADMIN_CORE_TOOLS` is a dead name and needs no edit. `npm run gen:canonical-tools` regenerates the byte-match artifact the build gate compares against.
+
+`platform/scripts/` is copied wholesale into the installer payload by `packages/create-maxy-code/scripts/bundle.js`, and both scripts are mode 755 in git, so they land executable by the same path `logs-read.sh` already uses.
+
+## Observability
+
+`[cpu-triage] op=start runId=<id> windowSec=<n>` from the start tool, `[cpu-triage] op=complete runId=<id> exit=<code> sustainedHotCores=<n>` from the wrapper. The completion line is written after the result file exists — a post-condition, not an intent. A run that starts and dies is visible as a start with no completion rather than as silence.
+
+## Tests
+
+Ephemeral, discarded after the sprint.
+
+1. `.meta.json` present, no `.status` → `running`
+2. `.status` `exit=0` plus valid JSON → `complete`, `perCore` populated, `sustainedHotCores` present
+3. `.status` `exit=2` → `failed`, exitCode 2, distinct from exit 0
+4. script absent → typed refusal naming the path
+5. wrapper against a stub exiting 2 → `.status` written and the log line appended, both after the result file
+
+Tests 1 and 2 are a pair. An implementation that always reports `complete` passes 2 and fails 1; neither alone pins the behaviour.
+
+## Scope boundaries
+
+**In scope:** the two tools, their registration, the wrapper, the JSON control field, result persistence and retention, shipping to devices.
+
+**Out of scope:** the measurement design, which is fixed and documented in the script header; the activity view (1810); escalation or alerting on a finding (1351); any remediation a report implies; remote targeting — this measures the box it runs on.

package/payload/platform/lib/storage-broker/dist/__tests__/audit.test.js

@@ -70,4 +70,48 @@ const clean = {
     strict_1.default.deepEqual(r.orphans, []);
     strict_1.default.deepEqual(r.phantoms, []);
 });
+// --- Data portal reconcile (Task 1704) ------------------------------------
+const row = (objectKey, ingested, uploadedAt) => ({
+    objectKey,
+    ingested,
+    uploadedAt,
+});
+(0, node_test_1.default)("reconcileDataPortal counts an object with no row as an orphan", () => {
+    const r = (0, audit_js_1.reconcileDataPortal)([{ key: "alice/a.pdf" }], [], 0);
+    strict_1.default.deepEqual(r.orphanObjects, ["alice/a.pdf"]);
+    strict_1.default.deepEqual(r.phantomRows, []);
+    strict_1.default.equal(r.distinctObjects, 1);
+    strict_1.default.equal(r.distinctRows, 0);
+});
+(0, node_test_1.default)("reconcileDataPortal counts a row with no object as a phantom", () => {
+    const r = (0, audit_js_1.reconcileDataPortal)([], [row("alice/a.pdf", 1, "2026-07-20T00:00:00Z")], 0);
+    strict_1.default.deepEqual(r.phantomRows, ["alice/a.pdf"]);
+    strict_1.default.deepEqual(r.orphanObjects, []);
+});
+(0, node_test_1.default)("reconcileDataPortal counts a matched pair as neither", () => {
+    const r = (0, audit_js_1.reconcileDataPortal)([{ key: "alice/a.pdf" }], [row("alice/a.pdf", 1, "2026-07-20T00:00:00Z")], 0);
+    strict_1.default.deepEqual(r.orphanObjects, []);
+    strict_1.default.deepEqual(r.phantomRows, []);
+});
+(0, node_test_1.default)("oldestUningestedHrs comes from the oldest ingested=0 row", () => {
+    const now = Date.parse("2026-07-20T12:00:00Z");
+    const r = (0, audit_js_1.reconcileDataPortal)([], [row("a", 0, "2026-07-20T09:00:00Z"), row("b", 0, "2026-07-20T11:00:00Z")], now);
+    strict_1.default.equal(r.oldestUningestedHrs, 3);
+});
+(0, node_test_1.default)("an ingested row is excluded from the backlog age", () => {
+    const now = Date.parse("2026-07-20T12:00:00Z");
+    const r = (0, audit_js_1.reconcileDataPortal)([], [row("a", 1, "2026-07-01T00:00:00Z"), row("b", 0, "2026-07-20T11:00:00Z")], now);
+    strict_1.default.equal(r.oldestUningestedHrs, 1);
+});
+(0, node_test_1.default)("oldestUningestedHrs is null when nothing is uningested", () => {
+    const r = (0, audit_js_1.reconcileDataPortal)([], [row("a", 1, "2026-07-20T00:00:00Z")], 0);
+    strict_1.default.equal(r.oldestUningestedHrs, null);
+});
+(0, node_test_1.default)("an unparseable uploadedAt never yields NaN", () => {
+    const now = Date.parse("2026-07-20T12:00:00Z");
+    const only = (0, audit_js_1.reconcileDataPortal)([], [row("a", 0, "not-a-date")], now);
+    strict_1.default.equal(only.oldestUningestedHrs, null);
+    const mixed = (0, audit_js_1.reconcileDataPortal)([], [row("a", 0, "not-a-date"), row("b", 0, "2026-07-20T11:00:00Z")], now);
+    strict_1.default.equal(mixed.oldestUningestedHrs, 1);
+});
 //# sourceMappingURL=audit.test.js.map

package/payload/platform/lib/storage-broker/dist/__tests__/audit.test.js.map

@@ -1 +1 @@
-{"version":3,"file":"audit.test.js","sourceRoot":"","sources":["../../src/__tests__/audit.test.ts"],"names":[],"mappings":";;;;;AAAA,0DAA6B;AAC7B,gEAAwC;AACxC,0CAA+E;AAE/E,MAAM,KAAK,GAAc;IACvB,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,WAAW,CAAC;IAC/B,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,EAAE;IACpB,eAAe,EAAE,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;IAC/D,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;QAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,2CAA2C,EAAE;KAC1F;CACF,CAAC;AAEF,IAAA,mBAAI,EAAC,mCAAmC,EAAE,KAAK,IAAI,EAAE;IACnD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC,KAAK,CAAC,CAAC;IACjD,gBAAM,CAAC,SAAS,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;AAC/B,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,mCAAmC,EAAE,KAAK,IAAI,EAAE;IACnD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,kBAAkB,CAAC;KACpD,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,iBAAiB,IAAI,CAAC,CAAC,MAAM,KAAK,kBAAkB,CAAC,CAAC,CAAC;AACjG,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,sCAAsC,EAAE,KAAK,IAAI,EAAE;IACtD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;YAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,+BAA+B,EAAE;SAC9E;KACF,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CACP,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,oBAAoB,IAAI,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAClF,CAAC;AACJ,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,gCAAgC,EAAE,KAAK,IAAI,EAAE;IAChD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;YAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,oCAAoC,EAAE;SACnF;KACF,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CACP,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,oBAAoB,IAAI,CAAC,CAAC,MAAM,KAAK,4BAA4B,CAAC,CACjG,CAAC;AACJ,CAAC,CAAC,CAAC;AAEH,4EAA4E;AAE5E,IAAA,mBAAI,EAAC,kEAAkE,EAAE,KAAK,IAAI,EAAE;IAClF,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,gBAAgB,EAAE,eAAe,CAAC;QAC3D,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,CAAC;KAClD,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC,gBAAgB,CAAC,CAAC,CAAC;IAChD,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;IACjC,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;IAC5B,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC;AAChC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,mFAAmF,EAAE,KAAK,IAAI,EAAE;IACnG,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,CAAC;QACzC,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,EAAE,cAAc,CAAC;KAClE,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;IAChC,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,cAAc,CAAC,CAAC,CAAC;AACjD,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,uEAAuE,EAAE,KAAK,IAAI,EAAE;IACvF,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,GAAG,EAAE,GAAG,CAAC;QAClC,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,GAAG,EAAE,GAAG,CAAC;KAC3C,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;IAChC,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;AACnC,CAAC,CAAC,CAAC"}
+{"version":3,"file":"audit.test.js","sourceRoot":"","sources":["../../src/__tests__/audit.test.ts"],"names":[],"mappings":";;;;;AAAA,0DAA6B;AAC7B,gEAAwC;AACxC,0CAAoG;AAEpG,MAAM,KAAK,GAAc;IACvB,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,WAAW,CAAC;IAC/B,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,EAAE;IACpB,eAAe,EAAE,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;IAC/D,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;QAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,2CAA2C,EAAE;KAC1F;CACF,CAAC;AAEF,IAAA,mBAAI,EAAC,mCAAmC,EAAE,KAAK,IAAI,EAAE;IACnD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC,KAAK,CAAC,CAAC;IACjD,gBAAM,CAAC,SAAS,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;AAC/B,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,mCAAmC,EAAE,KAAK,IAAI,EAAE;IACnD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,IAAI,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,kBAAkB,CAAC;KACpD,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,iBAAiB,IAAI,CAAC,CAAC,MAAM,KAAK,kBAAkB,CAAC,CAAC,CAAC;AACjG,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,sCAAsC,EAAE,KAAK,IAAI,EAAE;IACtD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;YAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,+BAA+B,EAAE;SAC9E;KACF,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CACP,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,oBAAoB,IAAI,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAClF,CAAC;AACJ,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,gCAAgC,EAAE,KAAK,IAAI,EAAE;IAChD,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,IAAA,2BAAgB,EAAC;QACxC,GAAG,KAAK;QACR,mBAAmB,EAAE,KAAK,IAAI,EAAE,CAAC;YAC/B,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,oCAAoC,EAAE;SACnF;KACF,CAAC,CAAC;IACH,gBAAM,CAAC,EAAE,CACP,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,oBAAoB,IAAI,CAAC,CAAC,MAAM,KAAK,4BAA4B,CAAC,CACjG,CAAC;AACJ,CAAC,CAAC,CAAC;AAEH,4EAA4E;AAE5E,IAAA,mBAAI,EAAC,kEAAkE,EAAE,KAAK,IAAI,EAAE;IAClF,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,gBAAgB,EAAE,eAAe,CAAC;QAC3D,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,CAAC;KAClD,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,CAAC,gBAAgB,CAAC,CAAC,CAAC;IAChD,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;IACjC,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;IAC5B,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC;AAChC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,mFAAmF,EAAE,KAAK,IAAI,EAAE;IACnG,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,CAAC;QACzC,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,eAAe,EAAE,cAAc,CAAC;KAClE,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;IAChC,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,cAAc,CAAC,CAAC,CAAC;AACjD,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,uEAAuE,EAAE,KAAK,IAAI,EAAE;IACvF,MAAM,CAAC,GAAG,MAAM,IAAA,yBAAc,EAAC;QAC7B,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,GAAG,EAAE,GAAG,CAAC;QAClC,kBAAkB,EAAE,KAAK,IAAI,EAAE,CAAC,CAAC,GAAG,EAAE,GAAG,CAAC;KAC3C,CAAC,CAAC;IACH,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;IAChC,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;AACnC,CAAC,CAAC,CAAC;AAEH,6EAA6E;AAE7E,MAAM,GAAG,GAAG,CAAC,SAAiB,EAAE,QAAgB,EAAE,UAAkB,EAAE,EAAE,CAAC,CAAC;IACxE,SAAS;IACT,QAAQ;IACR,UAAU;CACX,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,+DAA+D,EAAE,GAAG,EAAE;IACzE,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAAC,CAAC,EAAE,GAAG,EAAE,aAAa,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC;IAC/D,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,EAAE,CAAC,aAAa,CAAC,CAAC,CAAC;IACnD,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;IACpC,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,eAAe,EAAE,CAAC,CAAC,CAAC;IACnC,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC;AAClC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,8DAA8D,EAAE,GAAG,EAAE;IACxE,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAAC,EAAE,EAAE,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACtF,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,aAAa,CAAC,CAAC,CAAC;IACjD,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,EAAE,EAAE,CAAC,CAAC;AACxC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,sDAAsD,EAAE,GAAG,EAAE;IAChE,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAC3B,CAAC,EAAE,GAAG,EAAE,aAAa,EAAE,CAAC,EACxB,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAC/C,CAAC,CACF,CAAC;IACF,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,EAAE,EAAE,CAAC,CAAC;IACtC,gBAAM,CAAC,SAAS,CAAC,CAAC,CAAC,WAAW,EAAE,EAAE,CAAC,CAAC;AACtC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,0DAA0D,EAAE,GAAG,EAAE;IACpE,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;IAC/C,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAC3B,EAAE,EACF,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,EAAE,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAC1E,GAAG,CACJ,CAAC;IACF,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,mBAAmB,EAAE,CAAC,CAAC,CAAC;AACzC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,kDAAkD,EAAE,GAAG,EAAE;IAC5D,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;IAC/C,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAC3B,EAAE,EACF,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,EAAE,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAC1E,GAAG,CACJ,CAAC;IACF,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,mBAAmB,EAAE,CAAC,CAAC,CAAC;AACzC,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,wDAAwD,EAAE,GAAG,EAAE;IAClE,MAAM,CAAC,GAAG,IAAA,8BAAmB,EAAC,EAAE,EAAE,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC5E,gBAAM,CAAC,KAAK,CAAC,CAAC,CAAC,mBAAmB,EAAE,IAAI,CAAC,CAAC;AAC5C,CAAC,CAAC,CAAC;AAEH,IAAA,mBAAI,EAAC,4CAA4C,EAAE,GAAG,EAAE;IACtD,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;IAC/C,MAAM,IAAI,GAAG,IAAA,8BAAmB,EAAC,EAAE,EAAE,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,YAAY,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;IACvE,gBAAM,CAAC,KAAK,CAAC,IAAI,CAAC,mBAAmB,EAAE,IAAI,CAAC,CAAC;IAC7C,MAAM,KAAK,GAAG,IAAA,8BAAmB,EAC/B,EAAE,EACF,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,YAAY,CAAC,EAAE,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,sBAAsB,CAAC,CAAC,EAChE,GAAG,CACJ,CAAC;IACF,gBAAM,CAAC,KAAK,CAAC,KAAK,CAAC,mBAAmB,EAAE,CAAC,CAAC,CAAC;AAC7C,CAAC,CAAC,CAAC"}

package/payload/platform/lib/storage-broker/dist/audit.d.ts

@@ -3,9 +3,28 @@ export interface Stray {
     kind: StrayKind;
     detail: string;
 }
+/** CONCURRENCY CONTRACT — read before implementing these callbacks.
+ *
+ *  reconcileStorage invokes all five through ONE Promise.all. They overlap,
+ *  including the two registeredNames calls, which run at the same time as each
+ *  other. So no callback may share, with any other callback or with a second
+ *  invocation of itself, a resource that admits one operation at a time.
+ *
+ *  The resource that has actually caused this is a Neo4j session: it rejects a
+ *  query issued while one of its own is in flight. A caller that opened one
+ *  session in the enclosing scope and used it in registeredNames made the
+ *  second read reject, which left the standing storage audit reporting nothing
+ *  for 802 consecutive ticks on maxy-code and 295 on sitedesk-code (Task 1813).
+ *  Open the session inside the callback, one per invocation, and close it
+ *  there; driver sessions come from a pool, so this is cheap.
+ *
+ *  The contract is stated here rather than on reconcileStorage because this
+ *  interface is what a caller writing an implementation is looking at, and the
+ *  natural thing to write is the thing that breaks. */
 export interface AuditDeps {
     cfD1(): Promise<string[]>;
     cfR2(): Promise<string[]>;
+    /** Called twice, concurrently, once per kind. See the contract above. */
     registeredNames(kind: "d1" | "r2"): Promise<string[]>;
     accountSecretsFiles(): Promise<Array<{
         accountId: string;
@@ -16,6 +35,13 @@ export interface AuditDeps {
 export declare function reconcileStorage(deps: AuditDeps): Promise<{
     strays: Stray[];
 }>;
+/** Same concurrency contract as AuditDeps above: reconcilePages invokes both
+ *  callbacks through one Promise.all, so neither may share a single-operation
+ *  resource with the other.
+ *
+ *  That registeredProjects happens to issue exactly one query today is not the
+ *  property that makes a shared session safe here, and relying on it would put
+ *  the 1813 defect one added read away. */
 export interface PagesAuditDeps {
     cfProjects(): Promise<string[]>;
     registeredProjects(): Promise<string[]>;
@@ -27,4 +53,34 @@ export interface PagesReconcile {
     phantoms: string[];
 }
 export declare function reconcilePages(deps: PagesAuditDeps): Promise<PagesReconcile>;
+export interface DataPortalObject {
+    key: string;
+}
+export interface DataPortalRow {
+    objectKey: string;
+    ingested: number;
+    uploadedAt: string;
+}
+export interface DataPortalReconcile {
+    /** DISTINCT keys, which is not the same quantity as the length of the list
+     *  the read returned. The caller logs the lengths — "how many did I see" —
+     *  and these are "how many are distinct". They agree on a healthy store
+     *  (manifest.objectKey is UNIQUE and R2 keys are unique by construction), and
+     *  giving them one name would conflate two facts that diverge exactly when
+     *  the store is unhealthy, which is the case this audit is for. */
+    distinctObjects: number;
+    distinctRows: number;
+    orphanObjects: string[];
+    phantomRows: string[];
+    /** Null when nothing is uningested, or when no uningested row carries a
+     *  parseable timestamp. Rendered `na` by the caller. */
+    oldestUningestedHrs: number | null;
+}
+/** Pure over two lists and an injected clock, so the age arithmetic is
+ *  deterministic under test rather than dependent on when the suite runs.
+ *
+ *  Unlike its two siblings this takes lists rather than callbacks: both sides
+ *  are read per account by a caller that must catch each independently, so
+ *  there is no fan-out here to carry a concurrency contract. */
+export declare function reconcileDataPortal(objects: DataPortalObject[], rows: DataPortalRow[], nowMs: number): DataPortalReconcile;
 //# sourceMappingURL=audit.d.ts.map

package/payload/platform/lib/storage-broker/dist/audit.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"audit.d.ts","sourceRoot":"","sources":["../src/audit.ts"],"names":[],"mappings":"AAMA,MAAM,MAAM,SAAS,GAAG,iBAAiB,GAAG,iBAAiB,GAAG,oBAAoB,CAAC;AAErF,MAAM,WAAW,KAAK;IACpB,IAAI,EAAE,SAAS,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;CAChB;AAMD,MAAM,WAAW,SAAS;IACxB,IAAI,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1B,IAAI,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1B,eAAe,CAAC,IAAI,EAAE,IAAI,GAAG,IAAI,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IACtD,mBAAmB,IAAI,OAAO,CAAC,KAAK,CAAC;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC,CAAC;CAC9F;AAED,wBAAsB,gBAAgB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC;IAAE,MAAM,EAAE,KAAK,EAAE,CAAA;CAAE,CAAC,CA8BpF;AAmBD,MAAM,WAAW,cAAc;IAC7B,UAAU,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAChC,kBAAkB,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;CACzC;AAED,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB;AAED,wBAAsB,cAAc,CAAC,IAAI,EAAE,cAAc,GAAG,OAAO,CAAC,cAAc,CAAC,CAclF"}
+{"version":3,"file":"audit.d.ts","sourceRoot":"","sources":["../src/audit.ts"],"names":[],"mappings":"AAMA,MAAM,MAAM,SAAS,GAAG,iBAAiB,GAAG,iBAAiB,GAAG,oBAAoB,CAAC;AAErF,MAAM,WAAW,KAAK;IACpB,IAAI,EAAE,SAAS,CAAC;IAChB,MAAM,EAAE,MAAM,CAAC;CAChB;AAMD;;;;;;;;;;;;;;;;;uDAiBuD;AACvD,MAAM,WAAW,SAAS;IACxB,IAAI,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1B,IAAI,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1B,yEAAyE;IACzE,eAAe,CAAC,IAAI,EAAE,IAAI,GAAG,IAAI,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IACtD,mBAAmB,IAAI,OAAO,CAAC,KAAK,CAAC;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC,CAAC;CAC9F;AAED,wBAAsB,gBAAgB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC;IAAE,MAAM,EAAE,KAAK,EAAE,CAAA;CAAE,CAAC,CA8BpF;AAmBD;;;;;;2CAM2C;AAC3C,MAAM,WAAW,cAAc;IAC7B,UAAU,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;IAChC,kBAAkB,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;CACzC;AAED,MAAM,WAAW,cAAc;IAC7B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACpB;AAED,wBAAsB,cAAc,CAAC,IAAI,EAAE,cAAc,GAAG,OAAO,CAAC,cAAc,CAAC,CAclF;AAoBD,MAAM,WAAW,gBAAgB;IAC/B,GAAG,EAAE,MAAM,CAAC;CACb;AAED,MAAM,WAAW,aAAa;IAC5B,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,WAAW,mBAAmB;IAClC;;;;;uEAKmE;IACnE,eAAe,EAAE,MAAM,CAAC;IACxB,YAAY,EAAE,MAAM,CAAC;IACrB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB;4DACwD;IACxD,mBAAmB,EAAE,MAAM,GAAG,IAAI,CAAC;CACpC;AAED;;;;;gEAKgE;AAChE,wBAAgB,mBAAmB,CACjC,OAAO,EAAE,gBAAgB,EAAE,EAC3B,IAAI,EAAE,aAAa,EAAE,EACrB,KAAK,EAAE,MAAM,GACZ,mBAAmB,CA6BrB"}

package/payload/platform/lib/storage-broker/dist/audit.js

@@ -7,6 +7,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.reconcileStorage = reconcileStorage;
 exports.reconcilePages = reconcilePages;
+exports.reconcileDataPortal = reconcileDataPortal;
 // Keys that must never remain in a sub-account's cloudflare.env after
 // remediation. Each is an account-wide capability.
 const ACCOUNT_WIDE_KEYS = ["CLOUDFLARE_API_TOKEN", "CF_PAGES_D1_TOKEN", "CF_PAGES_TOKEN"];
@@ -54,4 +55,37 @@ async function reconcilePages(deps) {
         phantoms: [...regSet].filter((name) => !cfSet.has(name)),
     };
 }
+/** Pure over two lists and an injected clock, so the age arithmetic is
+ *  deterministic under test rather than dependent on when the suite runs.
+ *
+ *  Unlike its two siblings this takes lists rather than callbacks: both sides
+ *  are read per account by a caller that must catch each independently, so
+ *  there is no fan-out here to carry a concurrency contract. */
+function reconcileDataPortal(objects, rows, nowMs) {
+    const objectKeys = new Set(objects.map((o) => o.key));
+    const rowKeys = new Set(rows.map((r) => r.objectKey));
+    // Date.parse of a truthy non-ISO value is NaN, which slips past a `??` guard
+    // and renders as `oldestUningestedHrs=NaN` — the defect the availability
+    // loop's snapshotAgeMs already had to fix. Unparseable rows are excluded from
+    // the minimum; if that leaves none, the answer is null, never NaN.
+    const uningestedMs = rows
+        .filter((r) => r.ingested === 0)
+        .map((r) => Date.parse(r.uploadedAt))
+        .filter((ms) => !Number.isNaN(ms));
+    // Reduced, never `Math.min(...uningestedMs)`. A spread passes one argument per
+    // element and blows the call stack — measured: fine at 125k, RangeError at
+    // 130k. One element per uningested row, and an ingestion sweep that has
+    // stalled accumulates those without bound. A stalled sweep is one of the three
+    // conditions this audit exists to reveal, so the arithmetic must not be what
+    // goes dark once the backlog it is watching grows large. The throw would also
+    // escape the caller's per-side catches and cost every later account its line.
+    const oldestMs = uningestedMs.reduce((min, ms) => (ms < min ? ms : min), Infinity);
+    return {
+        distinctObjects: objectKeys.size,
+        distinctRows: rowKeys.size,
+        orphanObjects: [...objectKeys].filter((k) => !rowKeys.has(k)),
+        phantomRows: [...rowKeys].filter((k) => !objectKeys.has(k)),
+        oldestUningestedHrs: uningestedMs.length === 0 ? null : (nowMs - oldestMs) / 3_600_000,
+    };
+}
 //# sourceMappingURL=audit.js.map

package/payload/platform/lib/storage-broker/dist/audit.js.map

@@ -1 +1 @@
-{"version":3,"file":"audit.js","sourceRoot":"","sources":["../src/audit.ts"],"names":[],"mappings":";AAAA,+EAA+E;AAC/E,+EAA+E;AAC/E,8EAA8E;AAC9E,gFAAgF;AAChF,0DAA0D;;AAoB1D,4CA8BC;AA+BD,wCAcC;AAtFD,sEAAsE;AACtE,mDAAmD;AACnD,MAAM,iBAAiB,GAAG,CAAC,sBAAsB,EAAE,mBAAmB,EAAE,gBAAgB,CAAC,CAAC;AASnF,KAAK,UAAU,gBAAgB,CAAC,IAAe;IACpD,MAAM,MAAM,GAAY,EAAE,CAAC;IAE3B,MAAM,CAAC,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,YAAY,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC;QACjE,IAAI,CAAC,IAAI,EAAE;QACX,IAAI,CAAC,IAAI,EAAE;QACX,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC;QAC1B,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC;QAC1B,IAAI,CAAC,mBAAmB,EAAE;KAC3B,CAAC,CAAC;IAEH,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAChC,KAAK,MAAM,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,iBAAiB,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IACD,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAChC,KAAK,MAAM,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,iBAAiB,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IAED,KAAK,MAAM,IAAI,IAAI,YAAY,EAAE,CAAC;QAChC,KAAK,MAAM,IAAI,IAAI,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;YAC7C,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,+BAA+B,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;YAC7D,IAAI,GAAG,IAAI,iBAAiB,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;gBAC3C,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,oBAAoB,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,IAAI,GAAG,EAAE,EAAE,CAAC,CAAC;YAClF,CAAC;QACH,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,CAAC;AACpB,CAAC;AA+BM,KAAK,UAAU,cAAc,CAAC,IAAoB;IACvD,4EAA4E;IAC5E,0EAA0E;IAC1E,8EAA8E;IAC9E,+CAA+C;IAC/C,MAAM,CAAC,EAAE,EAAE,UAAU,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,UAAU,EAAE,EAAE,IAAI,CAAC,kBAAkB,EAAE,CAAC,CAAC,CAAC;IAC3F,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,EAAE,CAAC,CAAC;IAC1B,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,CAAC;IACnC,OAAO;QACL,QAAQ,EAAE,KAAK,CAAC,IAAI;QACpB,UAAU,EAAE,MAAM,CAAC,IAAI;QACvB,OAAO,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACvD,QAAQ,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;KACzD,CAAC;AACJ,CAAC"}
+{"version":3,"file":"audit.js","sourceRoot":"","sources":["../src/audit.ts"],"names":[],"mappings":";AAAA,+EAA+E;AAC/E,+EAA+E;AAC/E,8EAA8E;AAC9E,gFAAgF;AAChF,0DAA0D;;AAuC1D,4CA8BC;AAsCD,wCAcC;AAoDD,kDAiCC;AArMD,sEAAsE;AACtE,mDAAmD;AACnD,MAAM,iBAAiB,GAAG,CAAC,sBAAsB,EAAE,mBAAmB,EAAE,gBAAgB,CAAC,CAAC;AA4BnF,KAAK,UAAU,gBAAgB,CAAC,IAAe;IACpD,MAAM,MAAM,GAAY,EAAE,CAAC;IAE3B,MAAM,CAAC,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,YAAY,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC;QACjE,IAAI,CAAC,IAAI,EAAE;QACX,IAAI,CAAC,IAAI,EAAE;QACX,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC;QAC1B,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC;QAC1B,IAAI,CAAC,mBAAmB,EAAE;KAC3B,CAAC,CAAC;IAEH,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAChC,KAAK,MAAM,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,iBAAiB,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IACD,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAChC,KAAK,MAAM,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,iBAAiB,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IAED,KAAK,MAAM,IAAI,IAAI,YAAY,EAAE,CAAC;QAChC,KAAK,MAAM,IAAI,IAAI,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;YAC7C,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,+BAA+B,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;YAC7D,IAAI,GAAG,IAAI,iBAAiB,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;gBAC3C,MAAM,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,oBAAoB,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,IAAI,GAAG,EAAE,EAAE,CAAC,CAAC;YAClF,CAAC;QACH,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,CAAC;AACpB,CAAC;AAsCM,KAAK,UAAU,cAAc,CAAC,IAAoB;IACvD,4EAA4E;IAC5E,0EAA0E;IAC1E,8EAA8E;IAC9E,+CAA+C;IAC/C,MAAM,CAAC,EAAE,EAAE,UAAU,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,UAAU,EAAE,EAAE,IAAI,CAAC,kBAAkB,EAAE,CAAC,CAAC,CAAC;IAC3F,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,EAAE,CAAC,CAAC;IAC1B,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,CAAC;IACnC,OAAO;QACL,QAAQ,EAAE,KAAK,CAAC,IAAI;QACpB,UAAU,EAAE,MAAM,CAAC,IAAI;QACvB,OAAO,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACvD,QAAQ,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;KACzD,CAAC;AACJ,CAAC;AA8CD;;;;;gEAKgE;AAChE,SAAgB,mBAAmB,CACjC,OAA2B,EAC3B,IAAqB,EACrB,KAAa;IAEb,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACtD,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC;IAEtD,6EAA6E;IAC7E,yEAAyE;IACzE,8EAA8E;IAC9E,mEAAmE;IACnE,MAAM,YAAY,GAAG,IAAI;SACtB,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,QAAQ,KAAK,CAAC,CAAC;SAC/B,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC;SACpC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC;IAErC,+EAA+E;IAC/E,2EAA2E;IAC3E,wEAAwE;IACxE,+EAA+E;IAC/E,6EAA6E;IAC7E,8EAA8E;IAC9E,8EAA8E;IAC9E,MAAM,QAAQ,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,QAAQ,CAAC,CAAC;IAEnF,OAAO;QACL,eAAe,EAAE,UAAU,CAAC,IAAI;QAChC,YAAY,EAAE,OAAO,CAAC,IAAI;QAC1B,aAAa,EAAE,CAAC,GAAG,UAAU,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QAC7D,WAAW,EAAE,CAAC,GAAG,OAAO,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QAC3D,mBAAmB,EAAE,YAAY,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,GAAG,QAAQ,CAAC,GAAG,SAAS;KACvF,CAAC;AACJ,CAAC"}

package/payload/platform/lib/storage-broker/src/__tests__/audit.test.ts

@@ -1,6 +1,6 @@
 import test from "node:test";
 import assert from "node:assert/strict";
-import { reconcileStorage, reconcilePages, type AuditDeps } from "../audit.js";
+import { reconcileStorage, reconcilePages, reconcileDataPortal, type AuditDeps } from "../audit.js";
 
 const clean: AuditDeps = {
   cfD1: async () => ["gls-leads"],
@@ -78,3 +78,72 @@ test("reconcilePages is clean when every live project has exactly one owner", as
   assert.deepEqual(r.orphans, []);
   assert.deepEqual(r.phantoms, []);
 });
+
+// --- Data portal reconcile (Task 1704) ------------------------------------
+
+const row = (objectKey: string, ingested: number, uploadedAt: string) => ({
+  objectKey,
+  ingested,
+  uploadedAt,
+});
+
+test("reconcileDataPortal counts an object with no row as an orphan", () => {
+  const r = reconcileDataPortal([{ key: "alice/a.pdf" }], [], 0);
+  assert.deepEqual(r.orphanObjects, ["alice/a.pdf"]);
+  assert.deepEqual(r.phantomRows, []);
+  assert.equal(r.distinctObjects, 1);
+  assert.equal(r.distinctRows, 0);
+});
+
+test("reconcileDataPortal counts a row with no object as a phantom", () => {
+  const r = reconcileDataPortal([], [row("alice/a.pdf", 1, "2026-07-20T00:00:00Z")], 0);
+  assert.deepEqual(r.phantomRows, ["alice/a.pdf"]);
+  assert.deepEqual(r.orphanObjects, []);
+});
+
+test("reconcileDataPortal counts a matched pair as neither", () => {
+  const r = reconcileDataPortal(
+    [{ key: "alice/a.pdf" }],
+    [row("alice/a.pdf", 1, "2026-07-20T00:00:00Z")],
+    0,
+  );
+  assert.deepEqual(r.orphanObjects, []);
+  assert.deepEqual(r.phantomRows, []);
+});
+
+test("oldestUningestedHrs comes from the oldest ingested=0 row", () => {
+  const now = Date.parse("2026-07-20T12:00:00Z");
+  const r = reconcileDataPortal(
+    [],
+    [row("a", 0, "2026-07-20T09:00:00Z"), row("b", 0, "2026-07-20T11:00:00Z")],
+    now,
+  );
+  assert.equal(r.oldestUningestedHrs, 3);
+});
+
+test("an ingested row is excluded from the backlog age", () => {
+  const now = Date.parse("2026-07-20T12:00:00Z");
+  const r = reconcileDataPortal(
+    [],
+    [row("a", 1, "2026-07-01T00:00:00Z"), row("b", 0, "2026-07-20T11:00:00Z")],
+    now,
+  );
+  assert.equal(r.oldestUningestedHrs, 1);
+});
+
+test("oldestUningestedHrs is null when nothing is uningested", () => {
+  const r = reconcileDataPortal([], [row("a", 1, "2026-07-20T00:00:00Z")], 0);
+  assert.equal(r.oldestUningestedHrs, null);
+});
+
+test("an unparseable uploadedAt never yields NaN", () => {
+  const now = Date.parse("2026-07-20T12:00:00Z");
+  const only = reconcileDataPortal([], [row("a", 0, "not-a-date")], now);
+  assert.equal(only.oldestUningestedHrs, null);
+  const mixed = reconcileDataPortal(
+    [],
+    [row("a", 0, "not-a-date"), row("b", 0, "2026-07-20T11:00:00Z")],
+    now,
+  );
+  assert.equal(mixed.oldestUningestedHrs, 1);
+});