humanish 0.123.0 → 0.124.0

package/CHANGELOG.md

@@ -8,6 +8,19 @@ The Unreleased section holds the full notes for the next version until it is tag
 
 ## Unreleased
 
+## 0.124.0: Small Observer files for long studies, faster verify, closing reports under zero data retention (2026-10-12)
+
+humanish 0.124.0 keeps a long study's Observer files small, checks screenshots about eight times
+faster, and asks an `openai-computer-use` participant for its closing report at a scheduled end
+under zero data retention. `observer/observer-data.json` is now `humanish.observer-data.v2`: the
+run and each participant without their trace items and screenshot entries, which the Observer
+page carries compressed. On a synthetic run of 24 participants working 8 hours, observer-data.json
+fell from 155 MiB to 0.33 MiB and index.html from 94 MiB to 16 MiB, so verify reads both. A
+reader of observer-data.json that needs a participant's items reads the run's traces. verify of a
+synthetic 8-hour run with 8,895 real-size screenshots took 51 s, down from 282 s.
+
+[Release notes](https://github.com/danielgwilson/humanish/releases/tag/v0.124.0)
+
 ## 0.123.0: Run bundle v2 writes each trace once, bundle size estimate (2026-10-12)
 
 humanish 0.123.0 writes each participant's trace once, outside run.json, so long studies stay

package/CONTEXT.md

@@ -36,7 +36,7 @@ bundle and output fields keep older spellings; the second table maps them.
 | topology            | Whether participants share one instance of the subject. `route: shared-world` gives one for all; every other route gives each participant its own (`per-lane-worlds`). `run.json` records it as `sharedWorld.topology`.                                                                                                                                                                                                                                                                                                               | `route` in `StudyConfig` (`src/study/types.ts`); `isSharedWorldComposition` in `src/study/routing.ts`                                                                                                                                                                                                                                                                                                       |
 | plane               | The one instance of the subject that a shared-world study's participants use together. A `provisioned` plane is a `clone` or `local-tree` subject that humanish serves in a sandbox, with a declared subject state. An `external-public` plane is a deployment already running at `subject.appUrl`, whose owner the study names in `subject.publicTarget.owner`. Each plane has at least two participants.                                                                                                                            | `SharedWorldPlane` in `src/study/plan-types.ts`; `src/routes/shared-world/`                                                                                                                                                                                                                                                                                                                                 |
 | stream              | One live or recorded view of a participant, such as a browser or a terminal, in `run.json` and the Observer.                                                                                                                                                                                                                                                                                                                                                                                                                          | `RunStream` in `src/run/streams.ts`                                                                                                                                                                                                                                                                                                                                                                         |
-| Observer            | The single-file review page built from `observer/observer-data.json` (`humanish.observer-data.v1`).                                                                                                                                                                                                                                                                                                                                                                                                                                   | `observer/` (the page), `src/observer/` (data, render, serve)                                                                                                                                                                                                                                                                                                                                               |
+| Observer            | The single-file review page built from `observer/observer-data.json` (`humanish.observer-data.v2`).                                                                                                                                                                                                                                                                                                                                                                                                                                   | `observer/` (the page), `src/observer/` (data, render, serve)                                                                                                                                                                                                                                                                                                                                               |
 | evidence            | Files a run keeps: screenshots, traces, transcripts, recordings. `verify` checks them and grades share-safety.                                                                                                                                                                                                                                                                                                                                                                                                                        | `src/evidence/`; `verifyRun` in `src/verify/verify.ts`                                                                                                                                                                                                                                                                                                                                                      |
 | analysis            | An independent model reading of a finished run's retained evidence, kept apart from participant feedback. Supported live runs request it automatically, and `humanish analyze` runs it on demand. It records findings and observations as `humanish.study-analysis.v1`.                                                                                                                                                                                                                                                               | `src/analysis/`; `runAutomaticAnalysis` in `src/analysis/automatic.ts`; `analyzeRun` in `src/analysis/service.ts`; `ANALYSIS_SCHEMA` in `src/analysis/types.ts`                                                                                                                                                                                                                                             |
 | finding             | A result of analysis, with citations to evidence in the run. Each finding groups the observations that support it.                                                                                                                                                                                                                                                                                                                                                                                                                    | `src/analysis/`                                                                                                                                                                                                                                                                                                                                                                                             |

package/dist/actors/computer-use/loop/trace.js

@@ -65,7 +65,12 @@ export function statusForCompletionReason(reason) {
 }
 export function loopResult(session, stop, debrief) {
     const { provider, settings, trace: recorder, usage } = session;
-    const { completionReason, reason, stopCause } = stop;
+    const { completionReason, stopCause } = stop;
+    // A participant whose session ended on the clock has its closing account as its reason, as one
+    // that ended the session itself has its last message; the stop cause keeps the scheduled end.
+    const reason = stopCause === "scheduled_end" && debrief?.report !== undefined
+        ? debrief.report.summary
+        : stop.reason;
     const completedAtMs = session.now();
     const status = statusForCompletionReason(completionReason);
     const ids = {};

package/dist/actors/computer-use/openai-conversation.d.ts

@@ -65,12 +65,6 @@ export declare class OpenAiConversation {
      * from the request being sent. False when none are asked for, so the rejection stands.
      */
     dropReasoningSummaries(): boolean;
-    /**
-     * Whether the server holds the whole session, so a closing report threaded on it reads every
-     * turn. An explicit-context conversation summarizes its oldest turns past its budget, configured
-     * or switched to at runtime.
-     */
-    get serverHoldsSession(): boolean;
     /** The trace's account of how the conversation was carried (ActorConversation). */
     record(): ActorConversation;
     /**

package/dist/actors/computer-use/openai-conversation.js

@@ -72,14 +72,6 @@ export class OpenAiConversation {
         this.reasoningSummary = undefined;
         return true;
     }
-    /**
-     * Whether the server holds the whole session, so a closing report threaded on it reads every
-     * turn. An explicit-context conversation summarizes its oldest turns past its budget, configured
-     * or switched to at runtime.
-     */
-    get serverHoldsSession() {
-        return this.mode === "threaded" && this.lastResponseId !== undefined;
-    }
     /** The trace's account of how the conversation was carried (ActorConversation). */
     record() {
         return this.evidence.snapshot(this.mode);

package/dist/actors/computer-use/openai-provider.js

@@ -509,14 +509,11 @@ export function createOpenAiResponsesProvider(options) {
         get conversation() {
             return conversation.record();
         },
-        // A retrospective report reads the session the server holds; see serverHoldsSession.
-        get debrief() {
-            return conversation.serverHoldsSession
-                ? (req, signal) => requestTurn(req, signal, "report")
-                : undefined;
-        },
-        // Offered in both modes. An explicit_context conversation carries its newest screens whole and
-        // earlier turns as text, which serves impressions; a closing report needs the whole session.
+        // Both are offered in both modes. A threaded request reads the session the server holds; an
+        // explicit_context one carries its newest screens whole and its oldest turns summarized past
+        // the budget, which the trace's conversation record counts. A participant that worked a long
+        // shift writes its closing account from what it carries, as a person writes one from memory.
+        debrief: (req, signal) => requestTurn(req, signal, "report"),
         requestImpressions: (req, signal) => requestTurn(req, signal, "impressions"),
     };
 }

package/dist/evidence/image.js

@@ -1,3 +1,4 @@
+import { crc32, inflateSync } from "node:zlib";
 import { PNG } from "pngjs";
 const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
 const PNG_IHDR_LENGTH = 13;
@@ -145,12 +146,90 @@ export function screenshotEvidenceError(relativePath, bytes) {
     if (chunks?.some((chunk) => chunk.type === "IEND" && chunk.end - chunk.start !== 12)) {
         return "PNG IEND chunk must be empty";
     }
+    return chunks === null || !pngStructureHolds(bytes, chunks)
+        ? "could not decode PNG evidence"
+        : null;
+}
+/** The bit depths PNG allows for each color type. */
+const PNG_BIT_DEPTHS = {
+    0: [1, 2, 4, 8, 16],
+    2: [8, 16],
+    3: [1, 2, 4, 8],
+    4: [8, 16],
+    6: [8, 16],
+};
+const PNG_CHANNELS = { 0: 1, 2: 3, 3: 1, 4: 2, 6: 4 };
+/** The fewest bytes the decoder reads from a chunk; a shorter one fails to decode. */
+const PNG_MIN_CHUNK_BYTES = { gAMA: 4 };
+/**
+ * Whether a PNG that passed pngDecodeRefusal decodes, read from its structure: the file ends at an
+ * empty IEND, every chunk's CRC matches, IHDR comes once with fields PNG defines, the image data is
+ * one run of IDAT chunks, and it inflates, with zlib's native code, to exactly the
+ * bytes its rows need with nothing after the stream, each row starting with a filter type from 0
+ * to 4. For an image without a palette that is all a decode establishes, at about an eighth of
+ * the time of pngjs, which unfilters every pixel in JavaScript; a run holds a screenshot for each
+ * turn. A palette image is also decoded, since each pixel must index a palette that comes before
+ * the image data; in any other image a palette chunk is only a suggestion, wherever it is.
+ */
+function pngStructureHolds(bytes, chunks) {
+    const last = chunks.at(-1);
+    if (last?.type !== "IEND" || last.end !== bytes.length)
+        return false;
+    const crcHolds = (chunk) => crc32(bytes.subarray(chunk.start + 4, chunk.end - 4)) === bytes.readUInt32BE(chunk.end - 4);
+    if (!chunks.every(crcHolds))
+        return false;
+    const types = chunks.map((chunk) => chunk.type);
+    const firstIdat = types.indexOf("IDAT");
+    const idats = chunks.filter((chunk) => chunk.type === "IDAT");
+    if (types.lastIndexOf("IHDR") !== 0 ||
+        firstIdat < 0 ||
+        types.lastIndexOf("IDAT") !== firstIdat + idats.length - 1)
+        return false;
+    const [width, height] = [bytes.readUInt32BE(16), bytes.readUInt32BE(20)];
+    const [depth, colorType, compression, filter] = [bytes[24], bytes[25], bytes[26], bytes[27]];
+    if (!PNG_BIT_DEPTHS[colorType]?.includes(depth) || compression !== 0 || filter !== 0)
+        return false;
+    // The decoder sizes a row in 32-bit arithmetic and refuses one wider; no screenshot is.
+    const rowBits = width * PNG_CHANNELS[colorType] * depth;
+    if (rowBits + 7 > 0x7fffffff)
+        return false;
+    const trnsBytes = colorType === 0 ? 2 : colorType === 2 ? 6 : 0;
+    const tooShort = (chunk) => chunk.end - chunk.start - 12 <
+        (chunk.type === "tRNS" ? trnsBytes : (PNG_MIN_CHUNK_BYTES[chunk.type] ?? 0));
+    if (chunks.some(tooShort))
+        return false;
+    const rowBytes = 1 + Math.ceil(rowBits / 8);
+    const data = Buffer.concat(idats.map((chunk) => bytes.subarray(chunk.start + 8, chunk.end - 4)));
+    let rows;
     try {
-        const decoded = PNG.sync.read(bytes, { checkCRC: true });
-        return pngDimensionsError(decoded.width, decoded.height);
+        // A stream that would inflate past the rows the image needs stops there, so a small file
+        // cannot claim gigabytes.
+        const options = {
+            maxOutputLength: rowBytes * height,
+            info: true,
+        };
+        const inflated = inflateSync(data, options);
+        if (inflated.engine.bytesWritten !== data.length)
+            return false;
+        rows = inflated.buffer;
     }
     catch {
-        return "could not decode PNG evidence";
+        return false;
+    }
+    if (rows.length !== rowBytes * height)
+        return false;
+    for (let row = 0; row < rows.length; row += rowBytes)
+        if (rows[row] > 4)
+            return false;
+    return colorType !== 3 || pngDecodes(bytes);
+}
+function pngDecodes(bytes) {
+    try {
+        PNG.sync.read(bytes, { checkCRC: true });
+        return true;
+    }
+    catch {
+        return false;
     }
 }
 export function assertScreenshotEvidence(relativePath, bytes) {

package/dist/feedback/export-bundle.js

@@ -5,7 +5,6 @@ import { lstat, mkdir, mkdtemp, readdir, realpath, rename, rm, rmdir } from "nod
 import path from "node:path";
 import { tmpdir } from "node:os";
 import { renderObserver } from "../observer/render.js";
-import { buildObserverData } from "../observer/data.js";
 import { inventorySandboxIds, publicSandboxView, scrubSandboxIdBytes } from "../run/sandbox-ids.js";
 import { splitStreamTraces } from "../run/stream-trace.js";
 import { redactScreenshot, redactText } from "../evidence/redaction.js";
@@ -441,10 +440,7 @@ async function buildDerivative(stage, source, runId, bundle, maxBytes) {
         note: "A redacted copy of the same study, not a new attempt. Original pixels are not present. Text and findings still require human review before sharing.",
         files: transformed.entries,
     };
-    const transformedBundle = await loadRunBundlePrepared(stage.physicalPath, stagePaths);
-    if (!transformedBundle)
-        throw new Error("Derivative bundle could not be loaded.");
-    await writeContainedOutputFile(stagePaths, "observer/observer-data.json", jsonBytes(buildObserverData(transformedBundle.bundle)));
+    // Rendering writes observer-data.json and index.html from the derivative's run.json.
     const rendered = await renderObserver(stage.physicalPath, runId, { open: false });
     if (!rendered.ok)
         throw new Error("Derivative Observer could not be rebuilt.");

package/dist/feedback/export.js

@@ -13,7 +13,7 @@ import { createHash } from "node:crypto";
 import path from "node:path";
 import { renderObserver } from "../observer/render.js";
 import { renderObserverHtml } from "../observer/artifact.js";
-import { buildObserverData } from "../observer/data.js";
+import { buildObserverData, OBSERVER_DATA_SCHEMA, observerDataFromSaved, } from "../observer/data.js";
 import { resolveRunPath } from "../run/locate.js";
 import { loadRunTraces, StreamTraceRefusedError } from "../run/stream-trace.js";
 import { verifyRun } from "../verify/verify.js";
@@ -25,6 +25,7 @@ import { cli } from "../cli/invocation.js";
 import { analysisSharingProblems } from "../analysis/sharing.js";
 import { EVIDENCE_LIMITS, validateAnalysisEvidence } from "../analysis/evidence.js";
 import { readBoundedFile } from "../run/evidence-files.js";
+import { isRecord } from "../run/type-guards.js";
 import { shellQuote } from "../substrates/shell.js";
 import { plural } from "../run/text.js";
 const EXPORT_SCHEMA = "humanish.export-result.v1";
@@ -80,7 +81,7 @@ export async function exportRun(cwdInput, runInput, options = {}, deps = {}) {
     const analysis = await loadAnalysis(runPaths);
     const data = await currentObserverData(runPaths, analysis, slot);
     if (data === undefined)
-        return exportFailure(cwd, runInput, "HUMANISH_EXPORT_VERIFY_FAILED", "Source evidence changed while preparing the analysis export.");
+        return exportFailure(cwd, runInput, "HUMANISH_EXPORT_VERIFY_FAILED", "The run's evidence changed, or a participant's trace could not be read, while preparing the export.");
     const omittedRecording = omitRecordings(data.value);
     if (omittedRecording)
         warnings.push(MEDIA_OMISSION);
@@ -177,24 +178,42 @@ async function readObserverSlot(cwd, runInput, runRoot, runId, verified, deps, w
 /**
  * The Observer data to export. A current report must travel with the current source projection,
  * even if an older saved HTML was never refreshed; old recording-only exports keep the saved slot.
- * Undefined when run.json no longer matches the analysis.
+ * A page saved since observer-data v2 holds no participant's items in its slot, and verify does not
+ * read the page's compressed copy, so its export is the current projection of the run and its
+ * traces. Undefined when run.json no longer matches the analysis, or a v2 page's run cannot be read.
  */
 async function currentObserverData(runPaths, analysis, slot) {
     const saved = JSON.parse(slot);
-    if (analysis.state !== "ready" || !analysis.analysis)
-        return { value: saved };
-    const source = await readBoundedFile(runPaths, "run.json", EVIDENCE_LIMITS.sourceBytes);
-    if (!source ||
-        createHash("sha256").update(source).digest("hex") !== analysis.analysis.sourceRunSha256)
+    const source = async () => readBoundedFile(runPaths, "run.json", EVIDENCE_LIMITS.sourceBytes);
+    if (analysis.state !== "ready" || !analysis.analysis) {
+        const schema = saved?.schema;
+        if (schema === "humanish.observer-data.v1")
+            return { value: observerDataFromSaved(saved) };
+        if (schema !== OBSERVER_DATA_SCHEMA)
+            return { value: saved };
+        const bundle = await sourceTraces(runPaths, await source());
+        return bundle === undefined ? undefined : { value: buildObserverData(bundle) };
+    }
+    const bytes = await source();
+    if (!bytes ||
+        createHash("sha256").update(bytes).digest("hex") !== analysis.analysis.sourceRunSha256)
+        return undefined;
+    const bundle = await sourceTraces(runPaths, bytes);
+    return bundle === undefined ? undefined : { value: buildObserverData(bundle) };
+}
+/** The run with its traces read, or undefined when run.json is missing or a trace is refused. */
+async function sourceTraces(runPaths, bytes) {
+    if (!bytes)
+        return undefined;
+    const stored = JSON.parse(bytes.toString("utf8"));
+    if (!isRecord(stored) || !Array.isArray(stored.streams))
         return undefined;
-    const stored = JSON.parse(source.toString("utf8"));
     // A trace that no longer matches run.json's index is a source that no longer matches.
-    const bundle = await loadRunTraces(runPaths, stored).catch((error) => {
+    return loadRunTraces(runPaths, stored).catch((error) => {
         if (error instanceof StreamTraceRefusedError)
             return undefined;
         throw error;
     });
-    return bundle === undefined ? undefined : { value: buildObserverData(bundle) };
 }
 /** Portable HTML keeps the lightweight evidence; media remains in the source run. */
 function omitRecordings(data) {

package/dist/observer/artifact.js

@@ -7,6 +7,7 @@ import path from "node:path";
 import { fileURLToPath } from "node:url";
 import { projectShareCheckedAnalysis, analysisSharingProblems } from "../analysis/sharing.js";
 import { withObserverEndings } from "./data.js";
+import { escapeJsonScript, observerDataElements } from "./page-data.js";
 const OBSERVER_DATA_PLACEHOLDER = "__HUMANISH_OBSERVER_DATA__";
 const OBSERVER_DATA_SLOT = `<script id="observer-data" type="application/json">${OBSERVER_DATA_PLACEHOLDER}</script>`;
 let cachedObserverArtifact = null;
@@ -166,7 +167,7 @@ function renderObserverAppHtml(data, snapshot, analysis, notes, assets) {
     return (snapshot
         ? artifact.replace("</head>", '<meta name="humanish-observer-mode" content="snapshot"></head>')
         : artifact)
-        .replace(OBSERVER_DATA_SLOT, () => `<script id="observer-data" type="application/json">${escapeJsonScript(data)}</script>`)
+        .replace(OBSERVER_DATA_SLOT, () => observerDataElements(data))
         .replace(/<script id="study-analysis" type="application\/json">[\s\S]*?<\/script>/, () => `<script id="study-analysis" type="application/json">${escapeJsonScript(analysis)}</script>`)
         .replace(/<script id="run-notes" type="application\/json">[\s\S]*?<\/script>/, () => `<script id="run-notes" type="application/json">${escapeJsonScript(notes)}</script>`)
         .replace("</body>", () => `${renderExportAssets(assets)}</body>`)
@@ -241,11 +242,3 @@ function escapeHtml(value) {
         }
     });
 }
-function escapeJsonScript(value) {
-    return JSON.stringify(value)
-        .replace(/</g, "\\u003c")
-        .replace(/>/g, "\\u003e")
-        .replace(/&/g, "\\u0026")
-        .replace(/\u2028/g, "\\u2028")
-        .replace(/\u2029/g, "\\u2029");
-}

package/dist/observer/data.d.ts

@@ -2,7 +2,7 @@ import { type ActorEnding } from "../actors/stop-cause.js";
 import type { RunBundle, RunCostSummary, RunEvent, RunSimulation } from "../run/bundle.js";
 import { type RunDisplay } from "../run/display.js";
 import type { RunStream, RunStreamKind } from "../run/streams.js";
-export declare const OBSERVER_DATA_SCHEMA = "humanish.observer-data.v1";
+export declare const OBSERVER_DATA_SCHEMA = "humanish.observer-data.v2";
 interface ObserverArtifactLink {
     label: string;
     href: string;
@@ -10,7 +10,7 @@ interface ObserverArtifactLink {
 }
 export interface ObserverData {
     schema: typeof OBSERVER_DATA_SCHEMA;
-    schemaVersion: 1;
+    schemaVersion: 2;
     generatedAt: string;
     /** Served-only liveness observation. Never a replacement for the recorded evidence verdict. */
     runtime?: {
@@ -123,6 +123,13 @@ interface ObserverLaneGroup {
  * carried its outcome.
  */
 export declare function buildObserverData(bundle: RunBundle, generatedAt?: string, statusRecord?: unknown): ObserverData;
+/**
+ * A saved Observer projection as this version reads it, unchecked as before: an observer-data.json
+ * written before v2 holds every participant's items. A v2 one holds none, so it is read only when
+ * no participant has any: showing participants with no steps would say they did nothing, and the
+ * items are in the run's traces. Null for a v2 projection that leaves items out.
+ */
+export declare function observerDataFromSaved(value: unknown): ObserverData | null;
 /** Refresh presentation from recorded actor evidence, including older exported snapshots. */
 export declare function withObserverEndings(data: ObserverData): ObserverData;
 export {};

package/dist/observer/data.js

@@ -3,7 +3,8 @@ import { isCommsReceivingEvidence, receivingAnalysisContext } from "../comms/rec
 import { formatParticipantOutcomes, formatRunTaskFunnel, participantOutcomeDetails, withCuaReviewProvenance, } from "../run/outcomes.js";
 import { cuaGoalSource, CUA_COMPLETION_NOTE } from "../actors/goal-source.js";
 import { bundleDisplayFacts, runDisplay } from "../run/display.js";
-export const OBSERVER_DATA_SCHEMA = "humanish.observer-data.v1";
+import { joinObserverData, shellNeedsItems } from "./participant-items.js";
+export const OBSERVER_DATA_SCHEMA = "humanish.observer-data.v2";
 /** Discard a forged or stale runtime grant before projecting persisted evidence. */
 export function recordedStreamEmbed(embed) {
     if (!embed)
@@ -80,7 +81,7 @@ export function buildObserverData(bundle, generatedAt = new Date().toISOString()
     const active = streams.filter((stream) => stream.status === "running" || stream.status === "preparing").length;
     return withObserverEndings({
         schema: OBSERVER_DATA_SCHEMA,
-        schemaVersion: 1,
+        schemaVersion: 2,
         generatedAt,
         run: {
             runId: bundle.runId,
@@ -141,6 +142,20 @@ export function buildObserverData(bundle, generatedAt = new Date().toISOString()
         },
     });
 }
+/**
+ * A saved Observer projection as this version reads it, unchecked as before: an observer-data.json
+ * written before v2 holds every participant's items. A v2 one holds none, so it is read only when
+ * no participant has any: showing participants with no steps would say they did nothing, and the
+ * items are in the run's traces. Null for a v2 projection that leaves items out.
+ */
+export function observerDataFromSaved(value) {
+    const saved = value;
+    if (saved.schema === "humanish.observer-data.v1")
+        return { ...saved, schema: OBSERVER_DATA_SCHEMA, schemaVersion: 2 };
+    return shellNeedsItems(saved)
+        ? null
+        : joinObserverData(saved, null);
+}
 /** Refresh presentation from recorded actor evidence, including older exported snapshots. */
 export function withObserverEndings(data) {
     const streams = (data.streams ?? []).map(({ ending: _previousEnding, ...stream }) => {

package/dist/observer/page-data.d.ts

@@ -0,0 +1,5 @@
+import type { ObserverData } from "./data.js";
+/** The page's data elements for `data`, in place of the built page's data slot. */
+export declare function observerDataElements(data: ObserverData): string;
+/** JSON that stays inert inside a script element: no `</script>` breakout, no HTML entities. */
+export declare function escapeJsonScript(value: unknown): string;

package/dist/observer/page-data.js

@@ -0,0 +1,21 @@
+// How an Observer page carries a run's data: observer-data.json's content as JSON, and every
+// participant's trace items gzip-compressed in base64, which a file:// page decodes without a
+// fetch. The items are most of a long run's bytes and compress about eight times.
+import { gzipSync } from "node:zlib";
+import { splitObserverData } from "./participant-items.js";
+/** The page's data elements for `data`, in place of the built page's data slot. */
+export function observerDataElements(data) {
+    const { shell, items } = splitObserverData(data);
+    const packed = gzipSync(JSON.stringify(items)).toString("base64");
+    return (`<script id="observer-data" type="application/json">${escapeJsonScript(shell)}</script>` +
+        `<script id="observer-items" type="application/octet-stream" data-encoding="gzip">${packed}</script>`);
+}
+/** JSON that stays inert inside a script element: no `</script>` breakout, no HTML entities. */
+export function escapeJsonScript(value) {
+    return JSON.stringify(value)
+        .replace(/</g, "\\u003c")
+        .replace(/>/g, "\\u003e")
+        .replace(/&/g, "\\u0026")
+        .replace(/\u2028/g, "\\u2028")
+        .replace(/\u2029/g, "\\u2029");
+}

package/dist/observer/participant-items.d.ts

@@ -0,0 +1,45 @@
+import type { ActorTraceItem } from "../actors/contract.js";
+import type { ObserverData, ObserverStream } from "./data.js";
+export declare const OBSERVER_ITEMS_SCHEMA = "humanish.observer-items.v1";
+/** A stream's actor and live actor, each without its items. */
+type ObserverShellStream = Omit<ObserverStream, "actor" | "liveActor"> & {
+    actor?: Omit<NonNullable<ObserverStream["actor"]>, "items">;
+    liveActor?: Omit<NonNullable<ObserverStream["liveActor"]>, "items">;
+    /** How many screenshot entries of `artifacts` travel with the items, when any do. */
+    screenshotArtifacts?: number;
+};
+/** observer-data.json: the Observer's data with no participant's trace items. */
+export type ObserverShell = Omit<ObserverData, "streams"> & {
+    streams: ObserverShellStream[];
+};
+/**
+ * Each participant's trace items by stream id, with the stream's artifact list when it names
+ * screenshots: one per turn, so it grows with the items.
+ */
+export interface ObserverItems {
+    schema: typeof OBSERVER_ITEMS_SCHEMA;
+    runId: string;
+    streams: Record<string, {
+        actor?: ActorTraceItem[];
+        liveActor?: ActorTraceItem[];
+        artifacts?: ObserverStream["artifacts"];
+    }>;
+}
+/** The data as observer-data.json and the page's items carry it. */
+export declare function splitObserverData(data: ObserverData): {
+    shell: ObserverShell;
+    items: ObserverItems;
+};
+/**
+ * Whether `items` are this run's and hold every participant's items that the data leaves out. The
+ * page reads nothing short of that: participants shown with no steps would say they did nothing.
+ */
+export declare function itemsCoverShell(shell: ObserverShell, items: ObserverItems): boolean;
+/** Whether the data leaves out items or screenshot entries that only an items document holds. */
+export declare function shellNeedsItems(shell: ObserverShell): boolean;
+/**
+ * The data with each participant's items back in place. A stream the items do not cover keeps an
+ * empty list, so the page shows the participant with no recorded steps.
+ */
+export declare function joinObserverData(shell: ObserverShell, items: ObserverItems | null): ObserverData;
+export {};

package/dist/observer/participant-items.js

@@ -0,0 +1,85 @@
+// The Observer's data in two parts: observer-data.json, the run and every participant without
+// their trace items, and the items themselves. A participant working for hours records thousands
+// of items, so the items are most of the bytes; apart, the saved observer-data.json stays small
+// enough to verify, and the page carries the items compressed. The Observer bundles this module,
+// so it imports types only (docs/decisions/0002-observer-is-one-self-contained-file.md).
+export const OBSERVER_ITEMS_SCHEMA = "humanish.observer-items.v1";
+/** The data as observer-data.json and the page's items carry it. */
+export function splitObserverData(data) {
+    // No prototype, so a stream id such as `__proto__` is an entry like any other.
+    const items = Object.create(null);
+    const streams = data.streams.map((stream) => {
+        const { actor, liveActor, ...rest } = stream;
+        const entry = {};
+        const shell = rest;
+        // Saved projections are read unchecked, and one written by an older version may have no list.
+        if (Array.isArray(stream.artifacts) &&
+            stream.artifacts.some((artifact) => artifact.kind === "screenshot")) {
+            entry.artifacts = stream.artifacts;
+            shell.artifacts = stream.artifacts.filter((artifact) => artifact.kind !== "screenshot");
+            shell.screenshotArtifacts = stream.artifacts.length - shell.artifacts.length;
+        }
+        if (actor !== undefined) {
+            const { items: actorItems, ...summary } = actor;
+            entry.actor = actorItems;
+            shell.actor = summary;
+        }
+        if (liveActor !== undefined) {
+            const { items: liveItems, ...summary } = liveActor;
+            entry.liveActor = liveItems;
+            shell.liveActor = summary;
+        }
+        if (Object.keys(entry).length > 0)
+            items[stream.id] = entry;
+        return shell;
+    });
+    return {
+        shell: { ...data, streams },
+        items: { schema: OBSERVER_ITEMS_SCHEMA, runId: data.run.runId, streams: items },
+    };
+}
+/**
+ * Whether `items` are this run's and hold every participant's items that the data leaves out. The
+ * page reads nothing short of that: participants shown with no steps would say they did nothing.
+ */
+export function itemsCoverShell(shell, items) {
+    if (items.runId !== shell.run.runId)
+        return false;
+    return shell.streams.every((stream) => {
+        const entry = Object.hasOwn(items.streams, stream.id) ? items.streams[stream.id] : undefined;
+        return ((stream.actor === undefined || Array.isArray(entry?.actor)) &&
+            (stream.liveActor === undefined || Array.isArray(entry?.liveActor)) &&
+            (stream.screenshotArtifacts === undefined ||
+                (Array.isArray(entry?.artifacts) &&
+                    entry.artifacts.filter((artifact) => artifact.kind === "screenshot").length ===
+                        stream.screenshotArtifacts)));
+    });
+}
+/** Whether the data leaves out items or screenshot entries that only an items document holds. */
+export function shellNeedsItems(shell) {
+    return shell.streams.some((stream) => stream.actor !== undefined ||
+        stream.liveActor !== undefined ||
+        stream.screenshotArtifacts !== undefined);
+}
+/**
+ * The data with each participant's items back in place. A stream the items do not cover keeps an
+ * empty list, so the page shows the participant with no recorded steps.
+ */
+export function joinObserverData(shell, items) {
+    const byStream = items !== null && items.runId === shell.run.runId ? items.streams : {};
+    return {
+        ...shell,
+        streams: shell.streams.map((stream) => {
+            const entry = Object.hasOwn(byStream, stream.id) ? byStream[stream.id] : undefined;
+            const { actor, liveActor, screenshotArtifacts: _moved, ...rest } = stream;
+            return {
+                ...rest,
+                artifacts: entry?.artifacts ?? stream.artifacts,
+                ...(actor === undefined ? {} : { actor: { ...actor, items: entry?.actor ?? [] } }),
+                ...(liveActor === undefined
+                    ? {}
+                    : { liveActor: { ...liveActor, items: entry?.liveActor ?? [] } }),
+            };
+        }),
+    };
+}

package/dist/observer/render.js

@@ -17,6 +17,7 @@ import { RUN_STATUS_FILE } from "../run/status.js";
 import { verifyRunPrepared } from "../verify/verify.js";
 import { renderObserverHtml } from "./artifact.js";
 import { buildObserverData } from "./data.js";
+import { splitObserverData } from "./participant-items.js";
 import { buildServeSecurityHeaders, hostAllowed, parsePublicOrigin } from "./http.js";
 import { listenOnLoopback } from "./listen.js";
 import { NOTES_PATH } from "../run/note-shape.js";
@@ -116,7 +117,7 @@ export async function renderObserver(cwdInput, runInput, options = {}) {
         verifiedAt: new Date().toISOString(),
         reasons: verified.shareSafety.reasons.map((reason) => reason.code),
     };
-    await writeContainedOutputFile(preparedRunPaths, path.join("observer", "observer-data.json"), `${JSON.stringify(observerData, null, 2)}\n`, "utf8");
+    await writeContainedOutputFile(preparedRunPaths, path.join("observer", "observer-data.json"), `${JSON.stringify(splitObserverData(observerData).shell, null, 2)}\n`, "utf8");
     await writeContainedOutputFile(preparedRunPaths, path.join("observer", "study-analysis.json"), `${JSON.stringify(analysis, null, 2)}\n`, "utf8");
     await writeContainedOutputFile(preparedRunPaths, path.join("observer", "index.html"), renderObserverHtml(observerData, { analysis, notes: await readRunNotes(preparedRunPaths) }), "utf8");
     await validatePreparedRunArtifactPaths(preparedRunPaths);

package/dist/observer/run-routes.js

@@ -16,7 +16,8 @@ import { MAX_NOTE_FILE_BYTES, readRunNotes, RUN_NOTES_DIR, } from "../run/note-f
 import { analysisCostOf, runCost, runCostLabel } from "../run/run-cost.js";
 import { RUN_STATUS_FILE, RUN_STATUS_STALE_MS, isRunStatusRecord, } from "../run/status.js";
 import { renderObserverHtml } from "./artifact.js";
-import { buildObserverData, recordedStreamEmbed, withObserverEndings, } from "./data.js";
+import { splitObserverData } from "./participant-items.js";
+import { buildObserverData, observerDataFromSaved, recordedStreamEmbed, withObserverEndings, } from "./data.js";
 import { buildArtifactSecurityHeaders } from "./http.js";
 import { assertPinnedDirectory, openContainedFile, pinDirectChildDirectory, readContainedFile, relativeToRoot, sha256OfOpenedFile, } from "./pinned-files.js";
 /** The run's artifact paths, only when they are still the pinned directory being served. */
@@ -116,13 +117,19 @@ export async function serveRunPath(runRoot, relativePath, response, runtimeStrea
         }), "text/html; charset=utf-8");
         return;
     }
-    if (cleanedRelativePath === "observer/observer-data.json") {
+    // observer-data.json is the run without its participants' items, as the run saves it. The page
+    // polls observer-feed.json, the same data with the items, so one projection serves each poll.
+    if (cleanedRelativePath === "observer/observer-data.json" ||
+        cleanedRelativePath === "observer/observer-feed.json") {
         const observerData = await observerDataOrReason(runRoot, runtimeStreamUrls);
         if ("unavailable" in observerData) {
             writeResponse(response, 404, `Observer data not found: ${observerData.unavailable}`, "text/plain; charset=utf-8");
             return;
         }
-        writeResponse(response, 200, JSON.stringify(observerData, null, 2), "application/json; charset=utf-8");
+        const { shell, items } = splitObserverData(observerData);
+        writeResponse(response, 200, cleanedRelativePath === "observer/observer-data.json"
+            ? JSON.stringify(shell, null, 2)
+            : JSON.stringify({ data: shell, items }), "application/json; charset=utf-8");
         return;
     }
     if (cleanedRelativePath === "observer/study-analysis.json") {
@@ -214,7 +221,10 @@ async function savedObserverData(runRoot, record, runtimeStreamUrls) {
         const observerBytes = await readContainedFile(runRoot, path.join(runRoot.physicalPath, "observer", "observer-data.json"));
         if (!observerBytes)
             throw new Error("observer-data.json unavailable");
-        return withRuntimeStreamUrls(withLocalRunStatus(record, withObserverEndings(JSON.parse(observerBytes.toString("utf8")))), runtimeStreamUrls);
+        const saved = observerDataFromSaved(JSON.parse(observerBytes.toString("utf8")));
+        if (saved === null)
+            throw new Error("observer-data.json holds no participant's items");
+        return withRuntimeStreamUrls(withLocalRunStatus(record, withObserverEndings(saved)), runtimeStreamUrls);
     }
     catch {
         return { unavailable: "The run has written no run.json or Observer data yet." };