@kontourai/lookout 0.5.2 → 0.6.0

package/README.md

@@ -167,7 +167,7 @@ warnings.
 | `kind` | When | Extra fields |
 | --- | --- | --- |
 | `unchanged-304` | A validator-backed conditional request returned `304`; zero body transfer; the prior snapshot is not re-persisted. | `snapshotRef` |
-| `unchanged-hash` | A full body was fetched and persisted, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. | `priorSnapshotRef`, `currentSnapshotRef` |
+| `unchanged-hash` | A full body was fetched, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. A byte-identical repeat of the prior capture is not persisted again (see below). | `priorSnapshotRef`, `currentSnapshotRef` |
 | `changed` | The fresh body differs from the prior (`changeBasis: "hash"`), **or** it is the first successful observation (`changeBasis: "initial"`, `priorSnapshotRef: null`). | `priorSnapshotRef` (nullable), `currentSnapshotRef`, `changeBasis` |
 | `error` | Any operational failure — contained so the runner never rejects. | `origin` (`forage` \| `lookout`), `error` |
 
@@ -179,6 +179,14 @@ snapshot is persisted as the new baseline. (The `unchanged-304` path is already
 resource-scoped by Forage's validators, so only the hash path needs this
 guard.)
 
+A fetch that repeats the prior capture exactly is not appended to snapshot
+history: same URL, status, body bytes, redirects, render state, and `etag` /
+`last-modified` validators, with only the fetch time differing. Its
+`unchanged-hash` result names the stored capture as both `priorSnapshotRef` and
+`currentSnapshotRef`, and `checkedAt` records the check. Other response headers
+of the repeat, such as `Date`, are not kept. A stable source therefore adds no
+records, however often it is checked.
+
 `error` results preserve provenance: `origin: "forage"` carries Forage's
 discriminated `FetchError` verbatim (its `kind`, and `status` when present);
 `origin: "lookout"` carries a typed `kind` — `prior-read`, `persistence`,
@@ -189,7 +197,8 @@ are portable logical refs from Forage's `buildSnapshotSourceRef` — never
 filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by
 default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
-custom filenames or retention. `resolveLookoutSnapshot()` replays one exact
+custom filenames or retention; `createLookoutSnapshotStore(root, {
+maxHistoryFiles })` passes Forage's per-source record ceiling through. `resolveLookoutSnapshot()` replays one exact
 reference through an injected store or Lookout snapshot root, authenticates its
 body and replay metadata, and never fetches. References emitted before Forage's
 replay-envelope digest remain resolvable with `integrity: "body-and-identity"`;
@@ -257,6 +266,15 @@ Hachure-evidence-shaped; a consumer or product lifts them into a Hachure
 `TrustBundle` with `@kontourai/surface`'s `TrustBundleBuilder` when it wants
 Surface projection — lookout itself authors nothing in the trust layer.
 
+An observation id is the SHA-256 of the record's canonical JSON: object keys
+and proposals in UTF-16 code-unit order, never the host locale's collation.
+New records are `version: 2`. Records written before this change are
+`version: 1`, whose digest used the writing host's locale collation. They
+still load on a host whose locale collates their keys the same way, and the
+next commit replaces a `version: 1` prior with a `version: 2` record. Under a
+different locale, a `version: 1` record with non-ASCII or mixed-case keys or
+values can read as `corrupt-state`.
+
 ### Verified proposal head witnesses
 
 The concrete filesystem observation store additionally exposes
@@ -360,6 +378,10 @@ const composition = createObserveExtractDiff({
       // Caller-owned continuity and durable storage.
       return saveObservation(observation);
     },
+    async lastExtractedSnapshotRef(source) {
+      // The newest non-null extractedSnapshotRef(observation) you recorded.
+      return loadLastExtractedSnapshotRef(source.id);
+    },
   },
 });
 
@@ -367,7 +389,14 @@ const result = await composition.observe(source);
 ```
 
 `unchanged-304` and `unchanged-hash` are recorded as `unchanged` and never call
-the extraction capability, so they make zero preparation and provider calls.
+the extraction capability, so they make zero preparation and provider calls,
+when the capture has the same URL and body hash as the recorder's
+`lastExtractedSnapshotRef`. That is the last snapshot an observation fully
+handled: the current snapshot of a `completed`, `partial`, or `unchanged`
+observation, as `extractedSnapshotRef(observation)` returns. Otherwise the
+capture was persisted but never extracted (for example, a provider failed on the
+check that first saw it), so it is extracted now against that baseline instead
+of being reported as unchanged forever.
 Changed observations retain source and snapshot references, Traverse's prepared
 artifact identity, the proposal set, and the current/prior observation
 identities returned by the recorder. `partial`, `provider-failure`, mixed
@@ -442,10 +471,12 @@ behavior.
 ## Development
 
 ```sh
-npm ci
+pnpm install
 npm run verify   # content-boundary + decisions + typecheck + test + pack sanity
 ```
 
+The pnpm version is pinned in `package.json` (`packageManager`). Dependency install scripts are blocked by default; the only packages allowed to run one are listed under `allowBuilds` in `pnpm-workspace.yaml`, pinned by version. Scripts are still run with `npm run …` — that only invokes `package.json` scripts and does not depend on which tool installed `node_modules`.
+
 Individual gates: `npm test`, `npm run typecheck`, `npm run check:pack`,
 `npm run check:decisions`, `npm run check:content-boundary`.
 

package/dist/src/canonical-json.d.ts

@@ -0,0 +1,8 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export declare function compareCodeUnits(left: string, right: string): number;
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export declare function canonicalJson(value: unknown): string;

package/dist/src/canonical-json.js

@@ -0,0 +1,36 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export function compareCodeUnits(left, right) {
+    return left < right ? -1 : left > right ? 1 : 0;
+}
+function encode(input, key) {
+    // Mirrors JSON.stringify apart from key order: toJSON is honoured (a Date
+    // becomes its ISO string); undefined, functions and symbols have no encoding
+    // (omitted from objects, null in arrays); array holes and non-finite numbers
+    // become null; bigint throws.
+    const value = input !== null && typeof input === "object" && typeof input.toJSON === "function"
+        ? input.toJSON(key)
+        : input;
+    if (value === null || typeof value !== "object")
+        return JSON.stringify(value);
+    // Array.from visits holes; Array.prototype.map would skip them and write `[,1]`.
+    if (Array.isArray(value))
+        return `[${Array.from(value, (item, index) => encode(item, String(index)) ?? "null").join(",")}]`;
+    const members = [];
+    for (const key of Object.keys(value).sort(compareCodeUnits)) {
+        const item = encode(value[key], key);
+        if (item !== undefined)
+            members.push(`${JSON.stringify(key)}:${item}`);
+    }
+    return `{${members.join(",")}}`;
+}
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export function canonicalJson(value) {
+    const text = encode(value, "");
+    if (text === undefined)
+        throw new TypeError("Value has no JSON encoding");
+    return text;
+}

package/dist/src/check-runner.js

@@ -66,6 +66,13 @@ export function createCheckRunner(options) {
                 }
                 return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(prior) };
             }
+            // A byte-identical repeat of the stored capture is not appended: its only
+            // new information is the check time, which the result carries. Both
+            // refs then name the stored capture, which stays replayable.
+            if (prior !== undefined && isRepeatCapture(prior, snapshot)) {
+                const priorSnapshotRef = buildSnapshotSourceRef(prior);
+                return { ...base, kind: "unchanged-hash", priorSnapshotRef, currentSnapshotRef: priorSnapshotRef };
+            }
             try {
                 await options.store.put(snapshot);
             }
@@ -104,6 +111,22 @@ export function createCheckRunner(options) {
     }
     return { check, checkAll };
 }
+// Response headers other than the validators a later conditional request
+// reuses (e.g. Date) are not compared: they differ on nearly every response.
+const REVALIDATION_HEADERS = ["etag", "last-modified"];
+/** Same resource, same body, and nothing a later check reads differs. */
+function isRepeatCapture(prior, current) {
+    const priorEncoding = bodyEncoding(prior);
+    return priorEncoding !== undefined &&
+        priorEncoding === bodyEncoding(current) &&
+        prior.sourceId === current.sourceId &&
+        prior.url === current.url &&
+        prior.status === current.status &&
+        prior.bodyHash === current.bodyHash &&
+        prior.rendered === current.rendered &&
+        isDeepStrictEqual(prior.redirects, current.redirects) &&
+        REVALIDATION_HEADERS.every((name) => prior.headers?.[name] === current.headers?.[name]);
+}
 function bodyEncoding(snapshot) {
     const descriptor = Object.getOwnPropertyDescriptor(snapshot, "body");
     if (descriptor === undefined || !("value" in descriptor))

package/dist/src/drift-emission.d.ts

@@ -1,6 +1,6 @@
 import type { LookoutSource } from "./registry.js";
 import { type ProposalDiffEvent, type ProposalSetDiff, type ProposalSetDiffInput, type ProposalSetFacts, type ProposalSetObservation } from "./proposal-diff.js";
-import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservationV1 } from "./observation-store.js";
+import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservation } from "./observation-store.js";
 import type { SnapshotStore } from "@kontourai/forage";
 export interface BaselineEstablishedFact {
     readonly kind: "baseline-established";
@@ -25,7 +25,7 @@ export interface DriftSuccess {
     readonly facts: readonly DriftFact[];
     /** The prior observation this drift was diffed against, or null on a first-ever (baseline) observation. */
     readonly priorObservationId: string | null;
-    readonly committedObservation: StoredProposalObservationV1;
+    readonly committedObservation: StoredProposalObservation;
     readonly warnings: readonly string[];
 }
 export type DriftErrorKind = "invalid-input" | "prior-state-error" | "diff-error" | "persistence-error" | "serialization-error" | "unexpected";

package/dist/src/drift-emission.js

@@ -1,14 +1,15 @@
 import { diffProposalSets } from "./proposal-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 import { admitProposalObservation } from "./observation-admission.js";
 function stableJson(value) {
     if (Array.isArray(value))
         return `[${value.map(stableJson).join(",")}]`;
     if (value && typeof value === "object")
-        return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
+        return `{${Object.entries(value).sort(([a], [b]) => compareCodeUnits(a, b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
     return JSON.stringify(value) ?? "undefined";
 }
 function normalizeDiff(value) {
-    const sorted = (items) => [...items].sort((a, b) => stableJson(a).localeCompare(stableJson(b)));
+    const sorted = (items) => [...items].sort((a, b) => compareCodeUnits(stableJson(a), stableJson(b)));
     return {
         events: sorted(value.events),
         facts: {

package/dist/src/index.d.ts

@@ -10,7 +10,7 @@ export { inMemorySourceStore } from "./source-store.js";
 export type { SourceStore } from "./source-store.js";
 export type { ExtractableLookoutSource, LookoutRegistryDocument, LookoutSource, LookoutSourceKind, RenderPolicy, StructuredFileFormat, StructuredFileLookoutSource, } from "./registry.js";
 export { createLookoutSnapshotStore, resolveLookoutSnapshot, } from "./snapshot-store.js";
-export type { ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
+export type { LookoutSnapshotStoreOptions, ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
 export { admitProposalObservation } from "./observation-admission.js";
 export type { AdmitProposalObservationInput, AdmittedProposalObservation, AdmittedSnapshotIdentity, ObservationAdmissionError, ObservationAdmissionErrorKind, ObservationAdmissionResult } from "./observation-admission.js";
 export { admitSourceCapture, admitSourceCheck } from "./source-admission.js";
@@ -23,12 +23,12 @@ export type { KeyedMultisetFacts, KeyedMultisetOptions, StructuralComparison, St
 export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js";
 export type { FieldChangedEvent, FieldChangeKind, NewEntityAppearedEvent, ProposalDiffEvent, ProposalEvidence, ProposalIdentity, ProposalOccurrencePair, ProposalSetDiff, ProposalSetDiffInput, ProposalSetFacts, ProposalSetObservation, ProvenanceChangeFact, } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
-export type { CreateObservationStoreOptions, HeadWitnessComparison, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalHeadWitnessV1, ProposalObservationRecordInput, StoredProposalObservationV1, VerifiedHeadLimits, VerifiedHeadObservationStore, VerifiedHeadRead } from "./observation-store.js";
+export type { CreateObservationStoreOptions, HeadWitnessComparison, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalHeadWitnessV1, ProposalObservationRecordInput, StoredProposalObservation, StoredProposalObservationV1, StoredProposalObservationV2, VerifiedHeadLimits, VerifiedHeadObservationStore, VerifiedHeadRead } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export type { BaselineEstablishedFact, CreateDriftEmitterOptions, DriftEmitter, DriftError, DriftErrorKind, DriftFact, DriftResult, DriftSuccess, EmitDriftInput } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
 export type { SchemaCoverageGap, SchemaCoverageResult } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export type { ObserveExtractAcquisition, ObserveExtractAttempt, ObserveExtractDiff, ObserveExtractDiffOptions, ObserveExtractError, ObserveExtractErrorKind, ObserveExtractExtraction, ObserveExtractExtractionInput, ObserveExtractObservation, ObserveExtractObservationIdentity, ObserveExtractOutcome, ObserveExtractProviderFailure, ObserveExtractRecorder, ObserveExtractResult, ObserveExtractSource, ObserveExtractSourceSnapshot, } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";
 export type { BuildSemanticReviewWorkInput, SemanticClaimTarget, SemanticObservationIdentity, SemanticReviewCandidate, SemanticReviewChange, SemanticReviewItem, SemanticReviewKind, SemanticReviewWork, } from "./semantic-review-work.js";

package/dist/src/index.js

@@ -12,5 +12,5 @@ export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js
 export { createObservationStore } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";

package/dist/src/observation-admission.d.ts

@@ -1,6 +1,6 @@
 import type { SnapshotStore } from "@kontourai/forage";
 import type { LookoutSource } from "./registry.js";
-import type { StoredProposalObservationV1, ObservationCheckAnchor } from "./observation-store.js";
+import type { StoredProposalObservation, ObservationCheckAnchor } from "./observation-store.js";
 import type { ProposalSetObservation } from "./proposal-diff.js";
 /** A resolved snapshot identity suitable for durable observation metadata. */
 export interface AdmittedSnapshotIdentity {
@@ -44,7 +44,7 @@ export interface AdmitProposalObservationInput {
     readonly current: ProposalSetObservation;
     readonly check: ObservationCheckAnchor;
     /** The already-selected observation-store record; admission never selects or persists continuity. */
-    readonly prior: StoredProposalObservationV1 | null;
+    readonly prior: StoredProposalObservation | null;
     /** Explicit capability: admission never chooses a snapshot root or storage implementation. */
     readonly snapshotStore: SnapshotStore;
 }

package/dist/src/observation-store.d.ts

@@ -21,6 +21,15 @@ export interface StoredProposalObservationV1 {
     readonly check: ObservationCheckAnchor;
     readonly proposals: readonly ExtractionProposal[];
 }
+/**
+ * Same fields as version 1. The observationId is computed over code-unit-ordered
+ * canonical JSON, so it does not depend on the host locale. New records are
+ * always version 2.
+ */
+export interface StoredProposalObservationV2 extends Omit<StoredProposalObservationV1, "version"> {
+    readonly version: 2;
+}
+export type StoredProposalObservation = StoredProposalObservationV1 | StoredProposalObservationV2;
 export type ObservationStoreErrorKind = "invalid-input" | "corrupt-state" | "continuity-conflict" | "io-error";
 export interface ObservationStoreError {
     readonly kind: ObservationStoreErrorKind;
@@ -36,8 +45,8 @@ export type ObservationStoreResult<T> = {
     readonly error: ObservationStoreError;
 };
 export interface ObservationStore {
-    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservationV1 | null>>;
-    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservationV1>>;
+    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservation | null>>;
+    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservation>>;
 }
 /** Finite caller-controlled ceilings for a verified proposal-head read. */
 export interface VerifiedHeadLimits {

package/dist/src/observation-store.js

@@ -3,19 +3,27 @@ import { constants } from "node:fs";
 import { lstat, mkdir, open, opendir, readFile, readdir, realpath, rename, rm, unlink } from "node:fs/promises";
 import path from "node:path";
 import { types } from "node:util";
+import { canonicalJson, compareCodeUnits } from "./canonical-json.js";
 const VERIFIED_HEAD_MAX = Object.freeze({ maxEntries: 8, maxIndexBytes: 8 * 1024, maxPointerBytes: 8 * 1024, maxRecordBytes: 1024 * 1024 });
 function sourceKey(sourceId) {
     return `${encodeURIComponent(sourceId).replaceAll("%", "_").slice(0, 48)}-${createHash("sha256").update(sourceId).digest("hex").slice(0, 16)}`;
 }
-function stable(value) {
+function canonical(value) { return `${canonicalJson(value)}\n`; }
+function digest(body) {
+    return createHash("sha256").update(body.version === 1 ? legacyCanonical(body) : canonical(body)).digest("hex");
+}
+// Version 1 digests only. Version 1 records were hashed with keys sorted by the
+// host locale and then rebuilt with Object.fromEntries, which moves integer-like
+// keys first. A version 1 record therefore verifies only under a locale that
+// collates its keys the way the writing host did. Do not use this for new records.
+function legacyStable(value) {
     if (Array.isArray(value))
-        return value.map(stable);
+        return value.map(legacyStable);
     if (value && typeof value === "object")
-        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, stable(item)]));
+        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, legacyStable(item)]));
     return value;
 }
-function canonical(value) { return `${JSON.stringify(stable(value))}\n`; }
-function digest(body) { return createHash("sha256").update(canonical(body)).digest("hex"); }
+function legacyCanonical(value) { return `${JSON.stringify(legacyStable(value))}\n`; }
 function resolveHeadLimits(limits) {
     try {
         if (limits !== undefined && (!limits || typeof limits !== "object" || types.isProxy(limits) || Array.isArray(limits)))
@@ -219,8 +227,8 @@ function buildRecord(input) {
         if (!check || check.currentSnapshotRef !== observation.snapshotRef || typeof check.checkedAt !== "string" || (check.resultKind !== "changed" && check.resultKind !== "unchanged-hash") || typeof input.recordedAt !== "string") {
             return { ok: false, error: { kind: "invalid-input", message: "Check anchor must match the current observation snapshot" } };
         }
-        const proposals = [...observation.proposals].sort((a, b) => canonical(a).localeCompare(canonical(b)));
-        const body = { version: 1, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
+        const proposals = [...observation.proposals].sort((a, b) => compareCodeUnits(canonical(a), canonical(b)));
+        const body = { version: 2, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
         try {
             return { ok: true, value: { ...body, observationId: digest(body) } };
         }
@@ -249,7 +257,7 @@ function validate(value, expectedSourceId) {
     if (!value || typeof value !== "object" || Array.isArray(value))
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation is not an object" } };
     const item = value;
-    if (item.version !== 1 || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
+    if ((item.version !== 1 && item.version !== 2) || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation schema or continuity is invalid" } };
     }
     const { observationId, ...body } = item;
@@ -416,7 +424,7 @@ export function createObservationStore(options = {}) {
                     return null;
                 } }).filter((item) => item !== null);
                 const preserve = new Set([record.observationId, expectedPriorId].filter((item) => item !== null));
-                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => b.recordedAt.localeCompare(a.recordedAt) || b.id.localeCompare(a.id));
+                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => compareCodeUnits(b.recordedAt, a.recordedAt) || compareCodeUnits(b.id, a.id));
                 await Promise.all(extras.map((item) => rm(path.join(dir, item.name), { force: true })));
             }
             catch (cause) {

package/dist/src/observe-extract-diff.d.ts

@@ -64,7 +64,24 @@ export interface ObserveExtractObservationIdentity {
  */
 export interface ObserveExtractRecorder {
     record(observation: ObserveExtractObservation): Promise<ObserveExtractObservationIdentity>;
+    /**
+     * The snapshot reference of this source's most recent recorded observation
+     * for which `extractedSnapshotRef(observation)` is not null, or `null` when
+     * there is none.
+     *
+     * An unchanged check is only skipped when its capture has the same URL and
+     * body hash as this snapshot. Otherwise a change that acquisition already
+     * persisted, but that was never extracted (for example because a provider
+     * failed), would be reported as unchanged forever.
+     */
+    lastExtractedSnapshotRef(source: ObserveExtractSource): Promise<string | null>;
 }
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export declare function extractedSnapshotRef(observation: ObserveExtractObservation): string | null;
 export interface ObserveExtractDiffOptions {
     readonly acquisition: ObserveExtractAcquisition;
     readonly extraction: ObserveExtractExtraction;

package/dist/src/observe-extract-diff.js

@@ -1,4 +1,14 @@
 import { validatePreparedArtifact } from "@kontourai/traverse";
+import { parseSnapshotSourceRef } from "@kontourai/forage/fetch";
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export function extractedSnapshotRef(observation) {
+    const handled = observation.outcome === "completed" || observation.outcome === "partial" || observation.outcome === "unchanged";
+    return handled && observation.sourceSnapshot !== null ? observation.sourceSnapshot.currentSnapshotRef : null;
+}
 export function createObserveExtractDiff(options) {
     return {
         async observe(source) {
@@ -19,10 +29,26 @@ export function createObserveExtractDiff(options) {
                 if (check.kind === "error") {
                     return record(options.recorder, baseObservation(source, check, "acquisition-error", null, null, null, null));
                 }
+                let sourceSnapshot = snapshotFor(check);
                 if (check.kind === "unchanged-304" || check.kind === "unchanged-hash") {
-                    return record(options.recorder, baseObservation(source, check, "unchanged", snapshotFor(check), null, null, null));
+                    // "Unchanged" compares with the latest stored capture, which may never
+                    // have been extracted. Skip extraction only when the capture matches
+                    // the last one that was; otherwise extract it against that baseline.
+                    let extracted;
+                    try {
+                        extracted = await options.recorder.lastExtractedSnapshotRef(observationSource(source));
+                    }
+                    catch (cause) {
+                        return { ok: false, error: error("recording-failed", "Observation recorder could not report the last extracted snapshot", cause) };
+                    }
+                    if (extracted !== null && (typeof extracted !== "string" || extracted === "")) {
+                        return { ok: false, error: error("dependency-contract", "Observation recorder returned an invalid last extracted snapshot reference") };
+                    }
+                    if (extracted !== null && sameCapture(extracted, sourceSnapshot.currentSnapshotRef)) {
+                        return record(options.recorder, baseObservation(source, check, "unchanged", sourceSnapshot, null, null, null));
+                    }
+                    sourceSnapshot = { priorSnapshotRef: extracted, currentSnapshotRef: sourceSnapshot.currentSnapshotRef };
                 }
-                const sourceSnapshot = snapshotFor(check);
                 let extraction;
                 try {
                     extraction = await options.extraction.extract({ source, snapshotRef: sourceSnapshot.currentSnapshotRef });
@@ -65,7 +91,7 @@ export function createObserveExtractDiff(options) {
 }
 function baseObservation(source, check, outcome, sourceSnapshot, preparedArtifact, proposalSet, attempt) {
     return {
-        source: { id: source.id, url: source.url, kind: source.kind },
+        source: observationSource(source),
         check,
         outcome,
         sourceSnapshot,
@@ -93,6 +119,18 @@ async function record(recorder, observation) {
         return { ok: false, error: error("recording-failed", "Observation recorder failed", cause), observation };
     }
 }
+function observationSource(source) {
+    return { id: source.id, url: source.url, kind: source.kind };
+}
+/** Two references name the same capture content: same source, resource URL, and body hash. */
+function sameCapture(left, right) {
+    if (left === right)
+        return true;
+    const a = parseSnapshotSourceRef(left);
+    const b = parseSnapshotSourceRef(right);
+    return a !== undefined && b !== undefined &&
+        a.sourceId === b.sourceId && a.url === b.url && a.bodyHash === b.bodyHash;
+}
 function snapshotFor(check) {
     if (check.kind === "unchanged-304")
         return { priorSnapshotRef: check.snapshotRef, currentSnapshotRef: check.snapshotRef };

package/dist/src/proposal-diff.js

@@ -1,5 +1,6 @@
 import { canonicalValueKey, } from "./canonical-value.js";
 import { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 function callbackError(label, cause) {
     return { kind: "callback-threw", message: `${label} callback threw`, cause };
 }
@@ -90,7 +91,7 @@ function semanticFieldOrder(items, fieldIdentity) {
             return encoded;
         groups.set(fieldKey.value, [...(groups.get(fieldKey.value) ?? []), { item, key: encoded.key, index }]);
     }
-    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => left.key.localeCompare(right.key) || left.index - right.index));
+    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => compareCodeUnits(left.key, right.key) || left.index - right.index));
     return { ok: true, value: ordered.map(({ item }) => item) };
 }
 export function diffProposalSets(input) {

package/dist/src/snapshot-store.d.ts

@@ -1,6 +1,14 @@
 import type { SnapshotStore } from "@kontourai/forage";
 import { type SnapshotSourceRefResolution } from "@kontourai/forage/fetch";
-export declare function createLookoutSnapshotStore(root?: string): SnapshotStore;
+export interface LookoutSnapshotStoreOptions {
+    /**
+     * Per-source record ceiling, passed to Forage's filesystem store (1 to
+     * 10,000; Forage's default is 10,000). It cannot change after a source's
+     * store directory is initialized.
+     */
+    readonly maxHistoryFiles?: number;
+}
+export declare function createLookoutSnapshotStore(root?: string, options?: LookoutSnapshotStoreOptions): SnapshotStore;
 export type ResolveLookoutSnapshotOptions = {
     store: SnapshotStore;
     root?: never;

package/dist/src/snapshot-store.js

@@ -1,8 +1,11 @@
 import path from "node:path";
 import { createFilesystemSnapshotStore } from "@kontourai/forage";
 import { resolveSnapshotSourceRef, } from "@kontourai/forage/fetch";
-export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots")) {
-    return createFilesystemSnapshotStore({ root });
+export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots"), options = {}) {
+    return createFilesystemSnapshotStore({
+        root,
+        ...(options.maxHistoryFiles === undefined ? {} : { maxHistoryFiles: options.maxHistoryFiles }),
+    });
 }
 /** Replay one Lookout-emitted durable reference without any network access. */
 export async function resolveLookoutSnapshot(reference, options = {}) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kontourai/lookout",
-  "version": "0.5.2",
+  "version": "0.6.0",
   "description": "A small source registry and drift-check runner built on Forage snapshots.",
   "license": "Apache-2.0",
   "type": "module",
@@ -54,9 +54,13 @@
     "@kontourai/flow-agents": "5.9.0",
     "@kontourai/survey": "3.0.0",
     "@types/node": "^26.1.2",
+    "ajv": "8.20.0",
+    "ajv-formats": "3.0.1",
+    "hachure": "0.15.0",
     "typescript": "^7.0.2"
   },
   "engines": {
     "node": ">=22"
-  }
+  },
+  "packageManager": "pnpm@11.25.0"
 }