@kontourai/lookout 0.5.2 → 0.7.0

package/README.md

@@ -167,7 +167,7 @@ warnings.
 | `kind` | When | Extra fields |
 | --- | --- | --- |
 | `unchanged-304` | A validator-backed conditional request returned `304`; zero body transfer; the prior snapshot is not re-persisted. | `snapshotRef` |
-| `unchanged-hash` | A full body was fetched and persisted, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. | `priorSnapshotRef`, `currentSnapshotRef` |
+| `unchanged-hash` | A full body was fetched, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. A byte-identical repeat of the prior capture is not persisted again (see below). | `priorSnapshotRef`, `currentSnapshotRef` |
 | `changed` | The fresh body differs from the prior (`changeBasis: "hash"`), **or** it is the first successful observation (`changeBasis: "initial"`, `priorSnapshotRef: null`). | `priorSnapshotRef` (nullable), `currentSnapshotRef`, `changeBasis` |
 | `error` | Any operational failure — contained so the runner never rejects. | `origin` (`forage` \| `lookout`), `error` |
 
@@ -179,17 +179,55 @@ snapshot is persisted as the new baseline. (The `unchanged-304` path is already
 resource-scoped by Forage's validators, so only the hash path needs this
 guard.)
 
+A fetch that repeats the prior capture exactly is not appended to snapshot
+history: same URL, status, body bytes, text decoding, redirects, render state,
+and `etag` / `last-modified` validators, with only the fetch time differing. Its
+`unchanged-hash` result names the stored capture as both `priorSnapshotRef` and
+`currentSnapshotRef`, and `checkedAt` records the check. Other response headers
+of the repeat, such as `Date`, are not kept. A stable source therefore adds no
+records, however often it is checked. The text decoding is the encoding the
+declared `Content-Type` charset resolves to, so the same bytes under another
+charset (other text for an extractor) are a new capture, while `utf8` and
+`utf-8` are the same decoding.
+
+Changed captures still grow history. Pass `retention` to bound it with Forage's
+`prune`: after each stored capture, the runner keeps the newest `keepLast`
+snapshots, the latest, the prior capture its result names, and every reference
+`cited(sourceId)` returns. Cite what your observations, review rounds, and
+exported receipts still reference. If `cited` throws or returns anything but
+snapshot references, nothing is pruned and the result carries a warning.
+`cited` is read just before each prune, outside the store's lock, so record a
+citation before handing its reference out.
+
+```ts
+const runner = createCheckRunner({
+  store,
+  retention: { keepLast: 20, cited: (sourceId) => loadCitedSnapshotRefs(sourceId) },
+});
+```
+
+**Upgrading to 0.7 (Forage 1.0).** New text captures hash their exact bytes.
+A text page whose bytes are not plain UTF-8 (a non-UTF-8 charset, a byte-order
+mark, or invalid UTF-8) therefore reports `changed` once, on its first check
+after the upgrade, and is extracted again. Drift comes from proposal diffs, so
+a page whose extracted content is identical emits nothing. Plain UTF-8 pages
+keep their stored capture.
+
 `error` results preserve provenance: `origin: "forage"` carries Forage's
 discriminated `FetchError` verbatim (its `kind`, and `status` when present);
 `origin: "lookout"` carries a typed `kind` — `prior-read`, `persistence`,
-`dependency-contract`, or `unexpected`.
+`history-full`, `dependency-contract`, or `unexpected`. `history-full` means the
+source holds Forage's `maxHistoryFiles` records; with `retention` the runner
+prunes once and retries before reporting it.
 
 Snapshot references (`snapshotRef` / `priorSnapshotRef` / `currentSnapshotRef`)
 are portable logical refs from Forage's `buildSnapshotSourceRef` — never
 filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by
 default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
-custom filenames or retention. `resolveLookoutSnapshot()` replays one exact
+custom filenames or storage format; `createLookoutSnapshotStore(root, {
+maxHistoryFiles })` passes Forage's per-source record ceiling through and
+returns a store with Forage's `prune`. `resolveLookoutSnapshot()` replays one exact
 reference through an injected store or Lookout snapshot root, authenticates its
 body and replay metadata, and never fetches. References emitted before Forage's
 replay-envelope digest remain resolvable with `integrity: "body-and-identity"`;
@@ -257,6 +295,15 @@ Hachure-evidence-shaped; a consumer or product lifts them into a Hachure
 `TrustBundle` with `@kontourai/surface`'s `TrustBundleBuilder` when it wants
 Surface projection — lookout itself authors nothing in the trust layer.
 
+An observation id is the SHA-256 of the record's canonical JSON: object keys
+and proposals in UTF-16 code-unit order, never the host locale's collation.
+New records are `version: 2`. Records written before this change are
+`version: 1`, whose digest used the writing host's locale collation. They
+still load on a host whose locale collates their keys the same way, and the
+next commit replaces a `version: 1` prior with a `version: 2` record. Under a
+different locale, a `version: 1` record with non-ASCII or mixed-case keys or
+values can read as `corrupt-state`.
+
 ### Verified proposal head witnesses
 
 The concrete filesystem observation store additionally exposes
@@ -348,6 +395,8 @@ import { createObserveExtractDiff } from "@kontourai/lookout";
 
 const composition = createObserveExtractDiff({
   acquisition: { check: runner.check },
+  // The store acquisition persists to; used to compare captures by reference.
+  snapshots: store,
   extraction: {
     async extract({ source, snapshotRef }) {
       // Resolve `snapshotRef`, prepare content, and invoke a caller-selected
@@ -360,6 +409,10 @@ const composition = createObserveExtractDiff({
       // Caller-owned continuity and durable storage.
       return saveObservation(observation);
     },
+    async lastExtractedSnapshotRef(source) {
+      // The newest non-null extractedSnapshotRef(observation) you recorded.
+      return loadLastExtractedSnapshotRef(source.id);
+    },
   },
 });
 
@@ -367,7 +420,16 @@ const result = await composition.observe(source);
 ```
 
 `unchanged-304` and `unchanged-hash` are recorded as `unchanged` and never call
-the extraction capability, so they make zero preparation and provider calls.
+the extraction capability, so they make zero preparation and provider calls,
+when the capture has the same URL, body hash, and text decoding as the
+recorder's `lastExtractedSnapshotRef`. Both references are resolved through
+`snapshots`; one that does not resolve (for example, pruned) counts as a
+different capture and is extracted. That is the last snapshot an observation fully
+handled: the current snapshot of a `completed`, `partial`, or `unchanged`
+observation, as `extractedSnapshotRef(observation)` returns. Otherwise the
+capture was persisted but never extracted (for example, a provider failed on the
+check that first saw it), so it is extracted now against that baseline instead
+of being reported as unchanged forever.
 Changed observations retain source and snapshot references, Traverse's prepared
 artifact identity, the proposal set, and the current/prior observation
 identities returned by the recorder. `partial`, `provider-failure`, mixed
@@ -442,10 +504,12 @@ behavior.
 ## Development
 
 ```sh
-npm ci
+pnpm install
 npm run verify   # content-boundary + decisions + typecheck + test + pack sanity
 ```
 
+The pnpm version is pinned in `package.json` (`packageManager`). Dependency install scripts are blocked by default; the only packages allowed to run one are listed under `allowBuilds` in `pnpm-workspace.yaml`, pinned by version. Scripts are still run with `npm run …` — that only invokes `package.json` scripts and does not depend on which tool installed `node_modules`.
+
 Individual gates: `npm test`, `npm run typecheck`, `npm run check:pack`,
 `npm run check:decisions`, `npm run check:content-boundary`.
 

package/dist/src/canonical-json.d.ts

@@ -0,0 +1,8 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export declare function compareCodeUnits(left: string, right: string): number;
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export declare function canonicalJson(value: unknown): string;

package/dist/src/canonical-json.js

@@ -0,0 +1,36 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export function compareCodeUnits(left, right) {
+    return left < right ? -1 : left > right ? 1 : 0;
+}
+function encode(input, key) {
+    // Mirrors JSON.stringify apart from key order: toJSON is honoured (a Date
+    // becomes its ISO string); undefined, functions and symbols have no encoding
+    // (omitted from objects, null in arrays); array holes and non-finite numbers
+    // become null; bigint throws.
+    const value = input !== null && typeof input === "object" && typeof input.toJSON === "function"
+        ? input.toJSON(key)
+        : input;
+    if (value === null || typeof value !== "object")
+        return JSON.stringify(value);
+    // Array.from visits holes; Array.prototype.map would skip them and write `[,1]`.
+    if (Array.isArray(value))
+        return `[${Array.from(value, (item, index) => encode(item, String(index)) ?? "null").join(",")}]`;
+    const members = [];
+    for (const key of Object.keys(value).sort(compareCodeUnits)) {
+        const item = encode(value[key], key);
+        if (item !== undefined)
+            members.push(`${JSON.stringify(key)}:${item}`);
+    }
+    return `{${members.join(",")}}`;
+}
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export function canonicalJson(value) {
+    const text = encode(value, "");
+    if (text === undefined)
+        throw new TypeError("Value has no JSON encoding");
+    return text;
+}

package/dist/src/capture-decoding.d.ts

@@ -0,0 +1,12 @@
+import type { Snapshot } from "@kontourai/forage/fetch";
+/**
+ * The encoding a text capture's body was decoded with, or `null` for a binary
+ * capture. Two captures with the same bytes but a different decoding give the
+ * extractor different text, so they are different captures.
+ *
+ * Text captures that carry `bytes` were decoded with their declared charset;
+ * the label is resolved the way Forage decodes it, so `utf8` and `utf-8` are
+ * the same decoding. Text captures without `bytes` (the earlier Forage record
+ * format, and rendered pages) were decoded or serialized as UTF-8.
+ */
+export declare function captureDecoding(snapshot: Snapshot): string | null;

package/dist/src/capture-decoding.js

@@ -0,0 +1,18 @@
+import { decodeTextBody } from "@kontourai/forage/fetch";
+/**
+ * The encoding a text capture's body was decoded with, or `null` for a binary
+ * capture. Two captures with the same bytes but a different decoding give the
+ * extractor different text, so they are different captures.
+ *
+ * Text captures that carry `bytes` were decoded with their declared charset;
+ * the label is resolved the way Forage decodes it, so `utf8` and `utf-8` are
+ * the same decoding. Text captures without `bytes` (the earlier Forage record
+ * format, and rendered pages) were decoded or serialized as UTF-8.
+ */
+export function captureDecoding(snapshot) {
+    if (typeof snapshot.body !== "string")
+        return null;
+    if (snapshot.bytes === undefined)
+        return "utf-8";
+    return decodeTextBody(new Uint8Array(0), snapshot.declaredCharset ?? null).encoding;
+}

package/dist/src/check-result.d.ts

@@ -20,7 +20,7 @@ export interface ChangedResult extends CheckResultCommon {
     currentSnapshotRef: string;
     changeBasis: "initial" | "hash";
 }
-export type LookoutErrorKind = "prior-read" | "persistence" | "dependency-contract" | "unexpected";
+export type LookoutErrorKind = "prior-read" | "persistence" | "history-full" | "dependency-contract" | "unexpected";
 export interface ErrorResult extends CheckResultCommon {
     kind: "error";
     origin: "forage" | "lookout";

package/dist/src/check-runner.d.ts

@@ -3,8 +3,28 @@ import type { CheckResult } from "./check-result.js";
 import type { ProviderResolver } from "./provider-resolution.js";
 import type { LookoutSource } from "./registry.js";
 export type FetchSource = (config: SourceConfig, options?: FetchSourceOptions) => Promise<FetchResult>;
+/**
+ * Bounds each source's snapshot history with the store's `prune` capability.
+ * After every stored capture, all but the newest `keepLast` snapshots are
+ * removed, except the latest, the captures the result names, and every
+ * snapshot `cited` returns. `cited` is read just before the prune, outside the
+ * store's lock: a citation recorded after that read is not protected by that
+ * prune, so record a citation before handing its reference out.
+ */
+export interface SnapshotRetention {
+    /** Newest snapshots kept per source (a non-negative integer; the latest is always kept). */
+    readonly keepLast: number;
+    /**
+     * Snapshot references the caller still cites for this source: recorded
+     * observations, review rounds, exported receipts. Never pruned. When this
+     * throws or returns anything but canonical references, nothing is pruned.
+     */
+    readonly cited: (sourceId: string) => Promise<readonly string[]> | readonly string[];
+}
 export interface CreateCheckRunnerOptions {
     store: SnapshotStore;
+    /** Prune history after each stored capture. Without it, history is kept in full. */
+    retention?: SnapshotRetention;
     fetchSource?: FetchSource;
     fetchOptions?: Omit<FetchSourceOptions, "store">;
     clock?: () => string;

package/dist/src/check-runner.js

@@ -1,5 +1,6 @@
 import { isDeepStrictEqual } from "node:util";
-import { buildSnapshotSourceRef, fetchSource as forageFetchSource, } from "@kontourai/forage/fetch";
+import { buildSnapshotSourceRef, fetchSource as forageFetchSource, isSnapshotHistoryFullError, parseSnapshotSourceRef, } from "@kontourai/forage/fetch";
+import { captureDecoding } from "./capture-decoding.js";
 /**
  * The egress policy for lookout's registered-source fetches. Registered source
  * URLs are operator- / aggregator-supplied and not fully trusted, so a source
@@ -15,6 +16,40 @@ export function createCheckRunner(options) {
     const fetchImpl = options.fetchSource ?? forageFetchSource;
     const clock = options.clock ?? (() => new Date().toISOString());
     const fetchOptions = { ...options.fetchOptions };
+    const retention = options.retention;
+    if (retention !== undefined && (!Number.isSafeInteger(retention.keepLast) || retention.keepLast < 0 || typeof retention.cited !== "function")) {
+        throw new TypeError("retention requires a non-negative integer keepLast and a cited function");
+    }
+    /** Apply retention for one source. Returns a warning instead of throwing: the capture is already stored. */
+    async function prune(sourceId, keepAlso) {
+        if (retention === undefined)
+            return undefined;
+        if (typeof options.store.prune !== "function")
+            return "snapshot retention skipped: the store cannot prune";
+        const keep = keepAlso.map((snapshot) => lookupOf(buildSnapshotSourceRef(snapshot)));
+        try {
+            const cited = await retention.cited(sourceId);
+            if (!Array.isArray(cited))
+                return "snapshot retention skipped: cited references are not a list";
+            for (const reference of cited) {
+                const lookup = lookupOf(reference);
+                if (lookup === undefined)
+                    return "snapshot retention skipped: a cited reference is not a snapshot reference";
+                if (lookup.sourceId === sourceId)
+                    keep.push(lookup);
+            }
+        }
+        catch (error) {
+            return `snapshot retention skipped: cited references could not be read: ${error instanceof Error ? error.message : String(error)}`;
+        }
+        try {
+            await options.store.prune(sourceId, { keepLast: retention.keepLast, keep });
+            return undefined;
+        }
+        catch (error) {
+            return `snapshot retention failed: ${error instanceof Error ? error.message : String(error)}`;
+        }
+    }
     async function check(source) {
         const common = () => ({
             sourceId: source.id,
@@ -66,12 +101,43 @@ export function createCheckRunner(options) {
                 }
                 return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(prior) };
             }
+            // A byte-identical repeat of the stored capture is not appended: its only
+            // new information is the check time, which the result carries. Both
+            // refs then name the stored capture, which stays replayable.
+            if (prior !== undefined && isRepeatCapture(prior, snapshot)) {
+                const priorSnapshotRef = buildSnapshotSourceRef(prior);
+                return { ...base, kind: "unchanged-hash", priorSnapshotRef, currentSnapshotRef: priorSnapshotRef };
+            }
+            // The capture just stored is kept even when it is not the newest (clock
+            // skew, or another check storing a later capture first): its result names it.
+            const keepAlso = prior === undefined ? [snapshot] : [snapshot, prior];
             try {
                 await options.store.put(snapshot);
             }
             catch (error) {
-                return lookoutError(base, "persistence", error);
+                if (!isSnapshotHistoryFullError(error))
+                    return lookoutError(base, "persistence", error);
+                // A full history refuses every later capture. Retention frees space
+                // (for example when it is enabled on a store that already reached the
+                // ceiling), so prune once and retry; otherwise say what to do.
+                const warning = await prune(source.id, keepAlso);
+                if (retention === undefined || warning !== undefined) {
+                    if (warning !== undefined)
+                        base.warnings.push(warning);
+                    return historyFull(base, error.maxHistoryFiles);
+                }
+                try {
+                    await options.store.put(snapshot);
+                }
+                catch (retryError) {
+                    return isSnapshotHistoryFullError(retryError)
+                        ? historyFull(base, retryError.maxHistoryFiles)
+                        : lookoutError(base, "persistence", retryError);
+                }
             }
+            const retentionWarning = await prune(source.id, keepAlso);
+            if (retentionWarning !== undefined)
+                base.warnings.push(retentionWarning);
             const currentSnapshotRef = buildSnapshotSourceRef(snapshot);
             if (!prior) {
                 return {
@@ -104,6 +170,36 @@ export function createCheckRunner(options) {
     }
     return { check, checkAll };
 }
+// Response headers other than the validators a later conditional request
+// reuses (e.g. Date) are not compared: they differ on nearly every response.
+const REVALIDATION_HEADERS = ["etag", "last-modified"];
+/** Same resource, same body, and nothing a later check reads differs. */
+function isRepeatCapture(prior, current) {
+    const priorEncoding = bodyEncoding(prior);
+    return priorEncoding !== undefined &&
+        priorEncoding === bodyEncoding(current) &&
+        prior.sourceId === current.sourceId &&
+        prior.url === current.url &&
+        prior.status === current.status &&
+        prior.bodyHash === current.bodyHash &&
+        // Same bytes under another charset decode to other text: a new capture.
+        captureDecoding(prior) === captureDecoding(current) &&
+        prior.rendered === current.rendered &&
+        isDeepStrictEqual(prior.redirects, current.redirects) &&
+        REVALIDATION_HEADERS.every((name) => prior.headers?.[name] === current.headers?.[name]);
+}
+function lookupOf(reference) {
+    if (typeof reference !== "string")
+        return undefined;
+    const parsed = parseSnapshotSourceRef(reference);
+    if (parsed === undefined)
+        return undefined;
+    const { sourceId, url, bodyHash, fetchedAt, snapshotDigest } = parsed;
+    return { sourceId, url, bodyHash, fetchedAt, ...(snapshotDigest === undefined ? {} : { snapshotDigest }) };
+}
+function historyFull(common, maxHistoryFiles) {
+    return lookoutError(common, "history-full", `snapshot history for this source holds its maximum of ${maxHistoryFiles} records; configure snapshot retention or prune the store`);
+}
 function bodyEncoding(snapshot) {
     const descriptor = Object.getOwnPropertyDescriptor(snapshot, "body");
     if (descriptor === undefined || !("value" in descriptor))

package/dist/src/drift-emission.d.ts

@@ -1,6 +1,6 @@
 import type { LookoutSource } from "./registry.js";
 import { type ProposalDiffEvent, type ProposalSetDiff, type ProposalSetDiffInput, type ProposalSetFacts, type ProposalSetObservation } from "./proposal-diff.js";
-import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservationV1 } from "./observation-store.js";
+import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservation } from "./observation-store.js";
 import type { SnapshotStore } from "@kontourai/forage";
 export interface BaselineEstablishedFact {
     readonly kind: "baseline-established";
@@ -25,7 +25,7 @@ export interface DriftSuccess {
     readonly facts: readonly DriftFact[];
     /** The prior observation this drift was diffed against, or null on a first-ever (baseline) observation. */
     readonly priorObservationId: string | null;
-    readonly committedObservation: StoredProposalObservationV1;
+    readonly committedObservation: StoredProposalObservation;
     readonly warnings: readonly string[];
 }
 export type DriftErrorKind = "invalid-input" | "prior-state-error" | "diff-error" | "persistence-error" | "serialization-error" | "unexpected";

package/dist/src/drift-emission.js

@@ -1,14 +1,15 @@
 import { diffProposalSets } from "./proposal-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 import { admitProposalObservation } from "./observation-admission.js";
 function stableJson(value) {
     if (Array.isArray(value))
         return `[${value.map(stableJson).join(",")}]`;
     if (value && typeof value === "object")
-        return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
+        return `{${Object.entries(value).sort(([a], [b]) => compareCodeUnits(a, b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
     return JSON.stringify(value) ?? "undefined";
 }
 function normalizeDiff(value) {
-    const sorted = (items) => [...items].sort((a, b) => stableJson(a).localeCompare(stableJson(b)));
+    const sorted = (items) => [...items].sort((a, b) => compareCodeUnits(stableJson(a), stableJson(b)));
     return {
         events: sorted(value.events),
         facts: {

package/dist/src/index.d.ts

@@ -1,7 +1,7 @@
 export { runCli } from "./cli.js";
 export type { RunCliOptions } from "./cli.js";
 export { createCheckRunner } from "./check-runner.js";
-export type { CheckRunner, CreateCheckRunnerOptions, FetchSource } from "./check-runner.js";
+export type { CheckRunner, CreateCheckRunnerOptions, FetchSource, SnapshotRetention } from "./check-runner.js";
 export type { ChangedResult, CheckResult, CheckResultCommon, ErrorResult, LookoutErrorKind, Unchanged304Result, UnchangedHashResult, } from "./check-result.js";
 export { defaultProviderResolver } from "./provider-resolution.js";
 export type { ProviderResolver } from "./provider-resolution.js";
@@ -10,7 +10,7 @@ export { inMemorySourceStore } from "./source-store.js";
 export type { SourceStore } from "./source-store.js";
 export type { ExtractableLookoutSource, LookoutRegistryDocument, LookoutSource, LookoutSourceKind, RenderPolicy, StructuredFileFormat, StructuredFileLookoutSource, } from "./registry.js";
 export { createLookoutSnapshotStore, resolveLookoutSnapshot, } from "./snapshot-store.js";
-export type { ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
+export type { LookoutSnapshotStoreOptions, ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
 export { admitProposalObservation } from "./observation-admission.js";
 export type { AdmitProposalObservationInput, AdmittedProposalObservation, AdmittedSnapshotIdentity, ObservationAdmissionError, ObservationAdmissionErrorKind, ObservationAdmissionResult } from "./observation-admission.js";
 export { admitSourceCapture, admitSourceCheck } from "./source-admission.js";
@@ -23,12 +23,12 @@ export type { KeyedMultisetFacts, KeyedMultisetOptions, StructuralComparison, St
 export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js";
 export type { FieldChangedEvent, FieldChangeKind, NewEntityAppearedEvent, ProposalDiffEvent, ProposalEvidence, ProposalIdentity, ProposalOccurrencePair, ProposalSetDiff, ProposalSetDiffInput, ProposalSetFacts, ProposalSetObservation, ProvenanceChangeFact, } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
-export type { CreateObservationStoreOptions, HeadWitnessComparison, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalHeadWitnessV1, ProposalObservationRecordInput, StoredProposalObservationV1, VerifiedHeadLimits, VerifiedHeadObservationStore, VerifiedHeadRead } from "./observation-store.js";
+export type { CreateObservationStoreOptions, HeadWitnessComparison, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalHeadWitnessV1, ProposalObservationRecordInput, StoredProposalObservation, StoredProposalObservationV1, StoredProposalObservationV2, VerifiedHeadLimits, VerifiedHeadObservationStore, VerifiedHeadRead } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export type { BaselineEstablishedFact, CreateDriftEmitterOptions, DriftEmitter, DriftError, DriftErrorKind, DriftFact, DriftResult, DriftSuccess, EmitDriftInput } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
 export type { SchemaCoverageGap, SchemaCoverageResult } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export type { ObserveExtractAcquisition, ObserveExtractAttempt, ObserveExtractDiff, ObserveExtractDiffOptions, ObserveExtractError, ObserveExtractErrorKind, ObserveExtractExtraction, ObserveExtractExtractionInput, ObserveExtractObservation, ObserveExtractObservationIdentity, ObserveExtractOutcome, ObserveExtractProviderFailure, ObserveExtractRecorder, ObserveExtractResult, ObserveExtractSource, ObserveExtractSourceSnapshot, } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";
 export type { BuildSemanticReviewWorkInput, SemanticClaimTarget, SemanticObservationIdentity, SemanticReviewCandidate, SemanticReviewChange, SemanticReviewItem, SemanticReviewKind, SemanticReviewWork, } from "./semantic-review-work.js";

package/dist/src/index.js

@@ -12,5 +12,5 @@ export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js
 export { createObservationStore } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";

package/dist/src/observation-admission.d.ts

@@ -1,6 +1,6 @@
 import type { SnapshotStore } from "@kontourai/forage";
 import type { LookoutSource } from "./registry.js";
-import type { StoredProposalObservationV1, ObservationCheckAnchor } from "./observation-store.js";
+import type { StoredProposalObservation, ObservationCheckAnchor } from "./observation-store.js";
 import type { ProposalSetObservation } from "./proposal-diff.js";
 /** A resolved snapshot identity suitable for durable observation metadata. */
 export interface AdmittedSnapshotIdentity {
@@ -44,7 +44,7 @@ export interface AdmitProposalObservationInput {
     readonly current: ProposalSetObservation;
     readonly check: ObservationCheckAnchor;
     /** The already-selected observation-store record; admission never selects or persists continuity. */
-    readonly prior: StoredProposalObservationV1 | null;
+    readonly prior: StoredProposalObservation | null;
     /** Explicit capability: admission never chooses a snapshot root or storage implementation. */
     readonly snapshotStore: SnapshotStore;
 }

package/dist/src/observation-store.d.ts

@@ -21,6 +21,15 @@ export interface StoredProposalObservationV1 {
     readonly check: ObservationCheckAnchor;
     readonly proposals: readonly ExtractionProposal[];
 }
+/**
+ * Same fields as version 1. The observationId is computed over code-unit-ordered
+ * canonical JSON, so it does not depend on the host locale. New records are
+ * always version 2.
+ */
+export interface StoredProposalObservationV2 extends Omit<StoredProposalObservationV1, "version"> {
+    readonly version: 2;
+}
+export type StoredProposalObservation = StoredProposalObservationV1 | StoredProposalObservationV2;
 export type ObservationStoreErrorKind = "invalid-input" | "corrupt-state" | "continuity-conflict" | "io-error";
 export interface ObservationStoreError {
     readonly kind: ObservationStoreErrorKind;
@@ -36,8 +45,8 @@ export type ObservationStoreResult<T> = {
     readonly error: ObservationStoreError;
 };
 export interface ObservationStore {
-    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservationV1 | null>>;
-    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservationV1>>;
+    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservation | null>>;
+    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservation>>;
 }
 /** Finite caller-controlled ceilings for a verified proposal-head read. */
 export interface VerifiedHeadLimits {

package/dist/src/observation-store.js

@@ -3,19 +3,27 @@ import { constants } from "node:fs";
 import { lstat, mkdir, open, opendir, readFile, readdir, realpath, rename, rm, unlink } from "node:fs/promises";
 import path from "node:path";
 import { types } from "node:util";
+import { canonicalJson, compareCodeUnits } from "./canonical-json.js";
 const VERIFIED_HEAD_MAX = Object.freeze({ maxEntries: 8, maxIndexBytes: 8 * 1024, maxPointerBytes: 8 * 1024, maxRecordBytes: 1024 * 1024 });
 function sourceKey(sourceId) {
     return `${encodeURIComponent(sourceId).replaceAll("%", "_").slice(0, 48)}-${createHash("sha256").update(sourceId).digest("hex").slice(0, 16)}`;
 }
-function stable(value) {
+function canonical(value) { return `${canonicalJson(value)}\n`; }
+function digest(body) {
+    return createHash("sha256").update(body.version === 1 ? legacyCanonical(body) : canonical(body)).digest("hex");
+}
+// Version 1 digests only. Version 1 records were hashed with keys sorted by the
+// host locale and then rebuilt with Object.fromEntries, which moves integer-like
+// keys first. A version 1 record therefore verifies only under a locale that
+// collates its keys the way the writing host did. Do not use this for new records.
+function legacyStable(value) {
     if (Array.isArray(value))
-        return value.map(stable);
+        return value.map(legacyStable);
     if (value && typeof value === "object")
-        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, stable(item)]));
+        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, legacyStable(item)]));
     return value;
 }
-function canonical(value) { return `${JSON.stringify(stable(value))}\n`; }
-function digest(body) { return createHash("sha256").update(canonical(body)).digest("hex"); }
+function legacyCanonical(value) { return `${JSON.stringify(legacyStable(value))}\n`; }
 function resolveHeadLimits(limits) {
     try {
         if (limits !== undefined && (!limits || typeof limits !== "object" || types.isProxy(limits) || Array.isArray(limits)))
@@ -219,8 +227,8 @@ function buildRecord(input) {
         if (!check || check.currentSnapshotRef !== observation.snapshotRef || typeof check.checkedAt !== "string" || (check.resultKind !== "changed" && check.resultKind !== "unchanged-hash") || typeof input.recordedAt !== "string") {
             return { ok: false, error: { kind: "invalid-input", message: "Check anchor must match the current observation snapshot" } };
         }
-        const proposals = [...observation.proposals].sort((a, b) => canonical(a).localeCompare(canonical(b)));
-        const body = { version: 1, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
+        const proposals = [...observation.proposals].sort((a, b) => compareCodeUnits(canonical(a), canonical(b)));
+        const body = { version: 2, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
         try {
             return { ok: true, value: { ...body, observationId: digest(body) } };
         }
@@ -249,7 +257,7 @@ function validate(value, expectedSourceId) {
     if (!value || typeof value !== "object" || Array.isArray(value))
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation is not an object" } };
     const item = value;
-    if (item.version !== 1 || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
+    if ((item.version !== 1 && item.version !== 2) || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation schema or continuity is invalid" } };
     }
     const { observationId, ...body } = item;
@@ -416,7 +424,7 @@ export function createObservationStore(options = {}) {
                     return null;
                 } }).filter((item) => item !== null);
                 const preserve = new Set([record.observationId, expectedPriorId].filter((item) => item !== null));
-                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => b.recordedAt.localeCompare(a.recordedAt) || b.id.localeCompare(a.id));
+                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => compareCodeUnits(b.recordedAt, a.recordedAt) || compareCodeUnits(b.id, a.id));
                 await Promise.all(extras.map((item) => rm(path.join(dir, item.name), { force: true })));
             }
             catch (cause) {

package/dist/src/observe-extract-diff.d.ts

@@ -1,4 +1,5 @@
 import type { ExtractionPartial, ExtractionProviderFailure, ExtractionResult, PreparedArtifact } from "@kontourai/traverse";
+import type { SnapshotStore } from "@kontourai/forage/fetch";
 import type { CheckResult } from "./check-result.js";
 import type { ProposalSetObservation } from "./proposal-diff.js";
 import type { LookoutSource } from "./registry.js";
@@ -64,11 +65,36 @@ export interface ObserveExtractObservationIdentity {
  */
 export interface ObserveExtractRecorder {
     record(observation: ObserveExtractObservation): Promise<ObserveExtractObservationIdentity>;
+    /**
+     * The snapshot reference of this source's most recent recorded observation
+     * for which `extractedSnapshotRef(observation)` is not null, or `null` when
+     * there is none.
+     *
+     * An unchanged check is only skipped when its capture has the same URL, body
+     * hash, and text decoding as this snapshot. Otherwise a change that acquisition already
+     * persisted, but that was never extracted (for example because a provider
+     * failed), would be reported as unchanged forever.
+     */
+    lastExtractedSnapshotRef(source: ObserveExtractSource): Promise<string | null>;
 }
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export declare function extractedSnapshotRef(observation: ObserveExtractObservation): string | null;
 export interface ObserveExtractDiffOptions {
     readonly acquisition: ObserveExtractAcquisition;
     readonly extraction: ObserveExtractExtraction;
     readonly recorder: ObserveExtractRecorder;
+    /**
+     * The snapshot store acquisition persists to. An unchanged check is compared
+     * with the last extracted capture by resolving both references here, since a
+     * reference does not name the charset its text was decoded with. A reference
+     * that does not resolve counts as a different capture, so it is extracted.
+     * With snapshot retention, cite the last extracted reference to keep it.
+     */
+    readonly snapshots: SnapshotStore;
 }
 export type ObserveExtractErrorKind = "acquisition-threw" | "recording-failed" | "dependency-contract";
 export interface ObserveExtractError {

package/dist/src/observe-extract-diff.js

@@ -1,5 +1,21 @@
 import { validatePreparedArtifact } from "@kontourai/traverse";
+import { resolveSnapshotSourceRef } from "@kontourai/forage/fetch";
+import { captureDecoding } from "./capture-decoding.js";
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export function extractedSnapshotRef(observation) {
+    const handled = observation.outcome === "completed" || observation.outcome === "partial" || observation.outcome === "unchanged";
+    return handled && observation.sourceSnapshot !== null ? observation.sourceSnapshot.currentSnapshotRef : null;
+}
 export function createObserveExtractDiff(options) {
+    // Without a store every unchanged capture would silently count as new and be
+    // extracted again, so a missing one is refused up front.
+    if (typeof options?.snapshots?.findExact !== "function") {
+        throw new TypeError("createObserveExtractDiff requires snapshots: the snapshot store acquisition persists to, with exact lookup");
+    }
     return {
         async observe(source) {
             try {
@@ -19,10 +35,26 @@ export function createObserveExtractDiff(options) {
                 if (check.kind === "error") {
                     return record(options.recorder, baseObservation(source, check, "acquisition-error", null, null, null, null));
                 }
+                let sourceSnapshot = snapshotFor(check);
                 if (check.kind === "unchanged-304" || check.kind === "unchanged-hash") {
-                    return record(options.recorder, baseObservation(source, check, "unchanged", snapshotFor(check), null, null, null));
+                    // "Unchanged" compares with the latest stored capture, which may never
+                    // have been extracted. Skip extraction only when the capture matches
+                    // the last one that was; otherwise extract it against that baseline.
+                    let extracted;
+                    try {
+                        extracted = await options.recorder.lastExtractedSnapshotRef(observationSource(source));
+                    }
+                    catch (cause) {
+                        return { ok: false, error: error("recording-failed", "Observation recorder could not report the last extracted snapshot", cause) };
+                    }
+                    if (extracted !== null && (typeof extracted !== "string" || extracted === "")) {
+                        return { ok: false, error: error("dependency-contract", "Observation recorder returned an invalid last extracted snapshot reference") };
+                    }
+                    if (extracted !== null && await sameCapture(options.snapshots, extracted, sourceSnapshot.currentSnapshotRef)) {
+                        return record(options.recorder, baseObservation(source, check, "unchanged", sourceSnapshot, null, null, null));
+                    }
+                    sourceSnapshot = { priorSnapshotRef: extracted, currentSnapshotRef: sourceSnapshot.currentSnapshotRef };
                 }
-                const sourceSnapshot = snapshotFor(check);
                 let extraction;
                 try {
                     extraction = await options.extraction.extract({ source, snapshotRef: sourceSnapshot.currentSnapshotRef });
@@ -65,7 +97,7 @@ export function createObserveExtractDiff(options) {
 }
 function baseObservation(source, check, outcome, sourceSnapshot, preparedArtifact, proposalSet, attempt) {
     return {
-        source: { id: source.id, url: source.url, kind: source.kind },
+        source: observationSource(source),
         check,
         outcome,
         sourceSnapshot,
@@ -93,6 +125,23 @@ async function record(recorder, observation) {
         return { ok: false, error: error("recording-failed", "Observation recorder failed", cause), observation };
     }
 }
+function observationSource(source) {
+    return { id: source.id, url: source.url, kind: source.kind };
+}
+/**
+ * Two references name the same capture content: same source, resource URL,
+ * body hash, and text decoding. Same bytes under another declared charset are
+ * other text, so they are a different capture.
+ */
+async function sameCapture(store, left, right) {
+    if (left === right)
+        return true;
+    const [a, b] = await Promise.all([resolveSnapshotSourceRef(store, left), resolveSnapshotSourceRef(store, right)]);
+    if (!a.ok || !b.ok)
+        return false;
+    return a.snapshot.sourceId === b.snapshot.sourceId && a.snapshot.url === b.snapshot.url &&
+        a.snapshot.bodyHash === b.snapshot.bodyHash && captureDecoding(a.snapshot) === captureDecoding(b.snapshot);
+}
 function snapshotFor(check) {
     if (check.kind === "unchanged-304")
         return { priorSnapshotRef: check.snapshotRef, currentSnapshotRef: check.snapshotRef };

package/dist/src/proposal-diff.js

@@ -1,5 +1,6 @@
 import { canonicalValueKey, } from "./canonical-value.js";
 import { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 function callbackError(label, cause) {
     return { kind: "callback-threw", message: `${label} callback threw`, cause };
 }
@@ -90,7 +91,7 @@ function semanticFieldOrder(items, fieldIdentity) {
             return encoded;
         groups.set(fieldKey.value, [...(groups.get(fieldKey.value) ?? []), { item, key: encoded.key, index }]);
     }
-    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => left.key.localeCompare(right.key) || left.index - right.index));
+    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => compareCodeUnits(left.key, right.key) || left.index - right.index));
     return { ok: true, value: ordered.map(({ item }) => item) };
 }
 export function diffProposalSets(input) {

package/dist/src/snapshot-store.d.ts

@@ -1,6 +1,14 @@
-import type { SnapshotStore } from "@kontourai/forage";
+import type { ExactSnapshotStore, PrunableSnapshotStore, SnapshotStore } from "@kontourai/forage";
 import { type SnapshotSourceRefResolution } from "@kontourai/forage/fetch";
-export declare function createLookoutSnapshotStore(root?: string): SnapshotStore;
+export interface LookoutSnapshotStoreOptions {
+    /**
+     * Per-source record ceiling, passed to Forage's filesystem store (1 to
+     * 10,000; Forage's default is 10,000). It cannot change after a source's
+     * store directory is initialized.
+     */
+    readonly maxHistoryFiles?: number;
+}
+export declare function createLookoutSnapshotStore(root?: string, options?: LookoutSnapshotStoreOptions): ExactSnapshotStore & PrunableSnapshotStore;
 export type ResolveLookoutSnapshotOptions = {
     store: SnapshotStore;
     root?: never;

package/dist/src/snapshot-store.js

@@ -1,8 +1,11 @@
 import path from "node:path";
 import { createFilesystemSnapshotStore } from "@kontourai/forage";
 import { resolveSnapshotSourceRef, } from "@kontourai/forage/fetch";
-export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots")) {
-    return createFilesystemSnapshotStore({ root });
+export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots"), options = {}) {
+    return createFilesystemSnapshotStore({
+        root,
+        ...(options.maxHistoryFiles === undefined ? {} : { maxHistoryFiles: options.maxHistoryFiles }),
+    });
 }
 /** Replay one Lookout-emitted durable reference without any network access. */
 export async function resolveLookoutSnapshot(reference, options = {}) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kontourai/lookout",
-  "version": "0.5.2",
+  "version": "0.7.0",
   "description": "A small source registry and drift-check runner built on Forage snapshots.",
   "license": "Apache-2.0",
   "type": "module",
@@ -46,17 +46,21 @@
     "workflow:validate-artifacts": "flow-agents-validate-artifacts"
   },
   "dependencies": {
-    "@kontourai/datum": "0.7.0",
-    "@kontourai/forage": "0.6.1",
+    "@kontourai/datum": "0.8.0",
+    "@kontourai/forage": "1.0.0",
     "@kontourai/traverse": "0.25.1"
   },
   "devDependencies": {
-    "@kontourai/flow-agents": "5.9.0",
+    "@kontourai/flow-agents": "6.4.0",
     "@kontourai/survey": "3.0.0",
     "@types/node": "^26.1.2",
+    "ajv": "8.20.0",
+    "ajv-formats": "3.0.1",
+    "hachure": "0.15.0",
     "typescript": "^7.0.2"
   },
   "engines": {
     "node": ">=22"
-  }
+  },
+  "packageManager": "pnpm@11.25.0"
 }