@kontourai/lookout 0.5.1 → 0.6.0

package/README.md

@@ -167,7 +167,7 @@ warnings.
 | `kind` | When | Extra fields |
 | --- | --- | --- |
 | `unchanged-304` | A validator-backed conditional request returned `304`; zero body transfer; the prior snapshot is not re-persisted. | `snapshotRef` |
-| `unchanged-hash` | A full body was fetched and persisted, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. | `priorSnapshotRef`, `currentSnapshotRef` |
+| `unchanged-hash` | A full body was fetched, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. A byte-identical repeat of the prior capture is not persisted again (see below). | `priorSnapshotRef`, `currentSnapshotRef` |
 | `changed` | The fresh body differs from the prior (`changeBasis: "hash"`), **or** it is the first successful observation (`changeBasis: "initial"`, `priorSnapshotRef: null`). | `priorSnapshotRef` (nullable), `currentSnapshotRef`, `changeBasis` |
 | `error` | Any operational failure — contained so the runner never rejects. | `origin` (`forage` \| `lookout`), `error` |
 
@@ -179,6 +179,14 @@ snapshot is persisted as the new baseline. (The `unchanged-304` path is already
 resource-scoped by Forage's validators, so only the hash path needs this
 guard.)
 
+A fetch that repeats the prior capture exactly is not appended to snapshot
+history: same URL, status, body bytes, redirects, render state, and `etag` /
+`last-modified` validators, with only the fetch time differing. Its
+`unchanged-hash` result names the stored capture as both `priorSnapshotRef` and
+`currentSnapshotRef`, and `checkedAt` records the check. Other response headers
+of the repeat, such as `Date`, are not kept. A stable source therefore adds no
+records, however often it is checked.
+
 `error` results preserve provenance: `origin: "forage"` carries Forage's
 discriminated `FetchError` verbatim (its `kind`, and `status` when present);
 `origin: "lookout"` carries a typed `kind` — `prior-read`, `persistence`,
@@ -189,7 +197,8 @@ are portable logical refs from Forage's `buildSnapshotSourceRef` — never
 filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by
 default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
-custom filenames or retention. `resolveLookoutSnapshot()` replays one exact
+custom filenames or retention; `createLookoutSnapshotStore(root, {
+maxHistoryFiles })` passes Forage's per-source record ceiling through. `resolveLookoutSnapshot()` replays one exact
 reference through an injected store or Lookout snapshot root, authenticates its
 body and replay metadata, and never fetches. References emitted before Forage's
 replay-envelope digest remain resolvable with `integrity: "body-and-identity"`;
@@ -256,6 +265,32 @@ latest two valid observations. The emitted `events` are already
 Hachure-evidence-shaped; a consumer or product lifts them into a Hachure
 `TrustBundle` with `@kontourai/surface`'s `TrustBundleBuilder` when it wants
 Surface projection — lookout itself authors nothing in the trust layer.
+
+An observation id is the SHA-256 of the record's canonical JSON: object keys
+and proposals in UTF-16 code-unit order, never the host locale's collation.
+New records are `version: 2`. Records written before this change are
+`version: 1`, whose digest used the writing host's locale collation. They
+still load on a host whose locale collates their keys the same way, and the
+next commit replaces a `version: 1` prior with a `version: 2` record. Under a
+different locale, a `version: 1` record with non-ASCII or mixed-case keys or
+values can read as `corrupt-state`.
+
+### Verified proposal head witnesses
+
+The concrete filesystem observation store additionally exposes
+`readVerifiedHead(sourceId, limits)` and `compareHeadWitness(witness, limits)`.
+The first returns an authenticated observation id and snapshot reference with an
+opaque, versioned witness; the second reads only bounded head metadata and
+returns `matches`, `changed`, `missing`, `unavailable`, `corrupt`, or
+`unsupported`. It never reads proposal bytes during comparison.
+
+A witness is an as-of fence, not a freshness guarantee: it binds the source,
+current head, store scope, pointer, and filesystem metadata observed at capture.
+It detects ordinary older writers without requiring them to write a sidecar.
+Restarting the same store preserves a witness; copying or restoring the store
+can invalidate it. It does not prove proposal-body bit-rot or provide an atomic
+transaction across stores. Limits are finite hard caps on enumeration, pointer,
+and authenticated-record reads; callers may only narrow the defaults.
 Store paths refuse symbolic links. An existing source lock is not automatically
 broken: after confirming no writer is active, an operator may remove an
 abandoned `.lock` file and retry.
@@ -343,6 +378,10 @@ const composition = createObserveExtractDiff({
       // Caller-owned continuity and durable storage.
       return saveObservation(observation);
     },
+    async lastExtractedSnapshotRef(source) {
+      // The newest non-null extractedSnapshotRef(observation) you recorded.
+      return loadLastExtractedSnapshotRef(source.id);
+    },
   },
 });
 
@@ -350,7 +389,14 @@ const result = await composition.observe(source);
 ```
 
 `unchanged-304` and `unchanged-hash` are recorded as `unchanged` and never call
-the extraction capability, so they make zero preparation and provider calls.
+the extraction capability, so they make zero preparation and provider calls,
+when the capture has the same URL and body hash as the recorder's
+`lastExtractedSnapshotRef`. That is the last snapshot an observation fully
+handled: the current snapshot of a `completed`, `partial`, or `unchanged`
+observation, as `extractedSnapshotRef(observation)` returns. Otherwise the
+capture was persisted but never extracted (for example, a provider failed on the
+check that first saw it), so it is extracted now against that baseline instead
+of being reported as unchanged forever.
 Changed observations retain source and snapshot references, Traverse's prepared
 artifact identity, the proposal set, and the current/prior observation
 identities returned by the recorder. `partial`, `provider-failure`, mixed
@@ -425,10 +471,12 @@ behavior.
 ## Development
 
 ```sh
-npm ci
+pnpm install
 npm run verify   # content-boundary + decisions + typecheck + test + pack sanity
 ```
 
+The pnpm version is pinned in `package.json` (`packageManager`). Dependency install scripts are blocked by default; the only packages allowed to run one are listed under `allowBuilds` in `pnpm-workspace.yaml`, pinned by version. Scripts are still run with `npm run …` — that only invokes `package.json` scripts and does not depend on which tool installed `node_modules`.
+
 Individual gates: `npm test`, `npm run typecheck`, `npm run check:pack`,
 `npm run check:decisions`, `npm run check:content-boundary`.
 

package/dist/src/canonical-json.d.ts

@@ -0,0 +1,8 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export declare function compareCodeUnits(left: string, right: string): number;
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export declare function canonicalJson(value: unknown): string;

package/dist/src/canonical-json.js

@@ -0,0 +1,36 @@
+/** Orders strings by UTF-16 code unit. Unlike `localeCompare`, it ignores the host locale. */
+export function compareCodeUnits(left, right) {
+    return left < right ? -1 : left > right ? 1 : 0;
+}
+function encode(input, key) {
+    // Mirrors JSON.stringify apart from key order: toJSON is honoured (a Date
+    // becomes its ISO string); undefined, functions and symbols have no encoding
+    // (omitted from objects, null in arrays); array holes and non-finite numbers
+    // become null; bigint throws.
+    const value = input !== null && typeof input === "object" && typeof input.toJSON === "function"
+        ? input.toJSON(key)
+        : input;
+    if (value === null || typeof value !== "object")
+        return JSON.stringify(value);
+    // Array.from visits holes; Array.prototype.map would skip them and write `[,1]`.
+    if (Array.isArray(value))
+        return `[${Array.from(value, (item, index) => encode(item, String(index)) ?? "null").join(",")}]`;
+    const members = [];
+    for (const key of Object.keys(value).sort(compareCodeUnits)) {
+        const item = encode(value[key], key);
+        if (item !== undefined)
+            members.push(`${JSON.stringify(key)}:${item}`);
+    }
+    return `{${members.join(",")}}`;
+}
+/**
+ * JSON text with every object's keys in code-unit order, written directly.
+ * Rebuilding an object and calling JSON.stringify cannot give this order,
+ * because JavaScript always lists integer-like keys ("9", "10") first.
+ */
+export function canonicalJson(value) {
+    const text = encode(value, "");
+    if (text === undefined)
+        throw new TypeError("Value has no JSON encoding");
+    return text;
+}

package/dist/src/check-runner.js

@@ -66,6 +66,13 @@ export function createCheckRunner(options) {
                 }
                 return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(prior) };
             }
+            // A byte-identical repeat of the stored capture is not appended: its only
+            // new information is the check time, which the result carries. Both
+            // refs then name the stored capture, which stays replayable.
+            if (prior !== undefined && isRepeatCapture(prior, snapshot)) {
+                const priorSnapshotRef = buildSnapshotSourceRef(prior);
+                return { ...base, kind: "unchanged-hash", priorSnapshotRef, currentSnapshotRef: priorSnapshotRef };
+            }
             try {
                 await options.store.put(snapshot);
             }
@@ -104,6 +111,22 @@ export function createCheckRunner(options) {
     }
     return { check, checkAll };
 }
+// Response headers other than the validators a later conditional request
+// reuses (e.g. Date) are not compared: they differ on nearly every response.
+const REVALIDATION_HEADERS = ["etag", "last-modified"];
+/** Same resource, same body, and nothing a later check reads differs. */
+function isRepeatCapture(prior, current) {
+    const priorEncoding = bodyEncoding(prior);
+    return priorEncoding !== undefined &&
+        priorEncoding === bodyEncoding(current) &&
+        prior.sourceId === current.sourceId &&
+        prior.url === current.url &&
+        prior.status === current.status &&
+        prior.bodyHash === current.bodyHash &&
+        prior.rendered === current.rendered &&
+        isDeepStrictEqual(prior.redirects, current.redirects) &&
+        REVALIDATION_HEADERS.every((name) => prior.headers?.[name] === current.headers?.[name]);
+}
 function bodyEncoding(snapshot) {
     const descriptor = Object.getOwnPropertyDescriptor(snapshot, "body");
     if (descriptor === undefined || !("value" in descriptor))

package/dist/src/drift-emission.d.ts

@@ -1,6 +1,6 @@
 import type { LookoutSource } from "./registry.js";
 import { type ProposalDiffEvent, type ProposalSetDiff, type ProposalSetDiffInput, type ProposalSetFacts, type ProposalSetObservation } from "./proposal-diff.js";
-import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservationV1 } from "./observation-store.js";
+import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservation } from "./observation-store.js";
 import type { SnapshotStore } from "@kontourai/forage";
 export interface BaselineEstablishedFact {
     readonly kind: "baseline-established";
@@ -25,7 +25,7 @@ export interface DriftSuccess {
     readonly facts: readonly DriftFact[];
     /** The prior observation this drift was diffed against, or null on a first-ever (baseline) observation. */
     readonly priorObservationId: string | null;
-    readonly committedObservation: StoredProposalObservationV1;
+    readonly committedObservation: StoredProposalObservation;
     readonly warnings: readonly string[];
 }
 export type DriftErrorKind = "invalid-input" | "prior-state-error" | "diff-error" | "persistence-error" | "serialization-error" | "unexpected";

package/dist/src/drift-emission.js

@@ -1,14 +1,15 @@
 import { diffProposalSets } from "./proposal-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 import { admitProposalObservation } from "./observation-admission.js";
 function stableJson(value) {
     if (Array.isArray(value))
         return `[${value.map(stableJson).join(",")}]`;
     if (value && typeof value === "object")
-        return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
+        return `{${Object.entries(value).sort(([a], [b]) => compareCodeUnits(a, b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
     return JSON.stringify(value) ?? "undefined";
 }
 function normalizeDiff(value) {
-    const sorted = (items) => [...items].sort((a, b) => stableJson(a).localeCompare(stableJson(b)));
+    const sorted = (items) => [...items].sort((a, b) => compareCodeUnits(stableJson(a), stableJson(b)));
     return {
         events: sorted(value.events),
         facts: {

package/dist/src/index.d.ts

@@ -10,7 +10,7 @@ export { inMemorySourceStore } from "./source-store.js";
 export type { SourceStore } from "./source-store.js";
 export type { ExtractableLookoutSource, LookoutRegistryDocument, LookoutSource, LookoutSourceKind, RenderPolicy, StructuredFileFormat, StructuredFileLookoutSource, } from "./registry.js";
 export { createLookoutSnapshotStore, resolveLookoutSnapshot, } from "./snapshot-store.js";
-export type { ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
+export type { LookoutSnapshotStoreOptions, ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
 export { admitProposalObservation } from "./observation-admission.js";
 export type { AdmitProposalObservationInput, AdmittedProposalObservation, AdmittedSnapshotIdentity, ObservationAdmissionError, ObservationAdmissionErrorKind, ObservationAdmissionResult } from "./observation-admission.js";
 export { admitSourceCapture, admitSourceCheck } from "./source-admission.js";
@@ -23,12 +23,12 @@ export type { KeyedMultisetFacts, KeyedMultisetOptions, StructuralComparison, St
 export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js";
 export type { FieldChangedEvent, FieldChangeKind, NewEntityAppearedEvent, ProposalDiffEvent, ProposalEvidence, ProposalIdentity, ProposalOccurrencePair, ProposalSetDiff, ProposalSetDiffInput, ProposalSetFacts, ProposalSetObservation, ProvenanceChangeFact, } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
-export type { CreateObservationStoreOptions, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalObservationRecordInput, StoredProposalObservationV1 } from "./observation-store.js";
+export type { CreateObservationStoreOptions, HeadWitnessComparison, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalHeadWitnessV1, ProposalObservationRecordInput, StoredProposalObservation, StoredProposalObservationV1, StoredProposalObservationV2, VerifiedHeadLimits, VerifiedHeadObservationStore, VerifiedHeadRead } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export type { BaselineEstablishedFact, CreateDriftEmitterOptions, DriftEmitter, DriftError, DriftErrorKind, DriftFact, DriftResult, DriftSuccess, EmitDriftInput } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
 export type { SchemaCoverageGap, SchemaCoverageResult } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export type { ObserveExtractAcquisition, ObserveExtractAttempt, ObserveExtractDiff, ObserveExtractDiffOptions, ObserveExtractError, ObserveExtractErrorKind, ObserveExtractExtraction, ObserveExtractExtractionInput, ObserveExtractObservation, ObserveExtractObservationIdentity, ObserveExtractOutcome, ObserveExtractProviderFailure, ObserveExtractRecorder, ObserveExtractResult, ObserveExtractSource, ObserveExtractSourceSnapshot, } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";
 export type { BuildSemanticReviewWorkInput, SemanticClaimTarget, SemanticObservationIdentity, SemanticReviewCandidate, SemanticReviewChange, SemanticReviewItem, SemanticReviewKind, SemanticReviewWork, } from "./semantic-review-work.js";

package/dist/src/index.js

@@ -12,5 +12,5 @@ export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js
 export { createObservationStore } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
-export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export { createObserveExtractDiff, extractedSnapshotRef } from "./observe-extract-diff.js";
 export { buildSemanticReviewWork, semanticReviewApiVersion } from "./semantic-review-work.js";

package/dist/src/observation-admission.d.ts

@@ -1,6 +1,6 @@
 import type { SnapshotStore } from "@kontourai/forage";
 import type { LookoutSource } from "./registry.js";
-import type { StoredProposalObservationV1, ObservationCheckAnchor } from "./observation-store.js";
+import type { StoredProposalObservation, ObservationCheckAnchor } from "./observation-store.js";
 import type { ProposalSetObservation } from "./proposal-diff.js";
 /** A resolved snapshot identity suitable for durable observation metadata. */
 export interface AdmittedSnapshotIdentity {
@@ -44,7 +44,7 @@ export interface AdmitProposalObservationInput {
     readonly current: ProposalSetObservation;
     readonly check: ObservationCheckAnchor;
     /** The already-selected observation-store record; admission never selects or persists continuity. */
-    readonly prior: StoredProposalObservationV1 | null;
+    readonly prior: StoredProposalObservation | null;
     /** Explicit capability: admission never chooses a snapshot root or storage implementation. */
     readonly snapshotStore: SnapshotStore;
 }

package/dist/src/observation-store.d.ts

@@ -21,6 +21,15 @@ export interface StoredProposalObservationV1 {
     readonly check: ObservationCheckAnchor;
     readonly proposals: readonly ExtractionProposal[];
 }
+/**
+ * Same fields as version 1. The observationId is computed over code-unit-ordered
+ * canonical JSON, so it does not depend on the host locale. New records are
+ * always version 2.
+ */
+export interface StoredProposalObservationV2 extends Omit<StoredProposalObservationV1, "version"> {
+    readonly version: 2;
+}
+export type StoredProposalObservation = StoredProposalObservationV1 | StoredProposalObservationV2;
 export type ObservationStoreErrorKind = "invalid-input" | "corrupt-state" | "continuity-conflict" | "io-error";
 export interface ObservationStoreError {
     readonly kind: ObservationStoreErrorKind;
@@ -36,8 +45,44 @@ export type ObservationStoreResult<T> = {
     readonly error: ObservationStoreError;
 };
 export interface ObservationStore {
-    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservationV1 | null>>;
-    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservationV1>>;
+    loadLatest(sourceId: string): Promise<ObservationStoreResult<StoredProposalObservation | null>>;
+    commit(input: ProposalObservationRecordInput, expectedPriorId: string | null): Promise<ObservationStoreResult<StoredProposalObservation>>;
+}
+/** Finite caller-controlled ceilings for a verified proposal-head read. */
+export interface VerifiedHeadLimits {
+    readonly maxEntries?: number;
+    readonly maxIndexBytes?: number;
+    readonly maxPointerBytes?: number;
+    readonly maxRecordBytes?: number;
+}
+export interface ProposalHeadWitnessV1 {
+    readonly kind: "lookout.proposal-head-witness/v1";
+    readonly version: 1;
+    readonly sourceId: string;
+    readonly observationId: string;
+    /** Opaque binding of this store scope and its bounded metadata-as-of state. */
+    readonly token: string;
+}
+export type VerifiedHeadRead = {
+    readonly kind: "verified";
+    readonly sourceId: string;
+    readonly observationId: string;
+    readonly snapshotRef: string;
+    readonly witness: ProposalHeadWitnessV1;
+} | {
+    readonly kind: "missing" | "unavailable" | "corrupt" | "unsupported";
+};
+export type HeadWitnessComparison = {
+    readonly kind: "matches";
+    readonly sourceId: string;
+    readonly observationId: string;
+} | {
+    readonly kind: "changed" | "missing" | "unavailable" | "corrupt" | "unsupported";
+};
+/** Additive concrete-store capability. Generic ObservationStore implementations need not provide it. */
+export interface VerifiedHeadObservationStore {
+    readVerifiedHead(sourceId: string, limits?: VerifiedHeadLimits): Promise<VerifiedHeadRead>;
+    compareHeadWitness(witness: ProposalHeadWitnessV1, limits?: VerifiedHeadLimits): Promise<HeadWitnessComparison>;
 }
 export interface ObservationStoreFaults {
     readonly beforeSerialize?: () => void;
@@ -47,9 +92,13 @@ export interface ObservationStoreFaults {
     readonly beforePointerRename?: () => void;
     readonly beforeDirectorySync?: (kind: "record" | "pointer") => void;
     readonly beforePrune?: () => void;
+    /** Test-only fence hooks; production callers do not supply them. */
+    readonly beforeHeadStrongLoad?: () => void;
+    readonly beforeHeadRecordRead?: () => void;
+    readonly beforeHeadAfterFence?: () => void;
 }
 export interface CreateObservationStoreOptions {
     readonly root?: string;
     readonly faults?: ObservationStoreFaults;
 }
-export declare function createObservationStore(options?: CreateObservationStoreOptions): ObservationStore;
+export declare function createObservationStore(options?: CreateObservationStoreOptions): ObservationStore & VerifiedHeadObservationStore;

package/dist/src/observation-store.js

@@ -1,18 +1,223 @@
 import { createHash } from "node:crypto";
-import { lstat, mkdir, open, readFile, readdir, rename, rm, unlink } from "node:fs/promises";
+import { constants } from "node:fs";
+import { lstat, mkdir, open, opendir, readFile, readdir, realpath, rename, rm, unlink } from "node:fs/promises";
 import path from "node:path";
+import { types } from "node:util";
+import { canonicalJson, compareCodeUnits } from "./canonical-json.js";
+const VERIFIED_HEAD_MAX = Object.freeze({ maxEntries: 8, maxIndexBytes: 8 * 1024, maxPointerBytes: 8 * 1024, maxRecordBytes: 1024 * 1024 });
 function sourceKey(sourceId) {
     return `${encodeURIComponent(sourceId).replaceAll("%", "_").slice(0, 48)}-${createHash("sha256").update(sourceId).digest("hex").slice(0, 16)}`;
 }
-function stable(value) {
+function canonical(value) { return `${canonicalJson(value)}\n`; }
+function digest(body) {
+    return createHash("sha256").update(body.version === 1 ? legacyCanonical(body) : canonical(body)).digest("hex");
+}
+// Version 1 digests only. Version 1 records were hashed with keys sorted by the
+// host locale and then rebuilt with Object.fromEntries, which moves integer-like
+// keys first. A version 1 record therefore verifies only under a locale that
+// collates its keys the way the writing host did. Do not use this for new records.
+function legacyStable(value) {
     if (Array.isArray(value))
-        return value.map(stable);
+        return value.map(legacyStable);
     if (value && typeof value === "object")
-        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, stable(item)]));
+        return Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => [key, legacyStable(item)]));
     return value;
 }
-function canonical(value) { return `${JSON.stringify(stable(value))}\n`; }
-function digest(body) { return createHash("sha256").update(canonical(body)).digest("hex"); }
+function legacyCanonical(value) { return `${JSON.stringify(legacyStable(value))}\n`; }
+function resolveHeadLimits(limits) {
+    try {
+        if (limits !== undefined && (!limits || typeof limits !== "object" || types.isProxy(limits) || Array.isArray(limits)))
+            return null;
+    }
+    catch {
+        return null;
+    }
+    let supplied;
+    try {
+        const descriptors = limits === undefined ? {} : Object.getOwnPropertyDescriptors(limits);
+        if (Object.keys(descriptors).some((key) => !Object.hasOwn(VERIFIED_HEAD_MAX, key) || descriptors[key]?.get || descriptors[key]?.set))
+            return null;
+        supplied = Object.fromEntries(Object.entries(descriptors).map(([key, descriptor]) => [key, descriptor.value]));
+    }
+    catch {
+        return null;
+    }
+    const resolved = { ...VERIFIED_HEAD_MAX };
+    for (const key of Object.keys(VERIFIED_HEAD_MAX)) {
+        const value = supplied[key];
+        if (value === undefined)
+            continue;
+        if (typeof value !== "number" || !Number.isSafeInteger(value) || value <= 0 || value > VERIFIED_HEAD_MAX[key])
+            return null;
+        resolved[key] = value;
+    }
+    return resolved;
+}
+async function lstatHead(file) { return lstat(file, { bigint: true }); }
+function identity(stats) {
+    return { dev: String(stats.dev), ino: String(stats.ino), mode: String(stats.mode), size: String(stats.size), ctimeNs: String(stats.ctimeNs), mtimeNs: String(stats.mtimeNs) };
+}
+function sameIdentity(left, right) { return canonical(left) === canonical(right); }
+function headToken(rootRealpath, sourceId, metadata) {
+    return createHash("sha256").update(canonical({ kind: "lookout.proposal-head-witness/v1", version: 1, rootRealpath, sourceId, metadata })).digest("hex");
+}
+function errno(cause) { return cause?.code; }
+function boundedSourceId(sourceId) {
+    return typeof sourceId === "string" && sourceId.length > 0 && sourceId.length <= 256 && Buffer.byteLength(sourceId) <= 512;
+}
+async function boundedText(file, maximum) {
+    let handle;
+    try {
+        const before = await lstatHead(file);
+        if (before.isSymbolicLink() || !before.isFile())
+            return { kind: "corrupt" };
+        if (before.size > BigInt(maximum))
+            return { kind: "unavailable" };
+        handle = await open(file, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0));
+        const opened = await handle.stat({ bigint: true });
+        if (!opened.isFile() || !sameIdentity(identity(before), identity(opened)) || opened.size > BigInt(maximum))
+            return { kind: "unavailable" };
+        const buffer = Buffer.alloc(Number(opened.size) + 1);
+        const { bytesRead } = await handle.read(buffer, 0, buffer.length, 0);
+        const after = await handle.stat({ bigint: true });
+        if (!sameIdentity(identity(opened), identity(after)) || bytesRead !== Number(opened.size) || bytesRead > maximum)
+            return { kind: "unavailable" };
+        return { kind: "ok", text: buffer.subarray(0, bytesRead).toString("utf8"), identity: identity(after) };
+    }
+    catch (cause) {
+        return errno(cause) === "ENOENT" ? { kind: "unavailable" } : { kind: "unavailable" };
+    }
+    finally {
+        await handle?.close().catch(() => undefined);
+    }
+}
+async function inspectHead(root, sourceId, limits) {
+    if (!boundedSourceId(sourceId))
+        return { kind: "unsupported" };
+    let key;
+    try {
+        key = sourceKey(sourceId);
+    }
+    catch {
+        return { kind: "unsupported" };
+    }
+    let rootStats;
+    try {
+        rootStats = await lstatHead(root);
+    }
+    catch (cause) {
+        return errno(cause) === "ENOENT" ? { kind: "missing" } : { kind: "unavailable" };
+    }
+    if (rootStats.isSymbolicLink() || !rootStats.isDirectory())
+        return { kind: "corrupt" };
+    let rootRealpath;
+    try {
+        rootRealpath = await realpath(root);
+    }
+    catch {
+        return { kind: "unavailable" };
+    }
+    const dir = path.join(root, key);
+    let sourceStats;
+    try {
+        sourceStats = await lstatHead(dir);
+    }
+    catch (cause) {
+        return errno(cause) === "ENOENT" ? { kind: "missing" } : { kind: "unavailable" };
+    }
+    if (sourceStats.isSymbolicLink() || !sourceStats.isDirectory())
+        return { kind: "corrupt" };
+    const names = [];
+    let indexBytes = 0;
+    try {
+        const directory = await opendir(dir, { bufferSize: 1 });
+        try {
+            for await (const entry of directory) {
+                if (names.length >= limits.maxEntries)
+                    return { kind: "unavailable" };
+                indexBytes += Buffer.byteLength(entry.name);
+                if (indexBytes > limits.maxIndexBytes)
+                    return { kind: "unavailable" };
+                if (entry.name === ".lock" || entry.name.includes(".tmp-"))
+                    return { kind: "unavailable" };
+                if (entry.name !== "latest.json" && !/^[a-f0-9]{64}\.json$/.test(entry.name))
+                    return { kind: "corrupt" };
+                const entryStats = await lstatHead(path.join(dir, entry.name));
+                if (entryStats.isSymbolicLink() || !entryStats.isFile())
+                    return { kind: "corrupt" };
+                names.push(entry.name);
+            }
+        }
+        finally {
+            await directory.close().catch(() => undefined);
+        }
+    }
+    catch {
+        return { kind: "unavailable" };
+    }
+    names.sort();
+    if (!names.includes("latest.json"))
+        return names.length === 0 ? { kind: "missing" } : { kind: "corrupt" };
+    const pointerRead = await boundedText(path.join(dir, "latest.json"), limits.maxPointerBytes);
+    if (pointerRead.kind !== "ok")
+        return pointerRead;
+    let pointer;
+    try {
+        pointer = JSON.parse(pointerRead.text);
+    }
+    catch {
+        return { kind: "corrupt" };
+    }
+    if (!pointer || typeof pointer !== "object" || Array.isArray(pointer) || pointer.version !== 1 || pointer.sourceId !== sourceId || typeof pointer.observationId !== "string" || !/^[a-f0-9]{64}$/.test(pointer.observationId))
+        return { kind: "corrupt" };
+    const recordName = `${pointer.observationId}.json`;
+    if (!names.includes(recordName))
+        return { kind: "corrupt" };
+    let recordStats;
+    try {
+        recordStats = await lstatHead(path.join(dir, recordName));
+    }
+    catch {
+        return { kind: "unavailable" };
+    }
+    if (recordStats.isSymbolicLink() || !recordStats.isFile())
+        return { kind: "corrupt" };
+    if (recordStats.size > BigInt(limits.maxRecordBytes))
+        return { kind: "unavailable" };
+    let finalRoot;
+    let finalSource;
+    try {
+        finalRoot = await lstatHead(root);
+        finalSource = await lstatHead(dir);
+    }
+    catch {
+        return { kind: "unavailable" };
+    }
+    if (!sameIdentity(identity(rootStats), identity(finalRoot)) || !sameIdentity(identity(sourceStats), identity(finalSource)))
+        return { kind: "unavailable" };
+    return { kind: "ready", metadata: { root: identity(rootStats), rootRealpath, source: identity(sourceStats), names, pointer: pointerRead.text, pointerIdentity: pointerRead.identity, observationId: pointer.observationId, record: identity(recordStats) } };
+}
+function snapshotWitness(value) {
+    try {
+        if (!value || typeof value !== "object" || Array.isArray(value) || types.isProxy(value))
+            return { kind: "corrupt" };
+        const descriptors = Object.getOwnPropertyDescriptors(value);
+        const keys = ["kind", "version", "sourceId", "observationId", "token"];
+        if (Object.keys(descriptors).length !== keys.length || keys.some((key) => !Object.hasOwn(descriptors, key) || descriptors[key]?.get || descriptors[key]?.set))
+            return { kind: "corrupt" };
+        const witness = Object.fromEntries(keys.map((key) => [key, descriptors[key].value]));
+        if (witness.kind !== "lookout.proposal-head-witness/v1" || typeof witness.version !== "number")
+            return { kind: "unsupported" };
+        if (witness.version !== 1)
+            return { kind: "unsupported" };
+        if (!boundedSourceId(witness.sourceId) || typeof witness.observationId !== "string" || !/^[a-f0-9]{64}$/.test(witness.observationId) || typeof witness.token !== "string" || !/^[a-f0-9]{64}$/.test(witness.token))
+            return { kind: "corrupt" };
+        return { kind: "ok", witness: { kind: witness.kind, version: witness.version, sourceId: witness.sourceId, observationId: witness.observationId, token: witness.token } };
+    }
+    catch {
+        return { kind: "corrupt" };
+    }
+}
 function buildRecord(input) {
     try {
         const { observation, check } = input;
@@ -22,8 +227,8 @@ function buildRecord(input) {
         if (!check || check.currentSnapshotRef !== observation.snapshotRef || typeof check.checkedAt !== "string" || (check.resultKind !== "changed" && check.resultKind !== "unchanged-hash") || typeof input.recordedAt !== "string") {
             return { ok: false, error: { kind: "invalid-input", message: "Check anchor must match the current observation snapshot" } };
         }
-        const proposals = [...observation.proposals].sort((a, b) => canonical(a).localeCompare(canonical(b)));
-        const body = { version: 1, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
+        const proposals = [...observation.proposals].sort((a, b) => compareCodeUnits(canonical(a), canonical(b)));
+        const body = { version: 2, sourceKey: sourceKey(observation.sourceId), sourceId: observation.sourceId, snapshotRef: observation.snapshotRef, observedAt: observation.observedAt, recordedAt: input.recordedAt, check, proposals };
         try {
             return { ok: true, value: { ...body, observationId: digest(body) } };
         }
@@ -52,7 +257,7 @@ function validate(value, expectedSourceId) {
     if (!value || typeof value !== "object" || Array.isArray(value))
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation is not an object" } };
     const item = value;
-    if (item.version !== 1 || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
+    if ((item.version !== 1 && item.version !== 2) || item.sourceId !== expectedSourceId || item.sourceKey !== sourceKey(expectedSourceId) || typeof item.observationId !== "string" || !/^[a-f0-9]{64}$/.test(item.observationId) || typeof item.snapshotRef !== "string" || item.snapshotRef === "" || typeof item.observedAt !== "string" || item.observedAt === "" || typeof item.recordedAt !== "string" || item.recordedAt === "" || !Array.isArray(item.proposals) || item.proposals.some((proposal) => !validProposal(proposal)) || !item.check || typeof item.check !== "object" || typeof item.check.checkedAt !== "string" || item.check.checkedAt === "" || (item.check.resultKind !== "changed" && item.check.resultKind !== "unchanged-hash") || item.check.currentSnapshotRef !== item.snapshotRef) {
         return { ok: false, error: { kind: "corrupt-state", message: "Stored observation schema or continuity is invalid" } };
     }
     const { observationId, ...body } = item;
@@ -109,7 +314,7 @@ async function atomicWrite(file, bytes, kind, faults) {
     }
 }
 export function createObservationStore(options = {}) {
-    const root = options.root ?? path.join(process.cwd(), ".kontourai", "lookout", "observations");
+    const root = path.resolve(options.root ?? path.join(process.cwd(), ".kontourai", "lookout", "observations"));
     async function loadLatest(sourceId) {
         try {
             const dir = path.join(root, sourceKey(sourceId));
@@ -219,7 +424,7 @@ export function createObservationStore(options = {}) {
                     return null;
                 } }).filter((item) => item !== null);
                 const preserve = new Set([record.observationId, expectedPriorId].filter((item) => item !== null));
-                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => b.recordedAt.localeCompare(a.recordedAt) || b.id.localeCompare(a.id));
+                const extras = valid.filter((item) => !preserve.has(item.id)).sort((a, b) => compareCodeUnits(b.recordedAt, a.recordedAt) || compareCodeUnits(b.id, a.id));
                 await Promise.all(extras.map((item) => rm(path.join(dir, item.name), { force: true })));
             }
             catch (cause) {
@@ -237,5 +442,53 @@ export function createObservationStore(options = {}) {
             await rm(lockPath, { force: true }).catch(() => undefined);
         }
     }
-    return { loadLatest, commit };
+    async function readVerifiedHead(sourceId, suppliedLimits) {
+        const limits = resolveHeadLimits(suppliedLimits);
+        if (limits === null)
+            return { kind: "unsupported" };
+        const before = await inspectHead(root, sourceId, limits);
+        if (before.kind !== "ready")
+            return before;
+        // This is the only body read in the witness API. It reuses the concrete store's
+        // authenticated record validator and is bracketed by metadata fences below.
+        options.faults?.beforeHeadStrongLoad?.();
+        options.faults?.beforeHeadRecordRead?.();
+        const record = await boundedText(path.join(root, sourceKey(sourceId), `${before.metadata.observationId}.json`), limits.maxRecordBytes);
+        if (record.kind !== "ok")
+            return record;
+        let loaded;
+        try {
+            loaded = validate(JSON.parse(record.text), sourceId);
+        }
+        catch {
+            return { kind: "corrupt" };
+        }
+        if (!loaded.ok || loaded.value.observationId !== before.metadata.observationId)
+            return { kind: "corrupt" };
+        options.faults?.beforeHeadAfterFence?.();
+        const after = await inspectHead(root, sourceId, limits);
+        if (after.kind !== "ready")
+            return after;
+        if (canonical(before.metadata) !== canonical(after.metadata))
+            return { kind: "unavailable" };
+        const token = headToken(before.metadata.rootRealpath, sourceId, before.metadata);
+        return { kind: "verified", sourceId, observationId: loaded.value.observationId, snapshotRef: loaded.value.snapshotRef, witness: { kind: "lookout.proposal-head-witness/v1", version: 1, sourceId, observationId: loaded.value.observationId, token } };
+    }
+    async function compareHeadWitness(witness, suppliedLimits) {
+        const captured = snapshotWitness(witness);
+        if (captured.kind !== "ok")
+            return captured;
+        const limits = resolveHeadLimits(suppliedLimits);
+        if (limits === null)
+            return { kind: "unsupported" };
+        const inspected = await inspectHead(root, captured.witness.sourceId, limits);
+        if (inspected.kind !== "ready")
+            return inspected;
+        if (inspected.metadata.observationId !== captured.witness.observationId)
+            return { kind: "changed" };
+        return headToken(inspected.metadata.rootRealpath, captured.witness.sourceId, inspected.metadata) === captured.witness.token
+            ? { kind: "matches", sourceId: captured.witness.sourceId, observationId: captured.witness.observationId }
+            : { kind: "changed" };
+    }
+    return { loadLatest, commit, readVerifiedHead, compareHeadWitness };
 }

package/dist/src/observe-extract-diff.d.ts

@@ -64,7 +64,24 @@ export interface ObserveExtractObservationIdentity {
  */
 export interface ObserveExtractRecorder {
     record(observation: ObserveExtractObservation): Promise<ObserveExtractObservationIdentity>;
+    /**
+     * The snapshot reference of this source's most recent recorded observation
+     * for which `extractedSnapshotRef(observation)` is not null, or `null` when
+     * there is none.
+     *
+     * An unchanged check is only skipped when its capture has the same URL and
+     * body hash as this snapshot. Otherwise a change that acquisition already
+     * persisted, but that was never extracted (for example because a provider
+     * failed), would be reported as unchanged forever.
+     */
+    lastExtractedSnapshotRef(source: ObserveExtractSource): Promise<string | null>;
 }
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export declare function extractedSnapshotRef(observation: ObserveExtractObservation): string | null;
 export interface ObserveExtractDiffOptions {
     readonly acquisition: ObserveExtractAcquisition;
     readonly extraction: ObserveExtractExtraction;

package/dist/src/observe-extract-diff.js

@@ -1,4 +1,14 @@
 import { validatePreparedArtifact } from "@kontourai/traverse";
+import { parseSnapshotSourceRef } from "@kontourai/forage/fetch";
+/**
+ * The snapshot an observation's extraction fully handled: the current snapshot
+ * of a `completed`, `partial`, or `unchanged` observation, else `null`.
+ * Recorders use this to answer `lastExtractedSnapshotRef`.
+ */
+export function extractedSnapshotRef(observation) {
+    const handled = observation.outcome === "completed" || observation.outcome === "partial" || observation.outcome === "unchanged";
+    return handled && observation.sourceSnapshot !== null ? observation.sourceSnapshot.currentSnapshotRef : null;
+}
 export function createObserveExtractDiff(options) {
     return {
         async observe(source) {
@@ -19,10 +29,26 @@ export function createObserveExtractDiff(options) {
                 if (check.kind === "error") {
                     return record(options.recorder, baseObservation(source, check, "acquisition-error", null, null, null, null));
                 }
+                let sourceSnapshot = snapshotFor(check);
                 if (check.kind === "unchanged-304" || check.kind === "unchanged-hash") {
-                    return record(options.recorder, baseObservation(source, check, "unchanged", snapshotFor(check), null, null, null));
+                    // "Unchanged" compares with the latest stored capture, which may never
+                    // have been extracted. Skip extraction only when the capture matches
+                    // the last one that was; otherwise extract it against that baseline.
+                    let extracted;
+                    try {
+                        extracted = await options.recorder.lastExtractedSnapshotRef(observationSource(source));
+                    }
+                    catch (cause) {
+                        return { ok: false, error: error("recording-failed", "Observation recorder could not report the last extracted snapshot", cause) };
+                    }
+                    if (extracted !== null && (typeof extracted !== "string" || extracted === "")) {
+                        return { ok: false, error: error("dependency-contract", "Observation recorder returned an invalid last extracted snapshot reference") };
+                    }
+                    if (extracted !== null && sameCapture(extracted, sourceSnapshot.currentSnapshotRef)) {
+                        return record(options.recorder, baseObservation(source, check, "unchanged", sourceSnapshot, null, null, null));
+                    }
+                    sourceSnapshot = { priorSnapshotRef: extracted, currentSnapshotRef: sourceSnapshot.currentSnapshotRef };
                 }
-                const sourceSnapshot = snapshotFor(check);
                 let extraction;
                 try {
                     extraction = await options.extraction.extract({ source, snapshotRef: sourceSnapshot.currentSnapshotRef });
@@ -65,7 +91,7 @@ export function createObserveExtractDiff(options) {
 }
 function baseObservation(source, check, outcome, sourceSnapshot, preparedArtifact, proposalSet, attempt) {
     return {
-        source: { id: source.id, url: source.url, kind: source.kind },
+        source: observationSource(source),
         check,
         outcome,
         sourceSnapshot,
@@ -93,6 +119,18 @@ async function record(recorder, observation) {
         return { ok: false, error: error("recording-failed", "Observation recorder failed", cause), observation };
     }
 }
+function observationSource(source) {
+    return { id: source.id, url: source.url, kind: source.kind };
+}
+/** Two references name the same capture content: same source, resource URL, and body hash. */
+function sameCapture(left, right) {
+    if (left === right)
+        return true;
+    const a = parseSnapshotSourceRef(left);
+    const b = parseSnapshotSourceRef(right);
+    return a !== undefined && b !== undefined &&
+        a.sourceId === b.sourceId && a.url === b.url && a.bodyHash === b.bodyHash;
+}
 function snapshotFor(check) {
     if (check.kind === "unchanged-304")
         return { priorSnapshotRef: check.snapshotRef, currentSnapshotRef: check.snapshotRef };

package/dist/src/proposal-diff.js

@@ -1,5 +1,6 @@
 import { canonicalValueKey, } from "./canonical-value.js";
 import { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
+import { compareCodeUnits } from "./canonical-json.js";
 function callbackError(label, cause) {
     return { kind: "callback-threw", message: `${label} callback threw`, cause };
 }
@@ -90,7 +91,7 @@ function semanticFieldOrder(items, fieldIdentity) {
             return encoded;
         groups.set(fieldKey.value, [...(groups.get(fieldKey.value) ?? []), { item, key: encoded.key, index }]);
     }
-    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => left.key.localeCompare(right.key) || left.index - right.index));
+    const ordered = [...groups.values()].flatMap((group) => group.sort((left, right) => compareCodeUnits(left.key, right.key) || left.index - right.index));
     return { ok: true, value: ordered.map(({ item }) => item) };
 }
 export function diffProposalSets(input) {

package/dist/src/snapshot-store.d.ts

@@ -1,6 +1,14 @@
 import type { SnapshotStore } from "@kontourai/forage";
 import { type SnapshotSourceRefResolution } from "@kontourai/forage/fetch";
-export declare function createLookoutSnapshotStore(root?: string): SnapshotStore;
+export interface LookoutSnapshotStoreOptions {
+    /**
+     * Per-source record ceiling, passed to Forage's filesystem store (1 to
+     * 10,000; Forage's default is 10,000). It cannot change after a source's
+     * store directory is initialized.
+     */
+    readonly maxHistoryFiles?: number;
+}
+export declare function createLookoutSnapshotStore(root?: string, options?: LookoutSnapshotStoreOptions): SnapshotStore;
 export type ResolveLookoutSnapshotOptions = {
     store: SnapshotStore;
     root?: never;

package/dist/src/snapshot-store.js

@@ -1,8 +1,11 @@
 import path from "node:path";
 import { createFilesystemSnapshotStore } from "@kontourai/forage";
 import { resolveSnapshotSourceRef, } from "@kontourai/forage/fetch";
-export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots")) {
-    return createFilesystemSnapshotStore({ root });
+export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots"), options = {}) {
+    return createFilesystemSnapshotStore({
+        root,
+        ...(options.maxHistoryFiles === undefined ? {} : { maxHistoryFiles: options.maxHistoryFiles }),
+    });
 }
 /** Replay one Lookout-emitted durable reference without any network access. */
 export async function resolveLookoutSnapshot(reference, options = {}) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kontourai/lookout",
-  "version": "0.5.1",
+  "version": "0.6.0",
   "description": "A small source registry and drift-check runner built on Forage snapshots.",
   "license": "Apache-2.0",
   "type": "module",
@@ -54,9 +54,13 @@
     "@kontourai/flow-agents": "5.9.0",
     "@kontourai/survey": "3.0.0",
     "@types/node": "^26.1.2",
+    "ajv": "8.20.0",
+    "ajv-formats": "3.0.1",
+    "hachure": "0.15.0",
     "typescript": "^7.0.2"
   },
   "engines": {
     "node": ">=22"
-  }
+  },
+  "packageManager": "pnpm@11.25.0"
 }