@kontourai/lookout 0.3.0 → 0.3.1

package/README.md

@@ -3,12 +3,12 @@
 **Cheap, honest drift detection for content you re-check over time — "did this source change since we last looked?"**
 
 A small **source registry** and **non-throwing drift-check runner** built on
-[Traverse](https://github.com/kontourai/traverse) snapshots. It composes Traverse
-for fetching and snapshot storage, provides deterministic proposal diffing, and
+[Forage](https://github.com/kontourai/forage) snapshots. It composes Forage for
+fetching and snapshot storage, provides deterministic proposal diffing, and
 emits neutral, typed drift — its own vocabulary, in its own dependency-free
-package. It does **not** extract fields, author trust-layer records, project to
-Surface, review claims, notify, crawl, or schedule. Operational failures are
-returned as typed results.
+package. It does **not** implement acquisition or extraction, author trust-layer
+records, project to Surface, review claims, notify, crawl, or schedule.
+Operational failures are returned as typed results.
 
 ## Why it's different
 
@@ -21,7 +21,7 @@ as "no change"). Lookout is the opposite on all three:
 |---|---|---|
 | Cost | full re-download every check | **conditional `304`** — often no download at all |
 | Signal | raw bytes (ads/timestamps = "changed") | **proposal-identity diff** — "a *new entity appeared*" vs "a byte moved" |
-| Honesty | a crash or false-`304` silently reads as "unchanged" | **typed results, never throws**; hardened against false-`304` (the [traverse#49](https://github.com/kontourai/traverse/issues/49) validator-scoping pin) |
+| Honesty | a crash or false-`304` silently reads as "unchanged" | **typed results, never throws**; Forage scopes validators to the exact prior resource and Lookout verifies the returned capture identity |
 | Output | your problem to shape | **neutral, typed drift** — events already Hachure-evidence-shaped, ready for a consumer to lift into a trust bundle |
 
 So the point isn't "diffing" — it's *cheap + honest + semantic + review-ready*
@@ -37,8 +37,7 @@ One layer in a four-verb stack; each repo owns one verb and is usable alone:
 - **lookout** — CHANGE: did this registered source drift since last look?
 - **survey** — the SHAPE: what reviewed truth looks like (claims / review / resolution)
 
-Lookout composes the fetch/snapshot layer (today Traverse's `/fetch`; re-points at
-`forage` as its single-source fetch surface lands) for cheap `304`-aware re-checks,
+Lookout composes Forage's `/fetch` surface for cheap `304`-aware re-checks,
 and Traverse's `ExtractionProposal` identity for the semantic diff. Dependency
 arrows point only downward — no cycles. Lookout itself depends on nothing in the
 trust layer: its events are already Hachure-evidence-shaped (`snapshotRef` /
@@ -58,12 +57,11 @@ owns its transport and opts out of the default guard.
 ## Requirements
 
 - Node.js `>= 22`.
-- `@kontourai/traverse` `>= 0.14.1`. This is a hard floor, not a preference:
-  trustworthy `unchanged-304` classification depends on the validator-scoping fix
-  from [traverse#49](https://github.com/kontourai/traverse/issues/49), first
-  released in `0.14.1`. Lookout exact-pins `0.14.1`. On an earlier Traverse a
-  conditional request could reuse a prior snapshot's validators against a
-  different URL and report a **false** `304`, so do not downgrade.
+- `@kontourai/forage` `>= 0.4.0`. Exact durable-reference replay requires the
+  resolver first released in `0.4.0`; trustworthy `unchanged-304`
+  classification also depends on Forage's validator-scoped revalidation.
+  Lookout exact-pins Traverse separately for schema and extraction-proposal
+  contracts.
 
 ## Registry
 
@@ -81,6 +79,13 @@ override with the library path argument or the CLI `--registry` flag):
       "targetSchema": [{ "path": "title", "type": "string", "required": true }],
       "cadenceHint": "daily",
       "renderPolicy": "never"
+    },
+    {
+      "id": "published-results",
+      "kind": "structured-file",
+      "format": "yaml",
+      "url": "https://raw.githubusercontent.com/example/project/0123456789abcdef0123456789abcdef01234567/results.yml",
+      "cadenceHint": "weekly"
     }
   ]
 }
@@ -91,11 +96,17 @@ Fields per source:
 | Field | Meaning |
 | --- | --- |
 | `id` | Non-empty, unique within the document. Used for exact lookup and snapshot identity. |
-| `kind` | `web-page` or `api-record`. |
+| `kind` | `web-page`, `api-record`, or `structured-file`. |
 | `url` | Absolute HTTP(S) URL. |
-| `targetSchema` | Traverse `TargetFieldSchema[]`. Stored inert at L1 (no extraction yet). |
+| `format` | Required only for `structured-file`: `yaml`, `json`, or `csv`. Lookout does not parse it. |
+| `targetSchema` | Required for `web-page` and `api-record`: Traverse `TargetFieldSchema[]`. Stored inert at L1 (no extraction yet). |
 | `cadenceHint` | Non-empty string; advisory only — Lookout does not schedule. |
-| `renderPolicy` | `never` \| `on-shell-warning` \| `always`. **Inert** at L1 (never mapped to a fetch/render policy). |
+| `renderPolicy` | Required for `web-page` and `api-record`: `never` \| `on-shell-warning` \| `always`. **Inert** at L1. |
+
+Structured-file entries retain raw fetched bytes and use the same guarded fetch,
+snapshot, comparison, and replay path as every other source. Parsing and domain
+normalization stay downstream. Prefer immutable, commit-pinned artifact URLs so
+an upstream branch move cannot silently redefine the cited source location.
 
 Validation reports **every** deterministic issue at once (with index/id context)
 and never reads the network. Lookup is exact-id; listing preserves file order —
@@ -104,35 +115,39 @@ no merging, discovery, or remote registries.
 ## Check results
 
 A check classifies each source into exactly one of four kinds. Every result
-carries `sourceId`, the registered `sourceUrl`, `checkedAt`, and Traverse
-`warnings`.
+carries `sourceId`, the registered `sourceUrl`, `checkedAt`, and Forage fetch
+warnings.
 
 | `kind` | When | Extra fields |
 | --- | --- | --- |
 | `unchanged-304` | A validator-backed conditional request returned `304`; zero body transfer; the prior snapshot is not re-persisted. | `snapshotRef` |
 | `unchanged-hash` | A full body was fetched and persisted, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. | `priorSnapshotRef`, `currentSnapshotRef` |
 | `changed` | The fresh body differs from the prior (`changeBasis: "hash"`), **or** it is the first successful observation (`changeBasis: "initial"`, `priorSnapshotRef: null`). | `priorSnapshotRef` (nullable), `currentSnapshotRef`, `changeBasis` |
-| `error` | Any operational failure — contained so the runner never rejects. | `origin` (`traverse` \| `lookout`), `error` |
+| `error` | Any operational failure — contained so the runner never rejects. | `origin` (`forage` \| `lookout`), `error` |
 
 `unchanged-hash` asserts same-resource continuity, so it requires **both** an
 identical `bodyHash` **and** the same resource URL as the prior snapshot. If a
 source has moved — the prior snapshot's URL differs from this fetch's — the
 result is `changed` even when the bytes are byte-identical, and the fresh
 snapshot is persisted as the new baseline. (The `unchanged-304` path is already
-resource-scoped by Traverse's validators, so only the hash path needs this
+resource-scoped by Forage's validators, so only the hash path needs this
 guard.)
 
-`error` results preserve provenance: `origin: "traverse"` carries Traverse's
+`error` results preserve provenance: `origin: "forage"` carries Forage's
 discriminated `FetchError` verbatim (its `kind`, and `status` when present);
 `origin: "lookout"` carries a typed `kind` — `prior-read`, `persistence`,
 `dependency-contract`, or `unexpected`.
 
 Snapshot references (`snapshotRef` / `priorSnapshotRef` / `currentSnapshotRef`)
-are portable logical refs from Traverse's `buildSnapshotSourceRef` — never
-filesystem paths. Snapshots are stored via Traverse's filesystem store, rooted by
+are portable logical refs from Forage's `buildSnapshotSourceRef` — never
+filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by
 default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
-custom filenames or retention.
+custom filenames or retention. `resolveLookoutSnapshot()` replays one exact
+reference through an injected store or Lookout snapshot root, authenticates its
+body and replay metadata, and never fetches. References emitted before Forage's
+replay-envelope digest remain resolvable with `integrity: "body-and-identity"`;
+new references report `integrity: "snapshot-envelope"`.
 
 ## Schema coverage
 
@@ -144,6 +159,7 @@ can't catch this. `checkSchemaCoverage` is the static complement:
 ```ts
 import { checkSchemaCoverage } from "@kontourai/lookout";
 
+if (source.kind === "structured-file") throw new Error("structured sources are parsed downstream");
 const { covered, gaps } = checkSchemaCoverage(source.targetSchema, proposals);
 // covered: declared paths at least one proposal produced (schema order)
 // gaps:    declared paths that produced none, each with its `required` flag
@@ -213,17 +229,72 @@ const results = await runner.checkAll(registry.list());
 ```
 
 `createCheckRunner` accepts injected seams — `store`, `fetchSource` (defaults to
-Traverse's fetcher over forage's SSRF-guarded egress), `fetchOptions`, and
+Forage's SSRF-guarded fetcher), `fetchOptions`, and
 `clock` — so checks run with no live network or timers in tests. Injecting either
 `fetchSource` or `fetchOptions.fetch` overrides the default guarded transport.
 
+## Observe changed sources without re-extracting unchanged ones
+
+`createObserveExtractDiff` is an optional library composition for callers that
+want one typed observation per check. Supply acquisition, extraction, and
+recording capabilities; Lookout does not choose a content-preparation method,
+provider, or observation database.
+
+```ts
+import { createObserveExtractDiff } from "@kontourai/lookout";
+
+const composition = createObserveExtractDiff({
+  acquisition: { check: runner.check },
+  extraction: {
+    async extract({ source, snapshotRef }) {
+      // Resolve `snapshotRef`, prepare content, and invoke a caller-selected
+      // extraction implementation. Return its public Traverse result.
+      return extractChangedSource(source, snapshotRef);
+    },
+  },
+  recorder: {
+    async record(observation) {
+      // Caller-owned continuity and durable storage.
+      return saveObservation(observation);
+    },
+  },
+});
+
+const result = await composition.observe(source);
+```
+
+`unchanged-304` and `unchanged-hash` are recorded as `unchanged` and never call
+the extraction capability, so they make zero preparation and provider calls.
+Changed observations retain source and snapshot references, Traverse's prepared
+artifact identity, the proposal set, and the current/prior observation
+identities returned by the recorder. `partial`, `provider-failure`, mixed
+`partial-provider-failure`, and `extraction-failure` remain distinct typed
+outcomes. Provider failures are reduced to provider-neutral `kind` and
+`retryable` classifications; provider names, messages, native diagnostics,
+free-form extraction warnings, raw responses, and thrown-error text are not
+copied into the durable observation.
+A first changed observation
+has `priorObservationId: null`; it is a baseline observation, not a fabricated
+list of additions or removals.
+
+The composition validates that the check identifies the requested registered
+source and that Traverse validates the prepared artifact whose snapshot anchor
+matches the current check. This prevents mismatched capability results from
+being recorded as one observation. Proposal excerpts, source URLs, and
+acquisition check warnings can still contain sensitive source material; callers
+must apply their own retention and access policy before durable storage.
+
+The existing `createCheckRunner` and `createDriftEmitter` entrypoints remain
+available. Use `createDriftEmitter` when the caller wants deterministic
+proposal-diff events with its own identity callbacks.
+
 ## Non-goals
 
-Extraction, Surface projection, notifications, crawling, review/escalation or
-authority policy, rendered-fetch wiring
-(`renderPolicy` is inert pending traverse#50), and scheduling. Traverse retains
-all fetch politeness, robots, redirects, retries, timeouts, headers, user-agent,
-and rendered-fetch behavior.
+Acquisition and extraction implementations, Surface projection, notifications,
+crawling, review/escalation or authority policy, rendered-fetch wiring
+(`renderPolicy` is inert), and scheduling. Forage retains all fetch politeness,
+robots, redirects, retries, timeouts, headers, user-agent, and rendered-fetch
+behavior.
 
 ## Development
 

package/dist/src/check-result.d.ts

@@ -1,4 +1,4 @@
-import type { FetchError } from "@kontourai/traverse/fetch";
+import type { FetchError } from "@kontourai/forage/fetch";
 export interface CheckResultCommon {
     sourceId: string;
     sourceUrl: string;
@@ -23,7 +23,7 @@ export interface ChangedResult extends CheckResultCommon {
 export type LookoutErrorKind = "prior-read" | "persistence" | "dependency-contract" | "unexpected";
 export interface ErrorResult extends CheckResultCommon {
     kind: "error";
-    origin: "traverse" | "lookout";
+    origin: "forage" | "lookout";
     error: FetchError | {
         kind: LookoutErrorKind;
         message: string;

package/dist/src/check-runner.d.ts

@@ -1,4 +1,4 @@
-import type { FetchResult, FetchSourceOptions, SnapshotStore, SourceConfig } from "@kontourai/traverse/fetch";
+import type { FetchResult, FetchSourceOptions, SnapshotStore, SourceConfig } from "@kontourai/forage/fetch";
 import type { CheckResult } from "./check-result.js";
 import type { ProviderResolver } from "./provider-resolution.js";
 import type { LookoutSource } from "./registry.js";

package/dist/src/check-runner.js

@@ -1,30 +1,20 @@
-import { buildSnapshotSourceRef, fetchSource as traverseFetchSource, } from "@kontourai/traverse/fetch";
-import { createGuardedFetch } from "@kontourai/forage/egress";
+import { isDeepStrictEqual } from "node:util";
+import { buildSnapshotSourceRef, fetchSource as forageFetchSource, } from "@kontourai/forage/fetch";
 /**
- * The default egress transport for lookout's registered-source fetches: forage's
- * SSRF-pinned guarded fetch. Registered source URLs are operator- / aggregator-
- * supplied and not fully trusted, so a source pointing at a private, link-local,
- * loopback, or cloud-metadata host is refused before any connection — a drift
- * check can never be turned into an SSRF vector. Only wired when lookout uses the
- * default traverse fetcher and the caller injected no `fetch`; a caller that
- * supplies its own `fetchSource` or `fetchOptions.fetch` (e.g. tests) owns the
- * transport. Created lazily and cached so the guard is built once, not per check.
+ * The egress policy for lookout's registered-source fetches. Registered source
+ * URLs are operator- / aggregator-supplied and not fully trusted, so a source
+ * pointing at a private, link-local, loopback, or cloud-metadata host must be
+ * refused before any connection — a drift check can never be turned into an
+ * SSRF vector. `forage`'s `fetchSource` builds its own SSRF-pinned guarded
+ * transport from this policy whenever the caller doesn't inject a custom
+ * `fetch` (e.g. tests), so this is the single shared policy literal — never
+ * scattered per call site, never `{ guarded: false }`.
  */
-let cachedGuardedFetch;
-function defaultGuardedFetch() {
-    cachedGuardedFetch ??= createGuardedFetch();
-    return cachedGuardedFetch;
-}
+const GUARDED_EGRESS = { guarded: true };
 export function createCheckRunner(options) {
-    const usingDefaultFetcher = options.fetchSource === undefined;
-    const fetchImpl = options.fetchSource ?? traverseFetchSource;
+    const fetchImpl = options.fetchSource ?? forageFetchSource;
     const clock = options.clock ?? (() => new Date().toISOString());
-    // Guard the default traverse fetcher's egress with forage. A caller-supplied
-    // fetcher or `fetchOptions.fetch` owns its own transport and is left as-is.
     const fetchOptions = { ...options.fetchOptions };
-    if (usingDefaultFetcher && fetchOptions.fetch === undefined) {
-        fetchOptions.fetch = defaultGuardedFetch();
-    }
     async function check(source) {
         const common = () => ({
             sourceId: source.id,
@@ -41,17 +31,17 @@ export function createCheckRunner(options) {
         }
         let fetched;
         try {
-            fetched = await fetchImpl({ id: source.id, url: source.url, revalidate: true }, { ...fetchOptions, store: options.store });
+            fetched = await fetchImpl({ id: source.id, url: source.url, egress: GUARDED_EGRESS }, { ...fetchOptions, store: options.store });
         }
         catch (error) {
             return lookoutError(common(), "unexpected", error);
         }
         const base = { ...common(), warnings: Array.isArray(fetched?.warnings) ? [...fetched.warnings] : [] };
         if (!isFetchResult(fetched)) {
-            return lookoutError(base, "dependency-contract", "Traverse returned neither exactly one snapshot nor exactly one error");
+            return lookoutError(base, "dependency-contract", "Forage returned neither exactly one snapshot nor exactly one error");
         }
         if (fetched.error) {
-            return { ...base, kind: "error", origin: "traverse", error: fetched.error };
+            return { ...base, kind: "error", origin: "forage", error: fetched.error };
         }
         // Defense-in-depth: even if a future guard gap let a malformed snapshot
         // through, any stray throw in classification/ref-building becomes a typed
@@ -59,7 +49,22 @@ export function createCheckRunner(options) {
         try {
             const snapshot = fetched.snapshot;
             if (snapshot.notModified === true) {
-                return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(snapshot) };
+                const snapshotBodyEncoding = bodyEncoding(snapshot);
+                const priorBodyEncoding = prior === undefined ? undefined : bodyEncoding(prior);
+                if (prior === undefined ||
+                    snapshotBodyEncoding === undefined ||
+                    snapshotBodyEncoding !== priorBodyEncoding ||
+                    snapshot.sourceId !== prior.sourceId ||
+                    snapshot.url !== prior.url ||
+                    snapshot.status !== prior.status ||
+                    snapshot.fetchedAt !== prior.fetchedAt ||
+                    snapshot.bodyHash !== prior.bodyHash ||
+                    !isDeepStrictEqual(snapshot.headers, prior.headers) ||
+                    !isDeepStrictEqual(snapshot.redirects, prior.redirects) ||
+                    snapshot.rendered !== prior.rendered) {
+                    return lookoutError(base, "dependency-contract", "Forage returned a 304 snapshot without the matching prior capture");
+                }
+                return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(prior) };
             }
             try {
                 await options.store.put(snapshot);
@@ -99,6 +104,14 @@ export function createCheckRunner(options) {
     }
     return { check, checkAll };
 }
+function bodyEncoding(snapshot) {
+    const descriptor = Object.getOwnPropertyDescriptor(snapshot, "body");
+    if (descriptor === undefined || !("value" in descriptor))
+        return undefined;
+    if (typeof descriptor.value === "string")
+        return "utf8";
+    return descriptor.value instanceof Uint8Array ? "bytes" : undefined;
+}
 function isFetchResult(value) {
     if (typeof value !== "object" || value === null)
         return false;

package/dist/src/index.d.ts

@@ -6,8 +6,10 @@ export type { ChangedResult, CheckResult, CheckResultCommon, ErrorResult, Lookou
 export { defaultProviderResolver } from "./provider-resolution.js";
 export type { ProviderResolver } from "./provider-resolution.js";
 export { loadRegistry, LookoutRegistry, parseRegistry, RegistryValidationError, } from "./registry.js";
-export type { LookoutRegistryDocument, LookoutSource, LookoutSourceKind, RenderPolicy, } from "./registry.js";
-export { createLookoutSnapshotStore } from "./snapshot-store.js";
+export type { ExtractableLookoutSource, LookoutRegistryDocument, LookoutSource, LookoutSourceKind, RenderPolicy, StructuredFileFormat, StructuredFileLookoutSource, } from "./registry.js";
+export { createLookoutSnapshotStore, resolveLookoutSnapshot, } from "./snapshot-store.js";
+export type { ResolveLookoutSnapshotOptions } from "./snapshot-store.js";
+export type { SnapshotSourceRefResolution } from "@kontourai/forage/fetch";
 export { canonicalValueKey } from "./canonical-value.js";
 export type { CanonicalValueKey, CanonicalValueFormatVersion, CanonicalValueResult, DiffKernelError, DiffKernelErrorKind, DiffResult, IdentityResult, } from "./canonical-value.js";
 export { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
@@ -20,3 +22,5 @@ export { createDriftEmitter } from "./drift-emission.js";
 export type { BaselineEstablishedFact, CreateDriftEmitterOptions, DriftEmitter, DriftError, DriftErrorKind, DriftFact, DriftResult, DriftSuccess, EmitDriftInput } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
 export type { SchemaCoverageGap, SchemaCoverageResult } from "./coverage.js";
+export { createObserveExtractDiff } from "./observe-extract-diff.js";
+export type { ObserveExtractAcquisition, ObserveExtractAttempt, ObserveExtractDiff, ObserveExtractDiffOptions, ObserveExtractError, ObserveExtractErrorKind, ObserveExtractExtraction, ObserveExtractExtractionInput, ObserveExtractObservation, ObserveExtractObservationIdentity, ObserveExtractOutcome, ObserveExtractProviderFailure, ObserveExtractRecorder, ObserveExtractResult, ObserveExtractSource, ObserveExtractSourceSnapshot, } from "./observe-extract-diff.js";

package/dist/src/index.js

@@ -2,10 +2,11 @@ export { runCli } from "./cli.js";
 export { createCheckRunner } from "./check-runner.js";
 export { defaultProviderResolver } from "./provider-resolution.js";
 export { loadRegistry, LookoutRegistry, parseRegistry, RegistryValidationError, } from "./registry.js";
-export { createLookoutSnapshotStore } from "./snapshot-store.js";
+export { createLookoutSnapshotStore, resolveLookoutSnapshot, } from "./snapshot-store.js";
 export { canonicalValueKey } from "./canonical-value.js";
 export { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
 export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
 export { createDriftEmitter } from "./drift-emission.js";
 export { checkSchemaCoverage } from "./coverage.js";
+export { createObserveExtractDiff } from "./observe-extract-diff.js";

package/dist/src/observe-extract-diff.d.ts

@@ -0,0 +1,90 @@
+import type { ExtractionPartial, ExtractionProviderFailure, ExtractionResult, PreparedArtifact } from "@kontourai/traverse";
+import type { CheckResult } from "./check-result.js";
+import type { ProposalSetObservation } from "./proposal-diff.js";
+import type { LookoutSource } from "./registry.js";
+/** Acquisition is supplied by the caller; Lookout does not add another fetcher. */
+export interface ObserveExtractAcquisition {
+    check(source: LookoutSource): Promise<CheckResult>;
+}
+/**
+ * Extraction is supplied by the caller. The input carries only immutable source
+ * identity, leaving snapshot resolution, preparation, and provider selection
+ * outside Lookout.
+ */
+export interface ObserveExtractExtraction {
+    extract(input: ObserveExtractExtractionInput): Promise<ExtractionResult>;
+}
+export interface ObserveExtractExtractionInput {
+    readonly source: LookoutSource;
+    readonly snapshotRef: string;
+}
+export interface ObserveExtractAttempt {
+    readonly extractedAt: string;
+    readonly providerCalls: number;
+    readonly totalTokensUsed: number;
+    readonly partial?: ExtractionPartial;
+    readonly providerFailures?: readonly ObserveExtractProviderFailure[];
+}
+/** Provider-neutral durable classification; diagnostic payloads stay at the capability boundary. */
+export interface ObserveExtractProviderFailure {
+    readonly kind: ExtractionProviderFailure["kind"];
+    readonly retryable: boolean;
+}
+export interface ObserveExtractSourceSnapshot {
+    readonly priorSnapshotRef: string | null;
+    readonly currentSnapshotRef: string;
+}
+export interface ObserveExtractSource {
+    readonly id: string;
+    readonly url: string;
+    readonly kind: LookoutSource["kind"];
+}
+export type ObserveExtractOutcome = "acquisition-error" | "unchanged" | "completed" | "partial" | "partial-provider-failure" | "provider-failure" | "extraction-failure";
+/**
+ * One source observation, ready for caller-owned durable recording. Raw
+ * provider responses and source bodies are intentionally not copied here.
+ */
+export interface ObserveExtractObservation {
+    readonly source: ObserveExtractSource;
+    readonly check: CheckResult;
+    readonly outcome: ObserveExtractOutcome;
+    readonly sourceSnapshot: ObserveExtractSourceSnapshot | null;
+    readonly preparedArtifact: PreparedArtifact | null;
+    readonly proposalSet: ProposalSetObservation | null;
+    readonly attempt: ObserveExtractAttempt | null;
+}
+export interface ObserveExtractObservationIdentity {
+    readonly observationId: string;
+    readonly priorObservationId: string | null;
+}
+/**
+ * Lookout passes each completed observation to a caller-owned recorder. The
+ * recorder controls durable storage and continuity while this composition never
+ * supplies or configures the injected acquisition or extraction capabilities.
+ */
+export interface ObserveExtractRecorder {
+    record(observation: ObserveExtractObservation): Promise<ObserveExtractObservationIdentity>;
+}
+export interface ObserveExtractDiffOptions {
+    readonly acquisition: ObserveExtractAcquisition;
+    readonly extraction: ObserveExtractExtraction;
+    readonly recorder: ObserveExtractRecorder;
+}
+export type ObserveExtractErrorKind = "acquisition-threw" | "recording-failed" | "dependency-contract";
+export interface ObserveExtractError {
+    readonly kind: ObserveExtractErrorKind;
+    readonly message: string;
+    readonly cause?: unknown;
+}
+export type ObserveExtractResult = {
+    readonly ok: true;
+    readonly value: ObserveExtractObservation & ObserveExtractObservationIdentity;
+} | {
+    readonly ok: false;
+    readonly error: ObserveExtractError;
+    readonly observation?: ObserveExtractObservation;
+};
+export interface ObserveExtractDiff {
+    observe(source: LookoutSource): Promise<ObserveExtractResult>;
+}
+export declare function createObserveExtractDiff(options: ObserveExtractDiffOptions): ObserveExtractDiff;

package/dist/src/observe-extract-diff.js

@@ -0,0 +1,154 @@
+import { validatePreparedArtifact } from "@kontourai/traverse";
+export function createObserveExtractDiff(options) {
+    return {
+        async observe(source) {
+            try {
+                let check;
+                try {
+                    check = await options.acquisition.check(source);
+                }
+                catch (cause) {
+                    return { ok: false, error: error("acquisition-threw", "Acquisition capability threw", cause) };
+                }
+                if (!isCheckResult(check)) {
+                    return { ok: false, error: error("dependency-contract", "Acquisition capability returned an invalid check result") };
+                }
+                if (check.sourceId !== source.id || check.sourceUrl !== source.url) {
+                    return { ok: false, error: error("dependency-contract", "Acquisition result does not identify the requested source") };
+                }
+                if (check.kind === "error") {
+                    return record(options.recorder, baseObservation(source, check, "acquisition-error", null, null, null, null));
+                }
+                if (check.kind === "unchanged-304" || check.kind === "unchanged-hash") {
+                    return record(options.recorder, baseObservation(source, check, "unchanged", snapshotFor(check), null, null, null));
+                }
+                const sourceSnapshot = snapshotFor(check);
+                let extraction;
+                try {
+                    extraction = await options.extraction.extract({ source, snapshotRef: sourceSnapshot.currentSnapshotRef });
+                }
+                catch {
+                    return record(options.recorder, baseObservation(source, check, "extraction-failure", sourceSnapshot, null, null, null));
+                }
+                if (!isExtractionResult(extraction)) {
+                    return { ok: false, error: error("dependency-contract", "Extraction capability returned an invalid extraction result") };
+                }
+                const attempt = attemptFor(extraction);
+                const proposalSet = {
+                    sourceId: source.id,
+                    snapshotRef: sourceSnapshot.currentSnapshotRef,
+                    observedAt: extraction.extractedAt,
+                    proposals: extraction.proposals,
+                };
+                const outcome = outcomeFor(extraction);
+                if (outcome !== "extraction-failure") {
+                    if (extraction.preparedArtifact === undefined) {
+                        return { ok: false, error: error("dependency-contract", "Extraction result is missing its prepared artifact") };
+                    }
+                }
+                if (extraction.preparedArtifact !== undefined) {
+                    const validation = validatePreparedArtifact(extraction.preparedArtifact);
+                    if (validation.status !== "valid") {
+                        return { ok: false, error: error("dependency-contract", `Extraction result has an invalid prepared artifact: ${validation.status}`) };
+                    }
+                    if (validation.artifact.sourceSnapshotRef !== sourceSnapshot.currentSnapshotRef) {
+                        return { ok: false, error: error("dependency-contract", "Prepared artifact does not identify the current source snapshot") };
+                    }
+                }
+                return record(options.recorder, baseObservation(source, check, outcome, sourceSnapshot, extraction.preparedArtifact ?? null, proposalSet, attempt));
+            }
+            catch (cause) {
+                return { ok: false, error: error("dependency-contract", "Observe-extract composition could not inspect a capability result", cause) };
+            }
+        },
+    };
+}
+function baseObservation(source, check, outcome, sourceSnapshot, preparedArtifact, proposalSet, attempt) {
+    return {
+        source: { id: source.id, url: source.url, kind: source.kind },
+        check,
+        outcome,
+        sourceSnapshot,
+        preparedArtifact,
+        proposalSet,
+        attempt,
+    };
+}
+async function record(recorder, observation) {
+    try {
+        const identity = await recorder.record(observation);
+        if (!isIdentity(identity)) {
+            return { ok: false, error: error("dependency-contract", "Observation recorder returned an invalid identity"), observation };
+        }
+        return {
+            ok: true,
+            value: {
+                ...observation,
+                observationId: identity.observationId,
+                priorObservationId: identity.priorObservationId,
+            },
+        };
+    }
+    catch (cause) {
+        return { ok: false, error: error("recording-failed", "Observation recorder failed", cause), observation };
+    }
+}
+function snapshotFor(check) {
+    if (check.kind === "unchanged-304")
+        return { priorSnapshotRef: check.snapshotRef, currentSnapshotRef: check.snapshotRef };
+    return { priorSnapshotRef: check.priorSnapshotRef, currentSnapshotRef: check.currentSnapshotRef };
+}
+function attemptFor(result) {
+    return {
+        extractedAt: result.extractedAt,
+        providerCalls: result.providerCalls,
+        totalTokensUsed: result.totalTokensUsed,
+        ...(result.partial === undefined ? {} : { partial: result.partial }),
+        ...(result.providerFailures === undefined ? {} : {
+            providerFailures: result.providerFailures.map(({ kind, retryable }) => ({ kind, retryable })),
+        }),
+    };
+}
+function outcomeFor(result) {
+    const hasProviderFailures = (result.providerFailures?.length ?? 0) > 0;
+    if (result.partial !== undefined && hasProviderFailures)
+        return "partial-provider-failure";
+    if (hasProviderFailures)
+        return "provider-failure";
+    if (result.error !== undefined)
+        return "extraction-failure";
+    return result.partial === undefined ? "completed" : "partial";
+}
+function isCheckResult(value) {
+    if (typeof value !== "object" || value === null)
+        return false;
+    const result = value;
+    if (typeof result.sourceId !== "string" || typeof result.sourceUrl !== "string" ||
+        typeof result.checkedAt !== "string" || !Array.isArray(result.warnings))
+        return false;
+    if (result.kind === "unchanged-304")
+        return typeof result.snapshotRef === "string";
+    if (result.kind === "unchanged-hash")
+        return typeof result.priorSnapshotRef === "string" && typeof result.currentSnapshotRef === "string";
+    if (result.kind === "changed")
+        return (result.priorSnapshotRef === null || typeof result.priorSnapshotRef === "string") &&
+            typeof result.currentSnapshotRef === "string" && (result.changeBasis === "initial" || result.changeBasis === "hash");
+    return result.kind === "error" && (result.origin === "forage" || result.origin === "lookout") && result.error !== undefined;
+}
+function isExtractionResult(value) {
+    if (typeof value !== "object" || value === null)
+        return false;
+    const result = value;
+    return Array.isArray(result.proposals) && typeof result.extractedAt === "string" &&
+        Number.isFinite(result.providerCalls) && Number.isFinite(result.totalTokensUsed);
+}
+function isIdentity(value) {
+    if (typeof value !== "object" || value === null)
+        return false;
+    const identity = value;
+    return typeof identity.observationId === "string" && identity.observationId !== "" &&
+        (identity.priorObservationId === null || (typeof identity.priorObservationId === "string" && identity.priorObservationId !== ""));
+}
+function error(kind, text, cause) {
+    return { kind, message: text, ...(cause === undefined ? {} : { cause }) };
+}

package/dist/src/registry.d.ts

@@ -1,14 +1,25 @@
 import type { TargetFieldSchema } from "@kontourai/traverse";
-export type LookoutSourceKind = "web-page" | "api-record";
+export type LookoutSourceKind = "web-page" | "api-record" | "structured-file";
+export type StructuredFileFormat = "yaml" | "json" | "csv";
 export type RenderPolicy = "never" | "on-shell-warning" | "always";
-export interface LookoutSource {
+interface LookoutSourceBase {
     id: string;
-    kind: LookoutSourceKind;
     url: string;
-    targetSchema: TargetFieldSchema[];
     cadenceHint: string;
+}
+export interface ExtractableLookoutSource extends LookoutSourceBase {
+    kind: "web-page" | "api-record";
+    targetSchema: TargetFieldSchema[];
     renderPolicy: RenderPolicy;
+    format?: never;
+}
+export interface StructuredFileLookoutSource extends LookoutSourceBase {
+    kind: "structured-file";
+    format: StructuredFileFormat;
+    targetSchema?: never;
+    renderPolicy?: never;
 }
+export type LookoutSource = ExtractableLookoutSource | StructuredFileLookoutSource;
 export interface LookoutRegistryDocument {
     version: 1;
     sources: LookoutSource[];
@@ -27,3 +38,4 @@ export declare class LookoutRegistry {
 }
 export declare function loadRegistry(registryPath?: string): Promise<LookoutRegistry>;
 export declare function parseRegistry(value: unknown): LookoutRegistry;
+export {};

package/dist/src/registry.js

@@ -49,55 +49,120 @@ export function parseRegistry(value) {
     const ids = new Map();
     const sources = [];
     value.sources.forEach((candidate, index) => {
-        const context = `sources[${index}]`;
-        if (!isRecord(candidate)) {
-            issues.push(`${context} must be an object`);
-            return;
-        }
-        const id = candidate.id;
-        const label = typeof id === "string" && id.trim() ? `${context} (id "${id}")` : context;
-        if (typeof id !== "string" || id.trim() === "") {
-            issues.push(`${context}.id must be a non-empty string`);
-        }
-        else {
-            const first = ids.get(id);
-            if (first !== undefined)
-                issues.push(`${label}.id duplicates sources[${first}].id`);
-            else
-                ids.set(id, index);
-        }
-        if (candidate.kind !== "web-page" && candidate.kind !== "api-record") {
-            issues.push(`${label}.kind must be "web-page" or "api-record"`);
-        }
-        if (!isHttpUrl(candidate.url)) {
-            issues.push(`${label}.url must be an absolute HTTP(S) URL`);
-        }
-        if (typeof candidate.cadenceHint !== "string" || candidate.cadenceHint.trim() === "") {
-            issues.push(`${label}.cadenceHint must be a non-empty string`);
-        }
-        if (candidate.renderPolicy !== "never" &&
-            candidate.renderPolicy !== "on-shell-warning" &&
-            candidate.renderPolicy !== "always") {
-            issues.push(`${label}.renderPolicy must be "never", "on-shell-warning", or "always"`);
-        }
-        validateTargetSchema(candidate.targetSchema, label, issues);
-        if (typeof id === "string" && id.trim() !== "" &&
-            (candidate.kind === "web-page" || candidate.kind === "api-record") &&
-            isHttpUrl(candidate.url) &&
-            typeof candidate.cadenceHint === "string" && candidate.cadenceHint.trim() !== "" &&
-            (candidate.renderPolicy === "never" || candidate.renderPolicy === "on-shell-warning" || candidate.renderPolicy === "always") &&
-            Array.isArray(candidate.targetSchema)) {
-            sources.push(candidate);
-        }
+        const source = parseSource(candidate, index, ids, issues);
+        if (source !== undefined)
+            sources.push(source);
     });
     if (issues.length > 0)
         throw new RegistryValidationError(issues);
     return new LookoutRegistry(sources);
 }
+function parseSource(candidate, index, ids, issues) {
+    const context = `sources[${index}]`;
+    if (!isRecord(candidate)) {
+        issues.push(`${context} must be an object`);
+        return undefined;
+    }
+    const firstIssue = issues.length;
+    const common = validateCommonSource(candidate, index, ids, issues);
+    const source = candidate.kind === "structured-file"
+        ? parseStructuredFileSource(candidate, common, issues)
+        : candidate.kind === "web-page" || candidate.kind === "api-record"
+            ? parseExtractableSource(candidate, candidate.kind, common, issues)
+            : invalidSourceKind(candidate, common.label, issues);
+    return issues.length === firstIssue ? source : undefined;
+}
+function validateCommonSource(candidate, index, ids, issues) {
+    const context = `sources[${index}]`;
+    const id = typeof candidate.id === "string" && candidate.id.trim() !== "" ? candidate.id : undefined;
+    const label = id === undefined ? context : `${context} (id "${id}")`;
+    if (id === undefined) {
+        issues.push(`${context}.id must be a non-empty string`);
+    }
+    else {
+        const first = ids.get(id);
+        if (first !== undefined)
+            issues.push(`${label}.id duplicates sources[${first}].id`);
+        else
+            ids.set(id, index);
+    }
+    const url = isHttpUrl(candidate.url) ? candidate.url : undefined;
+    if (url === undefined) {
+        issues.push(`${label}.url must be an absolute HTTP(S) URL`);
+    }
+    const cadenceHint = typeof candidate.cadenceHint === "string" && candidate.cadenceHint.trim() !== ""
+        ? candidate.cadenceHint
+        : undefined;
+    if (cadenceHint === undefined) {
+        issues.push(`${label}.cadenceHint must be a non-empty string`);
+    }
+    return { id, url, cadenceHint, label };
+}
+function parseStructuredFileSource(candidate, common, issues) {
+    if (!isStructuredFileFormat(candidate.format)) {
+        issues.push(`${common.label}.format must be "yaml", "json", or "csv"`);
+    }
+    if (candidate.targetSchema !== undefined) {
+        issues.push(`${common.label}.targetSchema is not allowed for structured-file sources`);
+    }
+    if (candidate.renderPolicy !== undefined) {
+        issues.push(`${common.label}.renderPolicy is not allowed for structured-file sources`);
+    }
+    if (common.id === undefined || common.url === undefined || common.cadenceHint === undefined ||
+        !isStructuredFileFormat(candidate.format))
+        return undefined;
+    return {
+        id: common.id,
+        kind: "structured-file",
+        format: candidate.format,
+        url: common.url,
+        cadenceHint: common.cadenceHint,
+    };
+}
+function parseExtractableSource(candidate, kind, common, issues) {
+    if (!isRenderPolicy(candidate.renderPolicy)) {
+        issues.push(`${common.label}.renderPolicy must be "never", "on-shell-warning", or "always"`);
+    }
+    const targetSchema = validateTargetSchema(candidate.targetSchema, common.label, issues)
+        ? candidate.targetSchema
+        : undefined;
+    if (candidate.format !== undefined) {
+        issues.push(`${common.label}.format is only allowed for structured-file sources`);
+    }
+    if (common.id === undefined || common.url === undefined || common.cadenceHint === undefined ||
+        !isRenderPolicy(candidate.renderPolicy) || targetSchema === undefined)
+        return undefined;
+    return {
+        id: common.id,
+        kind,
+        url: common.url,
+        targetSchema,
+        cadenceHint: common.cadenceHint,
+        renderPolicy: candidate.renderPolicy,
+    };
+}
+function invalidSourceKind(candidate, label, issues) {
+    issues.push(`${label}.kind must be "web-page", "api-record", or "structured-file"`);
+    if (!isRenderPolicy(candidate.renderPolicy)) {
+        issues.push(`${label}.renderPolicy must be "never", "on-shell-warning", or "always"`);
+    }
+    validateTargetSchema(candidate.targetSchema, label, issues);
+    if (candidate.format !== undefined) {
+        issues.push(`${label}.format is only allowed for structured-file sources`);
+    }
+    return undefined;
+}
+function isStructuredFileFormat(value) {
+    return value === "yaml" || value === "json" || value === "csv";
+}
+function isRenderPolicy(value) {
+    return value === "never" || value === "on-shell-warning" || value === "always";
+}
 function validateTargetSchema(value, context, issues) {
+    const firstIssue = issues.length;
     if (!Array.isArray(value)) {
         issues.push(`${context}.targetSchema must be an array`);
-        return;
+        return false;
     }
     const allowedTypes = new Set(["string", "number", "boolean", "date", "enum", "array", "object"]);
     value.forEach((field, index) => {
@@ -124,6 +189,7 @@ function validateTargetSchema(value, context, issues) {
             issues.push(`${label}.inferenceType must be "explicit" or "inferred"`);
         }
     });
+    return issues.length === firstIssue;
 }
 function isRecord(value) {
     return typeof value === "object" && value !== null && !Array.isArray(value);

package/dist/src/snapshot-store.d.ts

@@ -1,2 +1,12 @@
-import type { SnapshotStore } from "@kontourai/traverse/fetch";
+import type { SnapshotStore } from "@kontourai/forage";
+import { type SnapshotSourceRefResolution } from "@kontourai/forage/fetch";
 export declare function createLookoutSnapshotStore(root?: string): SnapshotStore;
+export type ResolveLookoutSnapshotOptions = {
+    store: SnapshotStore;
+    root?: never;
+} | {
+    store?: never;
+    root?: string;
+};
+/** Replay one Lookout-emitted durable reference without any network access. */
+export declare function resolveLookoutSnapshot(reference: string, options?: ResolveLookoutSnapshotOptions): Promise<SnapshotSourceRefResolution>;

package/dist/src/snapshot-store.js

@@ -1,5 +1,22 @@
 import path from "node:path";
-import { createFilesystemSnapshotStore } from "@kontourai/traverse/fetch";
+import { createFilesystemSnapshotStore } from "@kontourai/forage";
+import { resolveSnapshotSourceRef, } from "@kontourai/forage/fetch";
 export function createLookoutSnapshotStore(root = path.join(process.cwd(), ".kontourai", "lookout", "snapshots")) {
     return createFilesystemSnapshotStore({ root });
 }
+/** Replay one Lookout-emitted durable reference without any network access. */
+export async function resolveLookoutSnapshot(reference, options = {}) {
+    try {
+        const store = options.store ?? createLookoutSnapshotStore(options.root);
+        return await resolveSnapshotSourceRef(store, reference);
+    }
+    catch {
+        return {
+            ok: false,
+            error: {
+                kind: "snapshot-store-error",
+                message: "the supplied snapshot store could not resolve the reference",
+            },
+        };
+    }
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@kontourai/lookout",
-  "version": "0.3.0",
-  "description": "A small source registry and drift-check runner built on Traverse snapshots.",
+  "version": "0.3.1",
+  "description": "A small source registry and drift-check runner built on Forage snapshots.",
   "license": "Apache-2.0",
   "type": "module",
   "private": false,
@@ -44,8 +44,8 @@
   },
   "dependencies": {
     "@kontourai/datum": "0.3.0",
-    "@kontourai/forage": "0.2.0",
-    "@kontourai/traverse": "0.14.1"
+    "@kontourai/forage": "0.4.1",
+    "@kontourai/traverse": "0.18.0"
   },
   "devDependencies": {
     "@types/node": "^25.6.0",