@kontourai/lookout 0.2.0 → 0.3.1

package/README.md

@@ -1,24 +1,67 @@
 # @kontourai/lookout
 
-A small **source registry** and **non-throwing drift-check runner** built on
-[Traverse](https://github.com/kontourai/traverse) snapshots. Lookout answers one
-question per registered source, cheaply and honestly: *did this source change
-since we last looked?*
+**Cheap, honest drift detection for content you re-check over time — "did this source change since we last looked?"**
 
-It composes Traverse for fetching and snapshot storage, provides deterministic
-proposal diffing, and authors provenance-bearing Survey inputs. It does **not**
-extract fields, project to Surface, review claims, notify, crawl, or schedule.
+A small **source registry** and **non-throwing drift-check runner** built on
+[Forage](https://github.com/kontourai/forage) snapshots. It composes Forage for
+fetching and snapshot storage, provides deterministic proposal diffing, and
+emits neutral, typed drift — its own vocabulary, in its own dependency-free
+package. It does **not** implement acquisition or extraction, author trust-layer
+records, project to Surface, review claims, notify, crawl, or schedule.
 Operational failures are returned as typed results.
 
+## Why it's different
+
+The naive way to answer "did it change?" is to re-crawl the source and diff the
+bytes. That's expensive (full re-download every time), noisy (a changed ad or
+timestamp reads as "changed"), and fragile (a thrown error mid-run looks the same
+as "no change"). Lookout is the opposite on all three:
+
+| | Re-crawl + byte-diff | `lookout` |
+|---|---|---|
+| Cost | full re-download every check | **conditional `304`** — often no download at all |
+| Signal | raw bytes (ads/timestamps = "changed") | **proposal-identity diff** — "a *new entity appeared*" vs "a byte moved" |
+| Honesty | a crash or false-`304` silently reads as "unchanged" | **typed results, never throws**; Forage scopes validators to the exact prior resource and Lookout verifies the returned capture identity |
+| Output | your problem to shape | **neutral, typed drift** — events already Hachure-evidence-shaped, ready for a consumer to lift into a trust bundle |
+
+So the point isn't "diffing" — it's *cheap + honest + semantic + review-ready*
+change detection, so a periodic re-check surfaces **only the real delta** (this
+provider is new, that listing changed) instead of re-reviewing everything.
+
+## Where it sits
+
+One layer in a four-verb stack; each repo owns one verb and is usable alone:
+
+- **[forage](https://github.com/kontourai/forage)** — CRAWL: fetch / frontier / SSRF-safe egress / snapshots
+- **[traverse](https://github.com/kontourai/traverse)** — EXTRACT: content + schema → reviewable proposals
+- **lookout** — CHANGE: did this registered source drift since last look?
+- **survey** — the SHAPE: what reviewed truth looks like (claims / review / resolution)
+
+Lookout composes Forage's `/fetch` surface for cheap `304`-aware re-checks,
+and Traverse's `ExtractionProposal` identity for the semantic diff. Dependency
+arrows point only downward — no cycles. Lookout itself depends on nothing in the
+trust layer: its events are already Hachure-evidence-shaped (`snapshotRef` /
+`locator` / `excerpt` / `fieldPath`), so a consumer or product lifts them into a
+Hachure `TrustBundle` via `@kontourai/surface`'s `TrustBundleBuilder` — the same
+pattern Traverse uses to match Survey's shape without importing Survey.
+
+**SSRF-safe egress.** Registered source URLs are operator- / aggregator-supplied
+and not fully trusted, so lookout routes its default fetch transport through
+[forage](https://github.com/kontourai/forage)'s SSRF-pinned guarded fetch
+(`@kontourai/forage/egress`). A registered source pointing at a private,
+link-local, loopback, or cloud-metadata host (e.g. `169.254.169.254`) is refused
+before any connection — a drift check can never be turned into an SSRF vector.
+A caller that injects its own `fetchSource` or `fetchOptions.fetch` (e.g. tests)
+owns its transport and opts out of the default guard.
+
 ## Requirements
 
 - Node.js `>= 22`.
-- `@kontourai/traverse` `>= 0.14.1`. This is a hard floor, not a preference:
-  trustworthy `unchanged-304` classification depends on the validator-scoping fix
-  from [traverse#49](https://github.com/kontourai/traverse/issues/49), first
-  released in `0.14.1`. Lookout exact-pins `0.14.1`. On an earlier Traverse a
-  conditional request could reuse a prior snapshot's validators against a
-  different URL and report a **false** `304`, so do not downgrade.
+- `@kontourai/forage` `>= 0.4.0`. Exact durable-reference replay requires the
+  resolver first released in `0.4.0`; trustworthy `unchanged-304`
+  classification also depends on Forage's validator-scoped revalidation.
+  Lookout exact-pins Traverse separately for schema and extraction-proposal
+  contracts.
 
 ## Registry
 
@@ -36,6 +79,13 @@ override with the library path argument or the CLI `--registry` flag):
       "targetSchema": [{ "path": "title", "type": "string", "required": true }],
       "cadenceHint": "daily",
       "renderPolicy": "never"
+    },
+    {
+      "id": "published-results",
+      "kind": "structured-file",
+      "format": "yaml",
+      "url": "https://raw.githubusercontent.com/example/project/0123456789abcdef0123456789abcdef01234567/results.yml",
+      "cadenceHint": "weekly"
     }
   ]
 }
@@ -46,11 +96,17 @@ Fields per source:
 | Field | Meaning |
 | --- | --- |
 | `id` | Non-empty, unique within the document. Used for exact lookup and snapshot identity. |
-| `kind` | `web-page` or `api-record`. |
+| `kind` | `web-page`, `api-record`, or `structured-file`. |
 | `url` | Absolute HTTP(S) URL. |
-| `targetSchema` | Traverse `TargetFieldSchema[]`. Stored inert at L1 (no extraction yet). |
+| `format` | Required only for `structured-file`: `yaml`, `json`, or `csv`. Lookout does not parse it. |
+| `targetSchema` | Required for `web-page` and `api-record`: Traverse `TargetFieldSchema[]`. Stored inert at L1 (no extraction yet). |
 | `cadenceHint` | Non-empty string; advisory only — Lookout does not schedule. |
-| `renderPolicy` | `never` \| `on-shell-warning` \| `always`. **Inert** at L1 (never mapped to a fetch/render policy). |
+| `renderPolicy` | Required for `web-page` and `api-record`: `never` \| `on-shell-warning` \| `always`. **Inert** at L1. |
+
+Structured-file entries retain raw fetched bytes and use the same guarded fetch,
+snapshot, comparison, and replay path as every other source. Parsing and domain
+normalization stay downstream. Prefer immutable, commit-pinned artifact URLs so
+an upstream branch move cannot silently redefine the cited source location.
 
 Validation reports **every** deterministic issue at once (with index/id context)
 and never reads the network. Lookup is exact-id; listing preserves file order —
@@ -59,42 +115,74 @@ no merging, discovery, or remote registries.
 ## Check results
 
 A check classifies each source into exactly one of four kinds. Every result
-carries `sourceId`, the registered `sourceUrl`, `checkedAt`, and Traverse
-`warnings`.
+carries `sourceId`, the registered `sourceUrl`, `checkedAt`, and Forage fetch
+warnings.
 
 | `kind` | When | Extra fields |
 | --- | --- | --- |
 | `unchanged-304` | A validator-backed conditional request returned `304`; zero body transfer; the prior snapshot is not re-persisted. | `snapshotRef` |
 | `unchanged-hash` | A full body was fetched and persisted, but its sha256 `bodyHash` equals the prior, **and** it came from the same resource URL — established by Lookout's own comparison. | `priorSnapshotRef`, `currentSnapshotRef` |
 | `changed` | The fresh body differs from the prior (`changeBasis: "hash"`), **or** it is the first successful observation (`changeBasis: "initial"`, `priorSnapshotRef: null`). | `priorSnapshotRef` (nullable), `currentSnapshotRef`, `changeBasis` |
-| `error` | Any operational failure — contained so the runner never rejects. | `origin` (`traverse` \| `lookout`), `error` |
+| `error` | Any operational failure — contained so the runner never rejects. | `origin` (`forage` \| `lookout`), `error` |
 
 `unchanged-hash` asserts same-resource continuity, so it requires **both** an
 identical `bodyHash` **and** the same resource URL as the prior snapshot. If a
 source has moved — the prior snapshot's URL differs from this fetch's — the
 result is `changed` even when the bytes are byte-identical, and the fresh
 snapshot is persisted as the new baseline. (The `unchanged-304` path is already
-resource-scoped by Traverse's validators, so only the hash path needs this
+resource-scoped by Forage's validators, so only the hash path needs this
 guard.)
 
-`error` results preserve provenance: `origin: "traverse"` carries Traverse's
+`error` results preserve provenance: `origin: "forage"` carries Forage's
 discriminated `FetchError` verbatim (its `kind`, and `status` when present);
 `origin: "lookout"` carries a typed `kind` — `prior-read`, `persistence`,
 `dependency-contract`, or `unexpected`.
 
 Snapshot references (`snapshotRef` / `priorSnapshotRef` / `currentSnapshotRef`)
-are portable logical refs from Traverse's `buildSnapshotSourceRef` — never
-filesystem paths. Snapshots are stored via Traverse's filesystem store, rooted by
+are portable logical refs from Forage's `buildSnapshotSourceRef` — never
+filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by
 default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
-custom filenames or retention.
+custom filenames or retention. `resolveLookoutSnapshot()` replays one exact
+reference through an injected store or Lookout snapshot root, authenticates its
+body and replay metadata, and never fetches. References emitted before Forage's
+replay-envelope digest remain resolvable with `integrity: "body-and-identity"`;
+new references report `integrity: "snapshot-envelope"`.
+
+## Schema coverage
+
+Drift isn't only "the bytes changed" — a source can reformat so that a field
+your schema *declares* silently stops being produced. The byte/proposal diffs
+above are temporal (they need a prior and say nothing on a first look), so they
+can't catch this. `checkSchemaCoverage` is the static complement:
+
+```ts
+import { checkSchemaCoverage } from "@kontourai/lookout";
+
+if (source.kind === "structured-file") throw new Error("structured sources are parsed downstream");
+const { covered, gaps } = checkSchemaCoverage(source.targetSchema, proposals);
+// covered: declared paths at least one proposal produced (schema order)
+// gaps:    declared paths that produced none, each with its `required` flag
+for (const gap of gaps) {
+  // escalate a missing required field harder than a missing optional one
+}
+```
+
+It reduces one declared `TargetFieldSchema[]` and one produced
+`ExtractionProposal[]` set into `{ covered, gaps }` by exact
+`proposal.fieldPath === field.path`. It is pure and total — no prior, no
+network, no throw — and answers on the very first observation. It measures the
+produced set you pass (it runs no post-verification filtering itself), reports
+only against the *declared* surface (an undeclared proposal path is neither
+covered nor a gap), and preserves schema order. Escalation — warning, review
+item, or hard failure — is the consumer's call.
 
 ## CLI
 
 ```
 lookout check <id> [--registry <path>] [--snapshot-root <path>]
 lookout check --all [--registry <path>] [--snapshot-root <path>]
-lookout emit-survey <id> --observation <path|-> [--registry <path>] [--observation-root <path>]
+lookout emit-drift <id> --observation <path|-> [--registry <path>] [--observation-root <path>]
 ```
 
 - Emits **exactly one compact JSON object per checked source, per stdout line**
@@ -110,14 +198,18 @@ lookout emit-survey <id> --observation <path|-> [--registry <path>] [--observati
 This exit/output contract is what makes `lookout check --all` safe to drive from
 an external scheduler.
 
-`emit-survey` is a separate composable command: its input is a JSON object with
+`emit-drift` is a separate composable command: its input is a JSON object with
 `observation` (a `ProposalSetObservation`) and its matching `check` anchor.
 Extraction is supplied by the caller. The first successful input commits a
-baseline fact and returns `surveyInput: null`; a genuine later change returns
-one unreviewed SurveyInput batch. State defaults to
-`.kontourai/lookout/observations`, uses immutable digest-addressed records and
-an atomic per-source pointer, and retains the latest two valid observations.
-Consumers pass the batch to Survey when they want Surface projection.
+baseline fact and returns `priorObservationId: null` with empty `events`; a
+genuine later change returns non-empty `events` and `facts` plus a
+`priorObservationId` pointing at the prior observation it was diffed against.
+State defaults to `.kontourai/lookout/observations`, uses immutable
+digest-addressed records and an atomic per-source pointer, and retains the
+latest two valid observations. The emitted `events` are already
+Hachure-evidence-shaped; a consumer or product lifts them into a Hachure
+`TrustBundle` with `@kontourai/surface`'s `TrustBundleBuilder` when it wants
+Surface projection — lookout itself authors nothing in the trust layer.
 Store paths refuse symbolic links. An existing source lock is not automatically
 broken: after confirming no writer is active, an operator may remove an
 abandoned `.lock` file and retry.
@@ -137,16 +229,72 @@ const results = await runner.checkAll(registry.list());
 ```
 
 `createCheckRunner` accepts injected seams — `store`, `fetchSource` (defaults to
-Traverse), `fetchOptions`, and `clock` — so checks run with no live network or
-timers in tests.
+Forage's SSRF-guarded fetcher), `fetchOptions`, and
+`clock` — so checks run with no live network or timers in tests. Injecting either
+`fetchSource` or `fetchOptions.fetch` overrides the default guarded transport.
+
+## Observe changed sources without re-extracting unchanged ones
+
+`createObserveExtractDiff` is an optional library composition for callers that
+want one typed observation per check. Supply acquisition, extraction, and
+recording capabilities; Lookout does not choose a content-preparation method,
+provider, or observation database.
+
+```ts
+import { createObserveExtractDiff } from "@kontourai/lookout";
+
+const composition = createObserveExtractDiff({
+  acquisition: { check: runner.check },
+  extraction: {
+    async extract({ source, snapshotRef }) {
+      // Resolve `snapshotRef`, prepare content, and invoke a caller-selected
+      // extraction implementation. Return its public Traverse result.
+      return extractChangedSource(source, snapshotRef);
+    },
+  },
+  recorder: {
+    async record(observation) {
+      // Caller-owned continuity and durable storage.
+      return saveObservation(observation);
+    },
+  },
+});
+
+const result = await composition.observe(source);
+```
+
+`unchanged-304` and `unchanged-hash` are recorded as `unchanged` and never call
+the extraction capability, so they make zero preparation and provider calls.
+Changed observations retain source and snapshot references, Traverse's prepared
+artifact identity, the proposal set, and the current/prior observation
+identities returned by the recorder. `partial`, `provider-failure`, mixed
+`partial-provider-failure`, and `extraction-failure` remain distinct typed
+outcomes. Provider failures are reduced to provider-neutral `kind` and
+`retryable` classifications; provider names, messages, native diagnostics,
+free-form extraction warnings, raw responses, and thrown-error text are not
+copied into the durable observation.
+A first changed observation
+has `priorObservationId: null`; it is a baseline observation, not a fabricated
+list of additions or removals.
+
+The composition validates that the check identifies the requested registered
+source and that Traverse validates the prepared artifact whose snapshot anchor
+matches the current check. This prevents mismatched capability results from
+being recorded as one observation. Proposal excerpts, source URLs, and
+acquisition check warnings can still contain sensitive source material; callers
+must apply their own retention and access policy before durable storage.
+
+The existing `createCheckRunner` and `createDriftEmitter` entrypoints remain
+available. Use `createDriftEmitter` when the caller wants deterministic
+proposal-diff events with its own identity callbacks.
 
 ## Non-goals
 
-Extraction, Surface projection, notifications, crawling, review/escalation or
-authority policy, rendered-fetch wiring
-(`renderPolicy` is inert pending traverse#50), and scheduling. Traverse retains
-all fetch politeness, robots, redirects, retries, timeouts, headers, user-agent,
-and rendered-fetch behavior.
+Acquisition and extraction implementations, Surface projection, notifications,
+crawling, review/escalation or authority policy, rendered-fetch wiring
+(`renderPolicy` is inert), and scheduling. Forage retains all fetch politeness,
+robots, redirects, retries, timeouts, headers, user-agent, and rendered-fetch
+behavior.
 
 ## Development
 

package/dist/src/check-result.d.ts

@@ -1,4 +1,4 @@
-import type { FetchError } from "@kontourai/traverse/fetch";
+import type { FetchError } from "@kontourai/forage/fetch";
 export interface CheckResultCommon {
     sourceId: string;
     sourceUrl: string;
@@ -23,7 +23,7 @@ export interface ChangedResult extends CheckResultCommon {
 export type LookoutErrorKind = "prior-read" | "persistence" | "dependency-contract" | "unexpected";
 export interface ErrorResult extends CheckResultCommon {
     kind: "error";
-    origin: "traverse" | "lookout";
+    origin: "forage" | "lookout";
     error: FetchError | {
         kind: LookoutErrorKind;
         message: string;

package/dist/src/check-runner.d.ts

@@ -1,4 +1,4 @@
-import type { FetchResult, FetchSourceOptions, SnapshotStore, SourceConfig } from "@kontourai/traverse/fetch";
+import type { FetchResult, FetchSourceOptions, SnapshotStore, SourceConfig } from "@kontourai/forage/fetch";
 import type { CheckResult } from "./check-result.js";
 import type { ProviderResolver } from "./provider-resolution.js";
 import type { LookoutSource } from "./registry.js";

package/dist/src/check-runner.js

@@ -1,7 +1,20 @@
-import { buildSnapshotSourceRef, fetchSource as traverseFetchSource, } from "@kontourai/traverse/fetch";
+import { isDeepStrictEqual } from "node:util";
+import { buildSnapshotSourceRef, fetchSource as forageFetchSource, } from "@kontourai/forage/fetch";
+/**
+ * The egress policy for lookout's registered-source fetches. Registered source
+ * URLs are operator- / aggregator-supplied and not fully trusted, so a source
+ * pointing at a private, link-local, loopback, or cloud-metadata host must be
+ * refused before any connection — a drift check can never be turned into an
+ * SSRF vector. `forage`'s `fetchSource` builds its own SSRF-pinned guarded
+ * transport from this policy whenever the caller doesn't inject a custom
+ * `fetch` (e.g. tests), so this is the single shared policy literal — never
+ * scattered per call site, never `{ guarded: false }`.
+ */
+const GUARDED_EGRESS = { guarded: true };
 export function createCheckRunner(options) {
-    const fetchImpl = options.fetchSource ?? traverseFetchSource;
+    const fetchImpl = options.fetchSource ?? forageFetchSource;
     const clock = options.clock ?? (() => new Date().toISOString());
+    const fetchOptions = { ...options.fetchOptions };
     async function check(source) {
         const common = () => ({
             sourceId: source.id,
@@ -18,17 +31,17 @@ export function createCheckRunner(options) {
         }
         let fetched;
         try {
-            fetched = await fetchImpl({ id: source.id, url: source.url, revalidate: true }, { ...options.fetchOptions, store: options.store });
+            fetched = await fetchImpl({ id: source.id, url: source.url, egress: GUARDED_EGRESS }, { ...fetchOptions, store: options.store });
         }
         catch (error) {
             return lookoutError(common(), "unexpected", error);
         }
         const base = { ...common(), warnings: Array.isArray(fetched?.warnings) ? [...fetched.warnings] : [] };
         if (!isFetchResult(fetched)) {
-            return lookoutError(base, "dependency-contract", "Traverse returned neither exactly one snapshot nor exactly one error");
+            return lookoutError(base, "dependency-contract", "Forage returned neither exactly one snapshot nor exactly one error");
         }
         if (fetched.error) {
-            return { ...base, kind: "error", origin: "traverse", error: fetched.error };
+            return { ...base, kind: "error", origin: "forage", error: fetched.error };
         }
         // Defense-in-depth: even if a future guard gap let a malformed snapshot
         // through, any stray throw in classification/ref-building becomes a typed
@@ -36,7 +49,22 @@ export function createCheckRunner(options) {
         try {
             const snapshot = fetched.snapshot;
             if (snapshot.notModified === true) {
-                return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(snapshot) };
+                const snapshotBodyEncoding = bodyEncoding(snapshot);
+                const priorBodyEncoding = prior === undefined ? undefined : bodyEncoding(prior);
+                if (prior === undefined ||
+                    snapshotBodyEncoding === undefined ||
+                    snapshotBodyEncoding !== priorBodyEncoding ||
+                    snapshot.sourceId !== prior.sourceId ||
+                    snapshot.url !== prior.url ||
+                    snapshot.status !== prior.status ||
+                    snapshot.fetchedAt !== prior.fetchedAt ||
+                    snapshot.bodyHash !== prior.bodyHash ||
+                    !isDeepStrictEqual(snapshot.headers, prior.headers) ||
+                    !isDeepStrictEqual(snapshot.redirects, prior.redirects) ||
+                    snapshot.rendered !== prior.rendered) {
+                    return lookoutError(base, "dependency-contract", "Forage returned a 304 snapshot without the matching prior capture");
+                }
+                return { ...base, kind: "unchanged-304", snapshotRef: buildSnapshotSourceRef(prior) };
             }
             try {
                 await options.store.put(snapshot);
@@ -76,6 +104,14 @@ export function createCheckRunner(options) {
     }
     return { check, checkAll };
 }
+function bodyEncoding(snapshot) {
+    const descriptor = Object.getOwnPropertyDescriptor(snapshot, "body");
+    if (descriptor === undefined || !("value" in descriptor))
+        return undefined;
+    if (typeof descriptor.value === "string")
+        return "utf8";
+    return descriptor.value instanceof Uint8Array ? "bytes" : undefined;
+}
 function isFetchResult(value) {
     if (typeof value !== "object" || value === null)
         return false;

package/dist/src/cli.d.ts

@@ -1,7 +1,7 @@
 import type { Writable } from "node:stream";
 import { type CheckRunner } from "./check-runner.js";
 import { type LookoutRegistry } from "./registry.js";
-import { type EmissionResult } from "./survey-emission.js";
+import { type DriftResult } from "./drift-emission.js";
 export interface RunCliOptions {
     argv?: string[];
     stdout?: Pick<Writable, "write">;
@@ -9,6 +9,6 @@ export interface RunCliOptions {
     loadRegistry?: (path?: string) => Promise<LookoutRegistry>;
     runner?: CheckRunner;
     readObservation?: (path: string) => Promise<unknown>;
-    emitSurvey?: (sourceId: string, value: unknown, registry: LookoutRegistry, observationRoot?: string) => Promise<EmissionResult>;
+    emitDrift?: (sourceId: string, value: unknown, registry: LookoutRegistry, observationRoot?: string) => Promise<DriftResult>;
 }
 export declare function runCli(options?: RunCliOptions): Promise<number>;

package/dist/src/cli.js

@@ -3,7 +3,7 @@ import { createCheckRunner } from "./check-runner.js";
 import { loadRegistry } from "./registry.js";
 import { createLookoutSnapshotStore } from "./snapshot-store.js";
 import { createObservationStore } from "./observation-store.js";
-import { createSurveyEmitter } from "./survey-emission.js";
+import { createDriftEmitter } from "./drift-emission.js";
 export async function runCli(options = {}) {
     const argv = options.argv ?? process.argv.slice(2);
     const stdout = options.stdout ?? process.stdout;
@@ -21,7 +21,7 @@ export async function runCli(options = {}) {
         stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
         return 1;
     }
-    if (parsed.command === "emit-survey") {
+    if (parsed.command === "emit-drift") {
         const source = registry.get(parsed.id);
         if (!source) {
             stderr.write(`Unknown source id: ${parsed.id}\n`);
@@ -35,7 +35,7 @@ export async function runCli(options = {}) {
             stderr.write(`Could not read observation: ${error instanceof Error ? error.message : String(error)}\n`);
             return 1;
         }
-        const result = await (options.emitSurvey ?? emitSurvey)(parsed.id, value, registry, parsed.observationRoot);
+        const result = await (options.emitDrift ?? emitDrift)(parsed.id, value, registry, parsed.observationRoot);
         if (!result.ok) {
             stderr.write(`${result.error.kind}: ${result.error.message}\n`);
             return 1;
@@ -61,10 +61,10 @@ export async function runCli(options = {}) {
     return 0;
 }
 function parseArgs(argv) {
-    if (argv[0] === "emit-survey")
+    if (argv[0] === "emit-drift")
         return parseEmitArgs(argv);
     if (argv[0] !== "check")
-        return "Usage: lookout check <id>|--all [--registry path] [--snapshot-root path]\n       lookout emit-survey <id> --observation <path|-> [--registry path] [--observation-root path]";
+        return "Usage: lookout check <id>|--all [--registry path] [--snapshot-root path]\n       lookout emit-drift <id> --observation <path|-> [--registry path] [--observation-root path]";
     const parsed = { command: "check", all: false };
     for (let index = 1; index < argv.length; index += 1) {
         const arg = argv[index];
@@ -125,10 +125,10 @@ function parseEmitArgs(argv) {
             id = arg;
     }
     if (!id)
-        return "emit-survey requires a source id";
+        return "emit-drift requires a source id";
     if (!observationPath)
-        return "emit-survey requires --observation <path|->";
-    return { command: "emit-survey", id, observationPath, registryPath, observationRoot };
+        return "emit-drift requires --observation <path|->";
+    return { command: "emit-drift", id, observationPath, registryPath, observationRoot };
 }
 async function readObservation(file) {
     const maxBytes = 1024 * 1024;
@@ -164,10 +164,10 @@ function cliEntities(observation) {
     }
     return [...grouped].map(([key, proposals]) => ({ key, proposals }));
 }
-async function emitSurvey(sourceId, value, registry, observationRoot) {
+async function emitDrift(sourceId, value, registry, observationRoot) {
     if (!value || typeof value !== "object" || Array.isArray(value))
         return { ok: false, error: { kind: "invalid-input", message: "Observation document must be an object" } };
     const document = value;
     const source = registry.get(sourceId);
-    return createSurveyEmitter({ store: createObservationStore({ root: observationRoot }) }).emit({ source, current: document.observation, check: document.check, callbacks: { selectEntities: cliEntities, entityIdentity: (entity) => entity.key, proposalsFor: (entity) => entity.proposals, fieldIdentity: (_entity, proposal) => proposal.fieldPath } });
+    return createDriftEmitter({ store: createObservationStore({ root: observationRoot }) }).emit({ source, current: document.observation, check: document.check, callbacks: { selectEntities: cliEntities, entityIdentity: (entity) => entity.key, proposalsFor: (entity) => entity.proposals, fieldIdentity: (_entity, proposal) => proposal.fieldPath } });
 }

package/dist/src/coverage.d.ts

@@ -0,0 +1,58 @@
+import type { ExtractionProposal, TargetFieldSchema } from "@kontourai/traverse";
+/**
+ * A declared schema field that the supplied observation did not cover: the
+ * extractor produced **zero** proposals whose `fieldPath` equals this field's
+ * declared `path`. `required` mirrors the schema field's own `required` flag
+ * (absent → `false`) so a consumer can escalate a missing required field harder
+ * than a missing optional one.
+ *
+ * The gap keys the declared field under `fieldPath` (not `path`) deliberately:
+ * it is the same string as the schema field's `path`, but named to line up with
+ * `ExtractionProposal.fieldPath` so a consumer can correlate a gap directly
+ * against the proposals it was measured over. `covered` needs no such wrapper —
+ * it carries neither the `required` flag nor a correlation need — so it stays a
+ * bare `string[]` of declared paths.
+ */
+export interface SchemaCoverageGap {
+    readonly fieldPath: string;
+    readonly required: boolean;
+}
+/**
+ * The result of checking one extraction observation against the declared
+ * schema. `covered` lists, in schema order, every declared `path` that at least
+ * one proposal produced; `gaps` lists, in schema order, every declared `path`
+ * that produced none. The two are disjoint and together cover the declared
+ * schema. `gaps` never mentions a proposal fieldPath that is absent from the
+ * schema — this is a coverage check of the *declared* surface, not an
+ * unexpected-field detector.
+ */
+export interface SchemaCoverageResult {
+    readonly covered: readonly string[];
+    readonly gaps: readonly SchemaCoverageGap[];
+}
+/**
+ * Static schema-coverage check: which declared schema fields did this single
+ * extraction observation fail to produce at all?
+ *
+ * This is the *static*, first-observation-capable complement to the temporal
+ * proposal diff (`diffProposalSets`). The diff answers "did the produced
+ * proposals change since we last looked?" and, by construction, says nothing on
+ * a first observation. Coverage answers a question that has an answer on the
+ * very first look and needs no prior: "did the source drift — reformat, rename a
+ * heading, reorder a table — such that a field the schema *declares* silently
+ * stopped being produced?". A declared field with no proposal is exactly that
+ * signal: a layout-drift regression that would otherwise surface only as a
+ * silent gap in the output.
+ *
+ * Membership is by exact `proposal.fieldPath === field.path`. Matching against
+ * the produced-proposal set (not any post-verification survivor set) is
+ * deliberate: a field the extractor *did* produce but a downstream step later
+ * dropped is explained by that step's own diagnostic and is not a coverage gap
+ * — the extractor covered it. Callers that want coverage measured against a
+ * post-verification set simply pass that set as `proposals`.
+ *
+ * Pure and total: no injected callbacks, no network, no throw. Declared fields
+ * are evaluated in schema order; a schema that repeats a `path` reports it once
+ * per occurrence (the schema, not this check, owns declared-field uniqueness).
+ */
+export declare function checkSchemaCoverage(schema: readonly TargetFieldSchema[], proposals: readonly ExtractionProposal[]): SchemaCoverageResult;

package/dist/src/coverage.js

@@ -0,0 +1,39 @@
+/**
+ * Static schema-coverage check: which declared schema fields did this single
+ * extraction observation fail to produce at all?
+ *
+ * This is the *static*, first-observation-capable complement to the temporal
+ * proposal diff (`diffProposalSets`). The diff answers "did the produced
+ * proposals change since we last looked?" and, by construction, says nothing on
+ * a first observation. Coverage answers a question that has an answer on the
+ * very first look and needs no prior: "did the source drift — reformat, rename a
+ * heading, reorder a table — such that a field the schema *declares* silently
+ * stopped being produced?". A declared field with no proposal is exactly that
+ * signal: a layout-drift regression that would otherwise surface only as a
+ * silent gap in the output.
+ *
+ * Membership is by exact `proposal.fieldPath === field.path`. Matching against
+ * the produced-proposal set (not any post-verification survivor set) is
+ * deliberate: a field the extractor *did* produce but a downstream step later
+ * dropped is explained by that step's own diagnostic and is not a coverage gap
+ * — the extractor covered it. Callers that want coverage measured against a
+ * post-verification set simply pass that set as `proposals`.
+ *
+ * Pure and total: no injected callbacks, no network, no throw. Declared fields
+ * are evaluated in schema order; a schema that repeats a `path` reports it once
+ * per occurrence (the schema, not this check, owns declared-field uniqueness).
+ */
+export function checkSchemaCoverage(schema, proposals) {
+    const producedPaths = new Set(proposals.map((proposal) => proposal.fieldPath));
+    const covered = [];
+    const gaps = [];
+    for (const field of schema) {
+        if (producedPaths.has(field.path)) {
+            covered.push(field.path);
+        }
+        else {
+            gaps.push({ fieldPath: field.path, required: field.required === true });
+        }
+    }
+    return { covered, gaps };
+}