@kontourai/lookout 0.1.0 → 0.3.0

package/README.md

@@ -1,14 +1,59 @@
 # @kontourai/lookout
 
-A small **source registry** and **non-throwing drift-check runner** built on
-[Traverse](https://github.com/kontourai/traverse) snapshots. Lookout answers one
-question per registered source, cheaply and honestly: *did this source change
-since we last looked?*
+**Cheap, honest drift detection for content you re-check over time — "did this source change since we last looked?"**
 
-It composes Traverse for fetching and snapshot storage, provides deterministic
-proposal diffing, and authors provenance-bearing Survey inputs. It does **not**
-extract fields, project to Surface, review claims, notify, crawl, or schedule.
-Operational failures are returned as typed results.
+A small **source registry** and **non-throwing drift-check runner** built on
+[Traverse](https://github.com/kontourai/traverse) snapshots. It composes Traverse
+for fetching and snapshot storage, provides deterministic proposal diffing, and
+emits neutral, typed drift — its own vocabulary, in its own dependency-free
+package. It does **not** extract fields, author trust-layer records, project to
+Surface, review claims, notify, crawl, or schedule. Operational failures are
+returned as typed results.
+
+## Why it's different
+
+The naive way to answer "did it change?" is to re-crawl the source and diff the
+bytes. That's expensive (full re-download every time), noisy (a changed ad or
+timestamp reads as "changed"), and fragile (a thrown error mid-run looks the same
+as "no change"). Lookout is the opposite on all three:
+
+| | Re-crawl + byte-diff | `lookout` |
+|---|---|---|
+| Cost | full re-download every check | **conditional `304`** — often no download at all |
+| Signal | raw bytes (ads/timestamps = "changed") | **proposal-identity diff** — "a *new entity appeared*" vs "a byte moved" |
+| Honesty | a crash or false-`304` silently reads as "unchanged" | **typed results, never throws**; hardened against false-`304` (the [traverse#49](https://github.com/kontourai/traverse/issues/49) validator-scoping pin) |
+| Output | your problem to shape | **neutral, typed drift** — events already Hachure-evidence-shaped, ready for a consumer to lift into a trust bundle |
+
+So the point isn't "diffing" — it's *cheap + honest + semantic + review-ready*
+change detection, so a periodic re-check surfaces **only the real delta** (this
+provider is new, that listing changed) instead of re-reviewing everything.
+
+## Where it sits
+
+One layer in a four-verb stack; each repo owns one verb and is usable alone:
+
+- **[forage](https://github.com/kontourai/forage)** — CRAWL: fetch / frontier / SSRF-safe egress / snapshots
+- **[traverse](https://github.com/kontourai/traverse)** — EXTRACT: content + schema → reviewable proposals
+- **lookout** — CHANGE: did this registered source drift since last look?
+- **survey** — the SHAPE: what reviewed truth looks like (claims / review / resolution)
+
+Lookout composes the fetch/snapshot layer (today Traverse's `/fetch`; re-points at
+`forage` as its single-source fetch surface lands) for cheap `304`-aware re-checks,
+and Traverse's `ExtractionProposal` identity for the semantic diff. Dependency
+arrows point only downward — no cycles. Lookout itself depends on nothing in the
+trust layer: its events are already Hachure-evidence-shaped (`snapshotRef` /
+`locator` / `excerpt` / `fieldPath`), so a consumer or product lifts them into a
+Hachure `TrustBundle` via `@kontourai/surface`'s `TrustBundleBuilder` — the same
+pattern Traverse uses to match Survey's shape without importing Survey.
+
+**SSRF-safe egress.** Registered source URLs are operator- / aggregator-supplied
+and not fully trusted, so lookout routes its default fetch transport through
+[forage](https://github.com/kontourai/forage)'s SSRF-pinned guarded fetch
+(`@kontourai/forage/egress`). A registered source pointing at a private,
+link-local, loopback, or cloud-metadata host (e.g. `169.254.169.254`) is refused
+before any connection — a drift check can never be turned into an SSRF vector.
+A caller that injects its own `fetchSource` or `fetchOptions.fetch` (e.g. tests)
+owns its transport and opts out of the default guard.
 
 ## Requirements
 
@@ -89,12 +134,39 @@ default at `<cwd>/.kontourai/lookout/snapshots` (override with the CLI
 `--snapshot-root` flag or by injecting a store in library use). Lookout adds no
 custom filenames or retention.
 
+## Schema coverage
+
+Drift isn't only "the bytes changed" — a source can reformat so that a field
+your schema *declares* silently stops being produced. The byte/proposal diffs
+above are temporal (they need a prior and say nothing on a first look), so they
+can't catch this. `checkSchemaCoverage` is the static complement:
+
+```ts
+import { checkSchemaCoverage } from "@kontourai/lookout";
+
+const { covered, gaps } = checkSchemaCoverage(source.targetSchema, proposals);
+// covered: declared paths at least one proposal produced (schema order)
+// gaps:    declared paths that produced none, each with its `required` flag
+for (const gap of gaps) {
+  // escalate a missing required field harder than a missing optional one
+}
+```
+
+It reduces one declared `TargetFieldSchema[]` and one produced
+`ExtractionProposal[]` set into `{ covered, gaps }` by exact
+`proposal.fieldPath === field.path`. It is pure and total — no prior, no
+network, no throw — and answers on the very first observation. It measures the
+produced set you pass (it runs no post-verification filtering itself), reports
+only against the *declared* surface (an undeclared proposal path is neither
+covered nor a gap), and preserves schema order. Escalation — warning, review
+item, or hard failure — is the consumer's call.
+
 ## CLI
 
 ```
 lookout check <id> [--registry <path>] [--snapshot-root <path>]
 lookout check --all [--registry <path>] [--snapshot-root <path>]
-lookout emit-survey <id> --observation <path|-> [--registry <path>] [--observation-root <path>]
+lookout emit-drift <id> --observation <path|-> [--registry <path>] [--observation-root <path>]
 ```
 
 - Emits **exactly one compact JSON object per checked source, per stdout line**
@@ -110,14 +182,18 @@ lookout emit-survey <id> --observation <path|-> [--registry <path>] [--observati
 This exit/output contract is what makes `lookout check --all` safe to drive from
 an external scheduler.
 
-`emit-survey` is a separate composable command: its input is a JSON object with
+`emit-drift` is a separate composable command: its input is a JSON object with
 `observation` (a `ProposalSetObservation`) and its matching `check` anchor.
 Extraction is supplied by the caller. The first successful input commits a
-baseline fact and returns `surveyInput: null`; a genuine later change returns
-one unreviewed SurveyInput batch. State defaults to
-`.kontourai/lookout/observations`, uses immutable digest-addressed records and
-an atomic per-source pointer, and retains the latest two valid observations.
-Consumers pass the batch to Survey when they want Surface projection.
+baseline fact and returns `priorObservationId: null` with empty `events`; a
+genuine later change returns non-empty `events` and `facts` plus a
+`priorObservationId` pointing at the prior observation it was diffed against.
+State defaults to `.kontourai/lookout/observations`, uses immutable
+digest-addressed records and an atomic per-source pointer, and retains the
+latest two valid observations. The emitted `events` are already
+Hachure-evidence-shaped; a consumer or product lifts them into a Hachure
+`TrustBundle` with `@kontourai/surface`'s `TrustBundleBuilder` when it wants
+Surface projection — lookout itself authors nothing in the trust layer.
 Store paths refuse symbolic links. An existing source lock is not automatically
 broken: after confirming no writer is active, an operator may remove an
 abandoned `.lock` file and retry.
@@ -137,8 +213,9 @@ const results = await runner.checkAll(registry.list());
 ```
 
 `createCheckRunner` accepts injected seams — `store`, `fetchSource` (defaults to
-Traverse), `fetchOptions`, and `clock` — so checks run with no live network or
-timers in tests.
+Traverse's fetcher over forage's SSRF-guarded egress), `fetchOptions`, and
+`clock` — so checks run with no live network or timers in tests. Injecting either
+`fetchSource` or `fetchOptions.fetch` overrides the default guarded transport.
 
 ## Non-goals
 

package/dist/src/check-runner.js

@@ -1,7 +1,30 @@
 import { buildSnapshotSourceRef, fetchSource as traverseFetchSource, } from "@kontourai/traverse/fetch";
+import { createGuardedFetch } from "@kontourai/forage/egress";
+/**
+ * The default egress transport for lookout's registered-source fetches: forage's
+ * SSRF-pinned guarded fetch. Registered source URLs are operator- / aggregator-
+ * supplied and not fully trusted, so a source pointing at a private, link-local,
+ * loopback, or cloud-metadata host is refused before any connection — a drift
+ * check can never be turned into an SSRF vector. Only wired when lookout uses the
+ * default traverse fetcher and the caller injected no `fetch`; a caller that
+ * supplies its own `fetchSource` or `fetchOptions.fetch` (e.g. tests) owns the
+ * transport. Created lazily and cached so the guard is built once, not per check.
+ */
+let cachedGuardedFetch;
+function defaultGuardedFetch() {
+    cachedGuardedFetch ??= createGuardedFetch();
+    return cachedGuardedFetch;
+}
 export function createCheckRunner(options) {
+    const usingDefaultFetcher = options.fetchSource === undefined;
     const fetchImpl = options.fetchSource ?? traverseFetchSource;
     const clock = options.clock ?? (() => new Date().toISOString());
+    // Guard the default traverse fetcher's egress with forage. A caller-supplied
+    // fetcher or `fetchOptions.fetch` owns its own transport and is left as-is.
+    const fetchOptions = { ...options.fetchOptions };
+    if (usingDefaultFetcher && fetchOptions.fetch === undefined) {
+        fetchOptions.fetch = defaultGuardedFetch();
+    }
     async function check(source) {
         const common = () => ({
             sourceId: source.id,
@@ -18,7 +41,7 @@ export function createCheckRunner(options) {
         }
         let fetched;
         try {
-            fetched = await fetchImpl({ id: source.id, url: source.url, revalidate: true }, { ...options.fetchOptions, store: options.store });
+            fetched = await fetchImpl({ id: source.id, url: source.url, revalidate: true }, { ...fetchOptions, store: options.store });
         }
         catch (error) {
             return lookoutError(common(), "unexpected", error);

package/dist/src/cli.d.ts

@@ -1,7 +1,7 @@
 import type { Writable } from "node:stream";
 import { type CheckRunner } from "./check-runner.js";
 import { type LookoutRegistry } from "./registry.js";
-import { type EmissionResult } from "./survey-emission.js";
+import { type DriftResult } from "./drift-emission.js";
 export interface RunCliOptions {
     argv?: string[];
     stdout?: Pick<Writable, "write">;
@@ -9,6 +9,6 @@ export interface RunCliOptions {
     loadRegistry?: (path?: string) => Promise<LookoutRegistry>;
     runner?: CheckRunner;
     readObservation?: (path: string) => Promise<unknown>;
-    emitSurvey?: (sourceId: string, value: unknown, registry: LookoutRegistry, observationRoot?: string) => Promise<EmissionResult>;
+    emitDrift?: (sourceId: string, value: unknown, registry: LookoutRegistry, observationRoot?: string) => Promise<DriftResult>;
 }
 export declare function runCli(options?: RunCliOptions): Promise<number>;

package/dist/src/cli.js

@@ -3,7 +3,7 @@ import { createCheckRunner } from "./check-runner.js";
 import { loadRegistry } from "./registry.js";
 import { createLookoutSnapshotStore } from "./snapshot-store.js";
 import { createObservationStore } from "./observation-store.js";
-import { createSurveyEmitter } from "./survey-emission.js";
+import { createDriftEmitter } from "./drift-emission.js";
 export async function runCli(options = {}) {
     const argv = options.argv ?? process.argv.slice(2);
     const stdout = options.stdout ?? process.stdout;
@@ -21,7 +21,7 @@ export async function runCli(options = {}) {
         stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
         return 1;
     }
-    if (parsed.command === "emit-survey") {
+    if (parsed.command === "emit-drift") {
         const source = registry.get(parsed.id);
         if (!source) {
             stderr.write(`Unknown source id: ${parsed.id}\n`);
@@ -35,7 +35,7 @@ export async function runCli(options = {}) {
             stderr.write(`Could not read observation: ${error instanceof Error ? error.message : String(error)}\n`);
             return 1;
         }
-        const result = await (options.emitSurvey ?? emitSurvey)(parsed.id, value, registry, parsed.observationRoot);
+        const result = await (options.emitDrift ?? emitDrift)(parsed.id, value, registry, parsed.observationRoot);
         if (!result.ok) {
             stderr.write(`${result.error.kind}: ${result.error.message}\n`);
             return 1;
@@ -61,10 +61,10 @@ export async function runCli(options = {}) {
     return 0;
 }
 function parseArgs(argv) {
-    if (argv[0] === "emit-survey")
+    if (argv[0] === "emit-drift")
         return parseEmitArgs(argv);
     if (argv[0] !== "check")
-        return "Usage: lookout check <id>|--all [--registry path] [--snapshot-root path]\n       lookout emit-survey <id> --observation <path|-> [--registry path] [--observation-root path]";
+        return "Usage: lookout check <id>|--all [--registry path] [--snapshot-root path]\n       lookout emit-drift <id> --observation <path|-> [--registry path] [--observation-root path]";
     const parsed = { command: "check", all: false };
     for (let index = 1; index < argv.length; index += 1) {
         const arg = argv[index];
@@ -125,10 +125,10 @@ function parseEmitArgs(argv) {
             id = arg;
     }
     if (!id)
-        return "emit-survey requires a source id";
+        return "emit-drift requires a source id";
     if (!observationPath)
-        return "emit-survey requires --observation <path|->";
-    return { command: "emit-survey", id, observationPath, registryPath, observationRoot };
+        return "emit-drift requires --observation <path|->";
+    return { command: "emit-drift", id, observationPath, registryPath, observationRoot };
 }
 async function readObservation(file) {
     const maxBytes = 1024 * 1024;
@@ -164,10 +164,10 @@ function cliEntities(observation) {
     }
     return [...grouped].map(([key, proposals]) => ({ key, proposals }));
 }
-async function emitSurvey(sourceId, value, registry, observationRoot) {
+async function emitDrift(sourceId, value, registry, observationRoot) {
     if (!value || typeof value !== "object" || Array.isArray(value))
         return { ok: false, error: { kind: "invalid-input", message: "Observation document must be an object" } };
     const document = value;
     const source = registry.get(sourceId);
-    return createSurveyEmitter({ store: createObservationStore({ root: observationRoot }) }).emit({ source, current: document.observation, check: document.check, callbacks: { selectEntities: cliEntities, entityIdentity: (entity) => entity.key, proposalsFor: (entity) => entity.proposals, fieldIdentity: (_entity, proposal) => proposal.fieldPath } });
+    return createDriftEmitter({ store: createObservationStore({ root: observationRoot }) }).emit({ source, current: document.observation, check: document.check, callbacks: { selectEntities: cliEntities, entityIdentity: (entity) => entity.key, proposalsFor: (entity) => entity.proposals, fieldIdentity: (_entity, proposal) => proposal.fieldPath } });
 }

package/dist/src/coverage.d.ts

@@ -0,0 +1,58 @@
+import type { ExtractionProposal, TargetFieldSchema } from "@kontourai/traverse";
+/**
+ * A declared schema field that the supplied observation did not cover: the
+ * extractor produced **zero** proposals whose `fieldPath` equals this field's
+ * declared `path`. `required` mirrors the schema field's own `required` flag
+ * (absent → `false`) so a consumer can escalate a missing required field harder
+ * than a missing optional one.
+ *
+ * The gap keys the declared field under `fieldPath` (not `path`) deliberately:
+ * it is the same string as the schema field's `path`, but named to line up with
+ * `ExtractionProposal.fieldPath` so a consumer can correlate a gap directly
+ * against the proposals it was measured over. `covered` needs no such wrapper —
+ * it carries neither the `required` flag nor a correlation need — so it stays a
+ * bare `string[]` of declared paths.
+ */
+export interface SchemaCoverageGap {
+    readonly fieldPath: string;
+    readonly required: boolean;
+}
+/**
+ * The result of checking one extraction observation against the declared
+ * schema. `covered` lists, in schema order, every declared `path` that at least
+ * one proposal produced; `gaps` lists, in schema order, every declared `path`
+ * that produced none. The two are disjoint and together cover the declared
+ * schema. `gaps` never mentions a proposal fieldPath that is absent from the
+ * schema — this is a coverage check of the *declared* surface, not an
+ * unexpected-field detector.
+ */
+export interface SchemaCoverageResult {
+    readonly covered: readonly string[];
+    readonly gaps: readonly SchemaCoverageGap[];
+}
+/**
+ * Static schema-coverage check: which declared schema fields did this single
+ * extraction observation fail to produce at all?
+ *
+ * This is the *static*, first-observation-capable complement to the temporal
+ * proposal diff (`diffProposalSets`). The diff answers "did the produced
+ * proposals change since we last looked?" and, by construction, says nothing on
+ * a first observation. Coverage answers a question that has an answer on the
+ * very first look and needs no prior: "did the source drift — reformat, rename a
+ * heading, reorder a table — such that a field the schema *declares* silently
+ * stopped being produced?". A declared field with no proposal is exactly that
+ * signal: a layout-drift regression that would otherwise surface only as a
+ * silent gap in the output.
+ *
+ * Membership is by exact `proposal.fieldPath === field.path`. Matching against
+ * the produced-proposal set (not any post-verification survivor set) is
+ * deliberate: a field the extractor *did* produce but a downstream step later
+ * dropped is explained by that step's own diagnostic and is not a coverage gap
+ * — the extractor covered it. Callers that want coverage measured against a
+ * post-verification set simply pass that set as `proposals`.
+ *
+ * Pure and total: no injected callbacks, no network, no throw. Declared fields
+ * are evaluated in schema order; a schema that repeats a `path` reports it once
+ * per occurrence (the schema, not this check, owns declared-field uniqueness).
+ */
+export declare function checkSchemaCoverage(schema: readonly TargetFieldSchema[], proposals: readonly ExtractionProposal[]): SchemaCoverageResult;

package/dist/src/coverage.js

@@ -0,0 +1,39 @@
+/**
+ * Static schema-coverage check: which declared schema fields did this single
+ * extraction observation fail to produce at all?
+ *
+ * This is the *static*, first-observation-capable complement to the temporal
+ * proposal diff (`diffProposalSets`). The diff answers "did the produced
+ * proposals change since we last looked?" and, by construction, says nothing on
+ * a first observation. Coverage answers a question that has an answer on the
+ * very first look and needs no prior: "did the source drift — reformat, rename a
+ * heading, reorder a table — such that a field the schema *declares* silently
+ * stopped being produced?". A declared field with no proposal is exactly that
+ * signal: a layout-drift regression that would otherwise surface only as a
+ * silent gap in the output.
+ *
+ * Membership is by exact `proposal.fieldPath === field.path`. Matching against
+ * the produced-proposal set (not any post-verification survivor set) is
+ * deliberate: a field the extractor *did* produce but a downstream step later
+ * dropped is explained by that step's own diagnostic and is not a coverage gap
+ * — the extractor covered it. Callers that want coverage measured against a
+ * post-verification set simply pass that set as `proposals`.
+ *
+ * Pure and total: no injected callbacks, no network, no throw. Declared fields
+ * are evaluated in schema order; a schema that repeats a `path` reports it once
+ * per occurrence (the schema, not this check, owns declared-field uniqueness).
+ */
+export function checkSchemaCoverage(schema, proposals) {
+    const producedPaths = new Set(proposals.map((proposal) => proposal.fieldPath));
+    const covered = [];
+    const gaps = [];
+    for (const field of schema) {
+        if (producedPaths.has(field.path)) {
+            covered.push(field.path);
+        }
+        else {
+            gaps.push({ fieldPath: field.path, required: field.required === true });
+        }
+    }
+    return { covered, gaps };
+}

package/dist/src/{survey-emission.d.ts → drift-emission.d.ts}

@@ -1,4 +1,3 @@
-import { type SurveyInput } from "@kontourai/survey";
 import type { LookoutSource } from "./registry.js";
 import { type ProposalDiffEvent, type ProposalSetDiff, type ProposalSetDiffInput, type ProposalSetFacts, type ProposalSetObservation } from "./proposal-diff.js";
 import type { ObservationCheckAnchor, ObservationStore, StoredProposalObservationV1 } from "./observation-store.js";
@@ -11,7 +10,7 @@ export interface BaselineEstablishedFact {
     readonly resolution: "observation";
     readonly proposalCount: number;
 }
-export type EmissionFact = BaselineEstablishedFact | {
+export type DriftFact = BaselineEstablishedFact | {
     readonly kind: "proposal-set-facts";
     readonly priorSnapshotRef: string;
     readonly currentSnapshotRef: string;
@@ -19,37 +18,38 @@ export type EmissionFact = BaselineEstablishedFact | {
     readonly resolution: "observation";
     readonly value: ProposalSetFacts;
 };
-export interface EmissionSuccess {
+export interface DriftSuccess {
     readonly sourceId: string;
     readonly events: readonly ProposalDiffEvent[];
-    readonly facts: readonly EmissionFact[];
-    readonly surveyInput: SurveyInput | null;
+    readonly facts: readonly DriftFact[];
+    /** The prior observation this drift was diffed against, or null on a first-ever (baseline) observation. */
+    readonly priorObservationId: string | null;
     readonly committedObservation: StoredProposalObservationV1;
     readonly warnings: readonly string[];
 }
-export type EmissionErrorKind = "invalid-input" | "prior-state-error" | "diff-error" | "authoring-error" | "persistence-error";
-export interface EmissionError {
-    readonly kind: EmissionErrorKind;
+export type DriftErrorKind = "invalid-input" | "prior-state-error" | "diff-error" | "persistence-error" | "serialization-error" | "unexpected";
+export interface DriftError {
+    readonly kind: DriftErrorKind;
     readonly message: string;
     readonly cause?: unknown;
 }
-export type EmissionResult = {
+export type DriftResult = {
     readonly ok: true;
-    readonly value: EmissionSuccess;
+    readonly value: DriftSuccess;
 } | {
     readonly ok: false;
-    readonly error: EmissionError;
+    readonly error: DriftError;
 };
-export interface EmitSurveyInput<E> {
+export interface EmitDriftInput<E> {
     readonly source: LookoutSource;
     readonly current: ProposalSetObservation;
     readonly check: ObservationCheckAnchor;
     readonly callbacks: Omit<ProposalSetDiffInput<E>, "prior" | "current">;
 }
-export interface SurveyEmitter<E> {
-    emit(input: EmitSurveyInput<E>): Promise<EmissionResult>;
+export interface DriftEmitter<E> {
+    emit(input: EmitDriftInput<E>): Promise<DriftResult>;
 }
-export interface CreateSurveyEmitterOptions<E> {
+export interface CreateDriftEmitterOptions<E> {
     readonly store: ObservationStore;
     readonly now?: () => string;
     readonly diff?: (input: ProposalSetDiffInput<E>) => {
@@ -61,8 +61,5 @@ export interface CreateSurveyEmitterOptions<E> {
             readonly message: string;
         };
     };
-    /** Test seam. Production callers must leave this as observation. */
-    readonly resolution?: "observation";
-    readonly transformSurveyInput?: (input: SurveyInput) => SurveyInput;
 }
-export declare function createSurveyEmitter<E>(options: CreateSurveyEmitterOptions<E>): SurveyEmitter<E>;
+export declare function createDriftEmitter<E>(options: CreateDriftEmitterOptions<E>): DriftEmitter<E>;

package/dist/src/drift-emission.js

@@ -0,0 +1,71 @@
+import { diffProposalSets } from "./proposal-diff.js";
+function stableJson(value) {
+    if (Array.isArray(value))
+        return `[${value.map(stableJson).join(",")}]`;
+    if (value && typeof value === "object")
+        return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
+    return JSON.stringify(value) ?? "undefined";
+}
+function normalizeDiff(value) {
+    const sorted = (items) => [...items].sort((a, b) => stableJson(a).localeCompare(stableJson(b)));
+    return {
+        events: sorted(value.events),
+        facts: {
+            retainedProposalOccurrences: sorted(value.facts.retainedProposalOccurrences),
+            addedProposalOccurrences: sorted(value.facts.addedProposalOccurrences),
+            removedProposalOccurrences: sorted(value.facts.removedProposalOccurrences),
+            provenanceChanges: sorted(value.facts.provenanceChanges),
+            removedEntities: [...value.facts.removedEntities].sort(),
+        },
+    };
+}
+export function createDriftEmitter(options) {
+    const now = options.now ?? (() => new Date().toISOString());
+    const diff = options.diff ?? diffProposalSets;
+    return {
+        async emit(input) {
+            try {
+                if (!input.source || input.source.id !== input.current?.sourceId || input.check?.currentSnapshotRef !== input.current?.snapshotRef) {
+                    return { ok: false, error: { kind: "invalid-input", message: "Registry source, observation, and check anchor must agree" } };
+                }
+                const loaded = await options.store.loadLatest(input.source.id);
+                if (!loaded.ok)
+                    return { ok: false, error: { kind: "prior-state-error", message: loaded.error.message, cause: loaded.error } };
+                const recordedAt = now();
+                const priorObservationId = loaded.value?.observationId ?? null;
+                let events = [];
+                let facts;
+                if (loaded.value === null) {
+                    facts = [{ kind: "baseline-established", sourceId: input.source.id, snapshotRef: input.current.snapshotRef, observedAt: input.current.observedAt, origin: input.source.kind, resolution: "observation", proposalCount: input.current.proposals.length }];
+                }
+                else {
+                    let derived;
+                    try {
+                        derived = diff({ prior: { sourceId: loaded.value.sourceId, snapshotRef: loaded.value.snapshotRef, observedAt: loaded.value.observedAt, proposals: loaded.value.proposals }, current: input.current, ...input.callbacks });
+                    }
+                    catch (cause) {
+                        return { ok: false, error: { kind: "diff-error", message: "Proposal diff threw", cause } };
+                    }
+                    if (!derived.ok)
+                        return { ok: false, error: { kind: "diff-error", message: derived.error.message, cause: derived.error } };
+                    const normalized = normalizeDiff(derived.value);
+                    events = normalized.events;
+                    facts = [{ kind: "proposal-set-facts", priorSnapshotRef: loaded.value.snapshotRef, currentSnapshotRef: input.current.snapshotRef, origin: input.source.kind, resolution: "observation", value: normalized.facts }];
+                }
+                try {
+                    JSON.stringify({ events, facts });
+                }
+                catch (cause) {
+                    return { ok: false, error: { kind: "serialization-error", message: "Drift result is not serializable", cause } };
+                }
+                const committed = await options.store.commit({ observation: input.current, recordedAt, check: input.check }, loaded.value?.observationId ?? null);
+                if (!committed.ok)
+                    return { ok: false, error: { kind: "persistence-error", message: committed.error.message, cause: committed.error } };
+                return { ok: true, value: { sourceId: input.source.id, events, facts, priorObservationId, committedObservation: committed.value, warnings: committed.warnings ?? [] } };
+            }
+            catch (cause) {
+                return { ok: false, error: { kind: "unexpected", message: "Drift emission failed", cause } };
+            }
+        },
+    };
+}

package/dist/src/index.d.ts

@@ -16,5 +16,7 @@ export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js
 export type { FieldChangedEvent, FieldChangeKind, NewEntityAppearedEvent, ProposalDiffEvent, ProposalEvidence, ProposalIdentity, ProposalOccurrencePair, ProposalSetDiff, ProposalSetDiffInput, ProposalSetFacts, ProposalSetObservation, ProvenanceChangeFact, } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
 export type { CreateObservationStoreOptions, ObservationCheckAnchor, ObservationStore, ObservationStoreError, ObservationStoreErrorKind, ObservationStoreResult, ProposalObservationRecordInput, StoredProposalObservationV1 } from "./observation-store.js";
-export { createSurveyEmitter } from "./survey-emission.js";
-export type { BaselineEstablishedFact, CreateSurveyEmitterOptions, EmissionError, EmissionErrorKind, EmissionFact, EmissionResult, EmissionSuccess, EmitSurveyInput, SurveyEmitter } from "./survey-emission.js";
+export { createDriftEmitter } from "./drift-emission.js";
+export type { BaselineEstablishedFact, CreateDriftEmitterOptions, DriftEmitter, DriftError, DriftErrorKind, DriftFact, DriftResult, DriftSuccess, EmitDriftInput } from "./drift-emission.js";
+export { checkSchemaCoverage } from "./coverage.js";
+export type { SchemaCoverageGap, SchemaCoverageResult } from "./coverage.js";

package/dist/src/index.js

@@ -7,4 +7,5 @@ export { canonicalValueKey } from "./canonical-value.js";
 export { compareStructural, diffKeyedMultiset } from "./structural-diff.js";
 export { diffProposalSets, extractionProposalIdentity } from "./proposal-diff.js";
 export { createObservationStore } from "./observation-store.js";
-export { createSurveyEmitter } from "./survey-emission.js";
+export { createDriftEmitter } from "./drift-emission.js";
+export { checkSchemaCoverage } from "./coverage.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kontourai/lookout",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "description": "A small source registry and drift-check runner built on Traverse snapshots.",
   "license": "Apache-2.0",
   "type": "module",
@@ -44,7 +44,7 @@
   },
   "dependencies": {
     "@kontourai/datum": "0.3.0",
-    "@kontourai/survey": "1.7.0",
+    "@kontourai/forage": "0.2.0",
     "@kontourai/traverse": "0.14.1"
   },
   "devDependencies": {

package/dist/src/survey-emission.js

@@ -1,138 +0,0 @@
-import { createHash } from "node:crypto";
-import { SurveyInputBuilder } from "@kontourai/survey";
-import { diffProposalSets } from "./proposal-diff.js";
-function id(...parts) { return createHash("sha256").update(JSON.stringify(parts)).digest("hex"); }
-function eventValue(event) {
-    return event.kind === "new-entity-appeared"
-        ? { kind: event.kind, entityKey: event.entityKey, currentValues: event.current.map((item) => ({ fieldKey: item.fieldKey, value: item.value })) }
-        : { kind: event.kind, changeKind: event.changeKind, entityKey: event.entityKey, fieldKey: event.fieldKey, ...(event.prior ? { priorValue: event.prior.value } : {}), ...(event.current ? { currentValue: event.current.value } : {}) };
-}
-function observationFor(source, prior, current, event, generatedAt, index) {
-    const eventId = id(source.id, prior.observationId, current.snapshotRef, event.kind, event.entityKey, event.kind === "field-changed" ? event.fieldKey : "", index);
-    const evidence = event.kind === "new-entity-appeared" ? event.current : [event.prior, event.current].filter((item) => item !== undefined);
-    const primary = event.kind === "new-entity-appeared" ? event.current[0] : event.current ?? event.prior;
-    const metadata = {
-        sourceId: source.id, eventKind: event.kind, priorObservationId: prior.observationId,
-        priorSnapshotRef: prior.snapshotRef, currentSnapshotRef: current.snapshotRef,
-        evidence: evidence.map((item) => ({ snapshotRef: item.snapshotRef, locator: item.provenance.locator, excerpt: item.provenance.excerpt, extractor: item.extractor, fieldPath: item.fieldPath })),
-        ...(event.kind === "field-changed" ? { changeKind: event.changeKind } : {}),
-    };
-    return {
-        id: `lookout-${eventId}`,
-        rawSource: { kind: source.kind, resolution: "observation", sourceRef: current.snapshotRef, observedAt: current.observedAt, locatorScheme: source.kind === "web-page" ? "html" : "structured-field", metadata },
-        extraction: { target: `lookout:${event.entityKey}:${event.kind === "field-changed" ? event.fieldKey : "appearance"}`, value: eventValue(event), confidence: primary?.confidence, locator: primary?.provenance.locator, excerpt: primary?.provenance.excerpt, extractor: primary?.extractor ?? "lookout:proposal-diff", extractedAt: generatedAt, metadata },
-        candidateSet: { status: "needs-review", metadata },
-        claim: { subjectType: "lookout.observed-entity", subjectId: event.entityKey, facet: "lookout.source-change", claimType: event.kind === "new-entity-appeared" ? "lookout.new-entity-appeared" : "lookout.field-changed", fieldOrBehavior: event.kind === "field-changed" ? event.fieldKey : event.entityKey, value: eventValue(event), status: "proposed", impactLevel: "low", collectedBy: "@kontourai/lookout", createdAt: generatedAt, evidenceMethod: "observation", metadata },
-    };
-}
-function containsAuthorizing(value) {
-    if (Array.isArray(value))
-        return value.some(containsAuthorizing);
-    if (!value || typeof value !== "object")
-        return false;
-    return Object.entries(value).some(([key, item]) => key === "authorizing" || containsAuthorizing(item));
-}
-function validateSurveyInput(input, source, current, prior, events) {
-    if (input.reviewOutcomes.length !== 0 || containsAuthorizing(input))
-        return false;
-    if (input.rawSources.length !== events.length || input.claims.length !== events.length)
-        return false;
-    for (const raw of input.rawSources)
-        if (raw.kind !== source.kind || raw.resolution !== "observation" || raw.sourceRef !== current.snapshotRef)
-            return false;
-    for (let index = 0; index < events.length; index += 1) {
-        const claim = input.claims[index];
-        const event = events[index];
-        if (!claim || !event || claim.status !== "proposed" || claim.metadata?.priorSnapshotRef !== prior.snapshotRef || claim.metadata?.currentSnapshotRef !== current.snapshotRef)
-            return false;
-        const evidence = claim.metadata.evidence;
-        if (!Array.isArray(evidence))
-            return false;
-        if (event.kind === "new-entity-appeared" && !event.current.every((side) => evidence.some((item) => item && typeof item === "object" && item.snapshotRef === side.snapshotRef)))
-            return false;
-        if (event.kind === "field-changed") {
-            if (event.prior && !evidence.some((item) => item && typeof item === "object" && item.snapshotRef === event.prior.snapshotRef))
-                return false;
-            if (event.current && !evidence.some((item) => item && typeof item === "object" && item.snapshotRef === event.current.snapshotRef))
-                return false;
-            if (!event.current && !input.rawSources[index] || input.rawSources[index]?.sourceRef !== current.snapshotRef)
-                return false;
-        }
-    }
-    return true;
-}
-function stableJson(value) {
-    if (Array.isArray(value))
-        return `[${value.map(stableJson).join(",")}]`;
-    if (value && typeof value === "object")
-        return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${stableJson(item)}`).join(",")}}`;
-    return JSON.stringify(value) ?? "undefined";
-}
-function normalizeDiff(value) {
-    const sorted = (items) => [...items].sort((a, b) => stableJson(a).localeCompare(stableJson(b)));
-    return { events: sorted(value.events), facts: { retainedProposalOccurrences: sorted(value.facts.retainedProposalOccurrences), addedProposalOccurrences: sorted(value.facts.addedProposalOccurrences), removedProposalOccurrences: sorted(value.facts.removedProposalOccurrences), provenanceChanges: sorted(value.facts.provenanceChanges), removedEntities: [...value.facts.removedEntities].sort() } };
-}
-export function createSurveyEmitter(options) {
-    const now = options.now ?? (() => new Date().toISOString());
-    const diff = options.diff ?? diffProposalSets;
-    return { async emit(input) {
-            try {
-                if (options.resolution !== undefined && options.resolution !== "observation")
-                    return { ok: false, error: { kind: "invalid-input", message: "Lookout may author only observation resolution" } };
-                if (!input.source || input.source.id !== input.current?.sourceId || input.check?.currentSnapshotRef !== input.current?.snapshotRef)
-                    return { ok: false, error: { kind: "invalid-input", message: "Registry source, observation, and check anchor must agree" } };
-                const loaded = await options.store.loadLatest(input.source.id);
-                if (!loaded.ok)
-                    return { ok: false, error: { kind: "prior-state-error", message: loaded.error.message, cause: loaded.error } };
-                const recordedAt = now();
-                let events = [];
-                let facts;
-                let surveyInput = null;
-                if (loaded.value === null) {
-                    facts = [{ kind: "baseline-established", sourceId: input.source.id, snapshotRef: input.current.snapshotRef, observedAt: input.current.observedAt, origin: input.source.kind, resolution: "observation", proposalCount: input.current.proposals.length }];
-                }
-                else {
-                    let derived;
-                    try {
-                        derived = diff({ prior: { sourceId: loaded.value.sourceId, snapshotRef: loaded.value.snapshotRef, observedAt: loaded.value.observedAt, proposals: loaded.value.proposals }, current: input.current, ...input.callbacks });
-                    }
-                    catch (cause) {
-                        return { ok: false, error: { kind: "diff-error", message: "Proposal diff threw", cause } };
-                    }
-                    if (!derived.ok)
-                        return { ok: false, error: { kind: "diff-error", message: derived.error.message, cause: derived.error } };
-                    const normalized = normalizeDiff(derived.value);
-                    events = normalized.events;
-                    facts = [{ kind: "proposal-set-facts", priorSnapshotRef: loaded.value.snapshotRef, currentSnapshotRef: input.current.snapshotRef, origin: input.source.kind, resolution: "observation", value: normalized.facts }];
-                    if (events.length > 0) {
-                        try {
-                            const builder = new SurveyInputBuilder({ source: "@kontourai/lookout", generatedAt: recordedAt });
-                            builder.addObservations(events.map((event, index) => observationFor(input.source, loaded.value, input.current, event, recordedAt, index)));
-                            surveyInput = builder.build();
-                            if (options.transformSurveyInput)
-                                surveyInput = options.transformSurveyInput(surveyInput);
-                            JSON.stringify(surveyInput);
-                            if (!validateSurveyInput(surveyInput, input.source, input.current, loaded.value, events))
-                                return { ok: false, error: { kind: "authoring-error", message: "SurveyInput violated observation-only provenance" } };
-                        }
-                        catch (cause) {
-                            return { ok: false, error: { kind: "authoring-error", message: "Could not author SurveyInput", cause } };
-                        }
-                    }
-                }
-                try {
-                    JSON.stringify({ events, facts, surveyInput });
-                }
-                catch (cause) {
-                    return { ok: false, error: { kind: "authoring-error", message: "Emission result is not serializable", cause } };
-                }
-                const committed = await options.store.commit({ observation: input.current, recordedAt, check: input.check }, loaded.value?.observationId ?? null);
-                if (!committed.ok)
-                    return { ok: false, error: { kind: "persistence-error", message: committed.error.message, cause: committed.error } };
-                return { ok: true, value: { sourceId: input.source.id, events, facts, surveyInput, committedObservation: committed.value, warnings: committed.warnings ?? [] } };
-            }
-            catch (cause) {
-                return { ok: false, error: { kind: "authoring-error", message: "Emission failed", cause } };
-            }
-        } };
-}