@konneal/engine 0.1.0

package/docs/sota-mechanisms.md

@@ -0,0 +1,184 @@
+# The SOTA mechanisms — the reference
+
+This is the canonical description of every mechanism in the serving
+stack, each with its staircase: what the layer below it cannot do. The
+staircase runs from raw text (fewest dimensions) to executable models
+(most) — every step adds structure, and every added structure unlocks
+questions the previous step could not answer.
+
+## The staircase of representations
+
+```
+plain text → clause-anchored documents → typed units (MKO)
+           → machine-readable models (Primmel) → execution
+```
+
+| Step | Carries | Can answer | Cannot answer |
+|---|---|---|---|
+| Plain text | words | topical lookup | anything with a citation anchor — half the chunks carry no clause reference at all |
+| Clause-anchored prose | words + location | "where does R 60 address creep?" with § | values inside tables, cross-references, anything typed |
+| Typed units (MKO) | + tables/formulas/figures as objects, cross-references as links | "which test validates this requirement" (a link, not a sentence); figures | computed conformance |
+| Machine models (Primmel) | + constraints, calculations, sequences, terms — machine-checkable | the machine limit itself; test order; instance parameters | executing them in an answer |
+| Execution | the check, evaluated | **"is D_max 26 000 v valid for E_max 30 000 v?" → INVALID, computed** | — (the frontier top) |
+
+Measured across the six comparison lanes (witness-graded, 18 rung-tagged
+probes): plain 2/18 · adoc 3/18 · MKO 6/18 · Primmel 13/18 · ablation
+12/18 · composed 14/18 · full serving 17–18/18.
+
+## The serving mechanisms
+
+### 1. Query understanding (meaning, never strings)
+One small model reads every question: language, named publication,
+edition, process intent, hypothetical answer, query variants,
+decomposition for complex asks, standalone reformulation of follow-ups.
+No keyword rules decide anything — the same judgements apply in every
+language. *Below this step:* fixed keyword routers, which answer
+differently depending on phrasing.
+
+### 2. Hybrid retrieval (dense + lexical, fused)
+The question's vector searches the corpus alongside a full-corpus BM25
+scan; RRF fusion merges them; a terminology graph adds
+concept→document candidates; multiple phrasings and sub-questions run
+as parallel lanes; a hypothetical answer serves as an extra search key
+(HyDE). *Below:* dense-only retrieval misses exact jargon ("n_LC");
+lexical-only misses paraphrase.
+
+### 3. Contextual enrichment (98.7% of non-synthetic chunks)
+Every chunk carries a model-written preamble stating where it sits in
+its document — written once at index time, its quality persists into
+every future retrieval. The preamble lives in the INDEX (plus a 30-day
+KV context cache), never in the build artifacts — a full index restore
+must replay it from the durable record (`scripts/replay_enrichment.py`,
+binding-embed, zero regeneration). Measured the hard way: a restore that
+skipped the replay dropped enrichment to 0% and identifier queries
+failed first ("what is OIML D 29?" — the preambles carry the
+identifier-rich context that ranks them). *Below:* bare chunks retrieve
+on local wording.
+
+### 4. Structural retrieval over the clause tree
+The corpus IS a tree (clause anchors chain parent→child). A hit's score
+blends its ancestors' and descendants' scores — a section whose clauses
+are collectively relevant rises; final evidence is presented in
+document reading order; same-chain near-duplicates collapse. *Below:*
+flat chunk retrieval — no notion of "the section around this clause".
+
+### 5. Section-summary units (multi-granularity)
+Depth-1 clause summaries are retrievable objects: a summary that ranks
+descends to its quotable child clauses and retires itself — citations
+always quote source text. *Below:* one grain per index — either too
+coarse to cite or too fine to orient.
+
+### 6. Vocabulary binding (the nomenclature bridge)
+Everyday words ("my output keeps drifting") rarely match defined terms
+("span stability"). A terminology index of the corpus's defined
+concepts links the question to candidate terms (dense candidates +
+cross-encoder rerank); the answer model adjudicates and leads with the
+corpus's own term, quoting its definition. *Below:* every representation
+measured fails the everyday-words→defined-term bridge — the one gap
+that is a vocabulary problem, not a structure problem.
+
+### 7. Reranking, edition steering, typed pin
+A cross-encoder orders candidates (a stronger model re-orders hard
+queries); family-relative steering demotes superseded editions while
+keeping them citable when only they carry content — family is the
+doctype+number, and the demotion is full-strength under either signal:
+the successor edition of the SAME identifier is in the pool, or the
+chunk's own status says superseded/unknown against the family's newest
+(the corpus carries the supersession the registry's missing successor
+links do not; in-force documents are exempt so a newer part-2 never
+demotes a current part-1). A document NAMED in the question scopes
+retrieval deterministically — read from the question text, never left
+to the understanding model's per-call mood ("What is OIML D 29?" was a
+coin flip before). Typed objects (tables/formulas/figures) are pinned a
+window slot so the answer contract can reference them — and a query
+that NAMES the artifact type ("figure", "table", "equation") dominates
+unit selection, so the pin lands the object the question is about
+(feeding multimodal generation for figures). *Below:* similarity-only
+ranking answers from stale editions, guesses at doc scope, and never
+surfaces a typed object.
+
+### 8. The answer contract (claims are checkable)
+Inline citations on every claim; normative values quoted verbatim from
+the cited passage; tables/formulas/figures rendered as typed objects
+exactly from the source, never re-typed; a deterministic post-check
+enforces all of it (quoted spans must be contained in served passages;
+a served table's data must carry its object reference) with one
+corrected retry. *Below:* generated prose whose correctness cannot be
+measured.
+
+### 9. The verdict engine (conformance by execution)
+When a question names a machine-checkable model object, the service
+executes it: the question's values bind to the rule's own parameters, a
+deterministic evaluator runs the check (OCL boolean expressions,
+threshold limits), and the verdict — pass, the standard's own violation
+word, or void naming the missing parameters — is attached as data the
+model must present faithfully. Counterfactuals are free: hypothetical
+values are just values. *Below:* quoting the rule and hoping the reader
+does the arithmetic.
+
+### 10. Provable absence
+"Does R 60 constrain packaging?" enumerates the standard's entire model
+plane and returns a certificate — absent, N nodes enumerated, 0
+matches, scope named — never a bare "I found nothing". *Below:* refusal,
+which asserts a search, not a proof.
+
+### 11. Answer verification
+Any answer can be checked against the corpus: verbatim-quote
+containment, object-reference resolution, citation presence
+(deterministic) plus a judged faithfulness score (labeled as judged).
+*Below:* trust me.
+
+### 12. Multimodal figure interpretation
+A pinned figure unit's pixels ride the generation call (R2 asset →
+base64 image part), so the model interprets the producer's drawing, not
+just its stored caption. Two measured invariants: assets must be
+vision-readable — vector-sourced rasters carrying black strokes on
+transparent alpha flatten to a solid black rectangle inside vision
+pipelines (browsers hid the defect by compositing on white;
+`scripts/fix_figure_assets.py` detects and re-uploads white-flattened) —
+and images ride their own short trailing user message, because long
+passage text + image parts in ONE message triggers nondeterministic
+provider 8005s that scale with payload. A third rule guards the blast
+radius: pixels attach only when the question WANTS a drawing (names a
+figure-ish artifact, or the pinned figure sits in the answering
+clause) — and when the multimodal call flakes down to a text-only
+fallback, the attach note drops with the pixels (a text model answers
+the note, not the question — observed in the wild). *Below:* caption-
+only answers that disclaim "the passage does not list" what the drawing
+plainly shows.
+
+### 13. Dataset scope and permission gating
+Three databases (OIML Publications · OIML SMART Models · ISO/IEC
+Conformity Assessment) are selectable per question: the sidebar toggles
+persist a scope, every ask carries it, and the server intersects it
+with what the SESSION may see — the ISO/IEC corpus requires the estate
+permission `ai-preview` (a role code set in id.oimlsmart.org);
+membership alone is not the bar, and the UI's lock is cosmetic. A
+corpus-scope stage runs last of the pool-assembly stages (after the
+lexical union refills the pool post-rerank) so disabling a database
+actually disables it at the hit level. *Below:* one monolithic corpus
+the user cannot narrow, gated (or not) on login alone.
+
+### 14. The measurement gates
+The golden suite (38 cases: doc-level, definitions, table values,
+refusals, filters, verdicts, auth, French) runs ×3 against the live
+service with witness-span containment grading; the annealment battery
+(18 rung-tagged probes) measures capability per representation; EIR
+(cited/retrieved) watches window precision; leakage probes gate every
+promotion; the deploy pipeline guards branch, tests, version bump and
+smoke. Current: golden 37–38/38 (97–100%), annealment 17–18/18 (mode 18/18 — the residuals are two single-case variances: the L5 cross-standard probe's retrieval and the L1 validity date's phrasing).
+
+## The vector adapter (one door)
+
+Every chunk crosses one boundary into any index: a pydantic wire schema
+(corpus registry, size caps, anchor sanity) and target gating — an
+index accepts only the corpora that belong to it. The wire schema
+mirrors the serving contract; new producers register corpora in one
+place, never ad hoc.
+
+Index equality with the canonical chunk set is an OPERATIONS loop, not
+a hope: reconciliation enumerates the index and deletes strays (upserts
+never delete — measured 49,553 live vs 31,512 canonical before the
+first run); the enrichment replay restores the contexts a full upsert
+overwrites; the asset sweep keeps unit images vision-readable. Each
+step is a script in `scripts/`, run between surgery and gate.

package/docs/spec-api.md

@@ -0,0 +1,77 @@
+# API surface spec
+
+The worker's complete HTTP inventory, mirrored 1:1 by the `ROUTES`
+registry in `workers/worker_public/src/index.ts` (dispatched by
+`lib/router.ts`'s `matchRoute`). Adding a route = one registry entry +
+its handler; this document changes with it. Request/response details
+beyond the inventory: [API.md](API.md).
+
+## Conventions
+
+- **Dual publication**: `/api/*` is the browser surface (anonymous, or
+  member via session cookie / bubble Bearer); `/v1/*` is the integrator
+  surface (`Authorization: Bearer oiml_<hex>` API key). Same handler,
+  tier derived from the path prefix.
+- **Admin routes** authenticate with `Bearer $ADMIN_TOKEN` (a worker
+  secret, not an API key).
+- Every response is JSON unless noted; errors are
+  `{ "error": { "code", "message" } }`. CORS for `*.oimlsmart.org`.
+- Trailing slashes are equivalent (segment-exact matching).
+
+## The inventory
+
+| Route | Method | Auth | Handler | Contract |
+|---|---|---|---|---|
+| `/`, `/api/`, `/index.html` | GET | none | `serveIndexPage` | The SPA shell from the assets binding, `must-revalidate` so deploys can never serve HTML referencing deleted fingerprinted assets. |
+| `/auth/login` | GET | none | `handleLogin` | OIDC redirect (PKCE); `?mode=bubble&origin=` starts the embedded-panel flow. |
+| `/auth/callback` | GET | none | `handleCallback` | OIDC callback; mints the session cookie; bubble mode postMessages the token to the validated origin. |
+| `/auth/me` | GET | session | `handleMe` | `{ authenticated, name, email, roles, tier }`. |
+| `/auth/logout` | GET, POST | none | `handleLogout` | Clears the session. |
+| `/api/conversations` | * | session | `conversationsRoute` → `handleConversations` | List (GET) / create (POST) / delete (DELETE) the signed-in user's conversations. |
+| `/api/conversations/:id` | * | session (owner) | `conversationsRoute` | One conversation (GET) / delete (DELETE). |
+| `/api/conversations/:id/messages` | POST | session (owner) | `appendMessageRoute` → `handleAppendMessage` | Append a message turn. |
+| `/api/conversations/:id/share` | POST | session (owner) | `shareRoute` → `handleShareConversation` | Publish a conversation to an unlisted share slug. (Was shadowed by the compound conversations branch pre-route-table — the pattern inventory made the collision visible and fixed it.) |
+| `/api/shared/:slug` | GET | none | `getSharedRoute` → `handleGetShared` | Read a shared conversation. |
+| `/api/datasets` | GET | none (session enriches) | `datasetsRoute` | The corpus catalog + starter questions (the UI's empty state — content from the API, never hardcoded in the client). Session-gated datasets carry `requires` (the estate permission, e.g. `the ai-preview permission (id.oimlsmart.org)`) and `enabled` reflects the permission, not just login. |
+| `/api/memories` | * | session | `memoriesRoute` → `handleMemories` | Personalized memory files (#171): GET list / POST create-update (≤10 files × 8k chars) / DELETE by id — every query owner-filtered by the session sub. Selected ids ride `/api/ask` as `memories: [id…]` (≤4 per ask, injected as one bounded trusted-user-facts note; the selection salts the answer cache). |
+| `/health` | GET | none | `healthRoute` | Liveness + deployed `index_version` (the deploy-drift guard reads this). |
+| `/api/ask`, `/v1/ask` | POST | tier | `askRoute` → `handleAsk` | The answer contract: streamed or JSON answer, citations, typed blocks, context echo. Quotas per tier. Optional `datasets: [id…]` narrows the corpora searched (server-intersected with session permissions; an explicitly-empty list is a 400); optional `memories: [id…]` selects the member's memory files to inject. Both selections salt the answer cache — a scoped or memory-flavored answer never serves a plain ask. |
+| `/api/absence`, `/v1/absence` | POST | tier | `absenceRoute` | Provable absence: exhaustive enumeration over a standard's model plane; `{ verdict: absent \| present, enumerated, matches }`. |
+| `/api/verify`, `/v1/verify` | POST | tier | `verifyRoute` | Self-verification battery over a supplied (query, answer): quote anchors, unit references, citations present + judged faithfulness. |
+| `/api/lane`, `/v1/lane` | POST | tier | `laneRoute` | Direct retrieval against a comparison index (dense + lexical fused), bypassing the full pipeline — for the annealment matrix and the /compare demo. |
+| `/api/search`, `/v1/search` | POST | tier | `searchRoute` → `handleSearch` | Passage search (retrieval without generation). |
+| `/api/feedback` | POST | none | `feedbackRoute` | Thumbs up/down keyed by query hash (privacy: hashes only). |
+| `/admin/enrich`, `/v1/admin/enrich` | POST | ADMIN_TOKEN | `handleEnrich` | Contextual enrichment / embed+upsert through the vector adapter (the ONLY wire-stage upsert door). |
+| `/admin/section`, `/v1/admin/section` | POST | ADMIN_TOKEN | `handleSectionUnit` | Build/serve depth-1 section-summary units. |
+| `/admin/vectors` | POST | ADMIN_TOKEN | `handleVectors` | Read/query vectors for ops. |
+| `/admin/caption` | POST | ADMIN_TOKEN | `handleCaption` | Figure captioning (vision lane). |
+| `/assets/*` | GET | none | `unitAssetRoute` | Immutable unit-keyed figure images (`u:<id>.<ext>`) from R2, 1-year immutable cache. |
+| `/api/research`, `/v1/research` | POST | session (member) | `researchRoute` → `handleResearch` | Deep-research dossier loop — members only (research spend stays with humans). |
+| `/admin/judge`, `/v1/admin/judge` | POST | ADMIN_TOKEN | `handleJudge` | LLM-as-judge scoring (promotion gates). |
+| `/v1/admin/keys` | POST | ADMIN_TOKEN | `handleCreateKey` | Issue an API key (`oiml_<hex>`, shown once). |
+| `/v1/admin/keys` | GET | ADMIN_TOKEN | `handleListKeys` | List keys (no secrets). |
+| `/v1/admin/stats` | GET | ADMIN_TOKEN | `adminStatsRoute` | 7-day telemetry: queries/day/tier, spend by model, feedback, error rate; prunes >90d rows. |
+| `/docs/:slug.html`, `/docs/:slug.anchors.json` | GET | public | `docsRoute` | Rendered publication documents (metanorma-mirror layer 1) served from R2 under `docs/`, immutable cache; the anchors map (clause number → heading anchor id) powers citation deep links. Public OIML content only. |
+| `/admin/enrich` | POST | ADMIN_TOKEN | `handleEnrich` | modes: default (context+embed+upsert in place), `context` (generate only, KV-cached), `ab` (generate at an explicit effort with NO side effects — the experiment lane; accepts an admin-gated prompt override for judged comparisons). |
+| anything else | any | — | — | `404 not_found`. |
+
+## Request semantics worth naming
+
+- **`effort` on ask** (`/api/ask`, `/v1/ask`): `"low"` (default, 1 quota unit) or `"medium"` (member lane, raises reasoning effort AND the output budget — `effortBudget()`; 2 quota units). Effort changes the answer, so it salts both answer caches alongside the datasets/memory selections (`requestEffort`/`answerEffort` in config.ts).
+- **`max_iterations` on research**: 1–3, clamped server-side.
+
+## Dispatch semantics
+
+`fetch` = OPTIONS preflight (204) → `matchRoute(ROUTES, method, path)` →
+handler with `{ env, req, ctx, url, path, params }` → 404. Patterns are
+segment-exact with `:param` capture and a trailing `*` wildcard
+(`/assets/*`); entry order is irrelevant because no pattern overlaps
+another. The `/api` and `/v1` twins deliberately share one handler so
+the tier split can never diverge between the two publications.
+
+## Isolation invariants (binding lint, `npm run lint:wrangler`)
+
+No route in this worker may reference the internal tier: no ISO index
+binding, no internal R2, no internal tokens. Federation happens through
+the `INTERNAL_SERVICE` binding inside the ask pipeline only, gated by a
+member session. The lint fails CI on any violation.

package/docs/spec-pipeline.md

@@ -0,0 +1,126 @@
+# Pipeline stage contracts
+
+The retrieval pipeline (`worker_public`) is a fixed-sequence registry of
+stages (`src/stages/index.ts`), each a self-contained module
+implementing the `Stage` interface (`src/stages/types.ts`). `retrieve()`
+(`src/pipeline.ts`) is composition only: it builds the context (the
+parallel embed/lexical prelude), runs the registry, and projects the
+result.
+
+## The contract vocabulary
+
+- **Context** (`PipelineContext`) — the mutable state every stage reads
+  and writes: `env`, `query` (the user's original wording), `rq` (the
+  retrieval query), `u` (query understanding, nullable), `filters` /
+  `filter` (the dense doc/edition scope), `vector`, `lexicalHits`,
+  `matches` (the candidate pool, Vectorize-match shape), `hits` (the
+  ranked pool), `finalHits` (the answer window), `glossary`, `opts`.
+- **Guard** (`when`) — a pure read of the context; absent = always runs.
+- **Failure mode** — `additive`: a throw is logged, the pipeline
+  continues with the context as the previous stage left it (the lane's
+  results were not written). `blocking` (default): the throw propagates
+  to the caller of `retrieve()`.
+
+Adding a stage = one file in `src/stages/` + one registry entry. No
+edits to existing stages, the runner, or `retrieve()`. A stage that
+needs a new context field declares it on `PipelineContext` (the field's
+owner is the stage that writes it).
+
+## Registry order and invariants
+
+Order is load-bearing. The phase structure:
+
+```
+candidate lanes → pool open → pool-level merges → refinement → window assembly
+```
+
+| # | Stage | Guard | Failure | Writes | Contract |
+|---|-------|-------|---------|--------|----------|
+| 1 | `dense` | — | blocking | `matches`, may mutate `filters.edition` | The primary Vectorize query. Three shapes: optimistic reuse (identical query + no filter only), filtered (with guessed-edition-pin drop and sparse-filter widen — the widen APPENDS unfiltered hits behind the filtered set), plain. Never unions optimistic hits into a diverged query. |
+| 2 | `hyde` | `u.hypothetical_answer && !filter` | additive | appends `matches` | Hypothetical-answer vector search; candidates enter at `hydeDiscount`. |
+| 3 | `glossary` | `env.GLOSSARY && vector` | additive | `glossary` | Vocabulary link: dense glossary candidates + cross-encoder rerank; top-3 distinct concepts. MUST run before the concept-graph and steering stages — they consume `glossary`. |
+| 4 | `concept-graph` | `glossary.length && env.DB && vector` | additive | appends `matches` | Linked terms → defining doc numbers (D1, parallel per-term) → metadata-filtered dense merge at `conceptGraphDiscount`. |
+| 5 | `graph-lane` | `opts.graphDocNumbers && vector` | additive | appends `matches` | Caller-resolved family/successor doc numbers merged at `graphLaneDiscount`. |
+| 6 | `multi-query` | `u.query_variants` | blocking (per-variant catches inside) | REPLACES `matches` | RRF-fuses the primary ranking with each variant's (k=60, top `retrieveK`). |
+| 7 | `sub-query` | `u.complexity === "complex" && u.sub_queries` | blocking (per-sub catches inside) | appends `matches` | Complementary-perspective union at `subQueryDiscount` (no RRF). |
+| 8 | `pool-open` | — | blocking | `hits` | Converts the candidate pool to the `Hit` shape. Last writer of `matches`; every later stage operates on `hits`. |
+| 9 | `lexical-union` | `lexicalHits.length` | blocking | appends `hits` | Full-corpus BM25 hits dense missed (dense metadata wins on id collision). |
+| 10 | `federate` | `opts.federate` | additive (callback catches transport) | appends `hits` | Internal ISO/IEC passages at `federateDiscount`. |
+| 11 | `seal` | `opts.sealScope` | blocking | filters `hits` | The declared context's hard cut: nothing outside the declared family reaches rerank. (The lexical lane is sealed at the SOURCE, in the prelude — not here.) |
+| 12 | `overview-demote` | — | blocking | mutates `hits[].score` | Overview boilerplate demotion (`overviewDemotion`). |
+| 13 | `family-boost` | boost only under `filter.doc_number`; sort always | blocking | mutates `hits[].score`, sorts | Family chunks decisively boosted for doc-scoped queries; the sort establishes the rerank-failure fallback order. |
+| 14 | `rerank` | `hits.length > 1` | additive | `hits[].rerank_score`, sorts, family pin | Cross-encoder scores; vector order is the designed fallback. Post-rerank family pin for doc-scoped queries. |
+| 15 | `lexical-rrf` | `hits.length > 1 && lexicalHits.length` | blocking | REPLACES `hits` order | RRF fusion with the full-corpus lexical ranking. Runs even when rerank failed (additive semantics preserve this). |
+| 16 | `std-ref-nudge` | query names ISO/IEC/ASTM/EN | blocking | `hits[].rerank_score`, sorts | Standard-reference nudge: chunks CARRYING such a citation get `stdRefNudgeSpread × spread` — the citing clause is the answer to "which standard does X invoke", and generic family prose otherwise fills the window (l5a measured). |
+| 17 | `edition-cover` | `!filters.edition && hits.length > 1` | additive | appends `hits` | Registry-driven cover: when a pool holds only stale editions of a document whose ACTIVE edition the documents registry knows, fetch the current edition's chunks and add at `editionCoverDiscount` — steering needs the successor present to demote. |
+| 18 | `corpus-scope` | `opts.datasetScope` | blocking | filters `hits` | Dataset scope (the sidebar toggles): drops hits whose corpus the request excludes. LAST of the pool-assembly stages — the lexical union above refills the pool after rerank, so filtering earlier let excluded corpora back in. Corpora the toggle model doesn't name pass untouched. |
+| 19 | `term-nudge` | `u.term` | blocking | `hits[].rerank_score`, sorts | Clause whose head IS the asked term gets `termNudgeSpread × spread` (decisive). |
+| 20 | `concept-steer` | `glossary.length && hits.length > 1` | blocking | `hits[].rerank_score`, sorts | Vocabulary-link families boosted (`conceptSteerSpread`). |
+| 21 | `edition-steer` | `!filters.edition && hits.length > 1` | blocking | `hits[].rerank_score`, sorts | Cross-pub recency boost + family-relative superseded-edition demotion (spread-scaled). |
+| 22 | `structural-propagate` | — | blocking | REPLACES `hits` | FABLE TreeExpansion: score blends along the clause tree. |
+| 23 | `diversity` | — | blocking | `finalHits` (from `hits`) | Per-publication caps (1 overview / 2–3 clauses; global overview cap 2/6); window cut to `rerankKeep`. FIRST writer of `finalHits`. |
+| 21 | `typed-pin` | pin families resolvable | blocking (inner parent-fetch additive) | `finalHits` | Answer-contract v2: one typed unit guaranteed a slot (+ small-to-big parent fetch at `smallToBigDiscount`). |
+| 22 | `section-descent` | a ranked depth-1 summary has children | additive | `finalHits` | Summary node → top child clauses at `sectionDescentDiscount`; the summary retires when children answer. |
+| 23 | `dedup` | — | blocking | `finalHits` | FABLE ancestor-descendant same-chain collapse (≥0.5 text overlap). |
+| 24 | `window-floor` | — | blocking | filters `finalHits` | Evidence-budget cut at `windowFloorFraction` of top; typed/family/unscored exempt; never fewer than two. |
+
+## Ordering dependencies (why the order is what it is)
+
+- **Lanes before the pool opens** (1–7): every candidate lane competes in
+  one pool; the pool is frozen at `pool-open`.
+- **`glossary` before `concept-graph`/`concept-steer`/`typed-pin`**: all
+  three consume the vocabulary link.
+- **`dense` before any lane that merges**: lanes are additive to the
+  primary ranking, never replacements (except `multi-query`, which
+  REPLACES `matches` — deliberate: RRF-fused variants subsume the
+  primary ranking).
+- **`seal` before `overview-demote`/`rerank`**: the declared context is
+  a hard scope, not a preference — steering and reranking happen WITHIN
+  it.
+- **`rerank` before every steering stage** (16–18): steering is
+  spread-scaled over rerank scores; steering before rerank would be
+  erased by the re-sort.
+- **`structural-propagate` after steering, before `diversity`**:
+  propagation re-scores the full ranked pool; diversity then reads the
+  final order.
+- **`diversity` is the FIRST writer of `finalHits`**: the window
+  assembly stages (21–24) operate only on the window.
+- **`window-floor` LAST**: it is the terminal budget cut; anything
+  appended after it would escape the evidence-budget principle.
+
+## Prefetch semantics (concurrent lane I/O)
+
+A stage may declare `prefetch(c)`: kick its INDEPENDENT I/O off into
+`c.lane[stage.name]` (a promise bag the stage owns). The runner invokes
+every stage's prefetch — guard-checked — BEFORE running any stage, so
+the independent lanes (hyde, glossary, graph-lane, multi-query,
+sub-query) overlap with each other and with the dense lane instead of
+serializing: ~5 summed round trips become ~1. `run()` awaits its own
+promise and merges.
+
+The invariants:
+
+- **Only pre-pipeline state** — prefetch may read `u`, `vector`, `opts`,
+  never another stage's output. concept-graph deliberately does NOT
+  prefetch (its D1 lookup consumes glossary's results).
+- **Merge order is registry order** — concurrency changes when I/O
+  completes, never when merges apply. Retrieval determinism is
+  untouched (the annealment gate re-verifies this).
+- **Failure semantics unchanged** — a rejected prefetch promise throws
+  at the await inside `run()`, where the stage's failure mode applies.
+
+## The prelude (not stages, deliberately)
+
+`retrieve()` resolves the query fold, runs the dense embed and the
+full-corpus lexical prefilter **in parallel**, and seal-filters the
+lexical lane at the source. This parallelism is load-bearing (Option C:
+the optimistic vector may already be resolved; embed and lexical must
+not serialize) — it stays in the composition layer, not in stages, so
+no stage boundary can introduce a serial round-trip on the hot path.
+
+## Verification
+
+- `tests/pipeline.test.ts` — the registry composition and each stage's
+  invariant over an in-memory fixture environment (no network).
+- `tests/golden/` ×3 + annealment ×6 — the live gates (every threshold
+  change re-runs them; see `config.ts` THRESHOLDS docstrings).

package/docs/vector-adapter.md

@@ -0,0 +1,88 @@
+# The vector adapter — producer formats → Vectorize, through one door
+
+`ingest/vector_adapter.py` is the single boundary every chunk crosses to
+become a vector. Producer projectors (`ingest/primmel.py`, `ingest/mko.py`,
+`ingest/parse.py`) turn producer formats into dicts; the adapter turns
+those dicts into the wire shape — validated, target-checked, and
+schema-locked.
+
+## Why one door (the failure modes it forecloses)
+
+Hand-rolled per-script metadata — no tie between a chunk and its TARGET
+index — fails in four characteristic ways, each of which the adapter
+makes structurally impossible:
+
+- **cross-index contamination** — a chunk written to an index its corpus
+  does not belong to (target gating refuses at adapt time and payload
+  time);
+- **corpus vocabulary drift** — the same corpus under different names,
+  or edition values in inconsistent forms (the registry and the schema
+  are the single source of truth);
+- **unciteable anchors** — producer UUID anchors reaching citations
+  (stripped at build time; serving hides any that remain as defense in
+  depth);
+- **wire-time size failures** — metadata over the index limit,
+  discovered only at the upsert call (validated before any wire call).
+
+## The contract
+
+### ChunkMetaModel — the wire schema
+
+Mirrors the serving contract (`workers/worker_public/src/pipeline.ts`
+`ChunkMeta`). Pydantic (the ecosystem serialization rule: framework, not
+hand-rolled `to_json`). `extra="forbid"` — an unknown field from a
+producer is a SCHEMA CHANGE, decided here, never an accident at the wire.
+
+Fields: `doc_id, docidentifier, doctype, doc_number, edition, language,
+clause_anchor, clause_title, tier, corpus, text_ref, status,
+superseded_by, unit_id, block, unit_hash, producer, source_lane,
+linked_clause, linked_document, section_summary, child_anchors, ctx,
+chunk_text`.
+
+Validators (build-time, before any wire call):
+
+- **corpus registry** — the value must be registered (`PRODUCTION_CORPORA`
+  ∪ `LANE_CORPORA`). New corpora are added HERE, never ad hoc in a script.
+- **clause_anchor sanity** — producer UUID anchors are stripped to `""`
+  (they are unciteable; serving treats them as garbage — this stops them
+  at the source).
+- **chunk_text cap** — 2,800 chars (the 40016 lesson).
+- **wire size** — serialized metadata ≤ 9,000 bytes (Vectorize hard limit
+  10KB).
+
+### Target gating — the structural guard
+
+`TARGET_CORPORA` maps every index to the corpora it may hold:
+
+| Target | Legal corpora |
+|---|---|
+| `production` (`idx_oiml_public_v2`) | oiml, dirty, clean, synthetic, smart-model |
+| `exp_plain` / `exp_adoc` / `exp_mko` / `exp_composed` | the same-named lane corpus |
+| `primmel` | primmel |
+| `primmel_flat` | primmel (the ablation indexes the same corpus, unlinked) |
+
+`normalize_chunk(raw, target=…)` refuses a chunk foreign to its target at
+ADAPT time; `chunk.upsert(vector, target)` refuses again at payload-build
+time. A lane corpus can no longer enter production through any code path
+that builds payloads through the adapter.
+
+### Producer remaps live here, not in scripts
+
+The production MKO flow remaps `corpus mko → oiml` (+`producer: mko`,
+`tier: curated`) inside `normalize_chunk(target="production")`; the lane
+MKO flow keeps its `exp_mko` namespace. Previously this remap lived in
+`ingest/enrich.py` — buried in a driver, duplicated by hand elsewhere.
+
+## Serving mirror
+
+The TypeScript `ChunkMeta` interface in `pipeline.ts` is the serving-side
+read of the same contract (plus its own optional fields). Change either
+side deliberately: adapter schema → wire shape → serving interface.
+
+## Call sites
+
+- `scripts/index_comparison_lanes.py` — every lane upsert builds its
+  payload through `normalize_chunk(...).upsert(..., target=ADAPTER_TARGET[lane])`.
+- Future producers (and the metanorma-ai deployment package's
+  document→vectors pipeline component) adopt the same door: project to
+  dicts, adapt through this module, never hand-roll a metadata dict again.

package/package.json

@@ -0,0 +1,70 @@
+{
+  "name": "@konneal/engine",
+  "workspaces": [
+    "workers/*"
+  ],
+  "scripts": {
+    "typecheck": "tsc -p workers/worker_public",
+    "deploy:public": "npm run deploy -w workers/worker_public",
+    "dev:public": "npm run dev -w workers/worker_public",
+    "deploy": "./scripts/deploy.sh",
+    "test:golden": "node scripts/eval.mjs",
+    "test:e2e": "node tests/e2e.mjs",
+    "lint:wrangler": "python3 scripts/lint_wrangler.py",
+    "test:ui": "node tests/ui.mjs",
+    "test:bridge": "node --test tests/bridge.test.mjs",
+    "test:units": "node --test --experimental-strip-types tests/*.test.ts",
+    "build": "rm -rf dist dist-types && esbuild workers/worker_public/src/index.ts workers/worker_public/src/config.ts workers/worker_public/src/refusal.ts workers/worker_public/src/profile.ts workers/worker_public/src/requestScope.ts --bundle --format=esm --splitting --platform=neutral --outdir=dist --loader:.md=text && tsc -p tsconfig.build.json && cp dist-types/worker_public/src/*.d.ts dist/ && mkdir -p dist/prompts && cp workers/worker_public/prompts/*.md dist/prompts/",
+    "prepare": "npm run build",
+    "lint:ports": "node scripts/lint-ports.mjs"
+  },
+  "devDependencies": {
+    "@cloudflare/workers-types": "^5.20260911.1",
+    "esbuild": "^0.28.2",
+    "jsdom": "^29.1.1",
+    "playwright": "^1.62.1",
+    "yaml": "^2.9.1"
+  },
+  "dependencies": {
+    "@astrojs/markdown-satteri": "^0.4.1",
+    "@astrojs/mdx": "^4.3.14"
+  },
+  "version": "0.1.0",
+  "description": "The Konneal engine: the publisher-agnostic build pipeline and API plane for standards intelligence (retrieval, answer contract, verdicts, evaluation).",
+  "license": "BSD-3-Clause",
+  "type": "module",
+  "main": "./dist/index.js",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    },
+    "./setProfile": {
+      "types": "./dist/profile.d.ts",
+      "default": "./dist/profile.js"
+    },
+    "./config": {
+      "types": "./dist/config.d.ts",
+      "default": "./dist/config.js"
+    },
+    "./refusal": {
+      "types": "./dist/refusal.d.ts",
+      "default": "./dist/refusal.js"
+    },
+    "./requestScope": {
+      "types": "./dist/requestScope.d.ts",
+      "default": "./dist/requestScope.js"
+    }
+  },
+  "files": [
+    "dist",
+    "workers",
+    "docs",
+    "profile",
+    "scripts/gen_profile.mjs",
+    "README.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/profile/corpora.yaml

@@ -0,0 +1,5 @@
+production: [pub, dirty, clean, synthetic, model]
+lanes:
+  exp_a: [exp_a]
+  exp_b: [exp_b]
+  glossary: [glossary]

package/profile/datasets.yaml

@@ -0,0 +1,14 @@
+datasets:
+  - id: pub
+    label: Fixture Publications
+    description: The fixture publisher's corpus
+    corpora: [pub, dirty, clean, synthetic]
+    note: >-
+      Some passages come from the fixture corpus — cite them the same
+      way as every other passage.
+  - id: internal
+    label: Internal corpus
+    description: An access-restricted corpus proving the permission gate
+    session: true
+    permission: preview
+    corpora: [internal]

package/profile/prompts.yaml

@@ -0,0 +1,5 @@
+vars:
+  assistant_identity: >-
+    the fixture assistant — a public service answering questions about
+    the fixture publisher's documents
+  refusal_sentence: I don't have information on this in the indexed fixture documents.

package/profile/publisher.yaml

@@ -0,0 +1,17 @@
+# The engine's fixture profile — the reference matrix starts here. A
+# real deployment (e.g. oimlsmart/ai) carries its own publisher facts;
+# this fixture exists so the engine's tests and CI run against
+# declared data, never hardcoded ones.
+id: fixture
+name: Fixture
+full_name: The Fixture Publisher
+product_name: Fixture Answers
+description: >-
+  A minimal publisher profile exercising every declared surface: an
+  open dataset, a permission-gated dataset, production and lane
+  corpora, prompt vars and retrieval vocabulary.
+domains:
+  public: fixture.example.org
+identity:
+  issuer: https://id.fixture.example.org
+codec: plain-slug

package/profile/retrieval.yaml

@@ -0,0 +1 @@
+process_expansion: " fixture certification system framework application evaluation"

package/profile/sources.yaml

@@ -0,0 +1,5 @@
+corpora:
+  clean: { repo: fixtures/corpus, note: the engine's fixture corpus }
+bibliography: {}
+terminology: {}
+models: {}

package/profile/ui.yaml

@@ -0,0 +1,7 @@
+suggestions:
+  - What is in the fixture corpus?
+  - Which documents does the fixture publisher issue?
+models_disclosure:
+  - { role: Answers, model: fixture-answer-model }
+smoke:
+  - { label: sanity, query: What is in the fixture corpus?, expect: fixture }