@konneal/engine 0.1.0

package/docs/INGEST-ARCHITECTURE.md

@@ -0,0 +1,158 @@
+# Ingest Architecture — Object Input Interface → RAG Data Preprocessor → RAG Worker
+
+*Formalizes the pipeline shape (2026-08-27). Companion pieces:
+`SOTA-STAGE-SPECS.md` (schemas), `METANORMA-AI-SERIALIZATION.md` (upstream
+serialization contract), `ROADMAP-SOTA.md` (phasing).*
+
+---
+
+## The principle: facts in source data, derivations in the preprocessor
+
+The relaton audit made this concrete. Relaton's `status` FIELD lies in 58
+records (claims in-force while carrying a successor edge) — because it is a
+**derived fact stored as data**, the classic copy-drift bug. The
+preprocessor therefore DERIVES status from successor edges and ignores the
+field; nothing needed "fixing" for serving to be correct.
+
+What derivation cannot invent: **missing edges**. 36 of 224 families have no
+active edition because a successor edge was never recorded (e.g. `B 18:2016`
+and `B 18:2017` both terminal). Those are genuine upstream gaps — they must
+be fixed in relaton-data-oiml; the preprocessor's job is to SURFACE them
+(the registry's no-active families are the worklist), not hide them.
+
+| Data problem | Fix where | Mechanism |
+|---|---|---|
+| Status field contradicts edges (58) | nowhere — derive | preprocessor: `derived_status` |
+| Language instances carry primaries | nowhere — collapse | preprocessor: canonical-id dedup |
+| Non-publication records (Bulletins) | nowhere — filter | preprocessor: series filter |
+| Missing successor edges (36 families) | **relaton-data-oiml** | registry worklist → upstream PRs |
+| Wrong/missing citation edges | **relaton-data-oiml** | graph build report → upstream PRs |
+
+The loop: preprocessor emits a data-quality report every build; the report
+drives upstream fixes; upstream fixes flow back on the next build. Data
+gets better because serving keeps score, not because anyone audits YAML by
+hand.
+
+## The three layers
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│ OBJECT INPUT INTERFACE (OII)                                     │
+│   The typed object contract. Four NATIVE object systems, each    │
+│   serialized from its own lutaml-model gem — never renderings:   │
+│     • Metanorma  — document content model  (metanorma-document)  │
+│     • Relaton    — bibliographic records   (relaton)             │
+│     • Glossarist — terminology concepts    (glossarist)          │
+│     • PubID      — canonical identifiers   (pubid)               │
+│   Contract: the AI-serialization node projection (documents +     │
+│   typed nodes + edges), schema_versioned, validated at entry.     │
+└──────────────────────────┬───────────────────────────────────────┘
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ RAG DATA PREPROCESSOR (RDP) — ingest/*, invariant-gated          │
+│   identity      pubid-canonical DocumentRecord → D1 registry     │
+│                 (derived status, active flags — the SSOT)         │
+│   structure     typed nodes → ChunkRecordV2 (per block type)      │
+│   derivation    registry + graph projection (relaton edges +      │
+│                 glossarist defines edges)                         │
+│   enrichment    contextual contexts (quality-first lane, KV-      │
+│                 cached, content-hash invalidated)                 │
+│   embedding     per-type embed_input → Vectorize (public/internal │
+│                 split — isolation is structural)                  │
+│   verification  invariants per stage + data-quality report →      │
+│                 upstream worklist                                 │
+│   orchestration INDEX_VERSION bump → cache flush protocol         │
+└──────────────────────────┬───────────────────────────────────────┘
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ RAG WORKERS — serving only, never parse source data              │
+│   rag-public   ask/search pipeline + registry notes + graph lane  │
+│   rag-internal ISO federation (isolation-enforced)                │
+│   rag-mcp      MCP tools for the agent ecosystem                  │
+│   They consume: Vectorize + D1 (registry, graph) + KV (caches)    │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+## Current state vs the target
+
+| Component | State |
+|---|---|
+| OII | de-facto: parse.py (HTML-first) + graph.py (relaton/glossarist). Target: consume the four native object serializations; the AI-serialization proposal (metanorma#592) is the upstream contract |
+| Identity SSOT | ✅ documents registry in D1 (derived status, active flags, supersession chains) |
+| Registry serving | ✅ edition notes injected into doc-scoped asks; `GET /api/documents?family=` public API |
+| Graph projection | ✅ 7,128 nodes / 6,486 edges in D1; query lane live (defined_terms → defines → candidates) |
+| Chunking | prose-only today; ChunkRecordV2 typed blocks = next build (tables/equations/requirements) |
+| Enrichment | ✅ 98.7% corpus (the residual ~400 are persistent empty enrichments), KV-cached, replayed from the durable record after full restores |
+| Verification | unit suites + contract tests in CI (`npm run test:units`, pydantic↔TS wire contract, schema-union drift); the data-quality report loop = next build |
+| Orchestration | manual commands today; one-command pipeline (`ingest run --doc X`) when the above land |
+
+## Corpus operations (the wire loop, as of 2026-09-08)
+
+```
+parse            corpora → artifacts/chunks.jsonl (identifiers sanitized,
+                 placeholders never win — language markers and :0000/:XXXX
+                 fall to the slug-derived identity)
+retrieval-plane  primmel export → artifacts/model_retrieval_chunks.jsonl
+                 (retrieval text/facets); the projection derives
+                 artifacts/model_typed_chunks.jsonl (the ONLY unit_id/
+                 block source) — separate files: one shared name had the
+                 two derivations overwriting each other
+embed            content-aware resume: artifacts/embed_text_hashes.json
+                 (id → text hash) — changed text re-embeds, untouched ids
+                 never do (model chunk ids are content-independent hashes;
+                 id-keyed resume alone would pair stale vectors with fresh
+                 metadata)
+upsert           the internal worker's /admin/sync binding route
+                 (rag-internal.<account>.workers.dev — no REST token
+                 needed); the CLI's REST path defaults to the production
+                 index idx_oiml_public_v2
+restore          scripts/restore_missing.py [sources...] [--batch N --pace S] —
+                 reconcile's twin: reconcile deletes strays, restore fills
+                 gaps (presence probe, then upsert only missing ids; paced
+                 under the rolling Vectorize upsert quota — sustained
+                 25-50-vector batches clamp after ~1k vectors; 10 × 25s
+                 sustains; skips hard-failed batches, rerun to pick them
+                 up)
+enrich-replay    scripts/replay_enrichment.py [--apply --pace S] — REQUIRED
+                 after every full upsert OR raw restore: enrichment lives
+                 only in the index (and the KV context cache), so both
+                 overwrite it with raw text; the replay re-embeds
+                 context+text from the durable record
+                 (artifacts/enriched-contexts.jsonl) via the binding —
+                 zero model generation, ≈$0.30
+reconcile        scripts/reconcile_index.py — enumerate the index (wrangler
+                 list-vectors), diff against the canonical chunk set
+                 declared ONCE in ingest/config.py
+                 (CANONICAL_CHUNK_SOURCES — four derivations: prose parse,
+                 retrieval plane, projection, MKO typed units), delete
+                 strays (--apply). Upserts never delete; this closes the
+                 loop.
+cache            scripts/invalidate_answer_cache.py (or wrangler kv put
+                 sys:corpus_gen) after any corpus surgery
+assets           scripts/fix_figure_assets.py [--apply] — unit assets must
+                 stay vision-readable (black-on-transparent rasters read
+                 as solid black after alpha flattening); detects and
+                 re-uploads white-flattened
+documents        scripts/upload_documents.py [--apply] — the rendered
+                 publications for citation deep links (R2 docs/<slug>.html
+                 + clause-anchor maps, served at /docs/*); rerun when the
+                 clean corpus's renderings change
+graph            ingest.cli graph (build) + graph --corpus apply (D1,
+                 wrangler; absolute --file path)
+gates            scripts/gates.sh — golden ×N + annealment ×M
+```
+
+The vector adapter (`ingest/vector_adapter.py`, contract in
+`docs/vector-adapter.md`) remains the ONLY door from any producer to any
+index: wire schema + target gating.
+
+## PubID as the identity backbone
+
+Node ids, registry keys, and chunk metadata all key on the canonical
+identifier (`OIML R 60-1:2021`). Today graph.py normalizes identifiers with
+its own regex; the target is the **pubid** gem's parser/generator as the
+single authority (parse any spelling → canonical form → round-trip), so
+`R60`, `OIML R60:2021 (E)`, and `r 60-1` all collapse to one identity
+before they touch the registry. This is also an upstream piece of the
+AI-serialization bundle: identifiers in the projection should BE pubid
+objects serialized.

package/docs/MCP.md

@@ -0,0 +1,92 @@
+# OIML SMART AI — MCP Server
+
+*Agent-native access to the OIML publications corpus via the Model
+Context Protocol. For chat/API usage see [API.md](API.md); MCP is the
+machine contract — including the publication registry, which is
+deliberately NOT part of the chatbot API.*
+
+---
+
+## 1. Connecting
+
+**Endpoint:** `https://rag-mcp.oimlsmart-06c.workers.dev/mcp`
+**Transport:** Streamable HTTP (MCP `2025-06-18`) — JSON-RPC 2.0 over
+`POST /mcp`, stateless. `GET /` returns a service descriptor.
+
+**Claude Desktop** (`claude_desktop_config.json`):
+```json
+{
+  "mcpServers": {
+    "oiml": {
+      "type": "http",
+      "url": "https://rag-mcp.oimlsmart-06c.workers.dev/mcp"
+    }
+  }
+}
+```
+**Any MCP client:** point it at the URL above; no session handshake
+state is required beyond the standard `initialize` call.
+
+**Authorization:** the server is public-tier by default. Setting the
+`RAG_API_KEY` secret (operator) raises its quota class; per-caller auth
+is passed through the underlying API key if configured.
+
+## 2. Tools
+
+### `oiml_search` — ranked passage retrieval
+```jsonc
+// input
+{ "query": "maximum permissible error load cell", "top_k": 3 }  // top_k 1–10
+```
+Returns passages as text blocks:
+```
+OIML R 76-1:2006 §2.5.4 — 2.5.4 Maximum permissible error, mpe (T.5.4.4)
+<snippet>
+```
+Use for: scanning the corpus, gathering context, cheap lookups (no
+generation cost).
+
+### `oiml_ask` — grounded, citation-linked answers
+```jsonc
+{ "query": "What does OIML R 60-1 require for verification intervals?" }
+```
+Returns the full answer plus a `Sources:` line of cited passages. Every
+claim cites the exact publication and clause; normative values carry
+verbatim quote anchors. Off-corpus questions return the canonical
+refusal sentence.
+
+### `oiml_documents` — publication registry (editions & status)
+```jsonc
+{ "family": "R-60" }   // series letter + number, e.g. "R-60", "B-18", "D-11"
+```
+Returns every edition of the family with derived status:
+```
+OIML R 60:1991 — superseded → superseded by OIML R 60:2000
+OIML R 60:2021 — in-force [ACTIVE]
+OIML R 60-1:2017 — superseded → superseded by OIML R 60-1:2021
+OIML R 60-1:2021 — in-force [ACTIVE]
+…
+```
+`[ACTIVE]` is DERIVED from successor edges (terminal node of the
+supersession chain), not from a status field — relaton's status field
+contradicts its own edges in ~9% of records. Use for: "current/latest
+edition", edition history, supersession questions. A family with no
+`[ACTIVE]` line means the bibliographic data has a gap (missing
+successor edge) — the registry surfaces it rather than guessing.
+
+## 3. Protocol notes
+
+- Notifications (no `id`) → `202`, no body
+- Unknown method → `-32601`
+- Tool failures → result with `isError: true` and the message (never a
+  dropped call)
+- `initialize` responds with capabilities `{tools:{}}`; `tools/list`
+  returns the three tool definitions with JSON schemas
+
+## 4. Implementation & isolation
+
+The server (`workers/worker_mcp`, `rag-mcp`) holds **no corpus access
+of its own**: `oiml_search`/`oiml_ask` proxy the rag-public API (so
+audience isolation stays enforced in exactly one place), and
+`oiml_documents` reads the derived D1 registry via a read-only binding
+(public OIML metadata only — no ISO/IEC content).

package/docs/METANORMA-AI-SERIALIZATION.md

@@ -0,0 +1,247 @@
+# Metanorma AI Serialization — Adapter Format Proposal
+
+*Proposed output flavor for `metanorma-document` (and the concrete answer to
+[metanorma/metanorma#592](https://github.com/metanorma/metanorma/issues/592)):
+a machine serialization of a Metanorma document designed for RAG ingestion
+and AI training corpora. Draft v1.0.0-proposal, 2026-08-26.*
+
+---
+
+## 1. Positioning
+
+*Evidence alignment (2025–26): the projection's shape is exactly what the
+normative-RAG literature converges on — [ETSI arXiv:2604.09868] recovers
+InfoUnits + parthood/citation edges from PDFs by heuristic and shows
+structure preservation is the retrieval win; [STC arXiv:2605.00318]
+quantifies typed table rows (MRR 0.36→0.59); [RDR2 arXiv:2510.04293],
+[SF-RAG arXiv:2602.13647], [SPIRE arXiv:2604.20849] all consume
+structure trees/natively-hierarchical indexes; [MAHA arXiv:2510.14592]
+indexes tables as structured objects and equations as LaTeX+description.
+Metanorma can emit natively what all of them recover heuristically.*
+
+
+Metanorma already has the right substrate: `metanorma-document`
+deserializes semantic XML into typed lutaml-model classes and serializes
+them to XML/YAML/JSON natively. This proposal does NOT invent a new
+document model — it defines a **projection flavor**: a stable, typed,
+chunk-ready node graph derived from the existing model.
+
+```
+adoc source ──compile──▶ semantic XML ──Root.from_xml──▶ metanorma-document MODEL
+                                                          │
+                          ┌───────────────────────────────┤──▶ HTML / PDF (renderers)
+                          └──▶ AiSerialization::Root.to_json / .to_yaml / .to_xml   ◀── THIS PROPOSAL
+```
+
+Properties:
+
+- **Derived output** (like renderings): lossy by design, never a source
+  format; the model and semantic XML remain canonical.
+- **Encoding-agnostic**: implemented as lutaml-model classes with
+  `mapping` blocks — JSON, YAML, and XML serializations of ONE schema.
+  No hand-rolled `to_h`/`to_json` anywhere (house rule).
+- **Chunk-ready**: every node is self-contained (document identity +
+  breadcrumb + typed payload), so RAG consumers embed nodes without
+  re-deriving structure.
+- **Graph-preserving**: a flat edge list exposes structure and semantics
+  (containment, requirements→conformance tests, term links, citations).
+- **Training-corpus friendly**: provenance, license fields, and stable
+  node hashes make the same export usable as a curated AI-training
+  corpus (the #592 ask).
+
+## 2. Envelope
+
+```jsonc
+{
+  "schema": "metanorma.ai-serialization",       // const identifier
+  "schema_version": "1.0.0",                     // semver
+  "generated": {
+    "tool": "metanorma-document",
+    "model_version": "x.y.z",                    // gem version
+    "flavor": "oiml",                            // document flavor used
+    "timestamp": "2026-08-26T00:00:00Z"
+  },
+  "document": { /* DocumentIdentity, §3 */ },
+  "nodes":  [ /* typed nodes, §4 */ ],
+  "edges":  [ /* flat edge list, §5 */ ],
+  "datasets": { /* optional Glossarist/Relaton embeds, §6 */ },
+  "rights": { "license": "...", "copyright": "..." }   // corpus-safe provenance
+}
+```
+
+## 3. Document identity node
+
+```jsonc
+"document": {
+  "docidentifier": "OIML R 60-1",     // authoritative, never slug-derived
+  "part": "1",                         // or "annex": "A"
+  "edition": "2017",
+  "language": "en",
+  "doctype": "R",
+  "status": "in-force",                // in-force | superseded | withdrawn | joint
+  "superseded_by": null,
+  "title": "Metrological regulation for load cells — Part 1",
+  "family": "R-60",                    // series node for part graphs
+  "source": { "file": "document.xml", "hash": "sha256:…" },
+  "canonical_id": "OIML R 60-1:2017:en"
+}
+```
+
+## 4. Typed nodes
+
+Node classes map 1:1 onto existing model classes (right column = source in
+`metanorma-document`):
+
+| Node type | Source model class | Notes |
+|---|---|---|
+| `clause` | `BasicDocument::Blocks::BasicBlock` (+subclasses) | carries `anchor`, `obligation`, breadcrumb |
+| `table` | `BasicDocument::Tables::TableBlock` | columns/units/rows payload |
+| `formula` | `BasicDocument::AncillaryBlocks::FormulaBlock` + `StemElement` | asciimath/latex/description |
+| `figure` | `BasicDocument::Figure` | alt text, image ref |
+| `term` | `StandardDocument` term entries | links to Glossarist concept |
+| `requirement` | `Document::Components::Blocks::RequirementModel` | ModSpec projection |
+| `permission` | `…/PermissionModel` | ModSpec projection |
+| `recommendation` | requirement variants | ModSpec projection |
+| `conformance_class` / `conformance_test` | ModSpec classes | `modspec-ruby` objects |
+| `reference` | `BibData`/bibitem entries | Relaton key |
+| `amendment`/`change` | `BasicDocument::Change` | editorial vs technical |
+
+**Common fields (all nodes):**
+```jsonc
+{
+  "id": "n:tbl-4.1.2-1",         // stable: "n:" + model anchor/id
+  "type": "table",
+  "parent": "n:cl-4.1.2",        // containment (also in edges)
+  "anchor": "tbl-4.1.2-1",
+  "breadcrumb": ["4 Metrological requirements", "4.1 Classification"],
+  "obligation": "normative",     // when the model carries it
+  "hash": "sha256:…",            // content hash — enrichment invalidation
+  "text": "…"                    // human-readable display text
+}
+```
+
+**Typed payloads:**
+```jsonc
+// table
+{ "columns": [{ "label": "Load m", "unit": "e", "scope": "per verification interval" }],
+  "rows": [["0 ≤ m ≤ 5·10³", "0.5e"]],
+  "caption": "Maximum permissible errors",
+  "embed_text": "Table: Maximum permissible errors; columns: Load m [e], MPE; row: 0 ≤ m ≤ 5·10³ | 0.5e" }
+
+// formula
+{ "asciimath": "n_LC <= …", "latex": "n_{LC} \\leq …",
+  "described": "limit on the number of verification intervals" }
+
+// term
+{ "concept": "load-cell", "designation": "load cell",
+  "definition": "…", "vocab": "oiml-complete", "glossarist_id": "…" }
+
+// requirement (ModSpec projection)
+{ "identifier": "/req/oiml-r60-1/classification/nlc-limit",
+  "class": "/req/oiml-r60-1/classification",
+  "obligation": "requirement",
+  "statement": "The number of load cell verification intervals n_LC shall be within…",
+  "inherits": [], "subject": "load cell", "inherits:": [] }
+
+// conformance_test
+{ "identifier": "/conf/oiml-r60-1/classification/nlc-limit",
+  "class": "/conf/oiml-r60-1/classification",
+  "tests": "/req/oiml-r60-1/classification/nlc-limit",
+  "method": "Inspect the marking and the accompanying documents…" }
+
+// reference
+{ "key": "IEC61000-4-2", "cited": "IEC 61000-4-2:2008", "relaton_id": "…" }
+```
+
+`embed_text` is a PRODUCED convenience (deterministic serialization of the
+payload for embedding); consumers may ignore it and serialize their own way.
+
+## 5. Edges
+
+```jsonc
+{ "from": "n:tbl-4.1.2-1", "to": "n:cl-4.1.2", "kind": "part_of" }
+```
+Kinds: `part_of` (containment), `cites` (reference node → external docid),
+`supersedes`, `defines` (term → concept), `tested_by` /
+`requirement_of` (requirement ↔ conformance test), `class_of`,
+`amends`, `variant_of` (language/edition variants).
+
+## 6. Dataset embeds (optional)
+
+```jsonc
+"datasets": {
+  "glossarist": { "id": "oiml-complete", "concepts": 6031, "embedded": false, "ref": "…" },
+  "relaton":    { "id": "relaton-data-oiml", "records": 5707, "embedded": false, "ref": "…" }
+}
+```
+Embedding the full datasets inline is OPTIONAL (size); the fields declare
+which datasets the document's edges/terms resolve against.
+
+## 7. Serializations
+
+One lutaml-model class set; three mappings (idiomatic examples):
+
+- **JSON / YAML**: attribute names as above (`Metanorma::AiSerialization::Root.to_json`)
+- **XML**: elements for complex parts, attributes for scalars —
+  `<node type="table" id="n:tbl-4.1.2-1" anchor="tbl-4.1.2-1"><caption>…</caption>…`
+
+Consumers pick an encoding; the schema is the contract.
+
+## 8. Conformance (producers MUST)
+
+1. Emit `schema`, `schema_version`, full document identity, and node
+   `id`/`type`/`anchor` for every node.
+2. Node ids stable across regenerations for unchanged content (anchor +
+   content hash basis).
+3. Tables carry columns+rows (never linearized strings only); formulas
+   carry at least two of {asciimath, latex, described}.
+4. Edges only reference present node ids or declared external keys.
+5. `rights` present when the source declares license/copyright.
+Consumers SHOULD treat unknown node types as opaque text nodes
+(forward compatibility).
+
+## 9. Our consumer mapping (rag ingest)
+
+| AI Serialization | ChunkRecordV2 (Stage 2 spec) |
+|---|---|
+| `document` | `DocumentRecord` |
+| `nodes[type=clause]` | chunk `block: clause` |
+| `nodes[type=table]` | chunk `block: table` + `table` payload |
+| `nodes[type=formula]` | `block: equation` |
+| `nodes[type=term]` | `block: definition` |
+| `requirement`/`conformance_test` | `block: requirement` |
+| `edges` (cites/supersedes/part_of/tested_by) | D1 graph projection |
+
+## 10. Implementation sketch (upstream)
+
+A `metanorma-document` flavor module — class set + a walk:
+
+```ruby
+module Metanorma
+  module AiSerialization
+    class Root < Lutaml::Model::Serializable
+      attribute :schema, :string
+      attribute :schema_version, :string
+      attribute :document, DocumentIdentity
+      attribute :nodes, Node, collection: true
+      attribute :edges, Edge, collection: true
+      json do |m| m.map "schema", to: :schema # … end
+      xml do |m| m.root "ai-serialization" # … end
+    end
+    module Project                     # the walk: model → nodes/edges
+      def self.call(doc_model) = …     # visits typed model classes only
+    end
+  end
+end
+```
+CLI: `metanorma compile doc.adoc -x ai-serialization,json` (a new output
+format alongside html/pdf/… — the #592 ask reduced to one flavor entry).
+
+## 11. Open questions for upstream
+
+1. Version stability contract for the projection (semver gates?).
+2. Whether `embed_text` serializations belong in the spec or a separate
+   "AI profile".
+3. Multi-document collections (per-document envelopes vs collection
+   envelope with cross-doc edges).
+4. License signaling for training-corpus redistribution.

package/docs/MKO-EXPORT-PIPELINE.md

@@ -0,0 +1,147 @@
+# MKO export pipeline — mn-samples-oiml → RAG
+
+**Status: producer + consumer validated end to end on the real corpus
+(2026-08-28).** This is the producer-native path that replaces HTML
+scraping for the clean corpus. Wire contract: MN 116 (metanorma/docs PR
+#9, sources/116).
+
+This pipeline is the first concrete realization of the ETSI-informed
+redesign gaps ([REDESIGN-NORMATIVE-RAG-ETSI.md](REDESIGN-NORMATIVE-RAG-ETSI.md)):
+native section `part_of` (G-ETSI-2), native `cites` edges (G-ETSI-3),
+atomic typed tables (G-ETSI-5) — from the producer, with no recovery
+heuristics.
+
+## What MKO is
+
+One Metanorma document → one `<short>.mko/` bundle of typed,
+addressable knowledge objects:
+
+| Component | Content |
+|---|---|
+| `manifest.json` | schema id, per-component SHA-256 hashes (verified at ingest) |
+| `document.json` | identity (parsed docids, doctype, edition, status, languages), relations, numbered structure tree |
+| `units.jsonl` | typed units — clause/annex/term/table/figure/formula/note/example/sourcecode/requirement/reference — each with anchor, number, breadcrumb, plain text, typed payload, content hash |
+| `edges.jsonl` | the graph: `part_of` containment, `cites` (reference **and** term units), `defines` (term→concept), `class_of`, document-level relations (`doc:<short> → ext:<id>`, Relaton type verbatim: obsoletes/hasPart/…) |
+| `bibdata.json` | the document's own record as **native Relaton JSON** |
+| `bibliography.jsonl` | every **cited document** as native objects: Relaton item + native pubid parse + pubid render |
+| `glossary.json` | term entries as **native Glossarist concepts** |
+| `identifiers.json` | the document's identifiers with **native pubid parses** |
+
+Semantic anchors over GUIDs, content-hash-stable unit ids, byte-identical
+determinism, zero flavor knowledge in the walk.
+
+## Pipeline for mn-samples-oiml
+
+The samples already carry compiled `document.xml` +
+`document.presentation.xml` in-tree (OIML compiles through the iso
+backend: `flavor="iso"`), so today the export is a one-liner per
+document (from the metanorma-document checkout, branch
+`feat/model-validation-l1-declarations`, PR #45):
+
+```bash
+# one command, gated (live since 2026-08-29):
+.venv/bin/python -m ingest.cli mko              # export → ingest → enrich → verify-gate → wire → fts → graph
+.venv/bin/python -m ingest.cli mko --dry        # counts + coverage check only
+```
+
+The pipeline enforces the stage order this path proved matters: the
+`verify` gate refuses to wire chunks that lack contextual preambles
+(the 2026-08-28 regression — un-enriched upserts dropped retrieval
+R@5 from 95% to 90%), and it stops before shipping (INDEX_VERSION
+bump + deploy stay human). Stage 5 (wire) needs
+`CLOUDFLARE_API_TOKEN` in the environment; enrichment upserts
+enriched vectors live as it runs, so wiring is idempotent
+bookkeeping afterwards. The underlying per-stage commands (for
+reference / partial runs):
+
+When the umbrella wiring lands (metanorma PR #591) this becomes
+`metanorma compile document.adoc -x mko` and the Ruby step disappears —
+the ingest side does not change.
+
+## Worked example: OIML R 60-1 (load cells)
+
+`oiml-r-60-1.mko` → ingest output (producer @e6918cb):
+
+```
+155 units (clause=72, term=59, table=6, figure=4, example=2, reference=12)
+  → 136 chunks, 59 terms, 12 cited docs, 541 graph rows
+canonical: OIML R 60-1 (edition 2)
+```
+
+**Full-corpus run (2026-08-29, live):** 36 documents → 3,020 enriched
+chunks (clause 1,913 / table 371 / term 113 / note 77 / formula 37 /
+annex 32 / example 8), 113 glossary concepts, 342 cited docs, 3,561
+section nodes + cites/defines edges in D1. Retrieval held at baseline
+(R@5 95%, AP 0.875, MRR 0.893) after widening the rerank window to 10
+for the enlarged clean lane.
+## Full-corpus validation (2026-08-28)
+
+All **37** compiled sample documents (R/D/B/G/E + OIML-CS admin + parts
++ amendments) export and ingest:
+
+```
+37/37 bundles → 3053 chunks, 113 native Glossarist concepts,
+342 cited documents (native Relaton + pubid), ~11k graph rows
+```
+
+Includes the amendment cases: R 60/A1 keeps its 17 terms in an annex
+(recovered into the glossary), and the OIML-CS admin documents keep
+their whole body in `<preface>` (walked as clauses). Clause text now
+carries list content in document order, not just paragraphs.
+
+*Counting notes (verified against the artifacts): the 539 graph rows are
+242 edges (147 `part_of`, 36 `cites`, 59 `defines`) + 297 node inserts;
+figures and references become bibliography/graph objects, not chunks, so
+135 chunks = clause 68 + term 59 + table 6 + example 2 (3 clause units
+merge/empty; the source XML carries 63 raw `<term>` tags, 4 of them
+nested/boilerplate).*
+
+- 59 Glossarist-native concepts (designations, definition, sources,
+  `language_code: eng`) — the glossary-heavy load-cell vocabulary lands
+  directly in the terminology lane, no scraping.
+- 6 tables as typed payloads (columns/rows), atomic — G-ETSI-5.
+- 539 graph rows: 147 `part_of` (section parthood — G-ETSI-2), 36
+  `cites` (12 references + term sources — G-ETSI-3), 59 `defines`
+  (edges total 242; the remaining rows are node inserts).
+- 12 cited documents (OIML V 1:2013, R 111, …) as native Relaton items —
+  the citation targets are addressable objects, not strings.
+
+## Answer contract v2 stages (live)
+
+- `to_payload_sql` → `artifacts/mko_unit_payloads.sql` → D1 `unit_payloads` (573 typed units; `mirror` stripped for serving size, 60KB cap)
+- Figure assets → R2 `rag-public-assets` at unit-keyed keys, served at `/assets/u:<id>.<ext>` (immutable); `unit_payloads.payload.uri` updated to the canonical route (producer `uri` is null today — upstream asset-contract ask, metanorma-document#50)
+
+## How the artifacts feed the existing stages
+
+`artifacts/mko_chunks.jsonl` uses the **same chunk schema** the
+parse/enrich/embed stages already consume (`chunk_text`, `doc_id`,
+`docidentifier`, `clause_anchor`, `clause_title`, `tier`, `corpus`,
+`status`, `superseded_by`, `text_ref`) plus new keys: `block` (unit
+type), `unit_id`/`unit_hash` (incremental re-index keys), and typed
+payloads (`metadata.table` with columns/rows, `metadata.formula` with
+description/asciimath, `metadata.term`, `metadata.requirement`).
+`artifacts/mko_graph.sql` targets the same D1 `graph_nodes`/`graph_edges`
+tables as `ingest/graph.py`. `mko_glossary.json` and
+`mko_bibliography.json` are new lanes (terminology exact-match; cited-doc
+registry).
+
+## Gaps / next steps
+
+1. **CLI**: `-x mko` is wired in metanorma PR #591 (unreleased). Until
+   it ships, use the Ruby export above (works against the in-tree
+   compiled XML).
+2. **pubid has no OIML flavor**: OIML identifiers ("OIML R 60-1:2017")
+   are not pubid-parsed → `pubid: null` in bibliography lines. Doc node
+   ids still normalize (`doc:OIML-R-60-1-2017` — same shape as
+   `graph.py#norm_id`, so the graphs join). Upstream ask filed:
+   pubid/pubid#342.
+3. **Document-level relations**: the samples' semantic XML embeds no
+   relaton `<relation>` elements, so `doc:` edges are empty for them;
+   the relaton-data-oiml join stays authoritative for
+   status/supersession until flavors embed relations at compile time.
+4. **Collections** (r060, r129, r138: parts + amendments via
+   `collection.yml`): export is per document today; a collection-level
+   manifest with cross-document edges is future MN 116 work.
+5. **Multilingual editions** (FR/AR/…): units carry `lang`; interlinear
+   alignment (same unit id across languages) is specified in MN 116 but
+   needs producer work on translated sources.