@konneal/engine 0.1.0

package/docs/paper-oiml-bulletin.md

@@ -0,0 +1,402 @@
+*Draft article for the OIML Bulletin — 2026-08-30. Style: Bulletin
+technical article (MS Word single-column on submission; this is the
+authoring source). Numbers are production-measured unless noted.*
+
+---
+
+## Abstract
+
+The OIML publishes close to 900 documents — Recommendations, Basic
+publications, Guides, Documents and Expert reports — that together form
+the reference library of legal metrology. Finding what they say has
+required knowing the corpus before querying it. We describe a
+question-answering service, **OIML SMART AI** (ai.oimlsmart.org), that
+answers natural-language questions from the indexed publications with
+clause-level citations, verbatim quote anchors for normative values, and
+typed renderings of the tables and equations the answers depend on — and
+it reads figures and user-supplied photographs directly, from pixels. The
+service measures every component it ships: a golden question set,
+retrieval metrics in the tradition of the information-retrieval
+literature, and judged faithfulness. We report the measured design
+decisions, the corpus-structure findings that shaped them, and the open
+path: producer-native ingestion of the Metanorma document model so that
+any Metanorma-authored corpus can be indexed the same way.
+
+---
+
+## 1. Introduction
+
+Legal metrology's knowledge lives in a large, cross-referential,
+multi-edition corpus. A typical working question — *"what is the maximum
+permissible error for a class III nonautomatic weighing instrument?"* —
+requires knowing which Recommendation applies (OIML R 76-1), which
+edition is current, which clause holds the table, and how to read the
+table's rows against the instrument's verification scale intervals.
+Every one of those steps assumes corpus knowledge the questioner may not
+have.
+
+Question answering over such corpora has been transformed by
+retrieval-augmented generation (RAG): a language model is given passages
+retrieved from a trusted corpus and may use only those. The engineering
+challenge is no longer whether a model can write fluent prose — it can —
+but whether the *right passages* are found, whether the answer stays
+*faithful* to them, and whether the reader can *verify* the result. Our
+service is built around those three properties.
+
+This article describes the architecture as deployed, the measurements
+that shaped it, and the corpus findings — including errors in the
+bibliographic record itself — that surfaced because the system keeps
+score. It closes with the producer-native ingestion path (Metanorma
+Knowledge Objects) and what it makes possible for other standards bodies.
+
+## 2. What the service does
+
+A user asks a question in any language. The service:
+
+1. **Understands the question** with a small language model — language,
+   named publication, edition, defined terms, complexity. There are no
+   keyword rules; the same model makes these judgements for every
+   question.
+2. **Retrieves in parallel** — three lanes leave the question at the
+   same instant: the vector index is queried with the question's
+   embedding while understanding is still running (the dense lane's
+   results are reused unchanged when understanding adds nothing), a
+   full-corpus keyword index catches exact identifiers, part numbers
+   and defined terms that vector similarity alone misses, and
+   understanding's refinements — a named publication, an edition —
+   arrive as metadata filters on the dense query.
+3. **Ranks** the fused candidates with a cross-encoder; complex
+   questions get a second, stronger re-ranking pass.
+4. **Answers from the retrieved passages only**, under a contract that
+   requires inline citations (`[OIML R 60-1:2021 §4.4.2]`) and verbatim
+   quote anchors for normative values.
+5. **Verifies** the answer after generation: a deterministic check that
+   every quoted phrase exists in a cited passage; a separate judge that
+   scores faithfulness; a correction-and-retry when either fails. An
+   answer that cannot be verified is never written to the answer cache.
+
+![Figure 1 — A question's path through the service. Three retrieval lanes run concurrently; every answer passes mechanical verification before it is served or cached. The edition registry derives publication status from supersession links, never from a status field.](paper-assets/architecture.svg)
+
+*Figure 1 — A question's path through the service. Three retrieval lanes run concurrently; every answer passes mechanical verification before it is served or cached. The edition registry derives publication status from supersession links, never from a status field.*
+
+
+When the question names a publication, answers are steered to the
+current edition by a registry derived from the bibliographic record's
+supersession links (§5). When the answer depends on a table or
+equation, the interface renders the *producer's* typed object — the
+actual rows and columns — never the model's re-typing of it (§6).
+
+If the corpus does not contain the answer, the system says so in one
+canonical sentence and redirects; refusals are never cached, because a
+refusal is a property of the moment, not of the question.
+
+## 3. Corpus and structure
+
+The indexed corpus comprises the English editions of the OIML
+publications (~900 documents, ~9.3 million words). Documents are chunked
+along clause boundaries — never arbitrary token windows — and each chunk
+carries provenance: publication identifier, edition, language, clause
+anchor, and bibliographic status.
+
+Three corpus-level findings shaped the design:
+
+**The bibliographic status field disagrees with its own record.** In 58
+of 224 publication families, the machine-readable status field claims a
+document is current while the same record's supersession links say
+otherwise. The service therefore *derives* status from the links and
+ignores the field. More consequentially, 36 of 224 families have no
+recorded successor link at all, so the current edition cannot be derived
+for them; the registry surfaces these gaps rather than guessing. Both
+numbers are the worklist for upstream bibliographic corrections — the
+kind that serving surfaces only because it keeps score.
+
+**Identifiers are identity, not display.** A cross-referenced audit of
+the dirty corpus found documents whose machine-readable identifier was
+a placeholder ("OIML D 0:0000", "OIML D X") or polluted with language
+markers — around 200 records — making them unreachable by
+doc-scoped retrieval. Identifiers are now derived from slug identity
+with placeholders never winning, and the index is reconciled against
+the canonical chunk set: the first census measured 49,553 live vectors
+against 31,512 canonical — 18,041 strays from every prior re-chunking,
+deleted in place (upserts never delete). A retrieval service that never
+enumerates its own index serves ghosts.
+
+**Tables are where the normative values live.** Maximum permissible
+errors, accuracy-class limits, verification interval bounds — the values
+practitioners ask for — are tabular. Flattening tables into prose loses
+the row/column geometry that makes them answerable. The index now treats
+tables as atomic typed objects (§6).
+
+**Structure is a retrieval signal.** Standards are hierarchically
+organized with extensive cross-references; the system indexes the
+section hierarchy and the citation graph as a queryable graph (7,128
+nodes, 6,486 edges) alongside the text, so a defined term can be
+resolved to the documents that define it even when the question's
+vocabulary does not match the corpus's.
+
+## 4. Retrieval: what the measurements showed
+
+We evaluate retrieval with a golden question set (now 29 questions
+including eight table-value questions with witness rows) using
+recall@5, average precision@5, and mean reciprocal rank@5 — the protocol
+of the standards-retrieval literature. Single runs proved noisy (±5
+percentage points); all reported numbers are means over three runs on a
+quiet account.
+
+The single largest measured improvement came from making keyword
+retrieval a **full-corpus first stage** rather than a re-scoring of the
+vector results: recall@5 rose from 86% to 95%. The reason is structural
+— standards vocabulary is exact ("n_LC", "R 60-3", "creep"), and dense
+embeddings alone miss exact identifiers that a lexical scan of the whole
+corpus recovers. Fusing both rankings (reciprocal rank fusion) lifted
+precision and MRR without costing recall.
+
+The second improvement came from **contextual enrichment**: a one-time
+pass that prepends a short model-written preamble to every chunk ("this
+clause defines the accuracy-class limits for load cells") before
+embedding and indexing. This is the contextual-retrieval recipe
+validated industry-wide; our corpus measurement confirmed it, and the
+one-time cost (~$0.0017 per chunk) is amortized over every future
+answer.
+
+With the typed-table lane live (§6), retrieval lanes running
+concurrently with query understanding, edition steering active (§5) and
+multimodal generation on (§6), the current baseline is:
+
+| Metric | Value |
+|---|---|
+| Recall@5 (mean of 3) | **94.3%** (range 93.1–96.6) |
+| Average precision@5 | 0.82 |
+| MRR@5 | 0.87 |
+| End-to-end golden cases | 14/14 |
+
+These numbers are on our own golden set, not on the benchmarks of the
+works we build on — the honest comparison is per-technique, on the same
+corpus, before and after. So read: the full-corpus lexical stage is the
+ETSI study's recipe (ref 1) — adopting it lifted recall@5 here from 86%
+to 95%; the contextual preambles are Anthropic's contextual retrieval
+(ref 3) — confirmed on this corpus at ~$0.0017 per chunk one-time; the
+clause-boundary, structure-preserving chunking follows the same
+structure-aware line as refs 2 and 7, which our typed-unit pin extends
+from "chunk better" to "guarantee the answering object a slot." Two
+elements of the deployed system have no counterpart in the cited work:
+the symbolic-reference contract, under which table data never passes
+through the model at all (refs 2's error-reduction approach still
+re-generates tables; we removed the corruption channel), and the
+mechanical post-generation verification of every answer (quote anchors,
+unit references, retyped-table detection) with a faithfulness judge that
+sees the passages actually used.
+
+Two structural additions complete the retrieval story. First, the
+corpus is a tree — every chunk carries its clause anchor — and the
+serving path uses the tree: a hit's score blends its neighbouring
+clauses' scores, evidence is presented to the answer model in document
+reading order, and a ranked section summary resolves to its quotable
+child clauses (adapted from FABLE/BEAR, ref 7, at none of its index-time
+cost, because Metanorma documents arrive as trees rather than needing
+one inferred). Second, everyday words rarely match defined terms — the
+one gap that failed for every corpus representation equally — so a
+terminology index of the corpus's defined concepts links a question's
+phrasing to candidate terms, and the answer model adjudicates and leads
+with the corpus's own term, quoting its definition ("my output keeps
+drifting" resolves to *span stability*, with the verbatim definition and
+its defining clause cited).
+
+## 5. Editions and trust
+
+Which edition applies is as consequential as what it says. The service's
+registry of publication families and editions is derived from
+supersession links, not from status fields (§3). Citations carry the
+edition and status of every source; superseded sources are marked.
+Answers to questions that name a publication are steered to the current
+edition; answers that must cite a superseded edition (the 2021 passages
+do not reproduce a table the 2017 edition carries) say so explicitly.
+
+Steering is measured, not assumed, and two mechanisms earned their place
+by fixing observed failures. Superseded editions match archaic phrasing
+strongly — a complex verification question once cited R 76-1:1992/1988
+while the current 2006 edition was crowded out of the passage window —
+so ranking now demotes older-edition chunks of a publication whenever a
+newer edition of the same publication is present in the evidence
+(cross-publication recency is deliberately untouched: a 1992 document
+that is still current must not lose to unrelated 2024 ones). And because
+query understanding can pin an edition for the wrong family, an edition
+pin that the corpus itself fails to corroborate is dropped to the
+document-level filter. After both fixes the probe cites R 76-1:2006 in
+force, with one legitimate 1988 tail citation for a clause only that
+edition carries; the window's measured average precision (0.85) and MRR
+(0.88) are the best the service has recorded.
+
+## 6. Tables, equations, and figures as typed objects
+
+The most consequential rendering decision: the language model never
+re-types normative data. When an answer depends on a table, the model
+emits a symbolic reference to the table's unit (`[[u:table-1]]`); the
+server validates the reference against the passages actually used and
+resolves it to the producer's typed payload — columns, rows, units. The
+interface renders the exact object. Small verbatim values in prose are
+permitted and mechanically verified against the source. The payload
+never passes through the model's output, so it cannot be corrupted by
+generation.
+
+The same principle extends from rendering to judgment: the model never
+judges what a machine can compute. Where the corpus's machine-readable
+models carry checkable objects — a constraint with an OCL expression, a
+requirement with a threshold limit — a question naming one gets it
+EXECUTED: the question's values bind to the rule's own parameters, the
+check evaluates deterministically, and the verdict (pass, or the
+standard's own violation word, with the arithmetic shown, or void naming
+exactly what is missing) is attached to the answer as computed data.
+Two sibling capabilities complete the set: provable absence (a topic
+that a standard does not constrain returns an enumeration certificate
+over its entire model plane — N nodes enumerated, zero matches — rather
+than a refusal) and answer verification (any answer checkable against
+the corpus: verbatim-quote containment, object-reference resolution, a
+judged faithfulness score labeled as judged).
+
+![Figure 2 — The answer contract. The model writes prose with inline citations and symbolic references; three mechanical checks run after generation; the server resolves surviving references to the producer's payloads and the client renders them.](paper-assets/contract-v2.svg)
+
+*Figure 2 — The answer contract. The model writes prose with inline citations and symbolic references; three mechanical checks run after generation; the server resolves surviving references to the producer's payloads and the client renders them.*
+
+
+Figures are described by a vision-capable model that reads the actual
+image pixels; equations are carried in their authoring-native formats
+(AsciiMath and MathML) with a plain-language description. Both arrive as typed objects with the same validation.
+
+Figures also flow the other way: when the retrieved passages contain
+figure units, their actual images are attached to the answer model's
+input, so descriptions and reasoning come from the drawing itself — the
+model reads labels that exist only in the pixels (the pixel-label
+probe — "what are the labeled example cases in the R 60-2 design-shapes
+figure?" — is answered A, B and C from the drawing, stable across
+repeated runs). Two measured invariants keep that true: assets must be
+readable by vision pipelines (vector-sourced rasters that draw black on
+transparent alpha flatten to a solid black rectangle inside a vision
+model, whatever a browser shows — a corpus sweep detects and repairs
+them), and images ride their own message in the generation call (long
+passage text and image parts in one message triggers provider errors
+that scale with payload). Users can likewise attach a photograph (an
+instrument nameplate, a scale dial, a schematic) to their question; the
+text still drives retrieval, and the image gives the model the visual
+context, under the same citation contract.
+
+## 7. Models, cost, and sovereignty
+
+All models are open-weight and served on a single cloud platform
+(Cloudflare Workers AI); no corpus data is sent to proprietary model
+providers. The answer model (GLM-5.3 Flash, natively multimodal) serves
+all tiers at approximately $0.001 per answer. The full one-time corpus
+preparation — contextual enrichment of 42,000 chunks — cost
+approximately $75. Daily serving at current traffic is under $5 per
+month including infrastructure. The model policy is deliberately
+cost-first on the serving path (every question pays it) and quality-first
+on one-time work (enrichment, captioning, evaluation), where quality
+persists into every future answer.
+
+One lesson generalizes: read the model card before wiring a model. Three
+separate live failures traced to defaults we never set — a
+reasoning-effort parameter that silently defaults to maximum (starving
+small output budgets), sampling defaults that let a thinking model loop
+(repetition consuming the token budget that carried the structured
+output), and a degraded non-thinking mode we were unknowingly paying
+for. Every call site now states its reasoning mode, its sampling, and a
+budget the reasoning cannot starve; the discipline costs nothing and
+removed an entire class of silent failure.
+
+## 8. The producer-native path: Metanorma Knowledge Objects
+
+OIML publications are authored in Metanorma, a model-driven document
+system: the source of truth is a typed document model, not any rendered
+PDF or HTML. The service's newest ingestion path consumes that model
+directly — one bundle per document containing typed units (clauses,
+tables, terms, equations, figures, requirements), a section graph, a
+native Glossarist glossary, and Relaton bibliographic objects — and
+replaces HTML scraping end to end for the cleanly-authored portion of
+the corpus. The format, Metanorma Knowledge Objects (MKO), is specified
+as Metanorma note 116 with the consumer contract (symbolic unit
+references and typed excerpts) contributed from this work.
+
+![Figure 3 — Producer-native ingestion. Each authored document exports as one MKO bundle; chunking, enrichment and indexing are derived from the bundle, and re-ingest is incremental by content hash.](paper-assets/mko-ingest.svg)
+
+*Figure 3 — Producer-native ingestion. Each authored document exports as one MKO bundle; chunking, enrichment and indexing are derived from the bundle, and re-ingest is incremental by content hash.*
+
+
+This path matters beyond OIML: any standards body whose publications are
+authored in Metanorma gets structured, table-aware, graph-connected
+question answering over its corpus without scraping renderings. The
+guidelines for building such a service from scratch accompany this
+article.
+
+## 9. Evaluation as a discipline
+
+The service's answers are evaluated on three axes, continuously:
+
+- **Retrieval** (recall, precision, MRR against the golden set with
+  witness citations)
+- **Faithfulness** (an independent judge scores whether each claim is
+  supported by the retrieved passages — with the important lesson that
+  the judge must see the passages the answer was actually built from,
+  not a fresh retrieval)
+- **User feedback** (thumbs up/down on every answer, logged to the same
+  evaluation loop)
+
+Every change ships through the same gate — literally one command:
+golden suite ×3 and the capability battery ×6 against the live service,
+where any failed run fails the command. The suites also run in CI and
+the service's cache versions flush on every retrieval change so no
+answer is served from a superseded index. A humility note from the
+measurement machinery itself: the capability battery's runner once
+omitted to set a non-zero exit code on failure, so a failing battery
+reported green through the gate — the same class of silent failure the
+cache-version discipline exists to prevent. Exit codes are part of the
+measurement contract now. The gate rejects as often as it accepts.
+A candidate change that unioned additional retrieval candidates into a
+rewritten query's pool — a plausible-looking "more evidence" improvement
+— dropped recall@5 from 94.3% to 89.7%: topically close but wrong
+documents outscored the right ones under the cross-encoder, and the
+per-case diff named the three failing questions. The fix (additive
+candidates may only replace an identical query, never dilute a rewritten
+one) is now a comment in the code. A second candidate — grading
+document-scoped retrieval more aggressively — was measured, found to buy
+nothing, and reverted the same day. Additive is not free; the suite, not
+the author, decides.
+
+## 10. Conclusions and outlook
+
+A grounded, citation-linked, typed-rendering question-answering service
+over the OIML corpus is measurable, economical, and — with the
+producer-native ingestion path — increasingly maintained by the
+documents' own structure rather than by scraping their renderings. The
+open work is equally concrete: upstream bibliographic corrections for
+the 36 families without derivable current editions; unit-level language
+tagging so bilingual annexes inside English editions stop masquerading
+as main text; a publisher-specific identifier flavor so citation joins
+become exact; collection-level bundles for cross-document reasoning; and
+interlingual unit alignment so the same clause can be answered in every
+OIML language. The requirements behind several of these belong to the
+authoring system itself, and we have filed them with the Metanorma
+document model team as the producer side of an AI-native publishing
+stack.
+
+The service is live at ai.oimlsmart.org. Try it, and tell us when it is
+wrong — the feedback button is the fastest path into the evaluation
+loop that everything above runs on.
+
+---
+
+### References
+
+1. Al Masoud, A., Arazzi, M., Germani, S., Nocera, A. *Exploring
+   Structural Complexity in Normative RAG with Graph-based approaches: A
+   case study on the ETSI Standards.* arXiv:2604.09868 (2026).
+2. Guttal, P., et al. *Structure-Aware Chunking for Tabular Data in
+   Retrieval-Augmented Generation.* arXiv:2605.00318 (2026).
+3. Anthropic. *Contextual Retrieval.* Engineering blog (2024).
+4. Günther, M., et al. *Late Chunking: Contextual Chunk Embeddings Using
+   Long-Context Embedding Models.* arXiv:2409.04701 (2024).
+5. Lewis, P., et al. *Retrieval-Augmented Generation for
+   Knowledge-Intensive NLP Tasks.* NeurIPS (2020).
+6. Metanorma. *MN 116: Metanorma Knowledge Objects (MKO) machine
+   serialization format.* Metanorma documentation (2026).
+7. Xu, L., et al. *Equipping Retrieval-Augmented LLMs with Document
+   Structure Awareness.* arXiv:2510.04293 (2025).