@cosyte/terminology 0.0.5 → 0.0.7

package/CHANGELOG.md

@@ -9,8 +9,298 @@ this file is maintained by hand (Changesets handles the version bump and publish
 
 ## [Unreleased]
 
+### Changed
+
+- **Breaking: `ExpansionDiagnostic.system` is replaced by `ExpansionDiagnostic.path`.** The field
+  used to carry the code system or value set URI the concern was about — consumer-supplied text of
+  unbounded length on a diagnostic surface. It now carries a value-free structural locus into your
+  own resource, built from integers and the engine's own field names: `"compose.include[2]"`,
+  `"compose.include[0].valueSet[1]"`, `"expansion"`. A diagnostic raised while expanding a
+  referenced value set is prefixed with the reference that reached it. This is strictly more precise
+  than the URI, which cannot distinguish two `include` components naming the same system. `expand`
+  and `validateCodeInValueSet` build the path identically, so a locus means the same thing in both;
+  they do **not** always emit the same _set_ of diagnostics, because membership can decide an answer
+  without evaluating a part expansion still has to report. Reading `d.system` is now a type error.
+  The rationale, and the other three sites, are under **Fixed** below.
+
 ### Fixed
 
+- **The UCUM unit table `loadUcumEssence()` hands out was `readonly` to TypeScript only, so a unit's
+  definition could be rewritten in place — and doing so fabricated a unit equivalence that outlived
+  putting the table back** (`TERMINOLOGY-MUTABLE-ESSENCE-TABLE`). One table is shared by every caller
+  and by the engine's own reducer, `essence.ts` froze nothing, and `reduce` memoizes each atom's
+  reduction against the atom object. So corrupting a loaded atom **before that atom's first
+  reduction** wrote the memo, and the reading survived restoring the table exactly as shipped.
+
+  Measured on `bf153cb`, and live on published `0.0.5` and `0.0.6` alike — `0.0.6` is that commit plus
+  a version bump. Defining the loaded litre as `1` with no prior
+  touch of `L`: `ucumEqual("L", "1")` and `ucumEqual("mg/L", "mg")` both answered `true` where the
+  same calls without it answered `false`, and `mmol/L` fell from `6.0221407599999985e+23 × m-3` to a
+  dimensionless `6.02214076e+20` — **a concentration comparing equal to a mass**. Two further routes
+  to the same readings on that build: writing the nested definition object in place
+  (`atom.value.unit = "1"`), and replacing an element of the `atoms` array, which is the one
+  `parseUcum` scans, so it hands the parser a forged atom without touching the memo at all.
+
+  This is the half that keying the memo on the atom **object** did not reach. That closed the route
+  where the atom was one you had _built_; this is the route where the atom is one you were _handed_,
+  and the consequence class is identical. Neither substitutes for the other.
+
+  The table is frozen when it is parsed, deeply: every atom, every atom's definition object, every
+  prefix, and both arrays. Shallow would have been worth nothing — freezing the atom alone leaves
+  `atom.value.unit`, which reproduces all three readings. `atomByCode` and `prefixByCode` refuse
+  `set`, `delete` and `clear`, which `Object.freeze` cannot do for a `Map`. `parseEssence` freezes
+  what it returns on the same rule.
+
+  **The guarantee is that the table does not change, not that a write throws.** `Object.freeze`
+  refuses a write by throwing in strict mode and ignoring it in sloppy mode, and this package ships a
+  CJS build a sloppy-mode caller can require; the table is the one that was parsed either way. The
+  tests assert the readings, and the `TypeError` only as how a strict-mode caller observes it.
+
+  **Two things this does not claim, both measured rather than reasoned.** `Map.prototype.set.call()`
+  can still force an entry into a lookup map — no `Map` can be made immutable in place — and what
+  that reaches is bounded and pinned in both directions: `parseUcum` resolves an atom by scanning
+  `atoms`, never through `atomByCode`, so a forced entry changes **no** reduction and no `ucumEqual`
+  answer, only what the caller's own `get` returns. And the values `parseUcum` and `reduce` hand back
+  are deliberately still per-call and mutable.
+
+  **No reduction changes.** 3,080 expressions built from the table — every atom bare, squared,
+  inverted and annotated; every prefix over rotating metric atoms; rotating ratios, products and
+  grouped powers; and the shapes the README and the tests name — plus 312 `ucumEqual` pairs, reduce
+  byte-identically on `bf153cb` and on this tree (3,393 output lines, one sha256).
+
+  One knock-on, stated because getting it wrong is easy: freezing the table removes **a** route to
+  `reduce`'s cyclic-definition guard, not the last one. Two routes that look like they should reach
+  it do not, and both are now pinned as tests — naming a table atom in a definition (it is resolved
+  against the loaded table, so your atom can _name_ one but never _be_ one; a self-referential `Pa`
+  taken off a table copy resolves to the shipped pascal and answers `1000 × g.m-1.s-2`), and
+  corrupting a loaded atom, which is what is now refused. The route that does reach it needs no table
+  at all: `UcumAtom.value` is `readonly` to TypeScript, which an **accessor** satisfies, and the
+  definition is read after the atom is marked in progress, so a getter can re-enter `reduce` for the
+  same atom. That is a supported public call, and the guard is pinned by it under a 100,000-byte atom
+  code — so all three of `reduce`'s messages are again asserted whole, against both `message` and
+  `err.stack`, with the caller's own code as the unbounded value in each.
+
+  Also here: the `/* v8 ignore next */` on `reduce`'s `atom.base` line is removed. It justified
+  itself with a property of the bundled table — every base atom there carries a `dim` — for a line
+  reachable through the exported surface with a caller-built `{ base: true }` atom that carries none.
+  That arm is answered on the atom's own code, is now covered by a test rather than excluded, and
+  changes nothing for the table's own base units.
+
+- **A `UnitNode` you assembled yourself could change how `reduce` and `ucumEqual` answer for a unit
+  parsed out of the bundled UCUM table, for the rest of the process**
+  (`TERMINOLOGY-ATOMMEMO-POISONING`). `reduce` is exported and takes a caller-built node, and
+  nothing checks that the atoms on it came from the bundled table. Its per-atom memo, and the guard
+  against a definition that leads back to itself, were keyed on `atom.code` — a **string** — so an
+  atom you built wrote under the key of whichever shipped atom happened to share its code, and every
+  later reduction of that shipped atom read what yours had left.
+
+  Measured on `0.0.5`: one `reduce()` over a value-less atom coded `L`, as the first touch of `L` in
+  the process, made `ucumEqual("L", "1")` and `ucumEqual("mg/L", "mg")` both answer `true` where the
+  same calls without it answered `false`, and dropped `mmol/L` from `6.02214076e23 × m-3` to a
+  dimensionless `6.02214076e20` — **a concentration comparing equal to a mass**, from an engine
+  whose contract is never to fabricate. Two further readings on the same build: an atom coded `N`
+  and defined as `N` left the cycle guard armed, so every later `ucumEqual("N", "kg.m/s2")` threw;
+  and the value returned for a term with no factors was the engine's own module-global dimensionless
+  object, so mutating a result you had been handed turned `kg.m/s2` into `999000 × m2.g.s-2` and
+  made `ucumEqual("N", "kg.m/s2")` answer `false`.
+
+  Four changes:
+  - The memo and the cycle guard key on the **atom object**, in a `WeakMap` / `WeakSet`. The atoms
+    `parseUcum` resolves are the unit table's cached singletons, so a parsed expression still gets
+    one memo entry per atom; an atom you built gets its own entry and is collected with it. Measured
+    warm over ten mixed expressions, best of seven trials of 2,000,000 `reduce` calls, on two
+    different machines: identity-keyed came in at or ahead of string-keyed both times. Read that as
+    no measured cost rather than as a speed-up; the absolute figures are machine- and load-dependent
+    and are deliberately not quoted as a pair.
+  - An atom carrying no linear definition, and which is neither a base nor an arbitrary unit, is now
+    **refused** with `UCUM atom has no linear definition` rather than answered — and memoized — as
+    dimensionless. No expression `parseUcum` accepts reaches this, though **not** because every atom
+    in the bundled table has a linear definition: 21 of its 312 atoms, the special units, have none.
+    It is that an expression containing a special unit short-circuits to the opaque `special` form
+    before linear reduction, base and arbitrary units are answered directly, and every remaining atom
+    resolves. A node you assemble can reach it — including by defining an atom in terms of a special
+    unit such as `Cel`, which lands on the table's own `Cel`.
+  - The cycle guard releases its atom in a `finally`, so a throw from inside the recursion no longer
+    wedges that atom for the rest of the process.
+  - `reduce` no longer returns the module-global dimensionless object. What you get back is per-call
+    state, and mutating it cannot reach the engine.
+
+  Keying on identity was chosen over validating that each atom came from the bundled table: it needs
+  no provenance check on the hot path and adds no failure mode for real input, and it is the
+  narrower change. The cost is that an atom you built is answered on its own terms only — the
+  intended contract, not a regression. **No unit obtained from `parseUcum` changes its answer**: the
+  530-case UCUM conformance suite and the rest of the suite are unchanged and green, and the new
+  refusals are reachable only through a node you assembled. **The claim rests on one property, and
+  it is the one this repo asserts: every atom in the bundled table reduces, and none of them reaches
+  a refusal** (`test/ucum/reduce-memo.test.ts`, last case, over all 312). That is enough to cover
+  every expression `parseUcum` accepts rather than sampling them — an expression containing a
+  special unit short-circuits before linear reduction, linear reduction is compositional over atoms,
+  and the table's own definitions close over that same set, so once every atom answers there is
+  nothing left for a bigger corpus to vary. `test/ucum/functional.test.ts` separately drives the
+  vendored UCUM functional-test suite's unit strings through `reduce`.
+  **Do not restate this as more evidence than that.** Three drafts of this paragraph each claimed
+  more than the tree holds — two cited generated-corpus sizes that were not reproducible from their
+  own descriptions, and the third attributed a three-clause sweep to a file that pins one clause of
+  it. A before/after differential is a two-tree measurement and **no in-tree test can pin one**, by
+  construction; do not add a guard to make a sentence here true, shrink the sentence. The three
+  guards are now covered by tests
+  rather than by a `/* v8 ignore */` block, and each message is pinned by whole-message equality
+  rather than by a substring match.
+
+- **Consumer-supplied strings reached `TerminologyError.message`, `err.stack` and an
+  `ExpansionDiagnostic`, contradicting this package's own "value-free by construction" claim**
+  (`PHI-WARNING-MESSAGE-LEAK`). Three sites, found by binding
+  `assertNoDiagnosticPhiLeak` from `@cosyte/test-utils@0.0.2` to a 52-slot table covering the
+  consumer-controlled positions across the readers and loaders that this repo has identified — which
+  is not a proof that none was missed, as the fourth site below shows; **14 of the 52 slots were red on the
+  base commit** — 4 CSV column-config slots, 1 `invertGem` slot, 4 in `expand.ts` and 5 in
+  `validate.ts` — and the table was run there first for exactly that reason. (An earlier draft of
+  this entry said 9. That was the measurement taken before the slot table was extended to the
+  **second copy** of the `src/valueset/` diagnostic factories, and it was never re-taken; the five
+  `validate.ts` slots are the difference. Re-measure after you widen a table.)
+  1. `csv.ts` `requireHeader` interpolated the caller's configured column name into the
+     `TERM_CODESYSTEM_MALFORMED` message. Measured against the published `0.0.5`: a 1,000,000-byte
+     `columns.code` produced a 1,000,063-byte `err.message`, and the same bytes landed in
+     `err.stack`. It now reads the message out of a frozen `MISSING_COLUMN_MESSAGE` table keyed on
+     the column's **role** (`code` / `display` / `status` / `property`) and takes no value
+     parameter. That is the `astm`/`transform` shape the ecosystem audit recorded for the designs
+     that do not leak, and it is what this factory now is — a property of **this** factory, not a
+     rule this package holds everywhere. Several factories here do take `string` parameters and are
+     safe because every argument reaching them is engine-owned; they are named below, and reading
+     the no-value-parameter shape as a package-wide rule is what made three drafts of this entry
+     wrong.
+  2. `invertGem` interpolated `GemMap.direction` into the `TERM_MAP_NOT_INVERTIBLE` message.
+     `GemDirection` is a two-value union, but nothing validates it on load and a JavaScript caller
+     can pass anything, so the echo was unbounded in practice. Now a closed-set `Map` lookup with a
+     direction-free fallback; the direction is kept where it is one of the two authored values,
+     because it tells the caller which file to load instead.
+  3. `ExpansionDiagnostic.system` carried the caller's `include.system`, referenced `valueSet` URL,
+     or `ValueSet.url` verbatim. **This site is not in the 2026-07-30 ecosystem audit, which
+     recorded `terminology` as "one site, a CSV column name".** It is the only one of the three
+     that is _document_-derived rather than caller-configuration. **Breaking:** the field is
+     replaced by `path`, a value-free index locus (`compose.include[2]`,
+     `compose.include[0].valueSet[1]`, `expansion`), built the same way in both `expand.ts` and
+     `validate.ts` so a locus means the same thing in each, and prefixed with the reference that
+     reached it when the diagnostic comes from a referenced value set. The two do **not** always
+     emit the same _set_ of diagnostics and the changeset no longer says they do: membership
+     short-circuits on a system mismatch and on an exclude that cannot change a decided verdict,
+     where expansion still has to report the part it could not compute. Not a truncation: an index path is
+     strictly more precise than the URI, which cannot distinguish two `include`s over one system.
+
+  **Severity is medium-low and stays medium-low.** Sites 1 and 2 echo the caller's own
+  configuration, not a patient's document; site 3 echoes a canonical URI out of reference content.
+  The engine's genuinely patient-derived inputs are its _query_ values (a code off a parsed
+  message, a unit off an observation, an NDC off a prescription, the `PatientContext` age/gender a
+  complex map branches on), and **no query value reaches a message or a stack anywhere**: the
+  engine throws only on an unusable source, never on a query. That is now asserted rather than
+  argued.
+
+  **A misfiled result field, corrected in the same docblock and shipped in `dist/index.d.ts`:
+  `MapProvenance.sourceSystem`.** A draft of the "which result fields carry verbatim text" list put
+  the whole of `MapProvenance` under _your loaded artifact_ — "reference data, not anything a
+  patient supplied". That is true of `.targetSystem`, `.conceptMapUrl` and `.conceptMapVersion`
+  (`group.target`, `map.url`, `map.version`) and **false of `.sourceSystem`**, which `translate`
+  fills from the `system` on the `Coding` **you** passed it, verbatim and unbounded, falling back to
+  the map's `group.source` only when the coding carries none. It is a query echo and is now listed
+  as one, so a reader applies the log-it-only-if-you-would-log-the-code rule to it. This is a
+  classification fix in prose: the value has always been the caller's own, it is not on a
+  diagnostic-message surface, and **nothing about the runtime behaviour changed** — bounding it is
+  not attempted here, and would be the same fabricate-a-miss trade-off as bounding any other echo.
+
+  **The `@param` contract this broke** is `TerminologyError`'s own: _"a **value-free** structural
+  description (path + fault; never an input value)"_. It now also tells a caller why, since the
+  string reaches `err.stack`.
+
+  **The "Value-free by construction" claim in `src/common/diagnostics.ts` was checked before being
+  rewritten, and it was false** on two counts: the two messages above echoed arbitrary input, and
+  "at most a code + system + version" described the leaking `ExpansionDiagnostic.system` as though
+  it were the safety property. The new wording states the mechanical fact — **every string a message
+  is assembled from is owned by the engine**: a literal, a frozen-table entry, or a locus this
+  package built (a `line`, a column count, an index path of integers and FHIR field names) — and, in
+  the same place, scopes it: it covers **diagnostics**, not **results**, which carry the caller's
+  values by design. A first draft of that replacement said instead that "no message is built from a
+  value parameter" and that every message is "a literal or a frozen-table entry". **That was false as
+  written**, of factories it did not touch: `malformed(path, fault)` in `conceptmap/load.ts` and
+  `valueset/load.ts`, and `cannotExpand` / `truncated` / `underPath` in `valueset/` — where
+  `expand.ts` and `validate.ts` each carry their own copy — all of which do interpolate
+  `string` parameters. **An earlier draft of this paragraph said "four", and the tree has seven**;
+  a count in prose is the thing that goes stale, so it is derived now, not written down.
+  The code is fine — every argument reaching them is engine-owned, each `fault`
+  a literal at the call site and each `path` built from indices — but the sentence was stronger than
+  the code, which is precisely the failure this entry exists to end. It now describes the arguments,
+  not the signatures.
+
+  **The mechanism is now stated as a general rule in exactly one place — the
+  `src/common/diagnostics.ts` docblock.** `README.md`, `docs-content/troubleshooting.md` and
+  `docs-content/concepts-archetype.md` carried the old "code + system + version" sentence, and the
+  `test/phi/diagnostic-surface.test.ts` header (new in this slice) carried the same doctrine in a
+  later wording; all four now state only the **consumer-facing property** — nothing you configured
+  and nothing your release or resource contained reaches a `message`, `detail`, `reason` or
+  `err.stack`, at any length — and none restates the general rule. An earlier draft of this slice
+  moved all four to a _restatement_ of the mechanism instead, in the wording already found false of
+  `malformed` / `cannotExpand` / `truncated`, and this entry claimed they had moved with the fix
+  when what they had moved to was the superseded draft. No gate in this repo can grade an English
+  sentence, which is the argument for one copy of the rule rather than four. **Statements scoped to
+  a single factory are not copies of the rule and deliberately stay put** — `csv.ts` still says its
+  own `MISSING_COLUMN_MESSAGE` lookup takes no value parameter, and `TerminologyError`'s `@param`
+  still tells a caller what may go into the string it is handed. The `diagnostics` guidance in
+  `troubleshooting.md`, which told readers to read a diagnostic for "which code system was missing",
+  points at `path`.
+
+  **What the gate proves is narrow, deliberately:** for each declared slot, no verbatim echo of
+  four or more bytes of the planted value on any swept surface, and the slot provably reached the
+  code it names. It does not prove the absence of a re-encoded echo, an echo under four bytes, or a
+  leak through a slot nobody declared. Two limits are written into the file rather than implied
+  away. First, `expectCode` shows the branch ran, not that the _marker_ ran it: about half the slots
+  are **model-only** and pair the marker with a second marker-free malformed element so a diagnostic
+  exists to sweep, and for those it is the sweep of **those diagnostics** that carries the proof. (An
+  earlier draft said "the whole rendered result". The runner sweeps exactly what the two selectors
+  return plus the thrown value, so for the `CodeSystem.url` slot the `url` itself is never swept —
+  correctly, since it is payload, but not swept.) Second, `getModelIdentifiers` returns the model's
+  real loci — a `RxNormLoadWarning.file`
+  token, an `ExpansionDiagnostic.path` — which are also swept as part of their diagnostic, so it is
+  redundant rather than load-bearing; it exists so the classification of every name-like string on
+  every exported model type is executed and reviewable instead of collapsing to a `[]` a reader has
+  to trust. `Property.code` and `RxNormEdge.predicate` are classified as payload on a checkable
+  test: an HL7 v2 `segment.type` is spec-bounded to three characters, so bounding it discards
+  nothing, while a FHIR `code` carries no `maxLength` and RxNorm's `RELA` is an open vocabulary, and
+  both are the keys the engine and the caller match on, so truncating either turns a hit into a
+  miss. The residual is
+  named: that protects the lookup, not a future consumer who builds a locus out of one.
+  `@cosyte/cli` is today the only package in the ecosystem that depends on this one, and it uses
+  `loadConceptMap` + `translate` only.
+
+  **A FOURTH site, found by the refuter after the three above: `reduce` echoed a forged atom's
+  `code` into a plain `Error.message` and `err.stack`** — a second ~1 MB leak. `reduce` is exported
+  and takes a caller-built
+  `UnitNode`, and nothing checks that the atoms on it came from the vendored UCUM table, so a
+  forged atom whose `value.unit` does not parse reaches the throw. Measured on `0.0.5`: a
+  1,000,000-byte atom code produced a **1,000,042-byte** `Error.message`, with the same bytes in
+  `err.stack`. Every `Error` message `reduce` can throw is a literal — two of them at that point,
+  three after the memo fix recorded at the top of this section. The site had been marked `/* v8 ignore */` as
+  "unreachable with the shipped data", and the throw was moved outside that block, with
+  `test/ucum/reduce.test.ts` pinning the message and the stack. **The two branches still inside the
+  block were not unreachable either, and the comment stopped saying they were.** A draft of that slice
+  relabelled them "genuinely unreachable through the public API"; they were not, because `inProgress`
+  and `atomMemo` keyed on the atom's `code` string and nothing checked that an atom came from the
+  vendored table, so a caller-forged atom whose code collided with a shipped one re-entered the
+  cyclic guard (measured: `reduce` over a forged `mol` defined as `mol` threw it) and a forged atom
+  with no `value` fell through the second, which threw nothing — it memoized dimensionless and
+  returned. The "with the shipped data" scope was the defensible one and was restored, stated
+  as a coverage exclusion over a corrupt-table guard rather than as an assertion of unreachability.
+  Neither branch leaked — the one message inside the block was a literal — so nothing was deferred **on
+  the diagnostic surface**. What was deferred is not merely that the branches were reachable: that
+  second branch wrote the forged atom's `code` into the module-global `atomMemo` before returning,
+  so one `reduce()` over a forged, value-less atom whose code collided with a shipped one poisoned
+  every later reduction of that atom in the process — enough to make `ucumEqual("mg/L", "mg")`
+  answer `true`. That is a **never-fabricate defect, not a diagnostic-surface one**, it was
+  byte-identical on `0.0.5`, and it was filed as its own item rather than folded in here. **It is
+  fixed in this same unreleased window — see the memo-poisoning entry at the top of this section —
+  and the `/* v8 ignore */` block is gone with it.** It is **not** in the slot table and
+  cannot be: it throws an un-coded `Error`, so it can carry no `expectCode`, and every slot in
+  that table names one. The 2026-07-30 ecosystem audit recorded this package as "one site"; with
+  `ExpansionDiagnostic.system` and this, it was **three**.
+
 - **`LICENSE` was unqualified MIT over a tree that also distributes third-party material**
   (`TERMINOLOGY-LICENSE-THIRD-PARTY`, residuals 1 and 2 of `TERMINOLOGY-RESIDUALS`). The README had
   already grown a "What is bundled" carve-out; the licence file had not, so the two consumer-facing

package/README.md

@@ -199,10 +199,19 @@ _names_, listed under [What is bundled](#what-is-bundled).
   **are** bundled are named, with their copyright, under [What is bundled](#what-is-bundled).
 - **Liberal load, conservative assertion.** Malformed input degrades to a typed diagnostic; a 1:many
   mapping returns the full candidate set, never collapsed to one.
-- **Value-free diagnostics.** A diagnostic carries a code + system + version, never patient context —
-  a code _in patient context_ can be PHI.
+- **Value-free diagnostic messages.** Nothing you configured and nothing your release or resource
+  contained reaches a `message`, `detail`, `reason` or `err.stack` — not a column name, not a
+  canonical URI, not a unit atom — at any length. What sits beside a message is a locus: a line
+  number, a column count, an index path into your own resource (`compose.include[2]`), or a fixed
+  token naming which file of a release it came from. The
+  **objects** are a different matter: a typed `unknown`/`unmapped` outcome names the code, unit or
+  NDC you asked about, so log the `code` and the locus rather than the whole result — a code _in
+  patient context_ can be PHI.
 - **Zero runtime dependencies. Dual ESM + CJS.** Node stdlib only; built with `tsup`, validated with
-  `attw`. Immutable by construction (every returned value is deep-frozen).
+  `attw`. The models the engine loads for you are frozen — the bundled unit table deeply so, atom by
+  atom and definition by definition, because it is shared with the engine's own reducer. The AST and
+  the reduction that `parseUcum` and `reduce` hand back are per-call values you own, and mutating one
+  changes nothing for anyone else.
 
 ## What is bundled