@cosyte/astm 0.0.7 → 0.0.9

package/CHANGELOG.md

@@ -11,6 +11,59 @@ this file is maintained by hand (Changesets handles the version bump and publish
 
 ### Added
 
+- **`ASTM_RECORD_FIELDS_UNSEPARATED`: a record that lost its fields no longer loses them in
+  silence** (`ASTM-TYPE-LETTER-SECOND-READER`, finding 2, the one that outranked the other).
+  A record carries its type letter and then its fields, separated by the field delimiter the header
+  declared, so a record that carries content beyond its type letter and still yields exactly **one**
+  field contains no field separator at all: the delimiters in force are not the set that record was
+  written with. Every modeled field of it is then absent. On an `R` that is the **value, the units
+  and the status at once**, and the result reads back as though it simply never carried them.
+
+  The reachable route is a header whose type letter the reader could not recognize. Delimiters are
+  re-read at every `H`, keyed on that same letter, so an unrecognized header does not re-scope them
+  and the whole following message is tokenized with the previous header's set. Measured: a
+  `99.9 mmol/L` final result read back with no value, no units and status `unspecified`, filed under
+  the previous message's patient, with the unrecognized-type warning as the **only** report on the
+  stream. That warning says a letter was unreadable; it never said a value had gone.
+
+  The detector is keyed on the **observed collapse**, not on that one cause of it, because
+  identifying the mangled header would itself require guessing which byte the sender meant. It
+  therefore also fires on the same silent collapse reached without any mangled header at all, which
+  reproduced on the previous release with **zero** warnings: a lone record written in another set
+  (`H|\^&` then `R*1*:::688*99.9*mmol/L**H**F`) parsed clean and answered `undefined` for the value.
+
+  **Reported, never repaired.** The fields are not re-split on a set no header declared, because
+  that would invent data; the raw line is surfaced intact and nothing is dropped. One warning per
+  affected record, each at its own position and carrying no field data. The code is **safety-critical
+  by construction** (the forbidden set is computed as every known code minus the tolerable
+  allow-list), so no profile can quiet it and `{ strict: true }` refuses. A header is exempt by
+  construction rather than by exception: it is always read with the set it declares itself. Content
+  after the type letter that is entirely whitespace is excluded, since no field is at stake there.
+
+  **Deliberately partial, and its absence certifies nothing.** This tests one of the four delimiter
+  roles, the **field** separator, and only in its total form, where that separator occurs nowhere in
+  the line. Two classes of the same loss stay silent and are documented as such on the code, in
+  `README.md` and in the quickstart rather than left to be inferred. A foreign set whose **field**
+  separator happens to occur anywhere in the line still splits, on the wrong boundaries: one stray
+  `|` inside an otherwise `*`-separated result loses the value, the units and the status with no
+  warning, and this can happen to one record **inside** a run of these warnings, so a run is not a
+  sweep of the records it spans. A set differing in the **repeat, component or escape** role usually
+  splits into fields normally, with the damage varying: a mis-split component can cost a test
+  identity while the value survives, but an **escape** character occurring literally in a record
+  merges every field after it and costs the value, the units and the status together (a lone `&`
+  under the canonical set leaves a nine-field `R` reading as four, in silence). All of them
+  reproduce identically on the previous release, and the last is recorded as a defect in its own
+  right. Widening the check would mean deciding which set a record ought to have had, which is the
+  same guess declined above, so the boundary is written down instead of chased.
+
+  The exported factory `fieldsUnseparated` joins the record registry, and `WARNING_CODES` goes from
+  15 members to 16. **That a record's fields are separated by the declared field delimiter is read
+  off this package's own emit contract and off every fixture in this repository, not off a normative
+  sentence, and no clause is claimed for it**: the relevant CLSI text is withheld from the free
+  sample and the surrounding standards are paywalled, and the redistributable reference corpus
+  hardcodes the canonical delimiters and never reads the declaration, so it cannot ground a
+  delimiter question either.
+
 - **`messages()`: read a stream as the sequence of messages it actually is**
   (`ASTM-PATIENT-RESULT-MISATTRIBUTION`). A parsed model has always been a whole record
   **stream**, and a stream may carry several messages back to back: a message runs from an `H`
@@ -129,6 +182,106 @@ phase 8` passes while `Phase 8` reds). An arm keyed on a following digit was wri
 
 ### Changed
 
+- **BREAKING: a record type letter the reader could not recognize no longer lets a host-query
+  request read as a result set** (`ASTM-TYPE-LETTER-SECOND-READER`, finding 1). `classifyMessage` counts `Q` / `R` /
+  `O` records **by letter**, and the `Q`-dominates guarantee (a message carrying a query is never
+  read as a result set) was stated on that count, so it only ever held while every letter was
+  legible. Measured: `H|\^&` plus a `Q` carrying one stray leading byte plus an `R` classified
+  `kind: "results"` with `isHostQueryRequest: false`, and the `ASTM_RECORD_AMBIGUOUS_MESSAGE_KIND`
+  warning that flags a `Q` + `R` contradiction vanished along with the `Q` it was counting. A stray
+  byte turned the fail-safe off.
+
+  The intended letter is still **never inferred**: that is the guess this package declines to make,
+  and inferring it would split a stream a different way just as silently. What the classifier does
+  instead is decline the positive answer. An unsupported record with no `Q` read alongside it now
+  yields `kind: "indeterminate"` rather than `"results"` or `"orders"`, and the new
+  `AstmMessageClassification.hasUnrecognized` reports why. A `Q` that **was** read still dominates,
+  because an unreadable letter can only ever add a kind, never remove a query already on the wire.
+
+  `hasQuery` / `hasResults` / `hasOrders` stay truthful, so a caller wanting the raw tally still has
+  it, and `results()` and the rest are untouched. Breaking for a consumer branching on `kind`: on a
+  stream carrying an unrecognized record letter, a previously positive `kind` now reads
+  `indeterminate`. That is the fix, and it moves in the fail-safe direction. A conformant stream,
+  where every letter is legible, is unchanged. Kept on the `0.0.x` pre-alpha ladder as a patch, per
+  the repo's version policy.
+
+  Two limits, stated rather than implied. `msg.classification` is still folded over the **whole
+  stream** (a known, separately-recorded defect), so one unrecognized letter anywhere now withholds
+  `kind` for the entire stream **unless a `Q` was read**, in which case the `Q` still dominates as
+  before. That widens the existing over-trigger; the per-message answer,
+  `classifyMessage(m.records)` on a `messages()` entry, is unaffected and is the reading to prefer.
+  And the fix lands on `kind`, not on `isHostQueryRequest`, which `README.md` calls the safety
+  surface: a mangled `Q` still reports `false` there, because the parser genuinely did not read a
+  query. `false` therefore means "no query was read", never "this is a result set", and both the
+  type doc and the quickstart now say so and point at `kind`.
+
+- **BREAKING: a profile may no longer tolerate `ASTM_RECORD_UNKNOWN_TYPE`**
+  (`ASTM-UNKNOWN-RECORD-REMERGE`). The code moves off the profile safety gate's tolerable
+  allow-list and onto the safety-critical set, so `defineAstmProfile({ tolerate: [...] })` naming
+  it now throws `AstmProfileDefinitionError` at definition time instead of building a profile, and
+  the gate is re-checked when a warning would be downgraded, so a profile assembled as a plain
+  object rather than through the factory gets the same answer (it declines to downgrade rather than
+  throwing, so a hand-authored profile still parses, it just cannot quiet a safety-critical code).
+  `TOLERABLE_CODES` goes from four members to three (`ASTM_NONSTANDARD_DELIMITERS`,
+  `ASTM_UNKNOWN_ESCAPE_SEQUENCE`, `ASTM_RECORD_UNINTERPRETED_QUERY_STATUS`) and
+  `SAFETY_CRITICAL_CODES` / `isSafetyCriticalCode()` answer accordingly. No built-in profile
+  tolerated the code, so nothing in this package changes shape; a consumer profile that named it
+  will now fail to define, loudly, at the point of definition.
+
+  **Why, and it is a wrong-patient path.** A message runs from an `H` header to its `L`
+  terminator, and `messages()` decides where one starts by reading each record's **type letter**.
+  A header the reader cannot recognize as a header (one carrying a stray leading byte, for
+  instance) is surfaced as an unsupported record, opens no message, and the two messages either
+  side of it are grouped as one. The merged message pairs a patient with results that arrived
+  under a different header. The `ASTM_RECORD_UNKNOWN_TYPE` warning is the entire report that this
+  happened, so a profile downgrading it to `PROFILE_QUIRK_APPLIED` quieted the whole signal, and a
+  `{ strict: true }` parse then accepted the stream with no objection at all.
+
+  **The type letter turned out to have two load-bearing readers, not one.** `classifyMessage`
+  counts `Q` / `R` / `O` records by the same letter, and the rule that a message carrying a query
+  is never read as a result set is stated on that count. An unrecognized `Q` costs a host-query
+  request that guarantee: the message reads as `kind: "results"`, and the
+  `ASTM_RECORD_AMBIGUOUS_MESSAGE_KIND` warning that flags a query-plus-result contradiction
+  disappears along with the `Q` it was counting. There too `ASTM_RECORD_UNKNOWN_TYPE` is the only
+  warning left on the stream. Both readers are measured in the regression test. The
+  mis-classification itself is not fixed here, for the same reason the merge is not: it is a
+  property of reading the letter, and inferring the intended letter is the guess this package
+  declines to make.
+
+  **The variant that reaches furthest carries no second patient.** When the merged second message
+  has its own `P`, the merged message holds two and the multi-patient guard throws. When the
+  second message is **result-only**, the merged message still holds exactly one `P`, so `patient()`
+  and `results()` both answer confidently and the first patient silently acquires the second
+  message's results. That is the fixture the regression test pins.
+
+  **The parser is deliberately unchanged.** Recognizing a mangled header _as_ a header means
+  guessing at a byte the sender did not send, and that guess would split a stream a different way
+  just as silently. A lenient parse still merges and still warns; a strict parse refuses; and the
+  refusal can no longer be configured away. `messages()`, the warning's own message text, the
+  `UnsupportedRecord` type, the README and the quickstart all now say to read this code as a
+  possible lost message boundary rather than as cosmetic noise.
+
+  **The admission test for the allow-list gained a second clause**, which is the durable part of
+  this change. A code was admitted on one question: can it alter, drop, or fabricate an extracted
+  value? `ASTM_RECORD_UNKNOWN_TYPE` still passes that one, and it was still wrong to tolerate,
+  because the condition it reports had quietly become something the library reads. The rule is now
+  that a code is tolerable only while **nothing else in this package reads the condition the
+  warning reports**, and that second clause is a claim about the whole library which can stop being
+  true without anyone editing `safety.ts`. It is a review obligation, not a mechanical check, and
+  the file says so rather than implying a gate that does not exist. The three surviving codes were
+  each re-derived against both readers of record structure, and each is measured by comparing the
+  same logical stream with the reported condition present and absent. That is deliberately not a
+  comparison of a parse with a profile against one without: the profile transform runs after the
+  records are built, so the second comparison is identical for every code and would have passed for
+  the one just removed. The comparison actually used is exercised on the mangled-header pair as a
+  negative control, where it has to fail.
+
+  **Evidence, labelled.** The `H` … `L` message unit is _verified primary_ and its grounding is
+  recorded on `messages()`, unchanged here. The allow-list itself is **this library's own policy,
+  not the standard's**: no clause governs which of our warning codes a profile may downgrade, and
+  none is claimed for it. The behaviour above (the merge, the accessors answering, the strict
+  acceptance under a tolerating profile) is _measured_, on the tree before and after the change.
+
 - **The em dash (U+2014) is gone from every tracked file, and a CI gate keeps it out**
   (`EMDASH-CONFORMANCE`). The brand rule bans the character outright across every cosyte surface
   and names commit messages explicitly; this package was the last one it had not reached, and the
@@ -225,6 +378,68 @@ phase 8` passes while `Phase 8` reds). An arm keyed on a following digit was wri
 
 ### Fixed
 
+- **One unescaped `&` in a value no longer swallows every field after it**
+  (`ASTM-UNESCAPED-ESCAPE-SWALLOWS-TAIL`; `PRE-EXISTING`, shipped `0.0.1` through `0.0.8`, not a
+  regression). The escape codec read an escape character with no closing escape character as the
+  opening of a sequence and copied from it to the end of the record, so every field after it merged
+  into the field it sat in, **and there was no warning code for the condition at all**. Measured on
+  `064c078`: `R|1|^^^687|28.6&|U/L||N||F` read back `value = "28.6&|U/L||N||F"` with `units`,
+  `abnormalFlags` and `flag` all `undefined` and `status` `unspecified` rather than `final`, on a
+  stream declaring the canonical `H|\^&`, with `warnings: []`. On the patient side
+  `P|1||LAB-0001||O&BRIEN^JANE||19800101|F` read the surname as the whole rest of the record and
+  came back with **no birth date and no sex**, again silently. A result that reads `28.6` with its
+  units gone is the harm this package exists to prevent, and an ampersand in a surname is not exotic.
+
+  **The round trip made it worse rather than surfacing it.** Emit re-escaped the merged text into
+  `R|1|^^^687|28.6&E&&F&U/L&F&&F&N&F&&F&F`: a spec-clean-looking line that re-parsed to the same
+  wrong value, with zero warnings, byte-stable across further trips. Silent re-read divergence, with
+  no malformed delimiter set anywhere in it.
+
+  **The fix is a bound, and it infers nothing.** An escape sequence is now exactly three characters
+  (the escape character, **one** body character, the escape character), which is all the four
+  mnemonics `&F&` `&S&` `&R&` `&E&` ever need, and both `splitEscapeAware` and `decodeEscapes` share
+  that one definition. An escape character that heads no such sequence is read as the **literal
+  character it is**: the value keeps the byte that arrived, it opens no atom, and nothing is
+  invented to close a sequence the sender did not open. The same fixture now
+  reads `value = "28.6&"` with `units: "U/L"` and `status.meaning: "final"`, and the patient keeps
+  their birth date and sex. Emit writes the literal character as `&E&`, so the emitted line is
+  spec-clean **and** structurally faithful, and re-parses to the same tree.
+
+  The bound also removes a non-local behavior: under the unbounded scan, whether a field kept its
+  value depended on whether some **later** field in the record happened to carry an escape character
+  too, which paired across the field separator between them. Two bare ampersands in one record now
+  cost nothing.
+
+  New warning code `ASTM_UNPAIRED_ESCAPE_CHARACTER` and exported factory `unpairedEscapeCharacter`
+  report each such character, one per occurrence, positioned on the record and field, carrying no
+  field text (PHI discipline). The code is **tolerable**: a profile may expect it, because reading
+  the character as a literal is unconditional and the parsed value is byte-identical with the profile
+  and without it. Untolerated, `{ strict: true }` refuses.
+
+  **The scope is the bare character, and only that.** The atom rule is unchanged, so an `&X&`
+  sequence whose body **is** a delimiter still swallows that delimiter and still costs the value, the
+  units and the status together: `R|1|^^^687|28.6&|&U/L||||F` reads `28.6&|&U/L` with no units and
+  status `unspecified`. It is never silent, but its only report is `ASTM_UNKNOWN_ESCAPE_SEQUENCE`,
+  which is tolerable, so a profile expecting that code lets `{ strict: true }` accept it. That case
+  is recorded as a known defect and deliberately not closed here: narrowing the atom would break the
+  guarantee it exists for, which is that `&F&` stays one token under a set naming `F` as a delimiter.
+  Both halves are pinned, so neither can drift.
+
+  `ASTM_UNKNOWN_ESCAPE_SEQUENCE` is unchanged for a single unrecognized body (`&Z&`). A multi-
+  character body is no longer treated as one atom: its bytes are all preserved, but a **delimiter
+  inside** such a body now splits, which moves every field after it, and the report becomes one
+  unpaired-character warning per loose escape character instead of one unknown-sequence warning. Over
+  an exhaustive corpus of bodies up to four characters in a result-value slot, that is 1,085 records
+  read correctly where the previous release read them wrongly, against 27 read differently in the
+  other direction.
+
+  Measured red on base: 10 of the 18 new tests fail against `064c078` extracted into a clean tree.
+  Of the 8 that pass, 6 measure behavior this change does not touch and 2 are the boundary pins on
+  what it deliberately does **not** fix.
+  `test/records/unseparated-fields.test.ts` had pinned this loss as a documented limit of
+  `ASTM_RECORD_FIELDS_UNSEPARATED`; that pin is inverted rather than deleted, and the prose stating
+  the limit moved with it in `README.md`, the quickstart and the warning code's own docs.
+
 - **Emit no longer writes a stream it cannot read back** (`ASTM-EMIT-RESIDUALS`; both gaps shipped
   `0.0.1` through `0.0.3`, `PRE-EXISTING`, neither a regression). Two residual emit-side holes, both
   recorded and deliberately deferred by `ASTM-MIXED-DELIMITER-EMIT` (#21) and again by

package/README.md

@@ -42,11 +42,14 @@ reference parser, [`@cosyte/hl7`](https://github.com/cosyte/hl7).
   and lost.
 - **Vendor profiles.** `defineAstmProfile()` builds a provenance-backed profile whose tolerances
   downgrade _expected_, non-safety-critical deviations to a `PROFILE_QUIRK_APPLIED` warning without
-  ever altering a value, behind a definition-time safety gate that refuses to tolerate any result
-  value, flag, status, range, or units warning, any patient or comment context, any message-kind
-  ambiguity, and any frame or transport integrity warning. A profile can never make a bad checksum
-  "ok" or a cancelled result read "final." Named per-vendor profiles await a public,
-  vendor-attributed quirk document.
+  ever altering a value, behind a safety gate that refuses to tolerate any result value, flag,
+  status, range, or units warning, any patient or comment context, any message-kind ambiguity, any
+  unrecognized record type, and any frame or transport integrity warning. The gate runs when a
+  profile is defined **and** again when a warning would be downgraded, so a profile assembled as a
+  plain object rather than through `defineAstmProfile()` gets the same answer. A profile can never
+  make a bad checksum "ok", a cancelled result read "final", or quiet the warning that says a
+  message boundary went unrecognized. Named per-vendor profiles await a public, vendor-attributed
+  quirk document.
 - **Terminology, bring your own.** `applyLivd(msg, catalog)` maps an analyzer's local test code to a
   LOINC from a consumer-supplied IICC LIVD catalog as an additive, advisory annotation that never
   mutates the raw code or value and never guesses a LOINC. No LOINC, SNOMED, or LIVD dictionary is
@@ -140,6 +143,30 @@ produce a confident wrong value: an embedded escaped delimiter reads as one comp
 record type is surfaced (never dropped), and a missing unit is flagged (never defaulted). A
 `{ strict: true }` mode escalates every tolerated deviation to a thrown error.
 
+### An unescaped ampersand does not cost you the rest of the record
+
+An escape sequence is the escape character, **one** body character, and the escape character again
+(`&F&` `&S&` `&R&` `&E&`). An escape character that heads no such sequence is not an escape: it is
+read as the literal character it is, it opens no atom, and `ASTM_UNPAIRED_ESCAPE_CHARACTER` reports
+it. So `R|1|^^^687|28.6&|U/L||N||F` reads a value of `28.6&` with units `U/L` and status `final`,
+and `O&Brien` in a surname keeps the patient's birth date and sex.
+
+The parser does not decide what the sender meant by the character: it keeps the byte that arrived
+and says so. The spec-clean way to send a literal escape character is `&E&`, which is what this
+package's serializer emits, so a stream it produced never trips the code. The code is **tolerable**,
+so a vendor profile can expect it on a feed that sends bare ampersands and still parse
+`{ strict: true }`.
+
+**One escape shape still costs a field boundary, and it is a different code.** A real three-character
+sequence is opaque by design, which is what keeps `&F&` one token under a set that names `F` as a
+delimiter. So where the body is itself a delimiter (`&|&` under the canonical set) that delimiter
+does not split, and every field after it shifts: `R|1|^^^687|28.6&|&U/L||||F` reads a value of
+`28.6&|&U/L` with no units and status `unspecified`. It is never silent, but the only report is
+`ASTM_UNKNOWN_ESCAPE_SEQUENCE`, which is **tolerable**, so a profile expecting that code (including
+the shipped `referenceCorpus`) will let a `{ strict: true }` parse accept it. Narrowing the atom to
+close this would break the guarantee the atom exists for, so it is written down rather than papered
+over.
+
 ### Several messages in one stream
 
 A message runs from its `H` header to its `L` terminator, so a stream can carry several. `messages()`
@@ -170,6 +197,58 @@ patient is unchanged, and so is a result-only message with no `P` at all, which
 `undefined`. `commentsFor()` is unchanged on every stream, because the parent record you hand it
 already names the message.
 
+Splitting reads each record's type letter, so check for an `ASTM_RECORD_UNKNOWN_TYPE` warning before
+you trust the split. A header the reader does not recognize as a header, one carrying a stray leading
+byte for instance, opens no message, and the messages either side of it merge back into one, so a
+patient can end up holding results that arrived under a different header. The
+parser warns on that record and a `{ strict: true }` parse refuses the stream. That warning is the
+only report the merge produces, so a profile is not allowed to tolerate it: the code is refused when
+a profile is defined, and a warning carrying it is not downgraded whatever profile is in force. Do
+not gate on the warning count, though, because the records that merged in can raise warnings of
+their own.
+
+Delimiters are re-read at each header too, so if the unrecognized one declared a different set, the
+records after it are read with the previous set and their fields can be lost rather than merely
+misfiled. `ASTM_RECORD_FIELDS_UNSEPARATED` reports a record that suffered the total form of that:
+the delimiters in force found no field separator in it at all, so the whole line read back as one
+field and none of its modeled fields survived. On a result record that is the value, the units and
+the status at once, so treat it as a lost result, not a formatting nit. The fields are never
+reconstructed, because the set the sender used is unknown and guessing at it would invent data. The
+code is safety-critical, and it does not need a mangled header to fire: a lone record written in
+another set trips it too.
+
+**Its absence does not certify that a record was read in its own set**, and this is the important
+half. The check tests one of the four delimiter roles, the **field** separator, and only in its
+total form, where no unescaped separator occurs in the line. Two classes of the same loss sit
+outside it:
+
+- A foreign set whose **field** separator happens to occur somewhere in the line still splits, on
+  the wrong boundaries and in silence. A single stray `|` in an otherwise `*`-separated result loses
+  the value, the units and the status with no warning at all, while the identical record without
+  that one byte is reported. This also happens **inside** a run of these warnings, so even a run
+  does not mean every record in it was checked.
+- A set differing in the **repeat, component or escape** role usually splits into fields normally,
+  and the damage then varies. A mis-split component can cost a test identity while the value and
+  units survive. The escape role's worst case has **narrowed, not gone**: a bare escape character no
+  longer merges the rest of the record (it reads as a literal and raises
+  `ASTM_UNPAIRED_ESCAPE_CHARACTER`), but an `&X&` sequence whose body is a delimiter is an opaque
+  atom, so that delimiter does not split and the value, the units and the status can still go
+  together. That one is reported only by the tolerable `ASTM_UNKNOWN_ESCAPE_SEQUENCE`.
+
+All are accepted limits, for two different reasons: widening the field-separator check would mean
+deciding which set a record ought to have had, which is the same guess the parser declines to make
+elsewhere, and narrowing the escape atom would break the guarantee it exists for. So they are
+written down rather than papered over. Read the warning as "this record definitely lost its
+fields", never as "no other record did". If
+delimiter drift is a real risk on your feed, parse with `{ strict: true }`, which refuses both an
+outright collapse and an unrecognized type letter, and treat `ASTM_RECORD_UNKNOWN_TYPE` as
+invalidating what follows it rather than expecting this warning to enumerate the damage.
+
+An unrecognized type letter also makes the message **kind** unknowable, because the letter that
+could not be read may have been the very `Q` that decides it. `classification.kind` is
+`indeterminate` in that case rather than `results` or `orders`, and `classification.hasUnrecognized`
+says why. A `Q` that was read still wins outright.
+
 ## Map local codes to LOINC (LIVD, bring-your-own)
 
 An analyzer sends a proprietary local test code in the Universal Test ID; a standard LOINC is mapped