@cosyte/astm 0.0.9 → 0.0.11

package/CHANGELOG.md

@@ -9,6 +9,193 @@ this file is maintained by hand (Changesets handles the version bump and publish
 
 ## [Unreleased]
 
+### Fixed
+
+- **A symbolic link under a scan root read CLEAN on BOTH of the PHI scanner's enumerating routes, so
+  a link pointing at a file full of real identifiers passed the gate twice over**
+  (`PHI-SCAN-SYMLINK-BLIND-ON-BOTH-ROUTES`). Development tooling only: `scripts/phi-scan.ts` ships in
+  no tarball and the package's public surface is unchanged.
+
+  Reproduced on `8e8012d` before any fix, with a synthetic name-bearing payload (an ASTM `P` record
+  carrying a patient name, a mother's maiden name and a birthdate, plus a dashed SSN and an email at
+  a non-test domain) placed outside the walk roots and a link to it at `src/leak.ts`. `pnpm phi-scan`
+  printed `OK: no hits` and exited **0**; `pnpm phi-scan --staged`, after `git add`, printed
+  `OK: no hits` and exited **0**. Naming the link's target explicitly on the command line reported
+  **six** hits and exited 1, so the payload was always detectable, and detectable by this package's
+  own structured `P`-record detectors rather than only by the cross-cutting SSN/email floor. The two
+  routes simply never looked at it.
+
+  The two blindnesses are separate mechanisms and needed separate fixes. The walk enumerates
+  `Dirent.isFile()`, which is an **lstat** answer, so a link is neither a file nor a directory and
+  fell out of the loop with no record that anything had been skipped, and `isDirectory()` answers
+  false for a linked directory too, so a whole subtree could disappear the same way. The `--staged`
+  route reads content with `git show :<path>`, and **git stores a symbolic link as its target path
+  under mode `120000`**, so it was handed the target's path text and scanned that. That second route
+  is this repo's `pre-commit` hook, which is exactly where the claim that `--staged` covers a link
+  would have been trusted.
+
+  **Neither route is made to follow the link.** Following would read bytes the enumeration does not
+  control (outside the repo, a loop, a device, a FIFO that blocks the gate forever) and git does not
+  carry those bytes anyway, so a hit on them would be a claim about something no commit contains.
+  Instead the enumeration is narrowed: an entry under a scan root that is not a regular file
+  **refuses the scan** (exit 2, the existing "could not complete" code), naming every offender rather
+  than the first. The walk classifies by `Dirent` (symbolic link, FIFO, socket, block device,
+  character device); `--staged` now reads `git diff --cached --raw -z` instead of `--name-only` so
+  the destination mode is visible, and refuses mode `120000` and `160000` (a gitlink) wherever that
+  route's own path scope reaches them. A `--raw` record that does not parse refuses as well, rather
+  than being skipped into a silently shortened list.
+
+  **`T` (typechange) is in the `--diff-filter`, and leaving it out would have made the mode check
+  unreachable whenever the file being replaced was already tracked.** Replacing a **tracked** regular
+  file with a link is neither an add nor a modify: git raises `:100644 120000 <sha> <sha> T`, so
+  `--diff-filter=AM` deletes the record before any mode can be read. Measured on git 2.39.5 on this
+  tree: with `AM` the raw output for that stage is empty, and the unfixed scanner passed that stage
+  green. Typechange carries a single path, exactly like `A` and `M`, so admitting it costs the record
+  stride nothing, and it means the reverse typechange, a link replaced by a real file, is now scanned
+  as the ordinary file it became.
+
+  **A refusal names the entry's own repo-relative path and an engine-owned token for its kind, and
+  never the link target**, because a target path is text off the working tree and can itself carry
+  PHI. That is asserted rather than argued: the pinning payload and the target's own filename both
+  carry a synthetic person name, and every refusal message is checked to contain none of it. 10 of
+  the file's 30 cases are red on `8e8012d`.
+
+  **What this does not cover, stated narrowly, each measured.** Explicit-path mode already read
+  through a link and reported the target's hits; it is unchanged. The `--staged` path scope is
+  unchanged, still `test/fixtures/**` and `src/**.ts`, so a staged link outside it is still not
+  looked at; narrowing what a scope admits is not widening the scope. That scope also bounds the
+  gitlink half: a submodule staged at `test/fixtures/nested` is refused, while one at `src/nested` is
+  a directory name that fails the `.ts` suffix and is not looked at. `R` (rename) and `C` (copy) are
+  still **not** enumerated by `--staged` at all, so a staged rename that also appends PHI passes that
+  route; pre-existing, unchanged here, and admitting it needs the two-path record shape handled,
+  which is a scope decision rather than this one. A tracked file **absent from the worktree** is
+  still caught at `git add` time only.
+
+  **And this scanner still has no rule that a sweep observing ZERO targets must refuse**, which is
+  worth naming rather than leaving to be discovered. Measured: run from a tree with no `src/` and no
+  `test/fixtures/`, all-mode prints `OK: no hits` and exits 0, so a clean report can be a report
+  about a tree the scan never looked at, and **nothing backstops it**: the all-mode sweep is the
+  route with the hole, so it cannot be its own compensating control, and neither the `pre-commit`
+  hook nor CI asserts that the sweep observed anything. Five of the sibling scanners carry that rule
+  (`ccda`, `hl7`, `mllp`, `ncpdp`, `synth`) and the rest, this one included, do not; do not read that
+  as a majority practice. It is unchanged by this slice in either direction, and adding it is its own
+  slice, because the threshold is a judgement about this repo's corpus rather than a consequence of
+  anything here.
+
+  **The enumerate-then-read race is deliberately NOT closed here, and the deferral is measured
+  rather than assumed.** The scanner still has no tolerance for a file that vanishes between
+  enumeration and read, so a transient appearing under a walk root can refuse a whole sweep (exit 2,
+  fails closed). This repo's walk roots are `test/fixtures/` and `src/`, and nothing was observed
+  landing in either: polling both roots continuously across a real `pnpm build` saw only the tracked
+  corpus, while `tsup.config.bundled_*.mjs` (the transient that blocked a sibling's publish) appeared
+  at the **repo root**, which is not a walk root here. This repo's own suite writes its throwaway
+  trees under the OS temp directory, not under a scan root. So the shape is present and its trigger
+  is not, which makes closing it a separate slice with its own argument rather than a rider on a PHI
+  blindness fix. It is a different defect and it fails in the safe direction.
+
+- **The frame encoder no longer writes a different character than the one it was handed**
+  (`ASTM-FRAME-BYTE-RESIDUALS`, closing known defect 7). `composeAstmFrames` turned a record given
+  as a `string` into bytes with `charCodeAt(i) & 0xff`, so any character above `U+00FF` was replaced
+  by its low byte. That low byte is another perfectly ordinary character, which is why the
+  substitution was **silent**: it reached the wire, checksummed clean, and came back with an empty
+  warnings array from **both** layers.
+
+  Measured, on this package's own round trip, three shapes on the canonical set:
+  - `R|1|^^^687|28.6|μmol/L||N||F` read back `28.6` in `¼mol/L` (`U+03BC` low byte `0xBC`). A lab
+    number whose **units** silently changed.
+  - `P|1||LAB-0001||ŁUKASZ^JAN||19800101|M` read back the name spelled `AUKASZ` (`U+0141` low byte
+    `0x41`).
+  - `P|1||LAB-0001||GRAżYNA^ANNA||19800101|F` read back as **ten fields where the record models
+    nine** (`U+017C` low byte `0x7C`, the field separator). Every field after the split shifted one
+    along, so `patient().sex` answered `19800101` and the birth date was gone. That is **silent
+    re-read divergence**: a different field tree, `warnings: []` at both layers.
+
+  The known-defect entry said this class fails loudly in every case measured, and for the cases it
+  had measured that was true: they all truncated onto `STX`/`ETX`/`ETB`/`CR`. Nothing had measured a
+  code point whose low byte is an ordinary character, and that is the larger and quieter class.
+
+  **The disposition is a refusal, and it invents nothing.** `composeAstmFrames` now throws
+  `AstmFrameEncodeError` with the new `ASTM_FRAME_UNENCODABLE_CHARACTER` code, carrying the record
+  index and the character's index, never the character and never its code point. The alternative,
+  encoding as UTF-8, was **considered and rejected**: turning a character into bytes requires a
+  character encoding, this library reads no character-set declaration from any ASTM record, and
+  picking one would be a guess at bytes the caller never supplied, in a package that declines to
+  infer a delimiter set, a record type letter or a code system. Emit also has no warning channel, so
+  a warning could only have been ignored while the wrong bytes still shipped. That is the same
+  argument already written for `ASTM_EMIT_UNENCODABLE_VALUE` and `ASTM_EMIT_INVALID_DELIMITERS`.
+
+  **What grounds the Latin-1 boundary**, since no clause is claimed for it and none was read. Two
+  things, both firsthand. (1) This package's own byte-to-string boundary is already Latin-1, on the
+  read side, deliberately: `parseAstmRecords` decodes with one `String.fromCharCode` per byte "so
+  every byte survives 1:1". The encoder is documented as the exact inverse of the decoder, and the
+  decoder can never produce a code point above `U+00FF`, so a string carrying one is outside the
+  codec's domain by construction. (2) The redistributable OSS reference implementation
+  (`kxepal/python-astm`, BSD) parameterizes the codec on `encoding` throughout
+  (`encode(records, encoding=ENCODING, ...)` / `decode(data, encoding=ENCODING)`) and defines
+  `ENCODING = 'latin-1'` in `astm/constants.py`: the wire code page there is caller-supplied
+  out-of-band knowledge with a latin-1 default, not something a reader discovers from the stream.
+  **The normative text was not reachable and no clause is cited**: LIS02-A2 is withheld from CLSI's
+  free sample and E1394-97 / LIS01-A2 are paywalled, so this is a reasoned choice, exactly as the
+  delimiter-scoping and surplus-declaration choices before it were.
+
+  **The refusal removes no capability.** `composeAstmFrames` already accepted `Uint8Array`, so a
+  consumer who knows their instrument's code page encodes with their own encoder and passes bytes,
+  which are written through untouched. A stream whose every character is Latin-1 is byte-identical
+  to before, the full `0x00`–`0xFF` range included. The **record** layer is deliberately unchanged:
+  `serializeAstmRecords` returns a `string`, which is not yet bytes, so the caller keeps the choice.
+
+  **Stated no wider than it was measured.** This closes the string-to-bytes step only. It is not a
+  claim that every accepted record reads back: a record already carrying a raw `STX`/`ETX`/`ETB`
+  byte is framed as given and re-decodes wrong (known defect 6), which this change does not touch.
+  **Both** of that defect's branches are now pinned as out of scope in
+  `test/frames/unencodable-character.test.ts`, the warned one and the **silent** one: where the two
+  bytes following the embedded control character happen to be that truncated frame's checksum, the
+  short frame verifies and a whole result record is absorbed into the previous record's text with
+  `warnings: []` at both layers. That branch was found grading this change, reproduces byte-identically
+  on the base, and is why known defect 6's "it fails loudly" ranking was withdrawn rather than restated. A surrogate pair needs no
+  separate rule, each half being above `U+00FF` itself. New export `AstmFrameEncodeErrorCode`;
+  `AstmFrameEncodeError.code` is now that union and gains a `characterIndex`. The property "either
+  refuse the record string or reproduce it byte for byte" is asserted over 2,000 generated cases,
+  with its bound written into the test and a non-vacuity check that both sides of it are reached.
+
+- **The `attw` publish gate now fails when the packed tarball carries no types, where the CLI exits
+  `0`** (`ATTW-FALSE-GREEN-PORT`). The `attw` script was the bare `attw --pack .`.
+  `@arethetypeswrong/cli` opens `getExitCode()` with `if (!analysis.types) return 0`, returning
+  before it ever reads the problem list, because an untyped package is a legitimate npm package and
+  the CLI treats "no types at all" as a description rather than a problem. No `--profile`,
+  `--ignore-rules` or config setting reaches that branch. For a package that ships types the same
+  result means the declarations were **not in the tarball**, which is a broken publish reported as a
+  pass. A false red costs an hour; a false green merges.
+
+  Measured on this package at `0.0.9`, on a quiet box with **no** concurrency, both states printing
+  "This package does not contain types." and exiting `0`: `rm -rf dist`, and
+  `rm -f dist/index.d.ts dist/index.d.cts`. The second is the realistic one, and the trigger is the
+  build rather than any race: timed on one real `tsup` run here, `dist/index.mjs` and
+  `dist/index.cjs` appeared at 1,176 ms and `dist/index.d.ts` and `dist/index.d.cts` at 3,063 ms, a
+  **1,887 ms window** in which `dist/` held JS and no declarations. A concurrent build or `clean` in
+  the same working tree lands the gate in that window. This is deliberately **not** answered with a
+  lock or a build queue: the gate has to be able to report that its own inputs were missing,
+  whatever removed them.
+
+  `attw` is now `node scripts/attw.mjs`, which keeps two nets because they catch different things. A
+  **preflight** checks that every relative path `package.json` promises (`main`, `module`, `types`,
+  `typings`, and every string leaf of `exports`) exists and is non-empty, and names the missing file
+  rather than leaving it to be inferred. A **post-check** promotes `attw`'s untyped sentence to a
+  failure, catching what the preflight structurally cannot: declarations present on disk but
+  excluded from the tarball by `files` or an `.npmignore`. No instance of that second case is on
+  record in this repo. Because the post-check reads a printed string, the routes that would hide it
+  are refused rather than tolerated: `--quiet`, `--format`, a `.attw.json` setting either, and
+  `--config-path` at a file setting either were each measured here to restore the exact exit `0`. The
+  refusal is by option name, wholesale, not by value, and the list is a record of what is closed
+  rather than a proof that the post-check cannot be blinded at all: `--definitely-typed` suppresses
+  the sentence by making the analysis typed, is equally true of the bare CLI, and is deliberately not
+  refused. The preflight is the net that reads no string. Other arguments are forwarded, so
+  `--profile node16` still works. `test/scripts/attw-gate.test.ts` pins the upstream exit `0` itself,
+  so an `attw` upgrade that fixes it or rewords the sentence reds the suite instead of letting the
+  net go quietly slack, and it carries a negative control plus a real `attw` failure, because a gate
+  that only ever fails is not a gate and one that swallows the status is not one either. **No
+  runtime code changed and the built output is identical.**
+
 ### Added
 
 - **`ASTM_RECORD_FIELDS_UNSEPARATED`: a record that lost its fields no longer loses them in
@@ -481,11 +668,11 @@ phase 8` passes while `Phase 8` reds). An arm keyed on a following digit was wri
   behind nothing but an `ASTM_FRAME_BAD_CHECKSUM`. Rather than enumerate the bytes each layer happens
   to reserve, and re-derive that list whenever a layer is added, no control character is carried at
   all. Two things that rule does **not** do, said here rather than left to be found: it is keyed on
-  the character while the frame layer's structure is keyed on the low byte (`charCodeAt(i) & 0xff`),
-  so a non-control character truncating onto `STX`/`ETX`/`ETB` still breaks framing, loudly, with a
-  typed error or an unknown-record-type warning, never silently; and refusing a control character in
-  a _surplus_ does not mean one cannot be a _delimiter role_, since only `CR`/`LF` are refused
-  there. Those are
+  the character, so a surplus character above `U+00FF` passes it (the frame layer is where that is
+  settled now, and refuses such a character rather than truncating it, see the entry above; when
+  this was written the frame encoder truncated to the low byte and the case broke framing instead);
+  and refusing a control character in a _surplus_ does not mean one cannot be a _delimiter role_,
+  since only `CR`/`LF` are refused there. Those are
   structural losses; dropping inert bytes is not.
 
   **2. A caller-supplied delimiter set was never validated.** `serializeAstmRecords(msg, d)` and its

package/README.md

@@ -39,7 +39,10 @@ reference parser, [`@cosyte/hl7`](https://github.com/cosyte/hl7).
   frame them with computed checksums, frame numbers, and the 240-byte split. Both layers round-trip by
   construction, and a delimiter set that fails any of the three conditions readback requires (one
   character per separator, no `CR`/`LF`, no two the same) is a typed error rather than bytes written
-  and lost.
+  and lost. A frame carries **bytes**, so a record handed to `composeAstmFrames` as a `string` is one
+  byte per character and a character above `U+00FF` is a typed error too: the encoder will not pick a
+  character encoding for you, and it never quietly writes a different character than the one you gave
+  it. Encode such content yourself and pass the `Uint8Array`.
 - **Vendor profiles.** `defineAstmProfile()` builds a provenance-backed profile whose tolerances
   downgrade _expected_, non-safety-critical deviations to a `PROFILE_QUIRK_APPLIED` warning without
   ever altering a value, behind a safety gate that refuses to tolerate any result value, flag,

package/dist/index.cjs

@@ -2226,18 +2226,37 @@ var AstmFrameEncodeError = class extends Error {
   code;
   /** Index of the offending record within the input, when applicable. */
   recordIndex;
+  /**
+   * Position of the offending character within that record, when applicable:
+   * its index in the string, never the character itself and never its code
+   * point. Enough to find it in the caller's own data.
+   */
+  characterIndex;
   /** @internal */
-  constructor(message, recordIndex) {
+  constructor(message, recordIndex, code = "ASTM_FRAME_EMPTY_RECORD", characterIndex) {
     super(message);
     this.name = "AstmFrameEncodeError";
-    this.code = "ASTM_FRAME_EMPTY_RECORD";
+    this.code = code;
     if (recordIndex !== void 0) this.recordIndex = recordIndex;
+    if (characterIndex !== void 0) this.characterIndex = characterIndex;
   }
 };
-function toBytes(input) {
+var MAX_SINGLE_BYTE_CODE_POINT = 255;
+function toBytes(input, recordIndex) {
   if (typeof input !== "string") return input;
   const out = new Uint8Array(input.length);
-  for (let i = 0; i < input.length; i += 1) out[i] = input.charCodeAt(i) & 255;
+  for (let i = 0; i < input.length; i += 1) {
+    const code = input.charCodeAt(i);
+    if (code > MAX_SINGLE_BYTE_CODE_POINT) {
+      throw new AstmFrameEncodeError(
+        "A record contains a character above U+00FF, which is not a single byte. Encode the record with the character encoding your instrument uses and pass the bytes.",
+        recordIndex,
+        "ASTM_FRAME_UNENCODABLE_CHARACTER",
+        i
+      );
+    }
+    out[i] = code;
+  }
   return out;
 }
 function encodeFrame(text, frameNumber, isFinal) {
@@ -2254,7 +2273,7 @@ function composeAstmFrames(records, options = {}) {
   const out = [];
   let frameNumber = options.startFrameNumber ?? FIRST_FRAME_NUMBER;
   records.forEach((record, recordIndex) => {
-    const bytes = toBytes(record);
+    const bytes = toBytes(record, recordIndex);
     if (bytes.length === 0) {
       throw new AstmFrameEncodeError(
         "Cannot frame an empty record: a frame must carry record bytes.",
@@ -2296,7 +2315,7 @@ function concatBytes3(chunks) {
 }
 
 // src/index.ts
-var VERSION = "0.0.9";
+var VERSION = "0.0.11";
 
 exports.ALL_ASTM_WARNING_CODES = ALL_ASTM_WARNING_CODES;
 exports.AMBIGUOUS_CODES = AMBIGUOUS_CODES;