@cosyte/dicom 0.0.17 → 0.0.19

package/CHANGELOG.md

@@ -1,5 +1,317 @@
 # Changelog
 
+## 0.0.19
+
+### Patch Changes
+
+- 58c9f2e: Stop the PHI gate constructing any `RegExp` at all, and stop its override log honouring a template
+  line as a live bypass. Both were disclosed by the previous slice and turn out to be one mechanism.
+  V8 keeps the last successful match on the `RegExp` constructor, so a matched subject stays readable
+  from `RegExp.input` and `RegExp.lastMatch` anywhere in the process. The scan route stopped handing
+  target bytes to a pattern last time; the gate's own configuration did not, and that residual was
+  disclosed as a number rather than a description, every clean column of the instrument reading
+  `input 3772`. Re-measured before building to it: that is `scripts/phi-allow-list.txt` in UTF-16 code
+  units, not the 3,774 bytes the file is on disk.
+
+  A runtime census of every pattern the script runs found five live sites, all four config parsers,
+  and `loadOverrideLog` held two of them. On the route that reads the override log the retained
+  subject IS the fence-blind template line, held verbatim, which is what made this one slice and not
+  two. All five are now forward scanners, so the carve-out sentence that was refused in three wordings
+  is deleted rather than worded again. A scrub was available and refused once more: a bound that holds
+  only from where a cleanup is called is not a bound.
+
+  `overrideLogPaths` is the one parser deliberately narrower than the pattern it replaces, in the
+  fail-closed direction, because a dropped entry makes `--allow-fixture` refuse the run at exit 2
+  rather than exempt the target.
+  It is fence-aware, so the committed `### <path>` template inside the "Format" block is no longer an
+  allow entry, and an all-whitespace heading no longer registers a lone space as a path. The template
+  entry's inertness was re-measured rather than inherited and holds twice over: neither gating route
+  can produce a target that normalizes to a root-level path, and it was live only where a caller names
+  the file itself.
+
+  Everything else is equivalence, measured: 0 differing bytes across 34 cells and 9,283 hit lines
+  against a mutation control differing in 17, and 5 of 6 config routes byte-identical with the sixth
+  required to differ. Developer-facing only, no public API change.
+
+- 7754a6c: Make the PHI gate's hit excerpt own its bytes, so a hit cannot retain the payload it was cut from.
+  `scripts/phi-scan.ts` bounded what a hit line prints, but not what a hit holds: V8 answers
+  `raw.slice(0, n)`, and a regexp match, with a string that points into its parent, so a 19 character
+  name matched inside an 8 MiB page kept that whole page alive for the rest of the run, and `hits`
+  lives for the whole run. Measured against `88be779`, retained memory grew by one whole file for
+  every hit-bearing file scanned and now does not grow at all: 178 MiB against 34 MiB over a 160 MiB
+  corpus, with a hit-free control of the same size reading 26 MiB on both.
+
+  The copy is inside the one factory that makes a hit value, so the bound is on the slot rather than
+  on a caller, and it is on both branches: a value already under the excerpt bound is a pointer too,
+  which is the route the text sweep's name recognizer takes. The round trip is `utf16le` rather than
+  `utf8` because `utf8` turns an unpaired surrogate into a replacement character, and a report naming
+  a value the file does not carry is the same class of wrong answer as a report printing too much.
+
+  The cost direction is stated rather than claimed away, and it is narrow: where one page is nothing
+  but violating tokens, a resident copy per hit replaces a resident pointer per hit, and the two are
+  within a few MiB of each other in either direction depending on token length.
+
+  Nothing printed changes: 42 of 42 grid cells across six routes, five value lengths and three cap
+  settings are byte-identical on exit code, stdout and stderr.
+  `pnpm measure:phi-scan-retention` re-derives both tables and the grid against any base.
+  Developer-facing only, no public API change.
+
+- 01d0983: Stop the PHI gate's scan route handing a target's bytes to a regular expression. The tag and text
+  sweeps now leave nothing in V8's legacy `RegExp` statics, where the whole of a scanned page and an
+  unexcerpted matched name were readable after a scan. Reported hits are unchanged, byte for byte.
+- 7fbc8e9: Spend the PHI gate's per-file print budget per recognizer, so one recognizer's flood cannot spend
+  another's share. `scripts/phi-scan.ts` printed the first `n` of a file's hits in scan order, and a
+  file's routes do not queue independently: the tag walk appends before the text sweep, so how many of
+  the text sweep's findings a reader saw was decided by how many the tag walk had already made. The
+  text sweep's PN recognizer is the only route that can see a caret-joined name in bytes the tag table
+  never typed, and it is the one at the end of the queue. Nothing was ever mis-counted, and this was
+  not a false green: the exit code, the total, the set of files named and the withheld count are
+  computed off the hits and were right throughout. What moved was which lines a reader saw.
+
+  The budget is now spent per entry of a closed recognizer table, per file, so whether a hit prints
+  depends on its own recognizer and its index among that file's hits from that recognizer, and on
+  nothing another recognizer found. The table is a type rather than an analysis of today's push sites:
+  budgeting on the printed reason string would have let one future recognizer interpolate a
+  payload-derived token into it and hand the payload a vote on how many classes exist. The `hits`
+  array is still deliberately not capped, because a cap there would make the totals a claim about what
+  was kept rather than about what was found.
+
+  Because a hit among a file's first `n` overall is among its own recognizer's first `n`, the printed
+  set is a superset of the previous scanner's at every cap: 72 grid cells over nine corpus shapes and
+  eight caps, zero cells where a line the old scanner printed goes unprinted, 29 cells printing
+  strictly more, and 23 cells where the old scanner printed nothing at all from a recognizer that had
+  hits. With the two scanners swapped the same grid reports 332 violations, so the zero is a
+  measurement rather than a blind check. `pnpm measure:phi-scan-monotonicity` re-derives it against
+  any base.
+
+  This is not monotonicity and is not described as it: under any finite budget `n+1` hits from one
+  entry print `n`, so adding a hit can always remove a line. Several sweeps also share an entry, which
+  costs more than the same sweep twice: the two text date passes share one, so over 200 ISO dates
+  followed by 200 compact ones the report prints 20 ISO lines and no compact one, and the compact pass
+  is the only route that sees a bare eight-digit date. That is measured, identical on the previous
+  scanner, unchanged here, and pinned by a test rather than argued away.
+
+  The cost is stated rather than claimed away: a file's report can be longer, bounded by the cap times
+  the closed table instead of by the cap, measured at 20 lines against 61 on the loudest grid shape at
+  the default budget. `--max-hit-lines 0` still prints every line. Developer-facing only, no public API
+  change.
+
+- 88be779: Bound the value a PHI-gate hit line echoes. `scripts/phi-scan.ts` printed the whole violating value
+  on one stderr line, and how large that was is chosen by the payload: an element declares its own
+  length, so a `(0010,0010)` claiming the rest of the object put the rest of the object on one line
+  to report that a name was not on the allow-list. It was the only unbounded payload-derived field on
+  the line; the others are a path, a rendered tag, an offset and a literal reason.
+
+  A hit now carries an excerpt of at most 194, bounded where the hit is made rather than where it is
+  printed: `Hit.value` is no longer a `string`, and one factory is the only way to make one. 194 is
+  PS3.5 2026c Table 6.2-1's PN arithmetic (three component groups of 64, two delimiters), but that
+  table measures in characters and this scanner measures a latin1 decode, so 194 bounds what the
+  report prints and says nothing about what the standard admits: a conformant 194-character PN under
+  `ISO_IR 192` is cut here, and a test pins it. Every cut states exactly how much was withheld,
+  outside the quotes.
+
+  Detection is unchanged: the exit code, the totals and the set of files named come from the hits and
+  not from what was printed. Developer-facing only, no public API change.
+
+## 0.0.18
+
+### Patch Changes
+
+- 21d42f5: Scan a file by its CONTENT in the repository's PHI gate (`scripts/phi-scan.ts`), which is developer
+  tooling and not part of the published surface.
+
+  `scanTarget` branched on the file's extension before it read a byte: `.json`, `.txt`, `.md` and
+  `.csv` ran the text sweep and the base64 decode and returned, so the DICOM-aware sweep never ran on
+  one whatever its bytes were. A file whose raw bytes are a Part 10 object was therefore never scanned
+  as one if it happened to be named `.md`, which a de-identification report, a bug repro or a fixture
+  saved under the wrong extension plausibly is.
+
+  What that cost is narrower than "a second opinion", and the fixture is built to show it: only the
+  tag table can see a **single-component** `(0010,0010)`, because that shape carries no caret for the
+  text sweep's person-name pass to match. Measured on `08ed3ee` over one object, the same bytes exited
+  1 as `.dcm`, `.bin` and `.dat`, and 0 with `OK - no hits` as `.md`, `.txt`, `.json` and `.csv`.
+  Preamble-less, the same.
+
+  The remedy is the DELETION of the branch, and that is what makes it safe rather than a net leak.
+  `scanDicom` gives up quietly at an undefined-length Sequence (PS3.5 2026c section 7.5.2 defines
+  `0xFFFFFFFF` as one of two Sequence delimitations, both of which decoders shall support, and section
+  7.1 orders tags ascending, so `(0008,1110)` precedes `(0010,0010)` in a conformant file), so routing
+  a file to it INSTEAD of the text sweep has previously taken a name from exit 1 to exit 0. The removed
+  branch's two calls were the text sweep and the base64 decode; the branch that replaces it makes the
+  same two unconditionally and adds one conditional DICOM sweep. Hits are only ever appended, so the
+  hit set, the totals and the exit code are a strict superset on every input.
+
+  That superset was checked mechanically over 11 objects times 7 extensions, 77 cells: 65 identical,
+  12 strictly more reported, 0 cells that lost an exit code or a reported value. Every fixture was
+  verified through this package's own parser before any zero it produced was believed. The scan of the
+  repository's real corpus is byte-identical, exit code and output alike.
+
+  Two residuals are unchanged and disclosed rather than closed: the DICOM sweep still reports nothing
+  about the bytes it never read, and the hit array is still unbounded in memory. The matrix, the
+  superset check and the reason the first of those is a separate slice are in
+  `documentation/agent-notes/dicom-phi-scan-name-dispatch.md`.
+
+- 028fe85: Close a silent halt in the repository's PHI gate (`scripts/phi-scan.ts`), which is developer
+  tooling and not part of the published surface.
+
+  `scanDicom` gives up quietly at the first Data Element header it cannot read, and an
+  undefined-length Sequence is one of those (PS3.5 2026c section 7.5.2 defines it as one of two
+  delimitations, both of which decoders shall support). The text sweep behind it ran only when
+  `isDicom` answered false, so a **preamble-ful** Part 10 object had nothing behind the halt: the
+  identical dataset was reported without a preamble and missed with one, and the gate printed
+  `OK - no hits` over a name-bearing `(0010,0010)`. The preamble is not part of the dataset, so it
+  cannot decide who is owed a text sweep.
+
+  The text sweep now runs on every binary target, and `scanEmbeddedObjects` runs it on the object it
+  decodes as well, where the base64 hides the name from the enclosing page. Both are additions beside
+  `scanDicom`, never replacements for it.
+
+  The accepted cost is false positives over binary values. Which recognizer produces them, and how
+  many, is a property of the payload's byte histogram rather than of the scanner: over 8 MiB of
+  synthetic pixel data the measured count runs from zero to five figures. The table, and the residuals
+  left open, are in `documentation/agent-notes/dicom-scandicom-silent-halt.md`.
+
+  Also closes a pre-existing refusal on the same route: the base64 run matcher was a greedy regular
+  expression, and one multi-megabyte run overflowed V8's backtrack stack, exiting 2 rather than
+  scanning. It is a forward scan now, yielding the same runs.
+
+- 99da18f: Make the repository's PHI gate (`scripts/phi-scan.ts`) report the bytes its DICOM sweep never read.
+  This is developer tooling and not part of the published surface.
+
+  `scanDicom` stops at the first Data Element header it cannot read and said nothing about what came
+  after it, so a run printed `OK - no hits` over a file whose tag table it had abandoned partway
+  through. The way in is conformant: PS3.5 2026c section 7.5.2 defines an undefined-length Sequence as
+  one of two delimitations, both of which decoders shall support, and section 7.1 orders tags
+  ascending, so `(0008,1110)` precedes `(0010,0010)`. Encapsulated pixel data has the same shape.
+
+  Each halt is now recorded per file and printed as a `PARTIAL:` line, and the word `OK` is withdrawn
+  from the clean line when any file has one. The line carries a path, two counts and a reason from a
+  closed table: no tag, no VR, no value and no byte of the object, because the bytes at a halt are
+  exactly the bytes that did not read as a header. The tally is bounded by the number of files rather
+  than by anything an object can choose, so it needs no cap and cannot be pushed off the report by a
+  file loud enough to bury its own hits.
+
+  The exit code deliberately does not move. Nothing was found, nothing refused the scan, the same
+  disclosure would fire on input section 7.5.2 makes legal, and folding it into exit 2 would mask a
+  real hit whenever both were present. A CI job that reads only the exit code therefore still cannot
+  see this, which is disclosed rather than claimed away.
+
+  Proved over 13 objects x 5 carriers x 3 cap settings: exit code and hit lines identical in all 195
+  cells, 60 outputs identical, 105 strictly larger, 30 differing only by the withdrawn `OK` line, 0
+  violations. The generator ships as `scripts/measure-phi-scan-unread.ts` so the table can be
+  re-derived. What the disclosure does and does not carry, and the residuals left open, are in
+  `documentation/agent-notes/dicom-phi-scan-unread-tail.md`.
+
+- 783f667: Let the repository's PHI gate (`scripts/phi-scan.ts`) finish writing its report before it exits.
+  This is developer tooling and not part of the published surface.
+
+  The script ended `process.exit(run())`. `process.exit()` tears the process down without waiting for
+  stdio that libuv has accepted but not yet written, and under every caller that matters this script's
+  stderr is a **pipe**: `spawnSync` in this repo's own suite, and the shell pipeline a CI job runs it
+  in. A pipe write that cannot complete immediately is queued and flushed on a later loop turn, which
+  `process.exit()` never allows, so `report()` returning was not the same as its bytes having left the
+  process.
+
+  **The failure mode is a gate defect, not a cosmetic one.** The exit code is computed from
+  `hits.length` and was always right, so a truncated report is a run that REFUSES while under-naming
+  what it found, and the bytes it drops are the END of the report: the last hit lines and the total,
+  which is the part a reader trusts to say how much there was.
+
+  This was red on `main`. `ci / verify (24, ubuntu-latest)` is a required check and it failed on
+  `08ed3ee` and `21d42f5`, both times on `countHitLines(...) === 200` reading 193, and on `#106`'s PR
+  head reading 191, with `verify (22)` green throughout and the exit code correct in every case.
+
+  The scan is deterministic, so the hit COUNT cannot vary between runs: the fixture is 200 distinct
+  tokens and the recognizers are plain global regexes over the whole file. What varied is what
+  arrived. Measured on `21d42f5` with the reader stalled, 5,000 small stderr writes: `process.exit(1)`
+  delivered 784 lines, about one 64 KiB pipe, on 3 of 3 runs; `process.exitCode = 1` delivered 5,000
+  on 3 of 3. Node 22.23.1 and 24.19.0 behaved identically there, so the version split is scheduling
+  and not stdio semantics. The CI failure itself reproduced on the real script and the real fixture
+  once the reader was made to lose (single-page pipes plus CPU contention): **30 of 60 runs truncated,
+  counts including 190, 191 and 192, exit 1 every time; 0 of 60 with this change.**
+
+  Not calling `process.exit()` has a cost, and it is paid here rather than discovered later: it makes
+  a late stdio error reachable. A write to a pipe whose reader has gone fails with `EPIPE` on a later
+  tick, after `run()` has returned, so `run()`'s try/catch cannot see it and Node's default
+  unhandled-`'error'` path exits 1, the one code that means "PHI was found". Without a listener a
+  clean corpus went 0 to 1 and an invocation error went 2 to 1. Both stdio streams now discard a late
+  error, which is exactly what `process.exit()` did by making it unreachable, so the exit code stays
+  exactly what `run()` returned. **The residual, disclosed and not closed: with a reader that never
+  drains at all this script now waits instead of exiting, where `process.exit()` ended it by dropping
+  the report.** That is introduced, on `node` and under `tsx` alike. A timeout would re-introduce the
+  defect this closes, so it is disclosed rather than answered.
+
+  It changes what ARRIVES, never what is WRITTEN. Over a draining reader, base and fixed emit
+  byte-identical stdout, stderr and exit code across 12 cases spanning all three of the contract's
+  exit codes, the per-file print cap at its default and at 0, 1 and 3, the net-leak control and both
+  cells of the finding that `report()` is not monotone in `hits` at the cap. Nothing about the cap or
+  its non-monotonicity is changed or claimed away here; the printed set simply arrives.
+
+  Still open and NOT fixed here: `test/helpers/run-script.ts` inherits `spawnSync`'s 1 MiB
+  `maxBuffer`, which truncates at roughly 15,400 lines in both exit modes. That is the parent dropping
+  bytes it was handed rather than the child failing to hand them over, and it is a different defect.
+
+  Detail, controls and the superset check: `documentation/agent-notes/dicom-phi-scan-exit-flush.md`.
+
+- b784c38: Name the frame a `DicomParseError`'s byte offset is counted in, and correct the universal that was
+  defending the bound around it.
+
+  `err.byteOffset` was the sole locator on the two Tier-3 messages that lost their remaining-bytes
+  count, and it is slice-relative inside a defined-length Sequence Item: on one fixture the same
+  defect reports `0`, `24` or `40` depending on where inside the Item the offending element sits, while
+  that Item's slice begins at absolute offset 202. Nothing on the class said which coordinate system
+  the number was in, and the wrong reading is not detectable by a consumer. It is a valid index into a
+  buffer that returns a real element, just not the one the diagnostic named. On a fixture engineered
+  for the earlier snippet fix, reading the item-relative number against the file lands inside a planted
+  `"MR BRAIN SMITHSON "`.
+
+  **`DicomParseError.offsetFrame` is new and required**, drawn from the new `OFFSET_FRAMES` export
+  (`"input"`, `"inflated-dataset"`, `"value-slice"`). **Only `"input"` means the offset indexes the
+  buffer you passed in.** The frame is in the `Error.message` suffix as well as on the class, so that
+  suffix reads `(offset=N frame=F)` on every Tier-3 fatal and every `{ strict: true }` escalation: a
+  string match on it stops matching. `err.code` is unchanged and which files throw is unchanged.
+
+  The frame's **name** is published and its **origin** is not. Where a slice begins is the sum of the
+  declared lengths that reached it, so an error naming its own frame's origin would hand back by
+  subtraction the wire field these same messages withhold. Internally the frame's bytes and its name
+  travel as one object, so a frame change is one assignment and no call site can move the bytes and
+  leave the label behind. It does not make a deliberately mismatched pair impossible, and is not
+  described as doing so.
+
+  **The universal used to justify withholding a declared length, in the README, two package docs and
+  two places in the parser, was false and is corrected rather than reworded a second time.** A fatal "about a length field that lies" does not fire
+  only when a length field is lying: a spec-clean object cut short by two bytes raises
+  `ELEMENT_LENGTH_EXCEEDS_BUFFER` with every declared length honest, and one cut short inside its File
+  Meta group raises `FILE_META_GROUP_LENGTH_OVERRUNS` with `(0002,0000)` untouched. Both are measured.
+  The bound is right in either reading because the withheld number is four bytes a sender wrote either
+  way; the universal was never what made it right.
+
+  Also replaces an arity pin that did not pin what it claimed. It read
+  `Function.prototype.length`, which stops counting at the first defaulted parameter, so a factory that
+  had grown exactly the slot the pin refuses would still have read `2` and stayed green.
+
+  `UNSUPPORTED_TRANSFER_SYNTAX` keeps the behaviour every released version has: its `snippet` slot
+  carries PS3.6's own name for the unsupported UID rather than raw bytes, so it is the one fatal whose
+  snippet is not a cut of the frame it names. `DicomParseWarning.position` and `Element.byteOffset`
+  still carry no frame, and `err.snippet` is still 16 raw source bytes. All three are pre-existing, disclosed, and not closed here. The measurements,
+  the residuals, and every clean result with the positive it is pinned beside are in
+  `documentation/agent-notes/dicom-diagnostic-locator-frame.md`.
+
+- 08ed3ee: Cap the repository's PHI gate (`scripts/phi-scan.ts`) at a per-file number of printed hit lines,
+  which is developer tooling and not part of the published surface.
+
+  `report()` wrote one stderr line per hit with no bound, and the preceding slice widened what reaches
+  it: the text sweep now runs over every Part 10 object's bytes, and its recognizers fire on image
+  noise at a rate that is a property of the payload's byte histogram. Measured on `b784c38` over 8 MiB
+  of synthetic pixel data uniform over `0x41-0x60`, one run wrote 71,447 hits as 6,037,715 bytes of
+  stderr.
+
+  The cap is on **printing only**. The exit code and the reported totals are derived from the hits and
+  not from what was printed, so a withheld line cannot turn a refusal into a clean run. It is applied
+  **per file**, never globally, so a loud file cannot push a later file's hits off the report. A file
+  with lines withheld says so, with the exact remainder. `--max-hit-lines <n>` sets the bound and
+  `--max-hit-lines 0` prints every line, reproducing the previous output byte for byte.
+
 ## 0.0.17
 
 ### Patch Changes

package/README.md

@@ -338,7 +338,9 @@ The library throws five typed errors, all exported from the package barrel. Warn
 
 ### `DicomParseError`
 
-Thrown by `parseDicom` on one of the 4 Tier-3 fatal codes. Carries the byte position, a registry-composed message, and a 16-byte hex `snippet` of the source. **The message is looked up in a frozen registry and its factories take no tag, no wire-length parameter and no count of the bytes left in the buffer**, so none of them can be interpolated: a fatal about a length field that lies fires precisely when bytes inside somebody's value are being read as a header, which is what used to make the tag and the wire length document content. The last of the three went in this release and **changes two message texts**, so a string match on them stops matching: a defined-length Sequence Item is parsed from a slice, so inside one the remaining-bytes count is that Item's own declared length less an offset the same message publishes. `err.code` is unchanged and `err.byteOffset` locates the element instead. The snippet is raw input and is cut in the frame that offset is counted in: treat it as PHI and redact it at your own boundary.
+Thrown by `parseDicom` on one of the 4 Tier-3 fatal codes. Carries the byte position, the **frame** that position is counted in, a registry-composed message, and a 16-byte hex `snippet` of the source. **The message is looked up in a frozen registry and its factories take no tag, no wire-length parameter and no count of the bytes left in the buffer**, so none of them can be interpolated. The reason is not that such a fatal fires only on a lying length field: it fires on an honestly truncated file too, where nothing is fabricated and the transport simply lost bytes. Measured on a spec-clean object cut short by two bytes, `ELEMENT_LENGTH_EXCEEDS_BUFFER` raises with every declared length in the file honest. The bound covers both because the withheld numbers are four bytes a sender wrote either way, and because on the desynchronized reading those four bytes are somebody's name: an under-declaring `ST` carrier holding `"MR BRAIN SMITHSON "` once rendered `declared length=1330858068`, which is `"THSO"`.
+
+`err.byteOffset` locates the element, and **`err.offsetFrame` says which coordinate system that number is counted in** (`OFFSET_FRAMES`: `"input"`, `"inflated-dataset"`, `"value-slice"`). Only in `"input"` is it an index into the buffer you passed in. A defined-length Sequence Item is parsed from a slice, so an offset raised inside one counts from that Item, and the same file reports `0`, `24` or `40` for the same defect depending on where in the Item it sits. **Where a slice begins is deliberately not published**, because the distance between two frames is a declared Value Length off the wire. The frame is in the `Error.message` suffix too, which now reads `(offset=N frame=F)`, so a string match on that suffix stops matching. `err.code` is unchanged. `DicomParseWarning.position` still carries no frame beyond its `deflated` flag, and `Element.byteOffset` carries none at all: both are pre-existing and neither is closed here. On every fatal but one the snippet is raw input, cut in the frame that offset is counted in: treat it as PHI and redact it at your own boundary. The exception is `UNSUPPORTED_TRANSFER_SYNTAX`, where the slot carries PS3.6's own name for the unsupported UID (`"RLE Lossless"`) when the registry publishes one, and 16 raw bytes only when it does not. That is deliberate, it predates the frame, and it is named here rather than left to a universal that would be false on the first code a compressed object reaches.
 
 ```ts
 import { parseDicom, DicomParseError, FATAL_CODES } from "@cosyte/dicom";