@cosyte/dicom 0.0.8 → 0.0.10

package/CHANGELOG.md

@@ -6,6 +6,241 @@ All notable changes to `@cosyte/dicom` will be documented in this file. The form
 
 ### Fixed
 
+- **🩺 A private value crossed the PS3.5 §7.8.1 reservation boundary and was retained under
+  `RetainSafePrivate`, in output stamped `PatientIdentityRemoved = YES`.** The **absorb** direction
+  is closed; the **eject** direction and the private-`SQ` carve-out are measured, pinned and **not**
+  closed (see below)
+  (`DICOM-PRIVATE-CREATOR-RESERVATION-LEAK`). **`PRE-EXISTING` and live on the published `0.0.9`**;
+  found by `#51`'s pass-6 `conformance-refuter` and measured **identical on `origin/main`**, so it
+  was never introduced by that branch.
+
+  **The shape.** An `(FFFE,E000)` Item that declares more bytes than its enclosing `SQ` element's
+  Value Length allows reads past the end of its sequence and absorbs the element that followed it.
+  With a Private Creator in the Item's genuine content and the private data element written at the
+  **root** - where no creator reserved its block, so the Basic Profile removes it - the absorbed
+  element lands beside that creator and is kept verbatim. Measured on `164eb39`, Explicit VR LE and
+  BE, `deidentify(ds, { retain: ["RetainSafePrivate"], profile: profiles.ge })`:
+  `report.removedPrivateTags` is `[]`, `SECRET-PRIVATE-PHI` is in the serialized output,
+  `(0012,0062)` is `YES`, and **the file parses with no warning on either channel**. A stamp that
+  outruns the redaction is the worst shape this class has, because a consumer trusting it has no
+  signal at all.
+
+  **The remedy is at the de-identify boundary and it removes; it does not downgrade the stamp.**
+  PS3.5 2026c §7.8.1, read from the vendored pin (`4dfd7b8c…`), one occurrence in the document:
+  "Items within a sequence are self contained Data Sets ..., any Item in the sequence that contains
+  Private Data Elements shall also have Private Creator Data Element reserving a block of Elements
+  for those Private Data Elements. The scope of the reservation is just within the Item. Items do
+  not inherit the Private Data Element reservations made by Private Creator Data Elements in the
+  Data Set in which the Item is nested." The reservation's scope **is** the Item's boundary, so a
+  file that contradicts itself about where the Item ends does not determine which reservation
+  covers an element. PS3.15 2026c §E.3.10 "Retain Safe Private Option": "Private Attributes that
+  are known by the de-identifier to be safe from identity leakage shall be retained, together with
+  the Private Creator IDs that are required to fully define the retained Private Attributes; all
+  other Private Attributes shall be removed **or processed in the element-specific manner recommended by Deidentification Action (0008,0307), if present within Private Data Element Characteristics Sequence (0008,0300)**". **The clause licenses two dispositions and the
+  first draft of this entry closed the quote at "removed"**, reading a permissive clause as an
+  absolute; this library does not implement `(0008,0307)`, so removal is the branch available to it.
+  Nothing there is _known_, so the standard's own
+  default applies. Every private element **the recursion reaches** in such an Item, and at every
+  depth below it, is removed and named in `report.removedPrivateTags` - **with one carve-out, stated
+  in the sentence rather than only in a later paragraph**: a private `SQ` the profile vouches for is
+  settled before the descent and is kept verbatim (see below).
+  **The unit is the SEQUENCE, not the individual Item**: `descendSequence` decides once and
+  applies it to every item, so a two-item sequence where only item 1 over-runs also loses a genuine
+  block in the honest item 0 (measured and pinned; the grid sweeps single-item fixtures, so that
+  extra cost is NOT inside the priced 20).
+  **The unit is the SEQUENCE, not the individual Item**: `descendSequence` decides once and
+  applies it to every item, so a two-item sequence where only item 1 over-runs also loses a genuine
+  block in the honest item 0 (measured and pinned; the grid sweeps single-item fixtures, so that
+  extra cost is NOT inside the priced 20).
+
+  **It is NOT a parser bound, and no reading changes.** An over-declaring Item and an
+  under-declaring Sequence are byte-identical - `#51` reached that conclusion three times and its
+  refuter could build no predicate separating them. `scripts/measure-sq-bound-grid.ts` over
+  **83,037 cells** against `164eb39`: **0 cells differ in any PARSE respect**, **0 cells whose
+  READING differs** (a new counter; the existing parse-respect count folds both warning channels
+  in), 0 new lenient fatals, 0 new strict fatals, 0 markers lost or gained from the parsed object,
+  0 wrong root `(0010,0020)`, 0 reports that lose an attribute, and **0 Implicit VR LE cells
+  changed** - a free control, because that path slices the item stream so no Item can over-run.
+
+  **The fail-safe-direction argument `#51` made in five artifacts is RETRACTED, not reworded.** The
+  direction is not a property of the two readings; it is a property of **where the sender put the
+  Private Creator relative to the disputed bytes**. Both **absorb** placements are pinned by tests
+  and both are refused now: the creator inside the Item with the data element absorbed in, and the
+  data element inside the Item with the creator absorbed in. The **eject** placement is pinned as a
+  residual and is not refused.
+
+  **The price, measured rather than asserted.** The grid gained a `priv|` family - the first
+  population in that harness that runs `RetainSafePrivate` at all, which is why three earlier
+  refuter passes read "0 PHI regressions" off it while this was live. **58 -> 0** cells keep a
+  private value inside an Item on a self-contradicting file, and **20 of those 58 were not leaking
+  anything**: the creator and the data element were both genuine Item content and the reservation
+  was real. They pay for the guarantee, because nothing on the wire distinguishes them. Retention on
+  a file whose length fields agree is **unchanged** - 6 -> 6 inside an Item, **9 -> 9** at the root -
+  and 0 -> 0 rows are kept with no creator in scope, on either tree.
+
+  **🩺 Two routes are NOT closed, found by the pass-1 `conformance-refuter` grade.** Both are
+  `PRE-EXISTING`, reproduce identically on `164eb39`, and both produce the same false attestation
+  (`removedPrivateTags: []`, the value in the output, `(0012,0062) = YES`).
+  **(a) The eject direction.** An Item that _under_-declares pushes its trailing elements **out**
+  into the enclosing Data Set, which is not narrowed, so a Private Creator that lands there reserves
+  a block for elements the sender never gave it. **It is NOT root-specific and an earlier draft of
+  this entry said it was** - pass 2 reproduced it one level down, where an inner sequence ejects a
+  creator into the still-usable enclosing Item. The item filed for this must be scoped to every
+  still-usable Data Set or it will be built to the wrong bar. Silent on every channel
+  and under `{ strict: true }`, Explicit VR LE and BE. **22 grid cells.** The widening was **built
+  and measured**: narrowing the flag whenever a Data Set _contains_ an over-running sequence costs
+  **24** root retentions and closes **2** of the 22, because the other 20 are Implicit VR LE where
+  the sequence records no over-run at all. A different mechanism, and its own item.
+  **(b) The private-`SQ` carve-out.** `keepsPrivate` decides **before** the descent, so a private
+  `SQ` the profile vouches for is kept **verbatim** and this rule is never consulted inside it. The
+  first draft of this entry said "every private element in such an Item ... is removed",
+  unconditionally, which is `#54`'s exact refusal repeated; the sentence is corrected rather than the
+  guard widened. Both routes are pinned as **residual tests that assert the leaking behaviour**, so
+  they go red when fixed.
+
+  **No new public surface**: no new warning code, no new report field, no snapshot change. The
+  action is already visible on `report.removedPrivateTags`, the channel this defect read `[]` on.
+  What is _not_ on any channel is the reason - a caller sees which private tags went, not that the
+  file contradicted itself. Disclosed rather than closed, because a new Tier-2 code would also have
+  to answer to `profiles.strict` escalation and to the per-element amplification bound `#48`
+  established.
+
+- **An unrecognized Explicit VR was read short-form, contrary to PS3.5 2026c §6.2's normative
+  "shall"** (`DICOM-UNRECOGNIZED-VR-SHORT-FORM`). It is now read - and written - long-form. **This
+  is a behaviour change on the read path**, and the trade is measured below rather than asserted.
+
+  **The clause, read from the vendored `vendor/nema/part05` pin (`4dfd7b8c…`), one occurrence in the
+  document.** §6.2 "Value Representation (VR)": "All new VRs defined in future versions of DICOM
+  **shall** be of the same Data Element Structure as defined in [§7.1.2] with reserved bytes after
+  the VR and a 32-bit unsigned integer VL (i.e., following the format for VRs such as OB or UT), and
+  may or may not permit Undefined Length." §6.2's **note** about an implementation choosing to
+  ignore unrecognized VRs is informative, and is deliberately not what this rests on.
+
+  **What a conformant future-VR file used to do: no sentence, deliberately.** Four attempts have
+  now been made to summarize that in one sentence and all four were wrong, the fourth by the first
+  draft of this entry. It is **shape-specific**: the short-form read took the carrier's length from
+  the two bytes §7.1.2 reserves (`0x0000`), got a zero-length value, and resumed inside the 32-bit
+  VL field - so what happened next depended on the payload's own bytes. Sometimes a whole-object
+  `INVALID_FILE_META`; sometimes a **clean parse into a tree the sender did not write**, with a
+  zero-length carrier plus an element manufactured out of the payload and, under Explicit VR BE,
+  **no warning on either channel**. `scripts/measure-unrecognized-vr.ts` is new here and prints the
+  per-shape table for both trees, including that case (`long-payload-tiles`). **Add a shape rather
+  than writing a summary.**
+
+  **What changed, and what deliberately did not.** `readExplicitElementHeader` and the File Meta
+  element reader take the 12-byte header for any VR outside the 34 this edition defines;
+  `serializeDicom` writes one. Everything after the header read is untouched - the same value read,
+  the same "declared length exceeds the remaining buffer" refusal, and the same undefined-length
+  refusal that `OB` / `UT` / `UN` already take. **This is one more VR class routed into an existing
+  bound, not a new bound**, which is the distinction four refusals in this family were about. No new
+  warning code was minted: a §6.2-conformant file is not something to warn about, and a new Tier-2
+  code would throw under `{ strict: true }` on exactly such a file.
+
+  **The reader and the writer had to move in one commit.** The short form's length field is 16 bits.
+  With the reader fixed alone, a 70,000-byte unrecognized-VR value would have been re-emitted
+  declaring **4,464**, silently. Pinned by a round-trip test at that size.
+
+  **The trade, on `scripts/measure-sq-bound-grid.ts` against `66f0c95` (76,611 cells).**
+  **1,221 recovered** (fatal before, parse now) against **932 newly refused**; 0 PHI regressions;
+  0 cells where the root `(0010,0020)` changes value; 0 reports that lose an attribute on a cell
+  that still parses; **0 Implicit VR LE cells changed at all** (a free control - there is no on-wire
+  VR there); leaking cells unmoved at 11 (`DICOM-BINARY-CARRIER-OVERDECLARE`, untouched here).
+  **All 932** of the newly refused had an unrecognized VR in their `66f0c95` parse tree, so every
+  file this refuses is one the old reader only "read" by manufacturing a Data Element header out of
+  the middle of somebody's value; the fabricated VRs across that population include `"CT"` - a
+  Modality value read as a VR.
+  The grid also reads **0 cells that parse on both trees and read differently**. Read that as a
+  statement about the grid's fixtures, **not** as "nothing is silently re-read": the
+  `long-payload-tiles` shape is exactly such a cell and it is outside anything the grid sweeps.
+  **The mirror shape is the honest cost**: a sender that ignores §6.2 and writes an unrecognized VR
+  in the 8-byte short form produced a readable object before and is refused now. That is pinned by a
+  test rather than left to be discovered.
+
+  **`deidentify()` is unchanged, and one of its claims was not.** The rule that empties an element
+  whose VR is not one of the 34 still fires - reading a header is not the same as knowing what its
+  value means, and Table E.1-1 acts per attribute. **Disclosed, not fixed**: it is the same
+  over-redaction trade the un-auditable-sequence rule makes, and re-deciding it is a product call.
+
+  **`#55` published "on a file conformant to PS3.5 2026c the cost is zero". It was never true**, and
+  no replacement account of the old reader belongs here: three were written while this entry was
+  drafted and the gate refuted all three. The harness prints what each shape does on each tree, and
+  the shape that refutes the shortest version of the story is a row in it
+  (`long-payload-tiles-future-vr`). **Add a shape rather than a sentence.**
+
+  **`DICOM_NONZERO_RESERVED_BYTES` no longer names a tag, and that is a PHI fix this slice owed.**
+  Reading an unrecognized VR long-form means landing on the two bytes §7.1.2 reserves, so this code
+  is newly reachable on a **fabricated** header - one built out of the middle of somebody's value.
+  Its message interpolated the four bytes it would call a tag: measured on an `ST` carrier holding
+  `"MR BRAIN  SMITHSON"` under-declared by 6, it streamed `Element (54495348) …` - the bytes
+  `"ITHS"` in wire order, four letters of the surname - where `66f0c95` emitted nothing naming that
+  element on the same file (its only warning there is `DICOM_ODD_LENGTH_VALUE_PADDED`, on a genuine
+  `(0010,0010)`). (Quote the payload **with** its tag: `"MR BRAIN SMITHSON "` produces `48544F53`
+  instead, and an earlier draft of this entry paired the two wrongly.) **The factory now takes no
+  tag parameter**, so the bound is the signature rather than a branch, and `position.byteOffset`
+  locates the element. This is exactly the remedy `#55` paid a blocker for in
+  `report.undefinedVrElements[].tag`, and the reason is the same: where the trigger is "these bytes
+  are not what they claim to be", `renderTag` checks shape and cannot refuse them. It also closes a
+  `PRE-EXISTING` instance on the same factory - a long-form VR fabricated the same way rendered its
+  tag on `66f0c95` too. Pinned with a name-bearing fixture and a non-vacuity assertion that the code
+  really fires.
+
+  **A second `DeidentifyReport` claim corrected rather than widened.** The report was described as
+  "value-free apart from `uidMap`". `removedPrivateTags` is a second exception: its entries are four
+  source bytes each, and on a malformed file those bytes can be document content. Measured
+  identically on **both** trees - an `OB` carrier holding `"SECRET-NOTE-"` followed by a well-formed
+  odd-group header reports `["41534342"]`, whose wire-order bytes read `"SABC"`. Reporting the tag
+  is the whole audit value of the field on a well-formed file, so it is documented as PHI rather
+  than withheld; narrowing it is a product decision that has not been made.
+
+- **A test asserted the machine it ran on rather than the code, and the script suites paid a
+  process start they did not need** (`PARSER-TESTTIMEOUT-ASSERTS-AN-IDLE-BOX`). This is a trim plus
+  two deletions and nothing else. Test and test-configuration only: no source, parser, de-identify,
+  serializer or public API change, and no number in this package moves.
+
+  **The trim.** Three suites spawned this repository's own TypeScript scripts through `tsx`. Node
+  strips types natively, so that process start bought nothing and was paid on every invocation.
+  They now go through `test/helpers/run-script.ts`, which runs them under `node`, measured with
+  byte-identical stdout and the same exit code from both runners. The suites are measurably faster;
+  no ratio is quoted here, because independent re-takes on one box do not agree with each other and
+  the gap moves with the workload as well as the box.
+
+  **The one script that cannot make that move, pinned rather than asserted.**
+  `scripts/generate-annex-e.ts` imports `"../src/dictionary/repeating-groups.js"`, and Node's ESM
+  resolver will not rewrite that `.js` specifier onto the `.ts` file; under `node` it exits 1 with
+  `ERR_MODULE_NOT_FOUND` at module resolution, before it opens a byte of DocBook or touches the
+  tracked `SHA.txt` this suite otherwise rewrites. It stays on `tsx`, and a new test asserts that
+  failure **by name**, so the carve-out goes red the day Node can resolve the specifier rather than
+  quietly outliving its reason. The rule and its single exception are stated once, in
+  `run-script.ts`, rather than copied into each suite: the sibling generator imports nothing from
+  `src/` and was measured green under `node` with byte-identical output, so the carve-out is one
+  script and not "the generators".
+
+  **Deletion one: a stopwatch that was already a false red.** `embedded-attribute.test.ts` carried
+  `expect(performance.now() - started).toBeLessThan(2_000)` over a 512 KiB value. It reads a couple
+  of hundred milliseconds on a quiet box, but on an ordinary loaded one it was measured at or over
+  that literal with the algorithm untouched. It was also the weakest cost claim in this package: its
+  fixture is half a megabyte of `0x41`, which yields zero tiling candidates, so every offset
+  short-circuits before the repertoire scan where the quadratic it named lived and the number it
+  produced was never a measurement of that cost. Deleted, and the test renamed to what it does
+  assert. The cost is still bounded, by `vitest.config.ts`'s per-test
+  timeout, which is the mechanism built for it.
+
+  **Deletion two: `hookTimeout: 10_000`, a verbatim no-op.** Measured rather than read: a `beforeAll`
+  that sleeps past it reds at "Hook timed out in 10000ms" on this repository's own Vitest with no
+  configuration at all, so the line restated the default exactly and said nothing. The one hook that
+  needs more (`docs-content.test.ts`, which shells out to `pnpm build`) already passes its own budget
+  at the call site.
+
+  **`testTimeout: 10_000` is UNCHANGED.**
+
+  **Disclosed residual, not fixed here.** The two `test/property/` fast-check suites can exceed that
+  10,000 ms global on a contended box, which reds them with the parser untouched. A per-test budget
+  for them was built and then **withdrawn**: the measurements offered for it did not survive
+  re-measurement, and the correct figures do not support the claim they were used to make. Those
+  suites therefore remain on the 10,000 ms global - **exactly where `main` is today** - so this
+  change neither fixes nor worsens that. It is a live false red and it needs its own slice, measured
+  from scratch.
+
 - **The `attw` publish gate exited 0 on a tarball carrying no type declarations, so a broken publish
   read as a pass** (`ATTW-FALSE-GREEN-PORT`). `pnpm attw` now runs `scripts/attw.mjs`, ported from
   `@cosyte/terminology` (#28, `bf153cb`), where the defect was diagnosed; `typecheck:exports` runs

package/README.md

@@ -47,7 +47,7 @@ That's the pitch: no config, no schema upload, no spec lookup. The parser accept
 - **Lazy typed value decode**: `element.value` decodes raw bytes into a discriminated `DicomValue` across all 34 VRs (numbers, `bigint`s, person names, dates/times, sequences, raw `binary`), honoring `(0008,0005)` Specific Character Set through nested items.
 - **Real-world tolerance, Postel's Law**: a lenient reader emits 25 stable warning codes for what it tolerated; only 4 truly-structural conditions are fatal. The serializer always writes spec-clean Part 10.
 - **Source/vendor profile system**: `defineProfile()` + 5 built-ins (`ge`, `siemens`, `philips`, `strict`, `lenient`) that only ever _tighten or annotate_ a parse, resolving vendor private tags by the file's live Private Creator string, never a wrong decode.
-- **Metadata-level de-identification**: `deidentify()` applies the PS3.15 Annex E Basic Profile + the nine metadata Options, returning a fresh dataset and an audit report that is value-free apart from the source UIDs in `report.uidMap`.
+- **Metadata-level de-identification**: `deidentify()` applies the PS3.15 Annex E Basic Profile + the nine metadata Options, returning a fresh dataset and an audit report built from static tables, with two fields that carry source bytes and are named as such on the type: `report.uidMap` (source UIDs) and `report.removedPrivateTags`.
 - **Spec-clean serializer**: `serializeDicom(ds)` round-trips a dataset back to Part 10 bytes in its source transfer syntax (no transcode), with correct File Meta group length, even-length padding, byte-exact sequence passthrough, and lossless File Meta: non-modeled `(0002,xxxx)` elements are preserved and re-emitted in tag order.
 - **Strict TypeScript, dual ESM + CJS, Node ≥ 22**: `noUncheckedIndexedAccess`, no `any`, JSDoc + `@example` on every public export feeding your editor's IntelliSense. Zero runtime dependencies today.
 
@@ -197,7 +197,7 @@ if (img.isEnhancedMultiFrame) {
 
 ### De-identify before sharing
 
-`deidentify()` applies the PS3.15 Annex E Basic Application Level Confidentiality Profile (replacing, emptying, or removing every attribute the standard lists as identifying) and returns a fresh dataset plus a report that is value-free apart from the source UIDs in `report.uidMap`.
+`deidentify()` applies the PS3.15 Annex E Basic Application Level Confidentiality Profile (replacing, emptying, or removing every attribute the standard lists as identifying) and returns a fresh dataset plus a report. Two of the report's fields are composed from source bytes rather than from static tables and are documented as such: `report.uidMap`, whose keys are the file's own UIDs, and `report.removedPrivateTags`, whose four-byte entries are the sender's own private tag numbers on a well-formed file and can be value bytes on a malformed one.
 
 ```ts
 import { parseDicom, deidentify, serializeDicom } from "@cosyte/dicom";
@@ -365,7 +365,8 @@ Thrown by `serializeDicom` for `MISSING_TRANSFER_SYNTAX` (the dataset names no t
 - **Pixel data.** No decode/decompression, no rendering, no measurements: Pixel Data is exposed as raw bytes. And v1 does not read a **compressed object at all**, not even structurally: a transfer syntax outside the four listed below is the fatal `UNSUPPORTED_TRANSFER_SYNTAX`, so JPEG / JPEG-LS / JPEG2000 / RLE / HTJ2K objects do not parse. → `@cosyte/dicom-pixel`.
 - **Burned-in PHI.** v1 **warns** it cannot remove burned-in annotation; a "de-identified" output is **metadata-de-identified only**.
 - **A sequence the parser could not open: content is dropped, not passed through.** `deidentify()` recurses only into a sequence whose items the parser materialized. A **standard** `SQ` element with no items is now **emptied**: its bytes are Data Sets by PS3.5 §7.5.1 and PS3.15 §E.1.1 obliges the de-identifier to reach the listed attributes inside them, so a run that cannot enumerate them must not ship them. `report.unauditableSequences` names the carrier and the byte length dropped (capped at 64 entries; the emptying itself is never capped), and `DICOM_DEIDENT_SEQUENCE_NOT_AUDITABLE` is raised. Expect **data loss** on such a file. The parse refusal is usually a sender-side encoding defect, but not always: a **conformant** file nested deeper than this library's own `NESTING_DEPTH_LIMIT` of 64 is refused the same way and loses that sequence too. **Two shapes are exempt and still leak.** A **private** `SQ` kept under `RetainSafePrivate` + a `Profile` is kept verbatim and unaudited, because the profile vouched for it. And an undefined-length **`UN`** the CP-246 descent could not read as a sequence keeps `vr === "UN"`, so the rule does not reach it and cannot be extended there: every ordinary `UN` element also has no items. For either, the reliable test remains `el.items === undefined` on the element itself.
-- **An element whose on-wire VR is not a VR: the value is dropped, not passed through.** Under an Explicit VR syntax the VR is two bytes the sender wrote and this parser trusts them. If those two bytes are not one of the 34 PS3.5 §6.2 defines, `deidentify()` **empties** the element: §6.2 requires every undefined VR to be long-form with a 32-bit VL, this parser reads it short-form, so those bytes are not a Value Field the library decoded and PS3.15 §E.1.1's obligation cannot be discharged inside them. `report.undefinedVrElements` names the **byte offset** and the byte length dropped (capped at 64 entries; the emptying itself is never capped) and `DICOM_DEIDENT_UNDEFINED_VR_NOT_AUDITABLE` is raised. It names **no tag**, deliberately and uniquely among the report's findings: this header _may_ have been fabricated out of the middle of some element's value, in which case its tag bytes are document content - and nothing distinguishes that case from an honestly-written unrecognized VR, so the tag is withheld either way. The usual cause is an **under**-declared Value Length earlier in the file, which desynchronizes the reader so that leftover value bytes are read as a Data Element header - and the element that genuinely followed becomes the fabricated element's value, which is how a `(0010,0020)` Patient ID used to reach de-identified output. **A file conformant to PS3.5 2026c never trips this**, and an Implicit VR LE file cannot: there the VR comes from the dictionary. (The edition matters: §6.2 exists precisely to say how a _future_ VR will be encoded.) `UN` is one of the 34 and is not affected. **There is no exemption** - unlike the sequence rule above, `RetainSafePrivate` does not keep such an element.
+- **An unrecognized Explicit VR is read (and written) long-form, and `deidentify()` still empties the element.** PS3.5 2026c §6.2: "All new VRs defined in future versions of DICOM **shall** be of the same Data Element Structure as defined in [§7.1.2] with reserved bytes after the VR and a 32-bit unsigned integer VL." So two on-wire VR bytes outside the 34 this release knows get a **12-byte header**, exactly as `OB` and `UT` do, on both the read and the write path. Before `0.0.9` the reader took the 8-byte form, which read the length out of the two bytes §6.2 reserves. What such a file did then was **shape-specific** - sometimes a whole-object refusal, sometimes a clean parse into a tree the sender did not write - which is why this package ships `scripts/measure-unrecognized-vr.ts` and no one-sentence account of it. The value is otherwise handled like any other: a declared length past the end of the buffer, or an undefined length, is refused exactly as it is for every other non-`SQ` VR. **The trade is stated rather than buried**: a sender that ignores §6.2 and writes an unrecognized VR short-form produced a readable object before and is refused now. `deidentify()` **still empties** such an element, because reading a header is not the same as knowing what its value means - Table E.1-1 acts per attribute, and nothing can say whether a VR from a later edition holds a name, a date or an opaque blob. `report.undefinedVrElements` names the **byte offset** and the byte length dropped (capped at 64 entries; the emptying itself is never capped) and `DICOM_DEIDENT_UNDEFINED_VR_NOT_AUDITABLE` is raised. It names **no tag**, deliberately and uniquely among the report's findings: the header may still have been fabricated out of the middle of some element's value, in which case its tag bytes are document content - and nothing distinguishes that case from an honestly-written unrecognized VR, so the tag is withheld either way. An Implicit VR LE file cannot reach any of this: there the VR comes from the dictionary. `UN` is one of the 34 and is not affected. **There is no exemption** - unlike the sequence rule above, `RetainSafePrivate` does not keep such an element.
+- **`RetainSafePrivate` retains nothing inside ANY Item of a sequence whose item stream over-runs its own declared length.** An Item that declares more bytes than its enclosing Sequence does absorbs whatever follows the sequence, so a private data element the sender wrote at the **root** - with no Private Creator reserving its block there - can land beside a creator that is genuine Item content and be kept verbatim. Through `0.0.9` it was, with `report.removedPrivateTags` reading `[]` and the object stamped `(0012,0062) Patient Identity Removed = YES`: a stamp that outran the redaction. PS3.5 2026c §7.8.1 scopes a private block reservation to the Item ("The scope of the reservation is just within the Item"), so a file that contradicts itself about where the Item ends does not determine which reservation covers an element; PS3.15 2026c §E.3.10 retains only what is **known** safe and requires that "all other Private Attributes shall be removed **or processed in the element-specific manner recommended by Deidentification Action (0008,0307), if present within Private Data Element Characteristics Sequence (0008,0300)**" (a two-branch clause; this library does not implement `(0008,0307)`, so removal is the branch available to it). So every private element the recursion **reaches** inside such a sequence, at every depth below it, is removed and named in `report.removedPrivateTags`. **The unit is the SEQUENCE, not the individual Item**: `descendSequence` decides once and applies it to every item, so a two-item sequence where only item 1 over-runs also loses a genuine block sitting in the honest item 0 (measured; the grid sweeps single-item fixtures, so that extra cost is NOT inside the priced 20). **Two routes are not closed and both still leak, measured and pinned:** an Item that _under_-declares **ejects** its trailing elements out into the enclosing Data Set, which is not narrowed - **at any depth, not only at the root** (the widening was built and measured at 24 retentions lost to close 2 of 22, so it is a separate slice); and a **private `SQ`** the profile vouches for is kept verbatim before the descent runs, which is the carve-out named two bullets above. **The direction is not fail-safe on its own and the opposite claim is retracted**: it depends on where the sender put the Private Creator. Both **absorb** placements are pinned by tests; the **eject** placement is pinned as a residual and still leaks. **The price is measured, not asserted**: on `scripts/measure-sq-bound-grid.ts`, 58 cells lose a private value and **20 of them were not leaking anything** (the creator and the data element were both genuine Item content). Retention on a file that does not contradict itself is unchanged, and **no reading changes** - no parser file is touched. This is not a parser bound: an over-declaring Item and an under-declaring Sequence are byte-identical, so nothing on the wire can separate them.
 - **An over-declared Value Length into a binary carrier still leaks, and it is a known trade.** The mirror case - an _over_-declared length that swallows the following element into an `OB`/`OW`/`US`/`UN` value - is **not** detected, and a `(0010,0020)` inside it reaches de-identified output with no warning and no report entry. Arbitrary bytes are exactly what those VRs are for, so no content test can decide it; the only candidate remedy was measured and empties conformant binary values (a legal LUT or blob deleted because 8 of its bytes read as a zero-length `(0010,0020)`). String carriers **are** covered, because there the same bytes are provably outside the VR's repertoire. If you accept files from a sender you do not control, treat a binary attribute's length as untrusted.
 - **Networking & web.** No DIMSE (C-STORE/FIND/MOVE, MWL, MPPS); no DICOMweb (QIDO/WADO/STOW). → `@cosyte/dicom-net`, `@cosyte/dicomweb`.
 - **Transcoding.** No transfer-syntax conversion. The serializer re-emits in the dataset's source syntax only.

package/dist/index.cjs

@@ -11,7 +11,7 @@ var __export = (target, all) => {
 };
 
 // src/version.ts
-var VERSION = "0.0.8";
+var VERSION = "0.0.10";
 
 // src/dictionary/index.ts
 var dictionary_exports = {};
@@ -11703,6 +11703,12 @@ var BE_VR_STRIDE = Object.freeze({
   UT: 0,
   SQ: 0
 });
+var KNOWN_VRS = Object.freeze(
+  new Set(Object.keys(BE_VR_STRIDE))
+);
+function isRecognizedVr(vr) {
+  return KNOWN_VRS.has(vr);
+}
 
 // src/parser/warnings.ts
 var WARNING_CODES = {
@@ -11753,7 +11759,11 @@ var WARNING_MESSAGES = Object.freeze({
   DICOM_VR_MISMATCH: "Element ({tag}) on-wire VR is {vr2}; dictionary lists {vr}. Trusting on-wire VR.",
   DICOM_PRIVATE_TAG_NO_CREATOR: "Private element ({tag}) has no Private Creator registered for its block; treating as VR=UN.",
   DICOM_GROUP_LENGTH_IN_DATASET: "Retired Group Length element ({tag}) encountered in dataset; preserved as-is.",
-  DICOM_NONZERO_RESERVED_BYTES: "Element ({tag}) has non-zero reserved bytes between VR and length (first byte {n}, second byte {n2}); ignoring.",
+  // The tag is deliberately absent, and this is the only *parser* message where
+  // that is true. See `nonzeroReservedBytes`. Kept terse for the same reason the
+  // de-identify messages are: it is emitted once per element, and element count
+  // is attacker-chosen.
+  DICOM_NONZERO_RESERVED_BYTES: "Non-zero reserved bytes between VR and length (first byte {n}, second byte {n2}); ignoring. Tag withheld; the byte offset identifies the element.",
   DICOM_SQ_NOT_DESCENDED: "Element ({tag}) resolved to VR=SQ from the dictionary but its defined-length value is not a valid item stream; kept as opaque bytes and NOT descended, so nothing nested inside it is visible to navigation, and deidentify() cannot audit it. See DICOM_DEIDENT_SEQUENCE_NOT_AUDITABLE.",
   DICOM_UN_PARSED_AS_SQ: "Element ({tag}) has VR=UN with undefined length; descended as Implicit VR LE sequence per CP-246.",
   DICOM_EMPTY_ITEM_IN_SEQUENCE: "Sequence ({tag}) contains an empty item (length=0); tolerated.",
@@ -11788,7 +11798,6 @@ var WARNING_MESSAGES = Object.freeze({
   DICOM_UNSUPPORTED_CHARSET: "(0008,0005) Specific Character Set value {n} names a defined term this build does not support; decoding text as UTF-8 best-effort. The term is not reproduced here - read the element if you need it.",
   DICOM_CHARSET_AMBIGUOUS_SEPARATOR: "(0008,0005) Specific Character Set value {n} is ambiguous under the active code extensions; decoding text as UTF-8 best-effort."
 });
-var KNOWN_VRS = new Set(Object.keys(BE_VR_STRIDE));
 var TAG_SHAPE = /^[0-9A-F]{8}$/u;
 function renderTag(tag) {
   return tag !== void 0 && TAG_SHAPE.test(tag) ? tag : WITHHELD;
@@ -11830,9 +11839,8 @@ function privateTagNoCreator(position, tag) {
 function groupLengthInDataset(position, tag) {
   return build(WARNING_CODES.DICOM_GROUP_LENGTH_IN_DATASET, position, { tag });
 }
-function nonzeroReservedBytes(position, tag, first, second) {
+function nonzeroReservedBytes(position, first, second) {
   return build(WARNING_CODES.DICOM_NONZERO_RESERVED_BYTES, position, {
-    tag,
     n: first,
     n2: second
   });
@@ -12036,12 +12044,12 @@ function readExplicitElementHeader(cursor, _ctx, emit) {
   const vr = vrSlice.toString("ascii");
   let length;
   let headerLength;
-  if (LONG_FORM_VRS.has(vr)) {
+  if (LONG_FORM_VRS.has(vr) || !isRecognizedVr(vr)) {
     const reserved0 = cursor.buffer[cursor.position];
     const reserved1 = cursor.buffer[cursor.position + 1];
     cursor.position += 2;
     if ((reserved0 ?? 0) !== 0 || (reserved1 ?? 0) !== 0) {
-      emit(nonzeroReservedBytes({ byteOffset: headerStart }, tag, reserved0 ?? 0, reserved1 ?? 0));
+      emit(nonzeroReservedBytes({ byteOffset: headerStart }, reserved0 ?? 0, reserved1 ?? 0));
     }
     length = cursor.readUInt32();
     headerLength = 12;
@@ -12179,7 +12187,7 @@ function readExplicitLeElement(cursor) {
   const vr = cursor.slice(2).toString("ascii");
   let length;
   let headerLength;
-  if (LONG_FORM_VRS.has(vr)) {
+  if (LONG_FORM_VRS.has(vr) || !isRecognizedVr(vr)) {
     cursor.slice(2);
     length = cursor.readUInt32();
     headerLength = 12;
@@ -13453,7 +13461,7 @@ function encodeDatasetElement(el, encoding) {
   }
   const littleEndian = encoding === "explicitLE";
   const vrBuf = buffer.Buffer.from(el.vr, "ascii");
-  if (LONG_FORM_VRS.has(el.vr)) {
+  if (LONG_FORM_VRS.has(el.vr) || !isRecognizedVr(el.vr)) {
     return buffer.Buffer.concat([
       uint16(group, littleEndian),
       uint16(element, littleEndian),
@@ -13504,7 +13512,7 @@ function encodeMetaElement(tag, vr, rawValue) {
   const elementBuf = buffer.Buffer.alloc(2);
   elementBuf.writeUInt16LE(element, 0);
   const vrBuf = buffer.Buffer.from(vr, "ascii");
-  if (LONG_FORM_VRS.has(vr)) {
+  if (LONG_FORM_VRS.has(vr) || !isRecognizedVr(vr)) {
     const lengthBuf2 = buffer.Buffer.alloc(4);
     lengthBuf2.writeUInt32LE(value.length, 0);
     return buffer.Buffer.concat([
@@ -14699,7 +14707,6 @@ function uidValueMultiplicity(rawBytes) {
 }
 
 // src/deident/embedded.ts
-var KNOWN_VRS2 = new Set(Object.keys(BE_VR_STRIDE));
 var UNDEFINED_LENGTH4 = 4294967295;
 var FFFE_GROUP = 65534;
 var SCANNABLE_CARRIER_VRS = /* @__PURE__ */ new Set([
@@ -14760,7 +14767,7 @@ function decodeAt(value, off, encoding) {
     return next2 > value.length ? void 0 : { next: next2, tag };
   }
   const vr = value.toString("latin1", off + 4, off + 6);
-  if (!KNOWN_VRS2.has(vr)) return void 0;
+  if (!KNOWN_VRS.has(vr)) return void 0;
   if (LONG_FORM_VRS.has(vr)) {
     if (off + 12 > value.length) return void 0;
     if (value[off + 6] !== 0 || value[off + 7] !== 0) return void 0;
@@ -14897,7 +14904,6 @@ var BODY_ENCODING2 = {
 };
 var MAX_UNAUDITABLE_SEQUENCE_FINDINGS = 64;
 var MAX_UNDEFINED_VR_FINDINGS = 64;
-var KNOWN_VRS3 = new Set(Object.keys(BE_VR_STRIDE));
 function validateRetain(retain) {
   const valid = new Set(DEIDENTIFY_OPTIONS);
   const active = /* @__PURE__ */ new Set();
@@ -15018,7 +15024,7 @@ function actsOnTag(tag, ctx) {
   return resolveAction(effectiveCode(action, ctx.active)) !== "K";
 }
 function hasUndefinedVr(el) {
-  return !KNOWN_VRS3.has(el.vr);
+  return !KNOWN_VRS.has(el.vr);
 }
 function keepOrEmpty(el, ctx, contextPath, out) {
   if (hasUndefinedVr(el)) {
@@ -15082,6 +15088,13 @@ function creatorFor(tag, creators) {
   const { group, element } = splitTag(tag);
   return creators.get(`${String(group)}:${String(element >> 8 & 255)}`);
 }
+var UNDEFINED_LENGTH5 = 4294967295;
+var EXPLICIT_SQ_HEADER_BYTES = 12;
+function itemStreamOverrunsSequence(el, encoding) {
+  if (el.vr !== "SQ" || el.length === UNDEFINED_LENGTH5) return false;
+  const headerBytes = encoding === "implicit" ? 0 : EXPLICIT_SQ_HEADER_BYTES;
+  return el.rawBytes.length - headerBytes > el.length;
+}
 function keepsPrivate(el, ctx, creators) {
   if (!ctx.active.has("RetainSafePrivate") || ctx.profile === void 0) return false;
   if (isPrivateCreatorElement(el.tag)) {
@@ -15091,7 +15104,7 @@ function keepsPrivate(el, ctx, creators) {
   if (creator === void 0) return false;
   return resolvePrivateTag(ctx.profile, el.tag, creator) !== void 0;
 }
-function processElements(source, ctx, contextPath) {
+function processElements(source, ctx, contextPath, reservationsUsable) {
   const out = {
     elements: /* @__PURE__ */ new Map(),
     attributes: [],
@@ -15104,7 +15117,8 @@ function processElements(source, ctx, contextPath) {
   const creators = creatorsInScope(source);
   for (const el of source) {
     if (isPrivateTag(el.tag)) {
-      if (keepsPrivate(el, ctx, creators)) keepOrEmpty(el, ctx, contextPath, out);
+      if (reservationsUsable && keepsPrivate(el, ctx, creators))
+        keepOrEmpty(el, ctx, contextPath, out);
       else out.removedPrivateTags.push(el.tag);
       continue;
     }
@@ -15112,7 +15126,8 @@ function processElements(source, ctx, contextPath) {
     if (action === void 0) {
       if (el.vr === "SQ") {
         if (isUnauditableSequence(el)) emptyUnauditableSequence(el, ctx, contextPath, out);
-        else out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out));
+        else
+          out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out, reservationsUsable));
       } else {
         keepOrEmpty(el, ctx, contextPath, out);
       }
@@ -15120,7 +15135,7 @@ function processElements(source, ctx, contextPath) {
     }
     const resolved = resolveAction(effectiveCode(action, ctx.active));
     if (el.vr === "SQ") {
-      applySequenceAction(el, resolved, action, ctx, contextPath, out);
+      applySequenceAction(el, resolved, action, ctx, contextPath, out, reservationsUsable);
       continue;
     }
     let applied;
@@ -15161,11 +15176,12 @@ function processElements(source, ctx, contextPath) {
   }
   return out;
 }
-function descendSequence(el, ctx, contextPath, out) {
+function descendSequence(el, ctx, contextPath, out, reservationsUsable) {
+  const childReservationsUsable = reservationsUsable && !itemStreamOverrunsSequence(el, ctx.encoding);
   const newItems = [];
   (el.items ?? []).forEach((item, index) => {
     const childPath = [...contextPath, `${el.tag}[${String(index)}]`];
-    const inner = processElements(item.elements(), ctx, childPath);
+    const inner = processElements(item.elements(), ctx, childPath, childReservationsUsable);
     out.attributes.push(...inner.attributes);
     out.removedPrivateTags.push(...inner.removedPrivateTags);
     out.embeddedAttributes.push(...inner.embeddedAttributes);
@@ -15176,7 +15192,7 @@ function descendSequence(el, ctx, contextPath, out) {
   });
   return rebuildSequence(el, newItems, ctx.encoding);
 }
-function applySequenceAction(el, resolved, action, ctx, contextPath, out) {
+function applySequenceAction(el, resolved, action, ctx, contextPath, out, reservationsUsable) {
   let applied;
   switch (resolved) {
     case "X":
@@ -15192,7 +15208,7 @@ function applySequenceAction(el, resolved, action, ctx, contextPath, out) {
         emptyUnauditableSequence(el, ctx, contextPath, out);
         applied = "emptied";
       } else {
-        out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out));
+        out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out, reservationsUsable));
         applied = "cleaned";
       }
       break;
@@ -15202,7 +15218,7 @@ function applySequenceAction(el, resolved, action, ctx, contextPath, out) {
         emptyUnauditableSequence(el, ctx, contextPath, out);
         applied = "emptied";
       } else {
-        out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out));
+        out.elements.set(el.tag, descendSequence(el, ctx, contextPath, out, reservationsUsable));
         applied = "kept";
       }
       break;
@@ -15253,7 +15269,7 @@ function deidentify(ds, options = {}) {
     encoding,
     budget: { unauditableSequences: 0, undefinedVrElements: 0 }
   };
-  const processed = processElements(ds.elements(), ctx, []);
+  const processed = processElements(ds.elements(), ctx, [], true);
   const {
     elements,
     attributes,