@cosyte/dicom 0.0.8 → 0.0.11

package/README.md

@@ -45,9 +45,9 @@ That's the pitch: no config, no schema upload, no spec lookup. The parser accept
 - **One-line metadata extraction**: `ds.patient`, `ds.study`, `ds.series`, `ds.image`: typed, fail-safe views over the safety-critical attributes. No `(group,element)` tags to memorise.
 - **Two access patterns**: named views, or structural `ds.get("00100010")` by 8-character `(group,element)` tag (resolve a keyword to its tag with `Dictionary.byKeyword`), plus `ds.elements()` to walk everything.
 - **Lazy typed value decode**: `element.value` decodes raw bytes into a discriminated `DicomValue` across all 34 VRs (numbers, `bigint`s, person names, dates/times, sequences, raw `binary`), honoring `(0008,0005)` Specific Character Set through nested items.
-- **Real-world tolerance, Postel's Law**: a lenient reader emits 25 stable warning codes for what it tolerated; only 4 truly-structural conditions are fatal. The serializer always writes spec-clean Part 10.
+- **Real-world tolerance, Postel's Law**: a lenient reader emits stable warning codes for what it tolerated; only 4 truly-structural conditions are fatal. The serializer always writes spec-clean Part 10. (The count used to be quoted here and had drifted: it read `25` against a `WARNING_CODES` of `28`. The locked snapshot in `test/property/__snapshots__/warning-codes.snapshot.test.ts.snap` is the pin, and it is measured on every run rather than narrated here.)
 - **Source/vendor profile system**: `defineProfile()` + 5 built-ins (`ge`, `siemens`, `philips`, `strict`, `lenient`) that only ever _tighten or annotate_ a parse, resolving vendor private tags by the file's live Private Creator string, never a wrong decode.
-- **Metadata-level de-identification**: `deidentify()` applies the PS3.15 Annex E Basic Profile + the nine metadata Options, returning a fresh dataset and an audit report that is value-free apart from the source UIDs in `report.uidMap`.
+- **Metadata-level de-identification**: `deidentify()` applies the PS3.15 Annex E Basic Profile + the nine metadata Options, returning a fresh dataset and an audit report built from static tables, with two fields that carry source bytes and are named as such on the type: `report.uidMap` (source UIDs) and `report.removedPrivateTags`.
 - **Spec-clean serializer**: `serializeDicom(ds)` round-trips a dataset back to Part 10 bytes in its source transfer syntax (no transcode), with correct File Meta group length, even-length padding, byte-exact sequence passthrough, and lossless File Meta: non-modeled `(0002,xxxx)` elements are preserved and re-emitted in tag order.
 - **Strict TypeScript, dual ESM + CJS, Node ≥ 22**: `noUncheckedIndexedAccess`, no `any`, JSDoc + `@example` on every public export feeding your editor's IntelliSense. Zero runtime dependencies today.
 
@@ -197,7 +197,7 @@ if (img.isEnhancedMultiFrame) {
 
 ### De-identify before sharing
 
-`deidentify()` applies the PS3.15 Annex E Basic Application Level Confidentiality Profile (replacing, emptying, or removing every attribute the standard lists as identifying) and returns a fresh dataset plus a report that is value-free apart from the source UIDs in `report.uidMap`.
+`deidentify()` applies the PS3.15 Annex E Basic Application Level Confidentiality Profile (replacing, emptying, or removing every attribute the standard lists as identifying) and returns a fresh dataset plus a report. Two of the report's fields are composed from source bytes rather than from static tables and are documented as such: `report.uidMap`, whose keys are the file's own UIDs, and `report.removedPrivateTags`, whose four-byte entries are the sender's own private tag numbers on a well-formed file and can be value bytes on a malformed one.
 
 ```ts
 import { parseDicom, deidentify, serializeDicom } from "@cosyte/dicom";
@@ -318,10 +318,14 @@ for (const w of ds.warnings) {
 }
 ```
 
-The 26 Tier-2 codes (`DICOM_MISSING_PREAMBLE`, `DICOM_FILE_META_GROUP_LENGTH_MISMATCH`, `DICOM_UN_PARSED_AS_SQ`, `DICOM_ODD_LENGTH_VALUE_PADDED`, `DICOM_PRIVATE_CREATOR_UNKNOWN`, `DICOM_VR_MISMATCH`, `DICOM_DA_LEGACY_FORMAT`, … ) live in [`src/parser/warnings.ts`](./src/parser/warnings.ts). Narrow on `w.code === WARNING_CODES.…` for typo-free comparisons, or pass `{ onWarning }` to `parseDicom` to stream them.
+The Tier-2 codes (`DICOM_MISSING_PREAMBLE`, `DICOM_FILE_META_GROUP_LENGTH_MISMATCH`, `DICOM_UN_PARSED_AS_SQ`, `DICOM_ODD_LENGTH_VALUE_PADDED`, `DICOM_PRIVATE_CREATOR_UNKNOWN`, `DICOM_VR_MISMATCH`, `DICOM_DA_LEGACY_FORMAT`, … ) live in [`src/parser/warnings.ts`](./src/parser/warnings.ts), which is the only place their number is worth reading: it said 26 here while the registry held 29, so the numeral is gone rather than corrected. Narrow on `w.code === WARNING_CODES.…` for typo-free comparisons, or pass `{ onWarning }` to `parseDicom` to stream them.
 
 The 4 Tier-3 fatal codes (`NOT_DICOM_PART_10`, `INVALID_FILE_META`, `UNSUPPORTED_TRANSFER_SYNTAX`, `EMPTY_INPUT`) always throw a `DicomParseError`; they represent input the parser cannot meaningfully recover.
 
+**Two Tier-2 codes report a LOSS rather than a tolerated deviation, and they are worth reading before you trust a parsed object.** A parsed Data Set is a map keyed by tag, so a file that carries one tag twice in the same Data Set loses the first element's value: the second replaces it, and the survivor looks exactly like an element the sender wrote once. `DICOM_DUPLICATE_TAG_IN_DATA_SET` is raised at the moment of the replacement, with the byte offset of the header that replaced. **That offset is the surviving element's own `Element.byteOffset`, which makes it a lookup only for a collision at the root** - inside a defined-length Sequence Item `Element.byteOffset` is relative to that item's own slice, so the same number can name an untouched root element and the warning does not tell you which Data Set it came from (`position.contextPath` is not populated by any parser warning). PS3.5 2026c §7.1 requires a tag to occur at most once in a Data Set and §7.5.1 requires the same inside an Item, so it cannot fire on a conformant file - the ordinary way to reach it is a length field that lies, which makes bytes inside somebody's value read as a Data Element header. **The reading is unchanged: last one read still wins, and nothing is guessed for the value that was replaced.** If you see this code, the object is missing something the file contained and no round trip will show you what; treat it as you would a fatal, and raise the file with the sender.
+
+**`DICOM_DUPLICATE_FILE_META_ELEMENT` is the same loss in the `(0002,xxxx)` group, and that group is the one that decides how every following byte is read.** The File Meta group is collected into an array rather than a map, so nothing is overwritten there - but the eight tags this library projects into typed `FileMeta` fields are answered by a **first-match** search and are excluded from `FileMeta.extraElements`, the verbatim residue that gives the group its byte-exact round trip. A second copy of one of those tags is in neither, so it left the object. Two copies of `(0002,0010)` Transfer Syntax UID carrying different UIDs are two different readings of the same file, and the order alone decides which you get. **The two codes resolve a repeat the opposite way round, deliberately, because the two readings do: the FIRST copy wins in the File Meta group, the LAST read wins in a Data Set.** Neither reading changed in this release. Unlike the Data Set code, `position.byteOffset` here is unambiguously file-absolute - the File Meta group is never nested - and it locates the copy that was **dropped**, not the survivor. A repeated `(0002,xxxx)` tag this library does not model is silent, because every copy of one is kept in `extraElements` and nothing is dropped - though note that `serializeDicom` then **re-emits both copies**, which is `PRE-EXISTING` and is where this package's round-trip promise and its spec-clean promise disagree. **Two bounds, both `PRE-EXISTING` and neither closed here.** The disclosure covers the group **as the parser delimits it**: a copy an intermediary appended past an honest `(0002,0000)` group length is never a File Meta element to this parser at all, and is relocated into the main Data Set silently. And an over-long or wrong group length is reported by its own codes, not this one.
+
 ---
 
 ## Error Handling
@@ -365,8 +369,11 @@ Thrown by `serializeDicom` for `MISSING_TRANSFER_SYNTAX` (the dataset names no t
 - **Pixel data.** No decode/decompression, no rendering, no measurements: Pixel Data is exposed as raw bytes. And v1 does not read a **compressed object at all**, not even structurally: a transfer syntax outside the four listed below is the fatal `UNSUPPORTED_TRANSFER_SYNTAX`, so JPEG / JPEG-LS / JPEG2000 / RLE / HTJ2K objects do not parse. → `@cosyte/dicom-pixel`.
 - **Burned-in PHI.** v1 **warns** it cannot remove burned-in annotation; a "de-identified" output is **metadata-de-identified only**.
 - **A sequence the parser could not open: content is dropped, not passed through.** `deidentify()` recurses only into a sequence whose items the parser materialized. A **standard** `SQ` element with no items is now **emptied**: its bytes are Data Sets by PS3.5 §7.5.1 and PS3.15 §E.1.1 obliges the de-identifier to reach the listed attributes inside them, so a run that cannot enumerate them must not ship them. `report.unauditableSequences` names the carrier and the byte length dropped (capped at 64 entries; the emptying itself is never capped), and `DICOM_DEIDENT_SEQUENCE_NOT_AUDITABLE` is raised. Expect **data loss** on such a file. The parse refusal is usually a sender-side encoding defect, but not always: a **conformant** file nested deeper than this library's own `NESTING_DEPTH_LIMIT` of 64 is refused the same way and loses that sequence too. **Two shapes are exempt and still leak.** A **private** `SQ` kept under `RetainSafePrivate` + a `Profile` is kept verbatim and unaudited, because the profile vouched for it. And an undefined-length **`UN`** the CP-246 descent could not read as a sequence keeps `vr === "UN"`, so the rule does not reach it and cannot be extended there: every ordinary `UN` element also has no items. For either, the reliable test remains `el.items === undefined` on the element itself.
-- **An element whose on-wire VR is not a VR: the value is dropped, not passed through.** Under an Explicit VR syntax the VR is two bytes the sender wrote and this parser trusts them. If those two bytes are not one of the 34 PS3.5 §6.2 defines, `deidentify()` **empties** the element: §6.2 requires every undefined VR to be long-form with a 32-bit VL, this parser reads it short-form, so those bytes are not a Value Field the library decoded and PS3.15 §E.1.1's obligation cannot be discharged inside them. `report.undefinedVrElements` names the **byte offset** and the byte length dropped (capped at 64 entries; the emptying itself is never capped) and `DICOM_DEIDENT_UNDEFINED_VR_NOT_AUDITABLE` is raised. It names **no tag**, deliberately and uniquely among the report's findings: this header _may_ have been fabricated out of the middle of some element's value, in which case its tag bytes are document content - and nothing distinguishes that case from an honestly-written unrecognized VR, so the tag is withheld either way. The usual cause is an **under**-declared Value Length earlier in the file, which desynchronizes the reader so that leftover value bytes are read as a Data Element header - and the element that genuinely followed becomes the fabricated element's value, which is how a `(0010,0020)` Patient ID used to reach de-identified output. **A file conformant to PS3.5 2026c never trips this**, and an Implicit VR LE file cannot: there the VR comes from the dictionary. (The edition matters: §6.2 exists precisely to say how a _future_ VR will be encoded.) `UN` is one of the 34 and is not affected. **There is no exemption** - unlike the sequence rule above, `RetainSafePrivate` does not keep such an element.
+- **An unrecognized Explicit VR is read (and written) long-form, and `deidentify()` still empties the element.** PS3.5 2026c §6.2: "All new VRs defined in future versions of DICOM **shall** be of the same Data Element Structure as defined in [§7.1.2] with reserved bytes after the VR and a 32-bit unsigned integer VL." So two on-wire VR bytes outside the 34 this release knows get a **12-byte header**, exactly as `OB` and `UT` do, on both the read and the write path. Before `0.0.9` the reader took the 8-byte form, which read the length out of the two bytes §6.2 reserves. What such a file did then was **shape-specific** - sometimes a whole-object refusal, sometimes a clean parse into a tree the sender did not write - which is why this package ships `scripts/measure-unrecognized-vr.ts` and no one-sentence account of it. The value is otherwise handled like any other: a declared length past the end of the buffer, or an undefined length, is refused exactly as it is for every other non-`SQ` VR. **The trade is stated rather than buried**: a sender that ignores §6.2 and writes an unrecognized VR short-form produced a readable object before and is refused now. `deidentify()` **still empties** such an element, because reading a header is not the same as knowing what its value means - Table E.1-1 acts per attribute, and nothing can say whether a VR from a later edition holds a name, a date or an opaque blob. `report.undefinedVrElements` names the **byte offset** and the byte length dropped (capped at 64 entries; the emptying itself is never capped) and `DICOM_DEIDENT_UNDEFINED_VR_NOT_AUDITABLE` is raised. It names **no tag**, deliberately and uniquely among the report's findings: the header may still have been fabricated out of the middle of some element's value, in which case its tag bytes are document content - and nothing distinguishes that case from an honestly-written unrecognized VR, so the tag is withheld either way. An Implicit VR LE file cannot reach any of this: there the VR comes from the dictionary. `UN` is one of the 34 and is not affected. **There is no exemption** - unlike the sequence rule above, `RetainSafePrivate` does not keep such an element.
+- **`RetainSafePrivate` retains nothing inside ANY Item of a sequence whose item stream over-runs its own declared length.** An Item that declares more bytes than its enclosing Sequence does absorbs whatever follows the sequence, so a private data element the sender wrote at the **root** - with no Private Creator reserving its block there - can land beside a creator that is genuine Item content and be kept verbatim. Measured on the published `0.0.8` tarball it was, with `report.removedPrivateTags` reading `[]`, the object stamped `(0012,0062) Patient Identity Removed = YES` and `ds.warnings` empty: a stamp that outran the redaction. (**There is no `0.0.9` on the registry** - `package.json` carried it and the publish never happened, an npm `E404` - so earlier wording here that said "through `0.0.9`" named a version that does not exist. Measured on the current published `0.0.10`, this direction is fixed.) PS3.5 2026c §7.8.1 scopes a private block reservation to the Item ("The scope of the reservation is just within the Item"), so a file that contradicts itself about where the Item ends does not determine which reservation covers an element; PS3.15 2026c §E.3.10 retains only what is **known** safe and requires that "all other Private Attributes shall be removed **or processed in the element-specific manner recommended by Deidentification Action (0008,0307), if present within Private Data Element Characteristics Sequence (0008,0300)**" (a two-branch clause; this library does not implement `(0008,0307)`, so removal is the branch available to it). So every private element the recursion **reaches** inside such a sequence, at every depth below it, is removed and named in `report.removedPrivateTags`. **The unit is the SEQUENCE, not the individual Item**: `descendSequence` decides once and applies it to every item, so a two-item sequence where only item 1 over-runs also loses a genuine block sitting in the honest item 0 (measured; the grid sweeps single-item fixtures, so that extra cost is NOT inside the priced 20). **One route is not closed and still leaks, measured and pinned:** a **private `SQ`** the profile vouches for is kept verbatim before the descent runs, which is the carve-out named two bullets above. **The direction is not fail-safe on its own and the opposite claim is retracted**: it depends on where the sender put the Private Creator. Both **absorb** placements are pinned by tests, and so is the **eject** placement (below). Treat `DICOM_ITEM_CROSSES_SEQUENCE_END` as "this file's structure is in dispute", never as a statement about what was retained. **The price is measured, not asserted**: on `scripts/measure-sq-bound-grid.ts`, 58 cells lose a private value and **20 of them were not leaking anything** (the creator and the data element were both genuine Item content). Retention on a file that does not contradict itself is unchanged, and **no reading changes** - no parser file is touched. This is not a parser bound: an over-declaring Item and an under-declaring Sequence are byte-identical, so nothing on the wire can separate them.
+- **`RetainSafePrivate` also retains nothing a Data Set holds AFTER a sequence whose own contents contradict the extent it declared.** The mirror direction: an Item that _under_-declares **ejects** its trailing elements out into the enclosing Data Set, and a Private Creator that lands there reserves a block for elements the sender never put beside it - the same false attestation. **Measured on the published `0.0.10` tarball, which is the current registry version**: `report.removedPrivateTags` `[]`, the value in the serialized output, `(0012,0062) = YES`, `ds.warnings` empty and no throw under `{ strict: true }`. It applies **in every Data Set, not only the root**: an inner sequence ejecting a creator into the still-usable Item that encloses it is measured and pinned. The cut is **positional**, so a reservation the sender wrote **ahead of** the offending sequence is untouched, and the same reservation written **after** it is refused. **Two predicates, because the parser records the same contradiction two ways**: under Explicit VR the item stream over-runs and the span shows up as `rawBytes.length > length`; under Implicit VR LE that path slices the item stream, so nothing over-runs at all and the descent is refused instead (`DICOM_SQ_NOT_DESCENDED`, `Element.items === undefined`). The second predicate is **broader than the ejection it is here for, deliberately**: it says the parser could not walk the sequence, which other unwalkable item streams also reach. **And two BOUNDS, not one, because a Data Set is a `Map<Tag, Element>`**: an ejected element whose tag the Data Set already holds overwrites in place and inherits the earlier element's position, ahead of the sequence it came out of, so `Element.byteOffset` is checked beside the index. **That collision also destroys the root's own value at parse time** - `PRE-EXISTING`, unchanged here, and pinned by a test so it is not mistaken for something this rule handles. It was silent when that was written; `DICOM_DUPLICATE_TAG_IN_DATA_SET` reports it now (below), and the destruction itself is still unchanged. **The price is measured**: on `scripts/measure-sq-bound-grid.ts` all **22** cells that retained a private value at the root on a self-contradicting file whose honest control removes it go to **0**, at the cost of **56** further root retentions on self-contradicting files whose honest control does keep the value (the column is `de-identified OUTPUT lost a marker (cost)`, reading 78; the parse-tree `LOST`/`GAINED a marker value` counters beside it read 0). Retention on a file that does not contradict itself is unchanged (9 at the root, 6 in an Item), and **no reading changes**: 0 cells differ in any parse respect, 0 cells whose reading differs, and no parser file is touched.
 - **An over-declared Value Length into a binary carrier still leaks, and it is a known trade.** The mirror case - an _over_-declared length that swallows the following element into an `OB`/`OW`/`US`/`UN` value - is **not** detected, and a `(0010,0020)` inside it reaches de-identified output with no warning and no report entry. Arbitrary bytes are exactly what those VRs are for, so no content test can decide it; the only candidate remedy was measured and empties conformant binary values (a legal LUT or blob deleted because 8 of its bytes read as a zero-length `(0010,0020)`). String carriers **are** covered, because there the same bytes are provably outside the VR's repertoire. If you accept files from a sender you do not control, treat a binary attribute's length as untrusted.
+- **`(0012,0063)` De-identification Method is ADDED TO, so a prior de-identifier's text is carried into the output.** PS3.15 2026c §E.1.1 says a text string describing the method "shall be inserted in or added to" that attribute, and this library used to **replace** it - destroying the provenance chain the attribute exists to carry, silently, on a file whose earlier pass may be the one a recipient was relying on. It now appends its own text as a further value of the `1-n` attribute, after a `\`, copying the prior bytes through verbatim. **The method string is itself a `1-n` value**: it is split on `\` and only the values not already recorded are added. **Trailing SPACE and NUL are padding rather than content** (PS3.5 2026c Table 6.2-1's `LO` row, which describes a **Value**, and `LO` is `1-n`), so they are ignored **per value, on both sides** when your string is matched against what is already recorded, and trimmed from the value written. Repeated de-identification is therefore a fixed point **from the first pass**, for every string: with or without a delimiter, and with a pad byte on any value, last or not. Leading spaces are yours and are written through untouched; a method string that is padding only records nothing. **One bound, and it is not cosmetic**: `LO` is a short-form VR whose Value Length field is 16 bits, so when the join would not fit, the prior value is **replaced** rather than added to and `report.warnings` carries `DICOM_DEIDENT_METHOD_NOT_ADDED`. Read that code as "the length ceiling was reached", never as "every fallback is disclosed": a `(0012,0063)` a file encoded under a VR other than `LO` is also replaced, and that one is silent (`PRE-EXISTING`). An element the serializer cannot encode would take the whole de-identified object down, and truncating the sender's earlier records instead would be a policy the standard does not state. A `deidentificationMethod` longer than that ceiling on its own is unchanged from every earlier release and still fails to serialize. **The cost is stated rather than glossed**: `(0012,0063)` is **not in Table E.1-1**, so the Basic Profile keeps it exactly as it keeps every other unlisted attribute, and a sender who wrote something identifying into it now sees that text in de-identified output where the replacement used to remove it as a side effect. That is the retained-by-omission posture of every unlisted attribute, not a rule this insertion states; it is pinned by a residual test that asserts the cost. **It is not silent**: `report.warnings` carries `DICOM_DEIDENT_METHOD_PRIOR_RETAINED` whenever prior bytes from the source file survive into `(0012,0063)`, because an object stamped `(0012,0062) = YES` carrying text nothing in the run inspected is a stamp that outran the redaction. It is **not** on `report.retained` - that field lists the Annex E option sets active for the run, and a kept attribute is not one of them. `(0012,0062)` Patient Identity Removed is still **replaced** with `YES` - the same paragraph of §E.1.1 uses the other verb for it ("shall be replaced or added to"), and the asymmetry is the standard's. A `(0012,0063)` a file encoded under a VR other than `LO` is replaced rather than appended to.
 - **Networking & web.** No DIMSE (C-STORE/FIND/MOVE, MWL, MPPS); no DICOMweb (QIDO/WADO/STOW). → `@cosyte/dicom-net`, `@cosyte/dicomweb`.
 - **Transcoding.** No transfer-syntax conversion. The serializer re-emits in the dataset's source syntax only.
 - **Terminology resolution.** Coded values are surfaced (designator + canonical source) but not validated against SNOMED/LOINC/etc.