@cosyte/dicom 0.0.3 → 0.0.4

package/CHANGELOG.md

@@ -6,6 +6,49 @@ All notable changes to `@cosyte/dicom` will be documented in this file. The form
 
 ### Added
 
+- **The element registry is now generated from NEMA's PS3.6 DocBook, the normative publication**
+  (`DICOM-UPSTREAM-EDITION-LAG`). `vendor/nema/part06/` pins `part06.xml` (PS3.6 **2026c**) by
+  SHA-256, and `scripts/generate-dictionary.ts` applies it as a **per-field overlay** over the
+  Innolitics mirror: for a tag both sources carry, PS3.6 wins on name, keyword, VR, VM and
+  retirement; a tag PS3.6 carries and the mirror does not is added; a tag the mirror carries and
+  PS3.6 does not is **kept**, because PS3.6 retires elements rather than deleting them, so an absence
+  is far more likely to be a parse gap here than a withdrawal there and dropping the entry would turn
+  a decoded element into an unknown one (that set is empty today, and the generator prints its size).
+  The overlay is scoped to the four registry tables (6-1, 7-1, 8-1, 9-1). UIDs stay on the
+  `sops.json` + curated path deliberately: `uids.ts` carries the four short forms every DICOM toolkit
+  uses instead of PS3.6's "...: Default Transfer Syntax for ..." clauses, and retirement as a
+  structured `retired` boolean rather than a trailing " (Retired)" glued into the name, and an
+  overlay would undo both. PS3.15 Annex E is a different part and a different generator, unchanged.
+  This is the durable fix for the edition lag recorded in `vendor/innolitics/README.md`: the mirror's
+  pin is exactly current against its own upstream, but that upstream last refreshed its data on
+  2024-04-18, so the tables sat on PS3.6 2024b against a current 2026c and no re-pinning discipline
+  could have moved them. The registry no longer comes from a source whose regeneration cadence is not
+  ours.
+  **The pin is a precondition, not a comment.** The generator recomputes the SHA-256 of the file it
+  reads and refuses to generate on a mismatch, reads the edition from the document's own
+  `<subtitle>` rather than asserting it, and fails loudly rather than emitting a thinner dictionary
+  on a row that is not six cells, a malformed `(gggg,eeee)` tag cell, a non-identifier keyword, an
+  unrecognized VR token, or a total under 5,000 registry rows. Two traps in the DocBook are handled
+  explicitly and covered by tests: the keyword column carries **13,470 ZERO WIDTH SPACE** line-break
+  hints in 2026c, one of which left in produces a keyword that looks right and never matches; and the
+  sixth column carries `DICOS` and `DICONDE` dictionary markers alongside `RET (edition)`, so reading
+  it as a boolean would retire **391** live tags. Only a leading `RET` retires. Row accounting is
+  the part that had to be got right rather than merely asserted: every `<tr>` in a registry table
+  must resolve to a matched body row or a header row, which covers unrecognized row markup, a second
+  `<tbody>`, and a row outside any `<tbody>` in one check. Counting row opens inside the already
+  truncated first-`<tbody>` slice, as an earlier draft did, structurally cannot see the rows it
+  dropped: splitting Table 6-1 in two made that version read 207 rows instead of 5,309. Cell text
+  is likewise stripped of markup to a fixpoint and then refused if any `<` or `>` survives, rather
+  than stripped in a single pass that can leave a residue which reassembles into another tag.
+  Every run prints the overlay it applied (shared / added / mirror-only, and the fields overridden
+  broken out by field, **with every VR override listed individually**), so a future re-pin of either
+  source shows what moved instead of a 5,000-line diff. There is deliberately **no staleness clock**:
+  a date-based gate would fire the day it was written, demand an action nobody can take on demand
+  (NEMA publishes when NEMA publishes), red unrelated pull requests, and train people to bump a date
+  instead of re-deriving anything. "Has NEMA published a new edition" is one command against
+  _content_, documented in `vendor/nema/README.md`. What is gated in CI is unchanged and offline:
+  the committed dictionary must be byte-for-byte what the pinned inputs produce.
+
 - **Em-dash brand gate in CI (`EMDASH-CONFORMANCE`).** The founder directive of 2026-07-24
   (`knowledgebase/06-brand/voice-and-tone.md`, "No em dashes. Ever.") bans `U+2014` outright across
   every cosyte surface and names commit messages explicitly, and the meta-repo's
@@ -70,8 +113,218 @@ All notable changes to `@cosyte/dicom` will be documented in this file. The form
   synthetic, base64-encoded Part 10 objects (invented patient, fake UIDs); no real PHI, no `.dcm`
   file on disk.
 
+### Fixed
+
+- **`deidentify()` never removed and never reported the repeating-group attributes PS3.15 marks `X`**
+  (`DICOM-REPEATING-GROUP-DEID-GAP`). Table E.1-1 states three of its rows as a group mask rather
+  than a single tag: `(50xx,xxxx)` Curve Data, `(60xx,3000)` Overlay Data and `(60xx,4000)` Overlay
+  Comments, all three marked **X** (remove). The matcher keyed attributes by exact tag, which cannot
+  express a mask, so all three rows were unreachable; `deidentify()` reads "not in the table" as
+  "not listed, keep", so a file carrying `(6000,4000)` free text came back with a **clean report and
+  the text still in it**. Overlay comments are a classic burned-in-identifier carrier, and
+  `DICOM_BURNED_IN_ANNOTATION_NOT_REMOVED` gives no coverage here: it keys on `(7FE0,0010)` +
+  `(0028,0301)`, which describe the image, not the overlay planes. This shipped at `0.0.3`, and
+  `deid`'s `/dicom` adapter delegates here, so it had the same hole.
+  The three rows are now generated from the pinned PS3.15 DocBook as pattern rules and consulted on
+  an exact-tag miss. A matched element is **removed and reported**: the report entry carries the
+  concrete tag, the family's attribute name, and a new `DeidentifiedAttribute.repeatingGroup` field
+  naming the mask that matched, so an audit distinguishes a mask hit from a single-tag hit. The
+  `CleanGraphics` column is honoured on these rows exactly as the table states it. **The exact-tag
+  table wins** over a mask that would also cover the tag, as the more specific statement the standard
+  makes about it; 2026c publishes no such overlap and the generator counts and prints it every run.
+  **The mask covers the groups PS3.5 bounds it to, not any four hex digits.** PS3.5 §7.6: repeating
+  groups "shall only be allowed in the even numbered Groups 6000-601E", and PS3.5-2004 §7.6, which
+  the current edition's note delegates to for curves, states "even Groups (5000-501E,eeee)". Sixteen
+  groups per mask, not 256, and the same section says of the odd ones that there is "no implication
+  of repeating semantics". Both bounds are load-bearing in opposite directions: a hex wildcard would
+  remove attributes the standard never marked (data loss on a call asked to be conservative), and an
+  exact tag matched nothing. Odd groups in the overlay range are private and still go through the
+  private-attribute path.
+  **The generator now refuses a masked row on a prefix PS3.5 does not define as a repeating group**,
+  rather than printing it and carrying on, which is what let these three rows go missing. Proven by
+  mutation rather than asserted: against a Table E.1-1 carrying an injected `(7Fxx,0010)` family row
+  the pre-remedy generator exits 0 and drops it, the post-remedy one exits 1 and names the prefix. A
+  second mutation moves the Overlay Comments Basic Profile code `X` -> `K` and asserts the emitted
+  rule moves with it, so the action codes are read from the document rather than assumed. Every run
+  prints the rules emitted, the concrete groups each covers, the private family row it does not
+  emit, and the shadowing count.
+- **`RetainLongitudinalTemporal` now documents which E.3.6 sub-option a caller actually gets.**
+  Behaviour is unchanged: PS3.15 §E.3.6 is two options and this package exposes one name carrying the
+  **full-dates** column, the less protective branch. It is `K` on all **169** rows where
+  modified-dates says `C`. The option's JSDoc and `docs-content/troubleshooting.md` now say so, and
+  say that date shifting is not performed at this layer. Splitting the option is a public-surface
+  change and is deliberately not made here.
+- **`deidentify()` retained 35 patient-identifying attributes and reported nothing**
+  (`DICOM-ANNEX-E-DEID-LAG`). The PS3.15 Annex E action table came from a third-party mirror alone,
+  pinned at a snapshot whose Table E.1-1 data is 2024b-era. Current PS3.15 publishes 652 concrete
+  attributes; that snapshot carried 617. Because `annexE()` returns `undefined` for a tag it does not
+  carry and `deidentify()` reads `undefined` as "not listed, keep", every missing attribute came
+  through the call verbatim, and the `DeidentifyReport`, whose whole job is to say what was done,
+  said nothing about them. The caller's only signal was a clean return. This shipped at `0.0.3`.
+  **32 of the 35 are marked X** (remove) by the standard; the other three are `(0040,B020)` Waveform
+  Annotation Sequence (`X/D`), `(0070,0006)` Unformatted Text Value (`D`), and `(300A,0054)` Table Top
+  Position Alignment UID (`U`). The removals include `(0010,0011)` through `(0010,0016)`,
+  the person-names-to-use and pronoun block, of which `(0010,0012)` **Name to Use is a patient's
+  preferred name and survived verbatim**; `(0010,0041)` through `(0010,0047)`, the gender-identity
+  and sex-parameters-for-clinical-use attributes; `(0010,2161)` `EthnicGroupCodeSequence` and
+  `(0010,2162)` `EthnicGroups`, the two attributes that replaced the retired `(0010,2160)`
+  `EthnicGroup`; the four `(0008,130x)` diagnosis code sequences; and the waveform, montage,
+  acquisition-context and display-URI attributes at `(003A,xxxx)`, `(0040,Axxx)`, `(0040,Bxxx)` and
+  `(0040,E012)`. `(0032,1033)` `RequestingService` also gains the `CleanDescriptors: C` column the
+  mirror had dropped.
+  The fix is the same authority rule the data dictionary already uses, pointed at PS3.15.
+  `vendor/nema/part15/` pins `part15.xml` (PS3.15 **2026c**) by SHA-256, and
+  `scripts/generate-annex-e.ts` applies Table E.1-1 as a **per-field overlay** over the mirror: for a
+  tag both sources carry, PS3.15 wins on attribute name, Basic Profile action code, and all nine
+  metadata-affecting option columns; a tag PS3.15 carries and the mirror does not is added; a tag the
+  mirror carries and PS3.15 does not is **kept**, because the standard retires rather than deletes,
+  so an absence is far more likely to be a parse gap here than a withdrawal there and dropping the
+  entry would turn an attribute the de-identifier acts on into one it silently keeps. That set is
+  empty today, and its size prints on every run so the assumption stays observable. Nothing is
+  hand-corrected: every value is derived from the fetched normative bytes and reproduced by every
+  regen. No Basic Profile code changed for a tag both sources carried, which is the reassuring half
+  of the finding: the mirror was not wrong about what it had, only silent about what it lacked.
+  **The pin is a precondition, not a comment.** The generator recomputes the SHA-256 of the file it
+  reads and refuses to generate on a mismatch, reads the edition from the document's own
+  `<subtitle>` rather than asserting it, and fails loudly rather than emitting a thinner table on a
+  header row whose 15 column labels are not where the generator's indices expect them, a body row that
+  is not 15 cells, a tag cell it does not recognize, an action code outside Table E.1-1a in any
+  non-empty action column, an empty Basic Profile cell, a `<tr>` that resolves to neither a matched
+  body row nor a header row, or a total under 600 concrete rows. The header check is the one that
+  catches a column **reorder**, which a cell count cannot see and which would read one option's action
+  code as another's. Every run prints the overlay
+  it applied, **with every action-code override listed individually**, because a changed action code
+  is the one difference that decides whether an identifier survives.
+  Two exclusions are deliberate and now **printed on every run** instead of being assumed. Four rows
+  of Table E.1-1 state a family rather than a single tag (`(50xx,xxxx)` Curve Data, `(60xx,3000)`
+  Overlay Data, `(60xx,4000)` Overlay Comments, and `(gggg,eeee) where gggg is odd` Private
+  Attributes); an exact-tag map cannot key them, private attributes are removed through their own
+  path, and the three repeating-group rows remain a stated gap. And PS3.15's E.3.6 is two options,
+  full dates and modified dates, against this package's single `RetainLongitudinalTemporal`, which
+  carries the full-dates column; the two columns diverge on **169** of the 652 rows in 2026c, and
+  that count now prints rather than living in a comment that guessed "usually not".
+  There is deliberately **no staleness clock**, for the same reasons as PS3.6: a date gate fires the
+  day it is written, demands an action nobody can take on demand, and reds unrelated pull requests.
+  What CI gates is unchanged and offline: the committed table must be byte-for-byte what the pinned
+  inputs produce.
+- **Three defects in the shipped data dictionary, all resolved from normative bytes**
+  (`DICOM-UPSTREAM-EDITION-LAG`). None is hand-corrected; each is what PS3.6 2026c prints, and each
+  is reproduced by every regen.
+  - **`(0010,2160)` `EthnicGroup` was marked current although PS3.6 retired it in 2025a**, and its
+    replacements were absent entirely. It is now `retired: true`, and `(0010,2161)`
+    `EthnicGroupCodeSequence` (`SQ`, VM 1) and `(0010,2162)` `EthnicGroups` (`UC`, VM 1-n) are
+    present. This is a real demographic-attribute change: an application reading only `EthnicGroup`
+    silently misses what current instances actually carry. Retired does not mean removed, so the
+    keyword still resolves and an older study still reads; the caller is told both what the tag is
+    and that it is no longer current.
+  - **`(3004,0012)` `DoseValue` was marked retired although PS3.6 defines it.** The `RET (2022d)`
+    marker belongs to the preceding row, `(3004,0010)` `RTDoseROISequence`, which remains retired.
+    `DoseValue` is now `retired: false`. A dose attribute wrongly flagged retired is the dangerous
+    direction of that error: it is the one that talks a caller out of reading a real RT dose.
+  - **`(003A,0320)` and `(003A,0325)` carried truncated keywords**, `SummarizedFilterLookupTable` and
+    `AnalogFilterType`, so `byKeyword` missed on the spelling PS3.6 publishes. They are now
+    `SummarizedFilterLookupTableSequence` and `AnalogFilterTypeCodeSequence`. The truncated forms are
+    **gone rather than kept as aliases**: they were never PS3.6 keywords, and carrying them would
+    preserve the defect under a nicer name.
+  - Alongside these, **180 tags PS3.6 has gained are now known**, so `Dictionary.lookup` names them
+    and Implicit VR parsing resolves their VR from the dictionary instead of falling back to `UN`.
+    The registry goes from **5,129 to 5,309** tags and `byKeyword` from **5,035 to 5,214** keywords.
+    Zero VR, name and VM differences existed on the 5,129 shared tags and zero tags were dropped, so
+    **no previously-decoded element decodes differently**; the change is corrections plus additions.
+
+- **174 SOP Class UID names in `UIDS` were wrong (`DICOM-DICT-CURRENCY`).** The dictionary generator
+  appended `" Storage"` to every name it took from the vendor `sops.json`, but that field is already
+  the full PS3.6 Table A-1 UID Name. The result was a doubled suffix on the 164 names that already
+  ended in "Storage", so `UIDS["1.2.840.10008.5.1.4.1.1.2"].name` read **"CT Image Storage Storage"**,
+  and an equally wrong tail on 10 of the other 11 ("Digital X-Ray Image Storage - For Presentation
+  Storage"). Every one shipped in `0.0.1` through `0.0.3`. The suffix is gone and the names now come
+  through verbatim.
+  The rule was wrong for all 175 entries it touched but landed on the right string for exactly one,
+  so removing it needed a compensating correction rather than a second blanket rule:
+  `1.2.840.10008.5.1.4.1.1.79.1` is the single UID whose vendor name genuinely lacks the suffix, and
+  it is now pinned in the generator's curated table to the "Macular Grid Thickness and Volume Report
+  Storage" that PS3.6 2026c Table A-1 gives it.
+  One further name is corrected: `1.2.840.10008.1.2.6.1`, the retired RFC 2557 transfer syntax, read
+  "RFC 2557 MIME Encapsulation" where PS3.6 prints "RFC 2557 MIME encapsulation".
+  Measured against PS3.6 2026c on the branch head: of the **261** UIDs shared with Table A-1,
+  **240 match the `UID Name` column byte for byte**; a further **17** differ only in that Table A-1
+  writes retirement into the name as a trailing " (Retired)" where this dictionary carries a
+  structured `retired` boolean, which gives **257** when those are read as matches and **zero**
+  retirement-flag disagreements; the remaining **4** are the deliberate transfer-syntax short forms
+  tabulated in `vendor/innolitics/README.md`; **0** are unexplained. All **7** well-known frames of
+  reference match Table A-2 byte for byte. `uids.ts` changes on 175 lines in total.
+  Name only: no UID value, `type`, or `retired` flag changed. Transfer-syntax dispatch keys off the
+  UID **value**, never the name, so no parse or de-identification behavior changes. The one place a
+  name reaches a caller at runtime is the human-readable `snippet` on the fatal
+  `UNSUPPORTED_TRANSFER_SYNTAX` error, which is improved by the correction rather than altered in
+  meaning.
+- **The byte-identical regen gate could go green while generating nothing (`DICOM-DICT-CURRENCY`).**
+  Two ways, both fixed by making the gate delete `src/dictionary/generated/` (except the hand-written
+  `README.md`) before running `pnpm gen:all`, so it measures what the generators produce instead of
+  what happens to be on disk. A stale orphan, an artifact a generator used to emit and no longer
+  does, was never rewritten and never diffed, so it passed indefinitely. And gutting `gen:all` to a
+  no-op wrote nothing, left every committed artifact untouched, and produced an empty `git diff`:
+  permanently green. `package.json` also joins the workflow's `paths` filters, without which that
+  second case never triggered the gate at all. Both now red, proved by seeding each defect and
+  watching the run fail. The delete is `-mindepth 1` rather than `-type f`, so a stale symlink or
+  subdirectory cannot become an orphan the gate is blind to, and the step asserts that only
+  `README.md` survives instead of printing an expectation nothing checks. `pnpm gen:clean` exposes
+  the same delete for local use but is deliberately **not** chained into `gen:all`: the generators
+  throw on malformed or missing vendor input, which is the normal state part-way through a re-pin,
+  and a chained clean would leave the working tree empty and the build broken. The gate does its own
+  delete rather than trusting the script under test.
+
 ### Changed
 
+- **The PS3.5 repeating-group bound is now derived from pinned normative documents rather than
+  transcribed (`DICOM-PS35-NOT-VENDORED`). No runtime impact: the emitted bound is identical and no
+  public API moves.** `deidentify()` expands PS3.15 Table E.1-1's `(50xx,xxxx)`, `(60xx,3000)` and
+  `(60xx,4000)` rows over the concrete groups PS3.5 section 7.6 admits, sixteen even groups per mask
+  (`5000`-`501E`, `6000`-`601E`), not 256. That bound is a safety limit in **both** directions, and it
+  was the last input here that lived as a **quotation in a source file** while PS3.6 and PS3.15 were
+  SHA-pinned and re-hashed before use, so the generator's guard could catch a new mask _prefix_ but
+  never a changed _bound_.
+  `vendor/nema/part05/` pins `part05.xml` (PS3.5 **2026c**) and `vendor/nema/part05-2004/` pins
+  `04_05pu.pdf`, both re-hashed as a precondition, and `scripts/generate-repeating-groups.ts` emits
+  `src/dictionary/generated/repeating-groups.ts`, which `src/dictionary/repeating-groups.ts`
+  re-exports. **Two documents, because the bound is split across two editions:** the current one
+  states the overlay bound normatively and excludes the odd `6001`-`601F`, but says nothing about the
+  curve bound, having retired curve encoding and delegated it to PS3.5-2004 by URL in section 7.6's
+  own Note. So the 2004 PDF is the authority the edition in force names, not a convenience copy. The
+  generator **proves that delegation** (section 7.6 must link exactly the vendored URL) instead of
+  assuming it, and because both editions state the overlay bound it **requires them to agree**.
+  `gen:repeating-groups` runs first in `gen:all`, since the Annex E generator expands family rows
+  through this bound.
+  **The values did not change; their falsifiability did.** Moving the overlay bound in either
+  vendored edition reds the cross-check, moving the curve bound in the 2004 edition moves the emitted
+  artifact (caught by the byte-identical regen gate), and removing the delegation link stops the 2004
+  document being used at all. Each was confirmed non-vacuous by disabling the guard and watching the
+  test fail. Reading a 2004 PDF needs a PDF reader; the one in that generator is deliberately minimal
+  (Node `zlib` only, one sentence recovered, matched against a precise expected shape) and is not to
+  be grown into a general parser. There is no staleness clock, for the same reasons as PS3.6 and
+  PS3.15.
+- **Recorded what the vendor pin is actually current against (`DICOM-DICT-CURRENCY`).**
+  `vendor/innolitics/README.md` gains a measured Currency section and drops the "re-pin monthly"
+  policy, which nothing ran and which measured the wrong thing. The pin is exactly current against
+  upstream: `git ls-remote` resolves `master` to the pinned SHA and all four vendored files are
+  byte-identical to upstream at it. Upstream itself is the stale link, having last changed
+  `attributes.json` on 2024-04-18 ("Update standard to rev2024b"), so the tag tables are grounded in
+  PS3.6 2024b while the current edition is PS3.6 2026c. The drift is now measured rather than
+  assumed, against the NEMA DocBook source for 2026c, and every non-additive difference is named in
+  that file. Figures on `86ab6c1` (`origin/main` at measurement time; this slice does not touch
+  `tags.ts` or `keywords.ts`): all **5,129** committed tags are shared with PS3.6 2026c, and across
+  them there are **zero VR differences**, zero name differences and zero VM differences, with 2
+  keyword and 2 retirement-status differences. **Every tag this dictionary carries is still in
+  PS3.6 2026c**; the drift is otherwise purely additive, 180 tags the standard has gained. VR is
+  what turns bytes into a value, so zero VR drift is the result that matters for reading a real
+  study. Nothing in the dictionary is hand-edited to close the gap: it is generated, and the
+  remaining differences are recorded for the next re-pin. One method note lives with the
+  measurement, because getting it wrong is easy and was got wrong on the first run: DICOM tag values
+  are hexadecimal and their case is not semantic, so tag keys must be compared case-insensitively.
+  PS3.6 prints them uniformly uppercase and `tags.ts` agrees on 1,540 of its 5,129 keys, but
+  `tags.ts` lowercases the 8 repeating-group keys whose trailing digits are hex letters and no
+  others, so a verbatim comparison mis-classifies exactly those 8.
+
 - **Removed the six em dashes the new gate found (`EMDASH-CONFORMANCE`).** Four tracked files, none
   of them markdown: `.github/CODEOWNERS` (2), `.github/workflows/release.yml` (2),
   `vendor/nema/SHA.txt` (1), and the npm `description` in `package.json` (1). The last is the only

package/README.md

@@ -210,6 +210,10 @@ deidentify(parseDicom(buf), {
 
 This is **metadata-level** de-identification. Pixel cleaning is out of scope: when a file carries burned-in annotation this layer cannot remove, you get a `DICOM_BURNED_IN_ANNOTATION_NOT_REMOVED` warning rather than a false sense of safety (pixel cleaning is deferred to `@cosyte/dicom-pixel`).
 
+The action table comes from NEMA's PS3.15 2026c DocBook, the normative publication of the standard, rather than from a third-party mirror of it, so the current edition's patient attributes are removed rather than quietly kept. That includes the three rows the standard states as a repeating-group mask rather than a single tag: `(50xx,xxxx)` Curve Data, `(60xx,3000)` Overlay Data and `(60xx,4000)` Overlay Comments are matched in every overlay or curve group the standard defines, removed, and named in the report with the mask that matched them. Overlay comments in particular are a common carrier for text typed onto a study, so a clean report on a file that still held them was worse than no report.
+
+The groups a mask covers are the sixteen even ones PS3.5 bounds it to (`6000`-`601E`, `5000`-`501E`), not any four hex digits. Reading `xx` as a wildcard would strip attributes the standard never marked, which is data loss on a call you asked to be conservative. That bound is read out of PS3.5 itself, pinned by SHA-256 the same way the action table is, rather than copied into the source by hand: the current edition states the overlay range, and the curve range comes from the 2004 edition its own note delegates to, with the two required to agree where they overlap.
+
 ### Bridge to FHIR / HL7 v2
 
 A common consulting ask is joining imaging to the rest of the record. The authoritative crosswalk is the FHIR [`ImagingStudy` "Mappings for DICOM"](https://build.fhir.org/imagingstudy-mappings.html) tab. The join keys a metadata parser must surface correctly: