dicom-dre 1.0.0.dev1__tar.gz → 1.0.0.dev3__tar.gz

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev3}/.devcontainer/devcontainer.json

@@ -16,7 +16,9 @@
         "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
         "extensions.verifySignature": false, // https://github.com/microsoft/vscode-remote-release/issues/9628
         "remote.autoForwardPorts": false,
-        "extensions.ignoreRecommendations": true
+        "extensions.ignoreRecommendations": true,
+        "git.autofetch": true,
+        "git.autofetchPeriod": 180
       },
       "extensions": [
         "bierner.markdown-mermaid",
@@ -43,6 +45,9 @@
     "ghcr.io/jsburckhardt/devcontainer-features/uv:1": {},
     "ghcr.io/devcontainers/features/github-cli:1": {}
   },
+  "mounts": [
+    "source=dicom-dre-home,target=/home/${localEnv:USER}${localEnv:USERNAME},type=volume"
+  ],
   "postCreateCommand": "bash ./.devcontainer/post-create.sh",
   "waitFor": "postCreateCommand"
 }

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev3}/.github/workflows/deploy.yml

@@ -23,15 +23,35 @@ on:
       - main
       - development
 
-# Prevent overlapping deploys on the same branch.
 concurrency:
   group: deploy-${{ github.ref_name }}
   cancel-in-progress: false
 
 jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    env:
+      # Exclude the docs group (a default group locally) from CI installs.
+      UV_NO_GROUP: docs
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run tests
+        run: uv run pytest
+
   version:
-    # Compute the next version without modifying the repository. Determines
-    # whether a release is warranted and the version/tag that would result.
+    needs: test
     runs-on: ubuntu-latest
     permissions:
       contents: read
@@ -138,9 +158,6 @@ jobs:
           path: ./dist/*.tar.gz
 
   release:
-    # First job that mutates the repository. Runs only after every distribution
-    # has built. Tags, commits the changelog, pushes, creates the GitHub
-    # Release, and uploads the built artifacts.
     needs: [version, build_wheels, build_sdist]
     if: needs.version.outputs.released == 'true'
     runs-on: ubuntu-latest

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev3}/.gitignore

@@ -17,6 +17,7 @@ docs/_build/
 
 # CFFI build scratch (generated by the JPEG-DCT build hook)
 src/dicom_dre/dicom_dre/
+src/dicom_dre/_jpeg_dct_accel.c
 
 # Virtual environments
 .venv/

dicom_dre-1.0.0.dev3/CHANGELOG.md

@@ -0,0 +1,23 @@
+# CHANGELOG
+
+<!-- version list -->
+
+## v1.0.0-dev.2 (2026-07-24)
+
+### Documentation
+
+- Initial rewrite from source repo
+  ([`80e26c9`](https://github.com/susom/dicom-dre/commit/80e26c99e5f270528dfa2ddb8551b0baf3fdc7b7))
+
+- Scaffolding
+  ([`c1db56a`](https://github.com/susom/dicom-dre/commit/c1db56a224050b8704f82ac99b857d6c0c43690f))
+
+### Performance Improvements
+
+- Redactor caching
+  ([`ce14989`](https://github.com/susom/dicom-dre/commit/ce14989512d0d688d6b726444c84f91cafc28757))
+
+
+## v1.0.0-dev.1 (2026-07-24)
+
+- Initial Release

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dicom-dre
-Version: 1.0.0.dev1
+Version: 1.0.0.dev3
 Summary: DICOM De-identification & Redaction Engine
 Project-URL: repository, https://github.com/susom/dicom-dre
 Author-email: Stanford <people@stanford.edu>
@@ -25,7 +25,7 @@ Description-Content-Type: text/markdown
 
 # DICOM-DRE
 
-A fast, deterministic DICOM de-identification and redaction engine.
+A fast, reproducible DICOM de-identification and redaction engine.
 
 `dicom-dre` removes protected health information (PHI) from DICOM instances in
 two places where it commonly hides:
@@ -39,22 +39,28 @@ two places where it commonly hides:
 - **Free-text metadata PHI.** Description fields that frequently carry PHI
   (`SeriesDescription`, `StudyDescription`, `ProtocolName`) are redacted
   token-by-token against an allowlist: any token not on the allowlist is masked,
-  while known clinical terms pass through.
+  while known clinical terms pass through. This approach works because the
+  distinct values in these short description fields are overwhelmingly derived
+  from structured entries in the source PACS, so a finite allowlist covers the
+  legitimate vocabulary. It is not intended for paragraphs of prose, general
+  clinical notes, or other entirely free-form fields with no discernible
+  pattern, where the allowlist model cannot bound the vocabulary or reliably
+  de-identify the text.
 
 Instance metadata is also scrubbed against a configurable de-identification
 profile, and instance/study/series UIDs are deterministically re-derived, so the
 same input plus the same parameters always yields the same output.
 
-## Deterministic by design
+## Reproducible by design
 
 The engine performs no hashing lookups, no network calls, and no randomization
 of its own. De-identification parameters (patient ID, accession number, UID
-root, salt, etc.) are consumed exactly as supplied. Given identical inputs and
-parameters, output is byte-reproducible. Callers are responsible for supplying
-already-hashed or already-mapped identifier values.
+root, salt, jitter, etc.) are consumed exactly as supplied. Given identical
+inputs and parameters, output is byte-for-byte reproducible. Callers are
+responsible for supplying already-hashed or already-mapped identifier values.
 
-This determinism is guarded by a regression test suite built from sampled DICOM
-studies. Each fixture records the technical matching tags and the expected
+This reproducibility is guarded by a regression test suite built from sampled
+DICOM studies. Each fixture records the technical matching tags and the expected
 catalog filtering and pixel-scrub decisions (with no PHI or pixel data), and the
 tests assert that the engine still reaches the recorded decision for every case,
 so catalog or profile changes that alter existing outcomes are caught.
@@ -151,13 +157,14 @@ site's vocabulary.
 ## Library usage
 
 ```python
-from dicom_dre import deidentify_paths, ProfileSpec
+from dicom_dre import DeidParameters, ProfileSpec, deidentify_paths
 
 for item in deidentify_paths(
     sources=["studies/"],
     output_dir="out/",
     recursive=True,
-    profile_spec=ProfileSpec(name="default", parameters={"PATIENT_ID": "TEST"}),
+    profile_spec=ProfileSpec(name="default"),
+    parameters=DeidParameters(patient_id="TEST"),
 ):
     print(item.input_file, item.result.outcome)
 ```

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev3}/README.md

@@ -1,6 +1,6 @@
 # DICOM-DRE
 
-A fast, deterministic DICOM de-identification and redaction engine.
+A fast, reproducible DICOM de-identification and redaction engine.
 
 `dicom-dre` removes protected health information (PHI) from DICOM instances in
 two places where it commonly hides:
@@ -14,22 +14,28 @@ two places where it commonly hides:
 - **Free-text metadata PHI.** Description fields that frequently carry PHI
   (`SeriesDescription`, `StudyDescription`, `ProtocolName`) are redacted
   token-by-token against an allowlist: any token not on the allowlist is masked,
-  while known clinical terms pass through.
+  while known clinical terms pass through. This approach works because the
+  distinct values in these short description fields are overwhelmingly derived
+  from structured entries in the source PACS, so a finite allowlist covers the
+  legitimate vocabulary. It is not intended for paragraphs of prose, general
+  clinical notes, or other entirely free-form fields with no discernible
+  pattern, where the allowlist model cannot bound the vocabulary or reliably
+  de-identify the text.
 
 Instance metadata is also scrubbed against a configurable de-identification
 profile, and instance/study/series UIDs are deterministically re-derived, so the
 same input plus the same parameters always yields the same output.
 
-## Deterministic by design
+## Reproducible by design
 
 The engine performs no hashing lookups, no network calls, and no randomization
 of its own. De-identification parameters (patient ID, accession number, UID
-root, salt, etc.) are consumed exactly as supplied. Given identical inputs and
-parameters, output is byte-reproducible. Callers are responsible for supplying
-already-hashed or already-mapped identifier values.
+root, salt, jitter, etc.) are consumed exactly as supplied. Given identical
+inputs and parameters, output is byte-for-byte reproducible. Callers are
+responsible for supplying already-hashed or already-mapped identifier values.
 
-This determinism is guarded by a regression test suite built from sampled DICOM
-studies. Each fixture records the technical matching tags and the expected
+This reproducibility is guarded by a regression test suite built from sampled
+DICOM studies. Each fixture records the technical matching tags and the expected
 catalog filtering and pixel-scrub decisions (with no PHI or pixel data), and the
 tests assert that the engine still reaches the recorded decision for every case,
 so catalog or profile changes that alter existing outcomes are caught.
@@ -126,13 +132,14 @@ site's vocabulary.
 ## Library usage
 
 ```python
-from dicom_dre import deidentify_paths, ProfileSpec
+from dicom_dre import DeidParameters, ProfileSpec, deidentify_paths
 
 for item in deidentify_paths(
     sources=["studies/"],
     output_dir="out/",
     recursive=True,
-    profile_spec=ProfileSpec(name="default", parameters={"PATIENT_ID": "TEST"}),
+    profile_spec=ProfileSpec(name="default"),
+    parameters=DeidParameters(patient_id="TEST"),
 ):
     print(item.input_file, item.result.outcome)
 ```

dicom_dre-1.0.0.dev3/docs/about/changelog.md

@@ -0,0 +1,8 @@
+# Changelog
+
+python-semantic-release generates the changelog at the repository root from
+Conventional Commit messages.
+
+```{include} ../../CHANGELOG.md
+:start-line: 2
+```

dicom_dre-1.0.0.dev3/docs/about/history.md

@@ -0,0 +1,46 @@
+# History
+
+`dicom-dre` grew out of a de-identification workflow built on the MIRC CTP
+(Clinical Trial Processor) engine. This page records that background. It is not
+a reference for configuring the engine; the device catalog and profiles
+documented elsewhere in these docs have replaced the behavior described here.
+
+## From CTP scripts to a device catalog
+
+The original workflow controlled DICOM image handling with two CTP script files
+and an XML metadata template:
+
+- **filter.script** decided which images to accept for processing. Images
+  matching a filter rule passed through; the engine quarantined images matching
+  no rule.
+- **pixel.script** defined rectangular regions to blank for images with
+  burned-in PHI annotations.
+
+Both scripts used a Java boolean expression syntax over DICOM attributes. Filter
+rules keyed on manufacturer, model, modality, software version, and image
+dimensions. Pixel rules paired a matching signature with one or more
+`(x, y, width, height)` scrub regions.
+
+Two problems motivated the rewrite:
+
+- The filter and pixel scripts encoded the same device signatures twice, once to
+  decide acceptance and again to place scrub regions, so the two files drifted
+  apart as maintainers added devices.
+- The expression syntax was code rather than data, which made the rule set hard
+  to inspect, test, and validate.
+
+`dicom-dre` replaces both script files with a single Python device catalog. Each
+catalog entry describes one device and binds its acquisition variants directly
+to the pixel regions to blank, so each device defines its filter decision and
+its scrub regions together. See [Device catalog](../concepts/device-catalog.md).
+
+## From an XML template to profiles
+
+De-identification profiles now express the metadata handling that a CTP
+anonymizer template previously expressed. See
+[Profiles](../concepts/profiles.md) for the profiles shipped in this repository.
+
+## Acknowledgement
+
+This project would not have been possible without years of working with and
+learning from the MIRC CTP engine.

dicom_dre-1.0.0.dev3/docs/about/provenance.md

@@ -0,0 +1,51 @@
+# Provenance and Portability
+
+The bundled device catalog and free-text allowlist come from studies on a single
+PACS at one medical research center. The catalog's device rules, pixel scrub
+regions, and allowlisted vocabulary reflect the scanner fleet and reporting
+conventions seen there. They are unlikely to be complete or correct for a
+different site.
+
+## What is site-specific
+
+The following ship with `dicom-dre` but encode assumptions about one site:
+
+- **Device rules.** Each catalog entry matches a specific device by
+  manufacturer, model, modality, software version, and image dimensions. Devices
+  absent from the observed fleet have no entry, and the engine will not
+  recognize them.
+- **Pixel scrub regions.** Blanking coordinates are fixed per device and
+  acquisition variant. If a device burns in text at a different location, or a
+  firmware revision moves the overlay, the engine will not scrub it correctly.
+- **Free-text allowlist.** The allowlisted vocabulary reflects the reporting
+  conventions and clinical terms seen in one site's description fields. The
+  engine masks local vocabulary that is absent from the allowlist; local PHI
+  patterns the allowlist does not anticipate may pass through.
+
+## Consequences of an unvalidated deployment
+
+- **Unmatched or mismatched devices** do not have their burned-in text blanked.
+  The engine filters an instance from an unknown device rather than emitting it
+  with unscrubbed PHI, but a partially matching rule with wrong coordinates can
+  emit an instance whose overlay is not fully covered.
+- **Over-redaction** masks legitimate terms when the allowlist omits vocabulary
+  used at your site, degrading the usefulness of description fields.
+- **Under-redaction** leaks PHI when tokens that should be masked resemble
+  allowlisted terms.
+
+## Before relying on `dicom-dre` elsewhere
+
+- Validate pixel scrubbing against your own devices. Confirm that each device
+  you process either matches a catalog entry with correct scrub regions or that
+  the engine intentionally filters it.
+- Review and extend the allowlist for your local vocabulary to reduce
+  over-redaction and under-redaction.
+- Run the regression suite after any catalog or allowlist change to confirm that
+  existing decisions still hold.
+
+The [Local validation](../guides/local-validation.md) guide describes these
+steps in detail. The [Extending the catalog](../guides/extending-the-catalog.md)
+guide covers adding new device entries.
+
+Treat the shipped catalog and allowlist as a starting point that requires local
+validation, not as a turnkey configuration.

dicom_dre-1.0.0.dev3/docs/about/testing.md

@@ -0,0 +1,62 @@
+# Testing
+
+The project uses pytest. Tests live in `tests/unit/` and mirror the structure of
+`src/dicom_dre/`.
+
+## Running the test suite
+
+Run all tests:
+
+```bash
+uv run pytest
+```
+
+Or use the justfile target:
+
+```bash
+just test
+```
+
+Run a single test file or pattern:
+
+```bash
+just test-one tests/unit/test_pipeline_preservation.py
+```
+
+## Coverage
+
+Generate a coverage report with per-line detail for uncovered lines:
+
+```bash
+just cov
+```
+
+This runs `uv run pytest --cov=src/dicom_dre --cov-report=term-missing`.
+
+## Pure-Python fallback
+
+The JPEG DCT-domain scrubber has an optional compiled C accelerator. To confirm
+that the pure-Python fallback works when the extension is absent, run:
+
+```bash
+just test-fallback
+```
+
+The target temporarily moves any compiled `_jpeg_dct_accel*.so` out of the
+package, runs the suite, and restores the extension afterward.
+
+## Regression fixtures
+
+`tests/unit/test_catalog_regression_fixtures.py` asserts that the catalog still
+reaches the recorded filtering and pixel-scrub decision for each sampled case.
+The fixtures record only the technical matching tags and the expected decision,
+with no PHI or pixel data. Run this suite after any catalog or profile change so
+that it catches alterations to existing outcomes.
+
+## Writing tests
+
+- Place tests in `tests/unit/`, mirroring the source structure.
+- Use descriptive test names that state what each test verifies.
+- Mock external services in unit tests, especially any that would make network
+  calls.
+- Use fixtures for common setup.

dicom_dre-1.0.0.dev3/docs/concepts/architecture.md

@@ -0,0 +1,153 @@
+# Architecture
+
+`dicom-dre` is a pure-Python DICOM de-identification engine. It filters images,
+blanks burned-in pixel regions, and rewrites metadata tags. For each input file
+it produces one of three outcomes by running three stages in sequence.
+
+## Overview
+
+For each input DICOM file the engine runs three stages:
+
+1. **Filter** — a device catalog decides whether to keep or reject the image.
+2. **Pixel scrub** — the engine blanks the burned-in text regions the catalog
+   identified.
+3. **Metadata** — the engine rewrites DICOM tags: it removes PHI, re-derives
+   UIDs, and shifts or keeps dates.
+
+Each run returns a {py:class}`~dicom_dre.result.DeidentifyResult` whose `outcome`
+is one of three {py:class}`~dicom_dre.result.Outcome` values:
+
+| Outcome | Cause |
+|---------|-------|
+| `DEIDENTIFIED` | The file passed the filter, and the engine wrote it out. |
+| `FILTERED` | A device or exclusion rule rejected the file; the result carries the matched reason. |
+| `QUARANTINED` | Processing raised an exception; the result carries the error text. |
+
+## Pipeline
+
+Every de-identification call enters through
+{py:func}`dicom_dre.pipeline.deidentify_file`, which composes the three stages and
+returns a `DeidentifyResult`.
+
+```{mermaid}
+flowchart LR
+    A["PHI DICOM"] --> F{"Filter<br>catalog.evaluate()"}
+    F -->|deny| X["FILTERED<br>(+ reason)"]
+    F -->|allow / scrub| S["Pixel scrub<br>blank_regions()"]
+    S --> M["Metadata<br>profile.apply()"]
+    M --> O["DEIDENTIFIED"]
+    F -.->|exception| Q["QUARANTINED"]
+```
+
+An exception raised in any stage does not propagate to the caller.
+`deidentify_file` catches it and returns a `QUARANTINED` result instead.
+
+## Stages
+
+### Filter
+
+`catalog.evaluate(DicomTags.from_file(path))` returns a
+{py:class}`~dicom_dre.catalog.CatalogDecision` with an action (`allow`, `deny`, or
+`scrub`), a human-readable reason, and a list of scrub regions. A `deny` action
+stops the pipeline and produces a `FILTERED` result whose `filter_reason` is the
+matched rule. Catalog evaluation reads tags directly with pydicom.
+
+See [Device Catalog](device-catalog.md).
+
+### Pixel scrub
+
+When the decision carries scrub regions,
+{py:func}`dicom_dre.pixel_blanker.blank_regions` blanks them. The engine edits
+JPEG Baseline images directly in the compressed bitstream with no re-encoding.
+For all other transfer syntaxes, it decodes with pydicom and numpy, blanks the
+regions, and writes uncompressed Explicit VR Little Endian.
+
+See [JPEG DCT Scrubbing](jpeg-dct.md).
+
+### Metadata
+
+The bound {py:class}`~dicom_dre.profile.DeidProfile` runs its tag rules on the
+dataset with `DeidProfile.apply(ds, params)`, where `params` is a
+{py:class}`~dicom_dre.parameters.DeidParameters` carrying the per-patient values.
+Rules are small closures defined in
+{py:mod}`dicom_dre.actions` (for example `remove`, `empty`, `hash_uid`,
+`jitter_date`, `keep`). The profile factories in {py:mod}`dicom_dre.profiles`
+compose the tag sets each profile applies them to.
+
+Each factory in `actions.py` closes over its build-time arguments at
+profile-construction time and returns a
+`(Dataset, BaseTag, DeidParameters) -> None` callable that reads per-patient
+values from the supplied parameters. The available actions:
+
+| Action | Effect |
+|--------|--------|
+| `keep()` | Leave the element unchanged. |
+| `remove()` | Delete the element from the dataset. |
+| `empty()` | Replace the value with a zero-length value (empty list for `SQ`). |
+| `set_value(value, create_if_missing=False)` | Replace the value with a literal string, optionally creating the element when absent. |
+| `set_param(field, default=None, fallback_field=None, create_if_missing=False)` | Write a per-patient value read from `DeidParameters`, with fallbacks. |
+| `hash_uid(root, use_study_salt=False)` | Re-derive a UID as an MD5 hash under `root`, optionally salted with the study identifier. |
+| `jitter_date()` | Shift the `DA`/`DT` date component forward by `params.jitter` days, preserving any time/timezone remainder. |
+| `append_value(text, create_if_missing=False)` | Append `text` to a multi-valued element using `\` as separator. |
+| `cap_age(threshold, replacement)` | Replace an `AS` age when its numeric part exceeds `threshold`. |
+| `if_exists(inner)` | Apply `inner` only when the element is present. |
+| `process()` | Mark an `SQ` element for recursive rule application by `DeidProfile`. |
+
+Profiles compose these actions rather than parsing a script.
+
+### Device-scoped private-tag preservation
+
+By default the engine removes every private data element that has no explicit
+tag rule. A device rule may name a small, reviewer-approved set of private
+elements to retain verbatim through its `preserved_private_tags` field (a tuple
+of `PrivateTagSpec`). When the catalog decision carries specs, the pipeline
+attaches them to the `DeidProfile`. At apply time, `DeidProfile.apply(ds, params)`
+resolves each spec's private-creator block (the block is runtime-assigned, not
+fixed) and keeps both the resolved data elements and their creator element. The
+engine removes all other private elements as usual. When preservation is active,
+the engine also stamps the De-identification Method Code Sequence `(0012,0064)`
+with the Retain Safe Private Option. It emits the sequence only for files that
+actually preserve private tags.
+
+See [Device Catalog](device-catalog.md#preserved-private-tags) and
+[Profiles](profiles.md).
+
+## Profiles
+
+The `default`, `lds`, `lds-no-dob`, and `pixels-only` profiles each map to a
+factory function in {py:mod}`dicom_dre.profiles`. A profile name selects which
+tags the engine keeps, removes, or shifts. [Profiles](profiles.md) describes the
+behavior of each profile.
+
+## Regression validation
+
+A regression fixture suite guards catalog and metadata behavior against known
+inputs. Each fixture pins the expected filter decision, scrub decision, scrub
+regions, and de-identified metadata for a representative instance, so the test
+run catches a change that alters any of them. The fixtures live in
+`tests/unit/test_catalog_regression_fixtures.py`.
+
+See [Testing](../about/testing.md).
+
+## Extending
+
+- To allow or deny a new device, or to blank a new pixel region, add a `device()`
+  entry to {py:mod}`dicom_dre.default_catalog`. See
+  [Extending the catalog](../guides/extending-the-catalog.md).
+- To change a tag rule or add a profile, compose actions in
+  {py:mod}`dicom_dre.profiles`, deriving from an existing profile with
+  `dataclasses.replace`.
+
+## Module reference
+
+| Module | Responsibility |
+|--------|----------------|
+| {py:mod}`dicom_dre.pipeline` | Orchestrator: composes the three stages into a `DeidentifyResult` |
+| {py:mod}`dicom_dre.result` | `DeidentifyResult` and `Outcome` result types |
+| {py:mod}`dicom_dre.profile` | `DeidProfile.apply()` metadata rewrite |
+| {py:mod}`dicom_dre.actions` | Tag-action factories (`remove`, `hash_uid`, `jitter_date`, ...) |
+| {py:mod}`dicom_dre.profiles` | `default`, `lds`, `lds-no-dob`, `pixels-only` factories and the profile builder |
+| {py:mod}`dicom_dre.catalog` | Catalog engine: `DeviceCatalog`, `CatalogDecision`, `DicomTags` |
+| {py:mod}`dicom_dre.default_catalog` | Device and exclusion rule data |
+| {py:mod}`dicom_dre.pixel_blanker` | `blank_regions()` pixel scrub dispatch |
+| {py:mod}`dicom_dre.jpeg_dct_scrubber` | DCT-domain JPEG blanking and C accelerator |