dicom-dre 1.0.0.dev1__tar.gz → 1.0.0.dev2__tar.gz

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev2}/.devcontainer/devcontainer.json

@@ -16,7 +16,9 @@
         "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
         "extensions.verifySignature": false, // https://github.com/microsoft/vscode-remote-release/issues/9628
         "remote.autoForwardPorts": false,
-        "extensions.ignoreRecommendations": true
+        "extensions.ignoreRecommendations": true,
+        "git.autofetch": true,
+        "git.autofetchPeriod": 180
       },
       "extensions": [
         "bierner.markdown-mermaid",
@@ -43,6 +45,9 @@
     "ghcr.io/jsburckhardt/devcontainer-features/uv:1": {},
     "ghcr.io/devcontainers/features/github-cli:1": {}
   },
+  "mounts": [
+    "source=dicom-dre-home,target=/home/${localEnv:USER}${localEnv:USERNAME},type=volume"
+  ],
   "postCreateCommand": "bash ./.devcontainer/post-create.sh",
   "waitFor": "postCreateCommand"
 }

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev2}/.github/workflows/deploy.yml

@@ -23,15 +23,35 @@ on:
       - main
       - development
 
-# Prevent overlapping deploys on the same branch.
 concurrency:
   group: deploy-${{ github.ref_name }}
   cancel-in-progress: false
 
 jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    env:
+      # Exclude the docs group (a default group locally) from CI installs.
+      UV_NO_GROUP: docs
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run tests
+        run: uv run pytest
+
   version:
-    # Compute the next version without modifying the repository. Determines
-    # whether a release is warranted and the version/tag that would result.
+    needs: test
     runs-on: ubuntu-latest
     permissions:
       contents: read
@@ -138,9 +158,6 @@ jobs:
           path: ./dist/*.tar.gz
 
   release:
-    # First job that mutates the repository. Runs only after every distribution
-    # has built. Tags, commits the changelog, pushes, creates the GitHub
-    # Release, and uploads the built artifacts.
     needs: [version, build_wheels, build_sdist]
     if: needs.version.outputs.released == 'true'
     runs-on: ubuntu-latest

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev2}/.gitignore

@@ -17,6 +17,7 @@ docs/_build/
 
 # CFFI build scratch (generated by the JPEG-DCT build hook)
 src/dicom_dre/dicom_dre/
+src/dicom_dre/_jpeg_dct_accel.c
 
 # Virtual environments
 .venv/

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev2}/CHANGELOG.md

@@ -5,7 +5,3 @@
 ## v1.0.0-dev.1 (2026-07-24)
 
 - Initial Release
-
-## v1.0.0-dev.1 (2026-07-24)
-
-- Initial Release

{dicom_dre-1.0.0.dev1 → dicom_dre-1.0.0.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dicom-dre
-Version: 1.0.0.dev1
+Version: 1.0.0.dev2
 Summary: DICOM De-identification & Redaction Engine
 Project-URL: repository, https://github.com/susom/dicom-dre
 Author-email: Stanford <people@stanford.edu>

dicom_dre-1.0.0.dev2/docs/about/changelog.md

@@ -0,0 +1,8 @@
+# Changelog
+
+The changelog is maintained at the repository root and generated by
+python-semantic-release from Conventional Commit messages.
+
+```{include} ../../CHANGELOG.md
+:start-line: 2
+```

dicom_dre-1.0.0.dev2/docs/about/history.md

@@ -0,0 +1,46 @@
+# History
+
+`dicom-dre` grew out of a de-identification workflow built on the MIRC CTP
+(Clinical Trial Processor) engine. This page records that background. It is not
+a reference for configuring the engine; the behavior described here has been
+replaced by the device catalog and profiles documented elsewhere in these docs.
+
+## From CTP scripts to a device catalog
+
+The original workflow controlled DICOM image handling with two CTP script files
+and an XML metadata template:
+
+- **filter.script** decided which images were accepted for processing. Images
+  matching a filter rule passed through; images matching no rule were
+  quarantined.
+- **pixel.script** defined rectangular regions to blank for images with
+  burned-in PHI annotations.
+
+Both scripts used a Java boolean expression syntax over DICOM attributes, with
+filter rules keyed on manufacturer, model, modality, software version, and image
+dimensions, and pixel rules pairing a matching signature with one or more
+`(x, y, width, height)` scrub regions.
+
+Two problems motivated the rewrite:
+
+- The filter and pixel scripts encoded the same device signatures twice, once to
+  decide acceptance and again to place scrub regions, so the two files drifted
+  apart as devices were added.
+- The expression syntax was code rather than data, which made the rule set hard
+  to inspect, test, and validate.
+
+`dicom-dre` replaces both script files with a single Python device catalog. Each
+catalog entry describes one device and binds its acquisition variants directly
+to the pixel regions that must be blanked, so a device's filter decision and its
+scrub regions are defined together. See [Device catalog](../concepts/device-catalog.md).
+
+## From an XML template to profiles
+
+Metadata handling that was previously expressed in a CTP anonymizer template is
+now expressed as de-identification profiles. See [Profiles](../concepts/profiles.md)
+for the profiles shipped in this repository.
+
+## Acknowledgement
+
+This project would not have been possible without years of working with and
+learning from the MIRC CTP engine.

dicom_dre-1.0.0.dev2/docs/about/provenance.md

@@ -0,0 +1,50 @@
+# Provenance and Portability
+
+The bundled device catalog and free-text allowlist were derived from studies on
+a single PACS at one medical research center. The catalog's device rules, pixel
+scrub regions, and allowlisted vocabulary reflect the scanner fleet and
+reporting conventions observed there. They are unlikely to be complete or
+correct for a different site.
+
+## What is site-specific
+
+The following ship with `dicom-dre` but encode assumptions about one site:
+
+- **Device rules.** Each catalog entry matches a specific device by
+  manufacturer, model, modality, software version, and image dimensions. Devices
+  absent from the observed fleet have no entry and will not be recognized.
+- **Pixel scrub regions.** Blanking coordinates are fixed per device and
+  acquisition variant. A device that burns in text at a different location, or a
+  firmware revision that moves the overlay, will not be scrubbed correctly.
+- **Free-text allowlist.** The allowlisted vocabulary reflects the reporting
+  conventions and clinical terms seen in one site's description fields. Local
+  vocabulary that is absent from the allowlist is masked; local PHI patterns not
+  anticipated by the allowlist may pass through.
+
+## Consequences of an unvalidated deployment
+
+- **Unmatched or mismatched devices** do not have their burned-in text blanked.
+  An instance from an unknown device is filtered rather than emitted with
+  unscrubbed PHI, but a partially matching rule with wrong coordinates can emit
+  an instance whose overlay is not fully covered.
+- **Over-redaction** masks legitimate terms when the allowlist omits vocabulary
+  used at your site, degrading the usefulness of description fields.
+- **Under-redaction** leaks PHI when tokens that should be masked resemble
+  allowlisted terms.
+
+## Before relying on `dicom-dre` elsewhere
+
+- Validate pixel scrubbing against your own devices. Confirm that each device
+  you process either matches a catalog entry with correct scrub regions or is
+  intentionally filtered.
+- Review and extend the allowlist for your local vocabulary to reduce
+  over-redaction and under-redaction.
+- Run the regression suite after any catalog or allowlist change to confirm that
+  existing decisions are preserved.
+
+The [Local validation](../guides/local-validation.md) guide describes these
+steps in detail. The [Extending the catalog](../guides/extending-the-catalog.md)
+guide covers adding new device entries.
+
+Treat the shipped catalog and allowlist as a starting point that requires local
+validation, not as a turnkey configuration.

dicom_dre-1.0.0.dev2/docs/about/testing.md

@@ -0,0 +1,62 @@
+# Testing
+
+The project uses pytest. Tests live in `tests/unit/` and mirror the structure of
+`src/dicom_dre/`.
+
+## Running the test suite
+
+Run all tests:
+
+```bash
+uv run pytest
+```
+
+Or use the justfile target:
+
+```bash
+just test
+```
+
+Run a single test file or pattern:
+
+```bash
+just test-one tests/unit/test_pipeline_preservation.py
+```
+
+## Coverage
+
+Generate a coverage report with per-line detail for uncovered lines:
+
+```bash
+just cov
+```
+
+This runs `uv run pytest --cov=src/dicom_dre --cov-report=term-missing`.
+
+## Pure-Python fallback
+
+The JPEG DCT-domain scrubber has an optional compiled C accelerator. To confirm
+that the pure-Python fallback works when the extension is absent, run:
+
+```bash
+just test-fallback
+```
+
+The target temporarily moves any compiled `_jpeg_dct_accel*.so` out of the
+package, runs the suite, and restores the extension afterward.
+
+## Regression fixtures
+
+`tests/unit/test_catalog_regression_fixtures.py` asserts that the catalog still
+reaches the recorded filtering and pixel-scrub decision for each sampled case.
+The fixtures record only the technical matching tags and the expected decision,
+with no PHI or pixel data. Run this suite after any catalog or profile change so
+that alterations to existing outcomes are caught.
+
+## Writing tests
+
+- Place tests in `tests/unit/`, mirroring the source structure.
+- Use descriptive test names that state what is being verified.
+- Mock external services in unit tests, especially any that would make network
+  calls.
+- Use fixtures for common setup.

dicom_dre-1.0.0.dev2/docs/concepts/architecture.md

@@ -0,0 +1,149 @@
+# Architecture
+
+`dicom-dre` is a pure-Python DICOM de-identification engine. It filters images,
+blanks burned-in pixel regions, and rewrites metadata tags. For each input file
+it produces one of three outcomes by running three stages in sequence.
+
+## Overview
+
+For each input DICOM file the engine runs three stages:
+
+1. **Filter** — a device catalog decides whether the image is kept or rejected.
+2. **Pixel scrub** — burned-in text regions identified by the catalog are
+   blanked.
+3. **Metadata** — DICOM tags are rewritten (PHI removed, UIDs re-derived, dates
+   shifted or kept).
+
+Each run returns a {py:class}`~dicom_dre.result.DeidentifyResult` whose `outcome`
+is one of three {py:class}`~dicom_dre.result.Outcome` values:
+
+| Outcome | Cause |
+|---------|-------|
+| `DEIDENTIFIED` | The file passed the filter and was written out. |
+| `FILTERED` | A device or exclusion rule rejected the file; the result carries the matched reason. |
+| `QUARANTINED` | Processing raised an exception; the result carries the error text. |
+
+## Pipeline
+
+Every de-identification call enters through
+{py:func}`dicom_dre.pipeline.deidentify_file`, which composes the three stages and
+returns a `DeidentifyResult`.
+
+```{mermaid}
+flowchart LR
+    A["PHI DICOM"] --> F{"Filter<br>catalog.evaluate()"}
+    F -->|deny| X["FILTERED<br>(+ reason)"]
+    F -->|allow / scrub| S["Pixel scrub<br>blank_regions()"]
+    S --> M["Metadata<br>profile.apply()"]
+    M --> O["DEIDENTIFIED"]
+    F -.->|exception| Q["QUARANTINED"]
+```
+
+An exception raised in any stage is caught by `deidentify_file` and returned as
+a `QUARANTINED` result rather than propagating to the caller.
+
+## Stages
+
+### Filter
+
+`catalog.evaluate(DicomTags.from_file(path))` returns a
+{py:class}`~dicom_dre.catalog.CatalogDecision` with an action (`allow`, `deny`, or
+`scrub`), a human-readable reason, and a list of scrub regions. A `deny` action
+stops the pipeline and produces a `FILTERED` result whose `filter_reason` is the
+matched rule. Catalog evaluation reads tags directly with pydicom.
+
+See [Device Catalog](device-catalog.md).
+
+### Pixel scrub
+
+When the decision carries scrub regions,
+{py:func}`dicom_dre.pixel_blanker.blank_regions` blanks them. JPEG Baseline images
+are edited directly in the compressed bitstream with no re-encoding; all other
+transfer syntaxes are decoded with pydicom and numpy, blanked, and written as
+uncompressed Explicit VR Little Endian.
+
+See [JPEG DCT Scrubbing](jpeg-dct.md).
+
+### Metadata
+
+The bound {py:class}`~dicom_dre.profile.DeidProfile` runs its tag rules on the
+dataset with `DeidProfile.apply(ds)`. Rules are small closures defined in
+{py:mod}`dicom_dre.actions` (for example `remove`, `empty`, `hash_uid`,
+`jitter_date`, `keep`); the tag sets each profile applies them to are composed
+by the profile factories in {py:mod}`dicom_dre.profiles`.
+
+Each factory in `actions.py` closes over its arguments at profile-construction
+time and returns a `(Dataset, BaseTag) -> None` callable. The available actions:
+
+| Action | Effect |
+|--------|--------|
+| `keep()` | Leave the element unchanged. |
+| `remove()` | Delete the element from the dataset. |
+| `empty()` | Replace the value with a zero-length value (empty list for `SQ`). |
+| `set_value(value, create_if_missing=False)` | Replace the value with a literal string, optionally creating the element when absent. |
+| `redact_text(redactor)` | Redact a free-text element token-by-token against an allowlist. |
+| `hash_uid(root, salt="")` | Re-derive a UID as an MD5 hash under `root`, optionally salted. |
+| `jitter_date(days)` | Shift the `DA`/`DT` date component forward by `days`, preserving any time/timezone remainder. |
+| `append_value(text, create_if_missing=False)` | Append `text` to a multi-valued element using `\` as separator. |
+| `cap_age(threshold, replacement)` | Replace an `AS` age when its numeric part exceeds `threshold`. |
+| `if_exists(inner)` | Apply `inner` only when the element is present. |
+| `process()` | Mark an `SQ` element for recursive rule application by `DeidProfile`. |
+
+Profiles compose these actions rather than parsing a script.
+
+### Device-scoped private-tag preservation
+
+By default the engine removes every private data element that has no explicit
+tag rule. A device rule may name a small, reviewer-approved set of private
+elements to retain verbatim through its `preserved_private_tags` field (a tuple
+of `PrivateTagSpec`). When the catalog decision carries specs, the pipeline
+attaches them to the `DeidProfile`, and `DeidProfile.apply(ds)` resolves each
+spec's private-creator block at apply time (the block is runtime-assigned, not
+fixed) and keeps both the resolved data elements and their creator element. All
+other private elements are removed as usual. When preservation is active the
+engine also stamps the De-identification Method Code Sequence `(0012,0064)` with
+the Retain Safe Private Option; the sequence is emitted only for files that
+actually preserve private tags.
+
+See [Device Catalog](device-catalog.md#preserved-private-tags) and
+[Profiles](profiles.md).
+
+## Profiles
+
+The `default`, `lds`, `lds-no-dob`, and `pixels-only` profiles each map to a
+factory function in {py:mod}`dicom_dre.profiles`. A profile name selects which tags
+are kept, removed, or shifted. The behavior of each profile is described in
+[Profiles](profiles.md).
+
+## Regression validation
+
+A regression fixture suite guards catalog and metadata behavior against known
+inputs. Each fixture pins the expected filter decision, scrub decision, scrub
+regions, and de-identified metadata for a representative instance, so a change
+that alters any of them is caught in the test run. The fixtures live in
+`tests/unit/test_catalog_regression_fixtures.py`.
+
+See [Testing](../about/testing.md).
+
+## Extending
+
+- To allow or deny a new device, or to blank a new pixel region, add a `device()`
+  entry to {py:mod}`dicom_dre.default_catalog`. See
+  [Extending the catalog](../guides/extending-the-catalog.md).
+- To change a tag rule or add a profile, compose actions in
+  {py:mod}`dicom_dre.profiles`, deriving from an existing profile with
+  `dataclasses.replace`.
+
+## Module reference
+
+| Module | Responsibility |
+|--------|----------------|
+| {py:mod}`dicom_dre.pipeline` | Orchestrator: composes the three stages into a `DeidentifyResult` |
+| {py:mod}`dicom_dre.result` | `DeidentifyResult` and `Outcome` result types |
+| {py:mod}`dicom_dre.profile` | `DeidProfile.apply()` metadata rewrite |
+| {py:mod}`dicom_dre.actions` | Tag-action factories (`remove`, `hash_uid`, `jitter_date`, ...) |
+| {py:mod}`dicom_dre.profiles` | `default`, `lds`, `lds-no-dob`, `pixels-only` factories and the profile builder |
+| {py:mod}`dicom_dre.catalog` | Catalog engine: `DeviceCatalog`, `CatalogDecision`, `DicomTags` |
+| {py:mod}`dicom_dre.default_catalog` | Device and exclusion rule data |
+| {py:mod}`dicom_dre.pixel_blanker` | `blank_regions()` pixel scrub dispatch |
+| {py:mod}`dicom_dre.jpeg_dct_scrubber` | DCT-domain JPEG blanking and C accelerator |

dicom_dre-1.0.0.dev2/docs/concepts/determinism.md

@@ -0,0 +1,33 @@
+# Determinism
+
+`dicom-dre` is deterministic by design. The engine performs no hashing lookups,
+no network calls, and no randomization of its own. De-identification parameters
+(patient ID, accession number, UID root, salt, jitter, and so on) are consumed
+exactly as supplied. Given identical inputs and parameters, the output is
+byte-reproducible.
+
+## What the caller supplies
+
+Because the engine does no lookups, the caller is responsible for supplying
+already-mapped identifier values:
+
+- Replacement identifiers (patient ID, accession number) are written verbatim.
+- UID re-derivation is a pure function of the UID root and salt passed in; the
+  same source UID plus the same root and salt always yields the same output UID.
+- Date shifting uses the caller-supplied jitter value, not a random draw.
+- Free-text description values, when provided, are written verbatim; when
+  omitted, redaction is a pure function of the field content and the allowlist.
+
+This keeps identifier mapping, salt management, and any cross-study consistency
+under the caller's control rather than hidden inside the engine. See
+[De-identification Profiles](profiles.md) and [Text redaction](text-redaction.md).
+
+## Regression guard
+
+Determinism is guarded by a regression test suite built from sampled DICOM
+studies. Each fixture records the technical matching tags and the expected
+catalog filtering and pixel-scrub decisions, with no PHI or pixel data, and the
+tests assert that the engine still reaches the recorded decision for every case.
+Catalog or profile changes that alter existing outcomes are caught by this
+suite. See [Testing](../about/testing.md) and
+[Local validation](../guides/local-validation.md).