dsimaging-admin 0.2.0__tar.gz → 0.3.0__tar.gz

dsimaging_admin-0.3.0/IMPROVEMENTS.md

@@ -0,0 +1,244 @@
+# IMPROVEMENTS
+
+Candidate improvements to make `dsimaging-admin` a complete administration tool
+for `dsimaging-store`, beyond the happy-path coverage shipped in **v0.2.0**.
+
+This is a roadmap, not a backlog. Items are prioritised by how much real
+operator pain they remove on the current `dsImaging` / `dsHPC` deployment.
+
+## Current coverage (v0.2.0)
+
+| Command   | What it does                                                           |
+|-----------|------------------------------------------------------------------------|
+| `init`    | Write `~/.dsimaging.yaml`.                                             |
+| `publish` | Local dir -> S3 (upload + indexes + manifest, optional Opal register). |
+| `rescan`  | Recompute indexes / manifest from current S3 contents.                 |
+| `list`    | Enumerate datasets; `published` if `manifest.yaml` exists.             |
+| `doctor`  | Check S3 connectivity, bucket existence, versioning, dataset count.    |
+
+Two structural observations that motivate most of what follows:
+
+1. The CLI does **not** talk to the `dsimaging-store` controller
+   (`:8080/health`, `:8080/datasets`). All status comes from S3 directly.
+2. `cli.py::_write_dataset_artifacts` and
+   `dsimaging-store/controller/controller.py::write_dataset_artifacts`
+   re-implement the same scan / hash / manifest pipeline in parallel. They
+   already differ subtly (extra-metadata handling). This is a drift hazard.
+
+---
+
+## Priority 1 — missing core admin verbs
+
+### 1.1 `delete <dataset-id>`
+
+Hard gap. Today the only way to drop a broken or obsolete dataset is
+`mc rm --recursive`.
+
+```
+dsimaging-admin delete <dataset-id> [--yes]
+                                    [--purge-versions]
+                                    [--opal-url ... --opal-project ... --opal-resource ...]
+```
+
+- Refuse without `--yes`.
+- By default, delete current objects only; prior versions stay (the bucket has
+  versioning on). `--purge-versions` issues `delete_object_versions` in pages
+  of 1000.
+- If `--opal-*` is provided, also remove the Opal resource.
+
+Effort: **low**. Pure `list_objects_v2` + `delete_objects` paginator work.
+
+### 1.2 `verify <dataset-id>`
+
+Validate that the dataset on S3 still matches its
+`content_hash_index.parquet`. The hash index is the whole reason content
+hashes are computed; there is no command to check it.
+
+```
+dsimaging-admin verify <dataset-id> [--sample-fraction 1.0]
+                                    [--output text|json]
+                                    [--quick]
+```
+
+- Reads `indexes/content_hash_index.parquet` (and the masks index if present).
+- Re-streams every object, recomputes sha256, reports
+  `missing` / `extra` / `mismatch` per sample.
+- `--quick` compares S3 ETag against the recorded etag first; only falls back
+  to full hashing on mismatch. Caveat: ETag == sha256 only for single-part
+  uploads, so `--quick` is an heuristic, not a proof.
+- Non-zero exit on drift; suitable for CI.
+
+Effort: **medium**. DICOM series must hash with the same composition rule as
+`controller.scan_s3_images` (sorted per-file sha256 fed back into a parent
+sha256) — another reason to share code (see P4).
+
+### 1.3 `opal {list,register,unregister}` standalone
+
+Today Opal registration is a side-effect of `publish`. Adding a fourth Opal
+site, or replaying registration after a wipe, forces either a full re-upload
+or hand-crafted Opal REST calls.
+
+```
+dsimaging-admin opal list       --opal-url ... [--project IMAGING]
+dsimaging-admin opal register    --dataset-id <id> --opal-url ... [--name ... --replace]
+dsimaging-admin opal unregister  --opal-url ... --project IMAGING --resource <name>
+```
+
+`register` rebuilds the `imaging+dataset://` URL from the dataset's
+manifest on S3, so it does not need access to the original local source.
+
+Implementation: extract `_register_opal_resource` from `cli.py` into an
+`opal.py` module and reuse from the new sub-commands and from `publish`.
+
+Effort: **low**.
+
+---
+
+## Priority 2 — observability & drift detection
+
+### 2.1 `doctor` extended for the full module
+
+The current `doctor` is bucket-scoped. Extend it to validate the deployment:
+
+- Ping controller `http://<host>:8080/health`, surface `dirty_datasets` and
+  `last_errors`.
+- Confirm the MinIO webhook is configured for `datasets/` (compare against
+  what the controller advertises).
+- Compare CLI-side `list` against controller `/datasets` and flag any
+  divergence.
+- Validate the init container ran (presence of `datasets/.keep`,
+  versioning enabled).
+
+New flags:
+
+```
+--controller-url http://127.0.0.1:8080   (default; from ~/.dsimaging.yaml)
+--skip-controller                        (for remote / firewalled stores)
+```
+
+### 2.2 `list` enriched with controller state
+
+When `--controller-url` is reachable, merge `dirty` and `last_error` into the
+table. Same network call as the extended `doctor`; gracefully degrade to
+S3-only when the controller is unreachable.
+
+### 2.3 `status <dataset-id>`
+
+Drill-down for a single dataset:
+
+- Object counts under `source/images/` and `source/masks/`.
+- Manifest `schema_version` + last-modified.
+- Row count of `content_hash_index.parquet` vs object count (cheap drift hint
+  short of a full `verify`).
+- Controller `last_reconcile` / `last_error` if reachable.
+
+---
+
+## Priority 3 — robustness of `publish`
+
+### 3.1 Atomic publish via staging prefix
+
+Today, a mid-upload failure leaves a partial set of objects under
+`datasets/<id>/source/`. The controller fires on each PUT and writes a
+partial manifest. Replace with a two-phase flow:
+
+1. Upload to `datasets/<id>/.staging-<uuid>/source/...`.
+2. On full success, server-side `CopyObject` into the canonical prefix and
+   delete the staging copy.
+3. Only then do the webhook events on the final prefix trigger the
+   controller; the controller never sees an in-progress dataset.
+
+Trade-off: intra-bucket copy adds IO but no egress. Acceptable for the
+target scale.
+
+### 3.2 Idempotent upload (skip-if-hash-matches)
+
+For larger public datasets (LUNG1, other TCIA cohorts), re-running `publish`
+should skip objects whose local sha256 already appears in
+`content_hash_index.parquet`. Add `--no-skip` to opt out.
+
+### 3.3 `--dry-run`
+
+Scan + report what would be uploaded, no S3 writes. Useful for sanity-checking
+metadata CSV joins (`--metadata`) before paying the upload cost.
+
+---
+
+## Priority 4 — internal refactor (drift killer)
+
+`cli.py::_write_dataset_artifacts` and
+`dsimaging-store/controller/controller.py::write_dataset_artifacts` are
+siblings that must stay in sync. They already diverge:
+
+- The controller has no concept of `extra_metadata` provided from outside;
+  it only preserves whatever was already in `metadata/samples.parquet`.
+- Mask handling code paths are similar but not identical.
+
+Two viable options:
+
+**Option A — shared core library (recommended).**
+Extract `manifest`, `hashing`, `schema`, `s3_layout` into a small package
+(`dsimaging-core` or similar) installed in both the CLI distribution and the
+controller container. CLI keeps `cli` + `opal`; controller keeps the HTTP
+layer + reconcile loop.
+
+**Option B — controller as sole writer.**
+CLI never writes indexes/manifests directly; after upload it issues
+`POST /reconcile/<dataset-id>` and waits. Single source of truth, but the
+CLI now hard-requires the controller stack — that breaks the use-case of
+publishing into a vanilla S3 bucket without the `dsimaging-store` services
+deployed.
+
+Recommendation: **A**. Keep `publish` self-sufficient.
+
+---
+
+## Priority 5 — nice-to-haves
+
+- `download <dataset-id> <dest>` — pull a dataset locally for inspection
+  / debugging.
+- `copy <src-id> <dst-id>` — server-side intra-bucket copy for renames or
+  forks.
+- Multi-profile config: `~/.dsimaging.yaml` with `profiles.<name>:` and a
+  `--profile <name>` flag (today only `default` is honoured).
+- `--output json` on `list`, `verify`, `status`, `doctor` for scripting.
+- Structured logging (`--log-format json`).
+- Pre-publish DICOM sanity checks (consistent SeriesInstanceUID per folder,
+  monotonic InstanceNumber, single Modality value).
+
+---
+
+## Non-goals (explicit)
+
+- Web UI. The MinIO console plus the CLI is enough for operator needs.
+- Authentication / RBAC inside the CLI. Inherited from S3 and Opal endpoints.
+- Image-level operations (DICOM de-identification, resampling, format
+  conversion). That is `dsImaging` / `dsRadiomics` territory, not admin.
+
+---
+
+## Open questions
+
+- Should `delete` cascade to `dsHPC` artifacts that reference the dataset?
+  No back-reference index exists today; probably out of scope for the admin
+  CLI.
+- Should `verify` write a tombstone (`indexes/last_verify.json`) on detected
+  drift so the controller can refuse to republish stale manifests?
+- Minimal `release.yml` for PyPI Trusted Publishing from
+  `isglobal-brge/dsimaging-admin`? Tracked separately; only blocks future
+  cuts.
+
+---
+
+## Sequencing suggestion
+
+**One full day of work**:
+
+1. Extract shared core (**P4 Option A**). Pays for everything below.
+2. `delete` + `opal *` standalone (**P1.1** + **P1.3**). Both small, both
+   unblock real operations.
+3. `verify` (**P1.2**). Turns the hash index from a hopeful artifact into a
+   real safety net.
+
+**An afternoon**: skip P4, do P1.1 + P1.3 only. Defer the shared-core
+refactor until the next time the manifest schema changes.

dsimaging_admin-0.3.0/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: dsimaging-admin
+Version: 0.3.0
+Summary: Admin CLI for managing medical imaging datasets in S3/MinIO for DataSHIELD
+Project-URL: Homepage, https://github.com/isglobal-brge/dsimaging-admin
+Project-URL: Repository, https://github.com/isglobal-brge/dsimaging-admin
+Author-email: David Sarrat Gonzalez <david.sarrat@isglobal.org>, Juan R Gonzalez <juanr.gonzalez@isglobal.org>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: datashield,dicom,medical-imaging,minio,radiomics,s3
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Healthcare Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: >=3.9
+Requires-Dist: boto3>=1.28.0
+Requires-Dist: click>=8.0.0
+Requires-Dist: pyarrow>=14.0.0
+Requires-Dist: pydicom>=3.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31.0
+Description-Content-Type: text/markdown
+
+# dsimaging-admin
+
+Admin CLI for creating and operating `dsimaging-store` deployments and the
+medical imaging datasets stored in them.
+
+## Install
+
+```bash
+pip install dsimaging-admin
+```
+
+## Create a store
+
+`dsimaging-admin store init` writes a Docker Compose project for MinIO plus the
+`dsimaging-store` controller.
+
+```bash
+dsimaging-admin store init ./study-store \
+  --controller-image davidsarratgonzalez/dsimaging-store:latest
+dsimaging-admin store up ./study-store
+dsimaging-admin store doctor ./study-store
+```
+
+For local controller development, build from a checked-out `dsimaging-store`
+repo instead of using an image:
+
+```bash
+dsimaging-admin store init ./study-store \
+  --store-source /path/to/dsimaging-store
+dsimaging-admin store up ./study-store
+```
+
+Use the generated connection details as a reusable CLI profile:
+
+```bash
+dsimaging-admin init \
+  --endpoint http://127.0.0.1:9000 \
+  --controller-url http://127.0.0.1:8080 \
+  --bucket imaging-data
+```
+
+## Dataset operations
+
+```bash
+# Publish with staging, publish lock, skip-if-hash-matches and DICOM checks.
+dsimaging-admin publish \
+  --dataset-id study_ct_v1 \
+  --source /data/study_ct \
+  --metadata /data/study_ct/clinical.csv \
+  --modality ct
+
+# Inspect and verify.
+dsimaging-admin list
+dsimaging-admin status study_ct_v1
+dsimaging-admin verify study_ct_v1
+dsimaging-admin doctor
+
+# Rebuild artifacts from S3, copy, download or delete.
+dsimaging-admin rescan study_ct_v1
+dsimaging-admin copy study_ct_v1 study_ct_v2 --yes
+dsimaging-admin download study_ct_v1 ./debug/study_ct_v1
+dsimaging-admin delete study_ct_v1 --yes --purge-versions
+```
+
+All reporting commands that are useful for automation support JSON output:
+
+```bash
+dsimaging-admin list --output json
+dsimaging-admin status study_ct_v1 --output json
+dsimaging-admin verify study_ct_v1 --output json
+dsimaging-admin doctor --output json
+```
+
+## What `publish` does
+
+1. Scans your local image directory and optional masks under `source/masks/`,
+   `masks/`, `source/labels/`, or `labels/`.
+2. Runs basic DICOM sanity checks for series UID, modality and instance order.
+3. Computes SHA-256 content hashes.
+4. Skips uploads that already match the current dataset hash indexes.
+5. Uploads through `datasets/<id>/.staging-*` and a `.publish-lock`, then copies
+   into `datasets/<id>/source/...`.
+6. Generates and uploads:
+   - `manifest.yaml`
+   - `indexes/content_hash_index.parquet`
+   - `indexes/masks_content_hash_index.parquet` when masks exist
+   - `metadata/sample_manifests.parquet`
+   - `metadata/samples.parquet`
+
+Use `--dry-run` to scan and show the upload plan without S3 writes, `--no-skip`
+to force uploads, or `--no-atomic` to disable staging.
+
+## Configuration
+
+`~/.dsimaging.yaml` supports multiple profiles:
+
+```yaml
+default_profile: default
+profiles:
+  default:
+    endpoint: http://127.0.0.1:9000
+    controller_url: http://127.0.0.1:8080
+    bucket: imaging-data
+    access_key: minioadmin
+    secret_key: minioadmin123
+    region: ""
+```
+
+Environment variables override profile values:
+
+| Variable | Default | Description |
+|---|---|---|
+| `DSIMAGING_PROFILE` | `default` | Config profile |
+| `DSIMAGING_ENDPOINT` | `http://127.0.0.1:9000` | S3/MinIO endpoint |
+| `DSIMAGING_CONTROLLER_URL` | (empty) | dsimaging-store controller URL |
+| `DSIMAGING_ACCESS_KEY` | `minioadmin` | S3 access key |
+| `DSIMAGING_SECRET_KEY` | `minioadmin123` | S3 secret key |
+| `DSIMAGING_BUCKET` | `imaging-data` | Bucket name |
+| `DSIMAGING_REGION` | (empty) | S3 region |
+
+## Dataset layout in S3
+
+```text
+s3://<bucket>/datasets/<dataset_id>/
+  manifest.yaml
+  metadata/
+    samples.parquet
+    sample_manifests.parquet
+  indexes/
+    content_hash_index.parquet
+    masks_content_hash_index.parquet
+  source/
+    images/
+    masks/
+  derived/
+  qc/
+```
+
+Store creation and dataset management only target dsimaging-store deployments.

dsimaging_admin-0.3.0/README.md

@@ -0,0 +1,139 @@
+# dsimaging-admin
+
+Admin CLI for creating and operating `dsimaging-store` deployments and the
+medical imaging datasets stored in them.
+
+## Install
+
+```bash
+pip install dsimaging-admin
+```
+
+## Create a store
+
+`dsimaging-admin store init` writes a Docker Compose project for MinIO plus the
+`dsimaging-store` controller.
+
+```bash
+dsimaging-admin store init ./study-store \
+  --controller-image davidsarratgonzalez/dsimaging-store:latest
+dsimaging-admin store up ./study-store
+dsimaging-admin store doctor ./study-store
+```
+
+For local controller development, build from a checked-out `dsimaging-store`
+repo instead of using an image:
+
+```bash
+dsimaging-admin store init ./study-store \
+  --store-source /path/to/dsimaging-store
+dsimaging-admin store up ./study-store
+```
+
+Use the generated connection details as a reusable CLI profile:
+
+```bash
+dsimaging-admin init \
+  --endpoint http://127.0.0.1:9000 \
+  --controller-url http://127.0.0.1:8080 \
+  --bucket imaging-data
+```
+
+## Dataset operations
+
+```bash
+# Publish with staging, publish lock, skip-if-hash-matches and DICOM checks.
+dsimaging-admin publish \
+  --dataset-id study_ct_v1 \
+  --source /data/study_ct \
+  --metadata /data/study_ct/clinical.csv \
+  --modality ct
+
+# Inspect and verify.
+dsimaging-admin list
+dsimaging-admin status study_ct_v1
+dsimaging-admin verify study_ct_v1
+dsimaging-admin doctor
+
+# Rebuild artifacts from S3, copy, download or delete.
+dsimaging-admin rescan study_ct_v1
+dsimaging-admin copy study_ct_v1 study_ct_v2 --yes
+dsimaging-admin download study_ct_v1 ./debug/study_ct_v1
+dsimaging-admin delete study_ct_v1 --yes --purge-versions
+```
+
+All reporting commands that are useful for automation support JSON output:
+
+```bash
+dsimaging-admin list --output json
+dsimaging-admin status study_ct_v1 --output json
+dsimaging-admin verify study_ct_v1 --output json
+dsimaging-admin doctor --output json
+```
+
+## What `publish` does
+
+1. Scans your local image directory and optional masks under `source/masks/`,
+   `masks/`, `source/labels/`, or `labels/`.
+2. Runs basic DICOM sanity checks for series UID, modality and instance order.
+3. Computes SHA-256 content hashes.
+4. Skips uploads that already match the current dataset hash indexes.
+5. Uploads through `datasets/<id>/.staging-*` and a `.publish-lock`, then copies
+   into `datasets/<id>/source/...`.
+6. Generates and uploads:
+   - `manifest.yaml`
+   - `indexes/content_hash_index.parquet`
+   - `indexes/masks_content_hash_index.parquet` when masks exist
+   - `metadata/sample_manifests.parquet`
+   - `metadata/samples.parquet`
+
+Use `--dry-run` to scan and show the upload plan without S3 writes, `--no-skip`
+to force uploads, or `--no-atomic` to disable staging.
+
+## Configuration
+
+`~/.dsimaging.yaml` supports multiple profiles:
+
+```yaml
+default_profile: default
+profiles:
+  default:
+    endpoint: http://127.0.0.1:9000
+    controller_url: http://127.0.0.1:8080
+    bucket: imaging-data
+    access_key: minioadmin
+    secret_key: minioadmin123
+    region: ""
+```
+
+Environment variables override profile values:
+
+| Variable | Default | Description |
+|---|---|---|
+| `DSIMAGING_PROFILE` | `default` | Config profile |
+| `DSIMAGING_ENDPOINT` | `http://127.0.0.1:9000` | S3/MinIO endpoint |
+| `DSIMAGING_CONTROLLER_URL` | (empty) | dsimaging-store controller URL |
+| `DSIMAGING_ACCESS_KEY` | `minioadmin` | S3 access key |
+| `DSIMAGING_SECRET_KEY` | `minioadmin123` | S3 secret key |
+| `DSIMAGING_BUCKET` | `imaging-data` | Bucket name |
+| `DSIMAGING_REGION` | (empty) | S3 region |
+
+## Dataset layout in S3
+
+```text
+s3://<bucket>/datasets/<dataset_id>/
+  manifest.yaml
+  metadata/
+    samples.parquet
+    sample_manifests.parquet
+  indexes/
+    content_hash_index.parquet
+    masks_content_hash_index.parquet
+  source/
+    images/
+    masks/
+  derived/
+  qc/
+```
+
+Store creation and dataset management only target dsimaging-store deployments.

{dsimaging_admin-0.2.0 → dsimaging_admin-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "dsimaging-admin"
-version = "0.2.0"
+version = "0.3.0"
 description = "Admin CLI for managing medical imaging datasets in S3/MinIO for DataSHIELD"
 readme = "README.md"
 license = "MIT"
@@ -28,6 +28,7 @@ dependencies = [
     "pyyaml>=6.0",
     "click>=8.0.0",
     "requests>=2.31.0",
+    "pydicom>=3.0.0",
 ]
 
 [project.scripts]

{dsimaging_admin-0.2.0 → dsimaging_admin-0.3.0}/src/dsimaging_admin/__init__.py

@@ -1,3 +1,3 @@
 """dsimaging-admin: Admin CLI for managing medical imaging datasets in S3/MinIO."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"