dsimaging-admin 0.3.2__tar.gz → 0.3.5__tar.gz

dsimaging_admin-0.3.5/IMPROVEMENTS.md

@@ -0,0 +1,85 @@
+# IMPROVEMENTS
+
+Validated status after the `dsimaging-admin` / `dsimaging-store` work for
+store-centric administration.
+
+## Product boundary
+
+`dsimaging-admin` manages `dsimaging-store` deployments and the datasets inside
+those stores. It must stay agnostic about clients that may consume those
+datasets later.
+
+That means Opal registration is not part of the public admin surface. Earlier
+notes in this file suggested an `opal` module and `--opal-*` publish flags; that
+was the wrong boundary for this project and has been intentionally removed.
+
+## Implemented
+
+### Store management
+
+- `store init` generates a ready-to-run `dsimaging-store` directory with
+  Docker Compose, `.env`, MinIO init script and controller settings.
+- `store up`, `store down`, `store ps`, `store logs`, `store config` and
+  `store doctor` operate that generated store from the admin CLI.
+- The default controller image is
+  `davidsarratgonzalez/dsimaging-store:latest`, with local-build support for
+  development.
+
+### Dataset administration
+
+- `delete <dataset-id> --yes [--purge-versions]` removes current dataset
+  objects, with optional version purging for versioned buckets.
+- `verify <dataset-id> [--quick] [--sample-fraction ...] [--output text|json]`
+  checks S3 objects against `content_hash_index.parquet` and mask indexes.
+- `status <dataset-id> [--output text|json]` reports object counts, manifest
+  state, hash-index rows and controller state when reachable.
+- `download <dataset-id> <dest>` pulls a dataset locally for inspection.
+- `copy <src-id> <dst-id> --yes` performs server-side S3 copies for renames or
+  forks.
+- `list --output text|json` now ignores deleted prefixes that only remain as
+  version history and can merge controller state.
+- `reconcile <dataset-id>` asks the store controller to rebuild artifacts for a
+  dataset.
+
+### Publish robustness
+
+- `publish` is atomic by default: uploads go to a staging prefix, then are
+  copied into the canonical dataset prefix after the upload completes.
+- A `.publish-lock` marker prevents the controller from reconciling a dataset
+  while the CLI is still publishing it.
+- `--no-atomic`, `--no-skip`, `--dry-run` and `--skip-dicom-checks` cover the
+  operational escape hatches.
+- Pre-publish DICOM checks validate consistent series UID, modality and
+  instance ordering when `pydicom` can parse the series.
+
+### Store observability
+
+- `doctor --output text|json` checks S3 connectivity, bucket existence,
+  versioning, init marker, controller health, webhook prefix/source prefixes and
+  controller/S3 dataset parity.
+- `--controller-url` and `--skip-controller` are global options.
+- Multi-profile config is supported through `profiles.<name>` in
+  `~/.dsimaging.yaml` and `--profile <name>`.
+
+### Shared scan/hash/manifest core
+
+The controller now imports the scan, hash-index and manifest builders from
+`dsimaging_admin.manifest` instead of carrying a divergent copy. This keeps
+CLI-side publish/rescan and controller-side reconcile on the same rules for
+images, masks, metadata and manifests.
+
+## Explicitly not implemented
+
+- Opal commands and `--opal-*` flags, by product decision.
+- A web UI; the CLI plus MinIO console remains the operating surface.
+- Image-level processing such as DICOM de-identification, resampling or format
+  conversion. Those belong in imaging/radiomics pipelines, not store admin.
+
+## Remaining follow-ups
+
+- Add CI release automation with Trusted Publishing so future PyPI releases do
+  not require a manual API token.
+- Decide whether a successful or failed `verify` should write an audit artifact
+  such as `indexes/last_verify.json`.
+- Add richer structured runtime logging if operators need machine-readable log
+  streams beyond the existing `--output json` command outputs.

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dsimaging-admin
-Version: 0.3.2
+Version: 0.3.5
 Summary: Admin CLI for managing medical imaging datasets in S3/MinIO for DataSHIELD
 Project-URL: Homepage, https://github.com/isglobal-brge/dsimaging-admin
 Project-URL: Repository, https://github.com/isglobal-brge/dsimaging-admin

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "dsimaging-admin"
-version = "0.3.2"
+version = "0.3.5"
 description = "Admin CLI for managing medical imaging datasets in S3/MinIO for DataSHIELD"
 readme = "README.md"
 license = "MIT"

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/src/dsimaging_admin/__init__.py

@@ -1,3 +1,3 @@
 """dsimaging-admin: Admin CLI for managing medical imaging datasets in S3/MinIO."""
 
-__version__ = "0.3.2"
+__version__ = "0.3.5"

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/src/dsimaging_admin/s3.py

@@ -33,6 +33,8 @@ def list_datasets(s3, bucket: str) -> list[dict]:
     for page in paginator.paginate(Bucket=bucket, Prefix="datasets/", Delimiter="/"):
         for cp in page.get("CommonPrefixes", []):
             ds_id = cp["Prefix"].strip("/").split("/")[-1]
+            if not _prefix_has_current_objects(s3, bucket, f"datasets/{ds_id}/"):
+                continue
             has_manifest = _object_exists(s3, bucket, f"datasets/{ds_id}/manifest.yaml")
             datasets.append({
                 "dataset_id": ds_id,
@@ -175,3 +177,12 @@ def _object_exists(s3, bucket: str, key: str) -> bool:
         return True
     except Exception:
         return False
+
+
+def _prefix_has_current_objects(s3, bucket: str, prefix: str) -> bool:
+    paginator = s3.get_paginator("list_objects_v2")
+    for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
+        for obj in page.get("Contents", []):
+            if _object_exists(s3, bucket, obj["Key"]):
+                return True
+    return False

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/src/dsimaging_admin/store.py

@@ -251,6 +251,8 @@ services:
       MINIO_ROOT_USER: ${{MINIO_ROOT_USER:-minioadmin}}
       MINIO_ROOT_PASSWORD: ${{MINIO_ROOT_PASSWORD:-minioadmin123}}
       BUCKET_NAME: ${{BUCKET_NAME:-imaging-data}}
+      WEBHOOK_RETRIES: ${{WEBHOOK_RETRIES:-12}}
+      WEBHOOK_RETRY_SECONDS: ${{WEBHOOK_RETRY_SECONDS:-5}}
     depends_on:
       minio:
         condition: service_healthy
@@ -269,6 +271,8 @@ MINIO_CONSOLE_PORT={console_port}
 CONTROLLER_PORT={controller_port}
 BUCKET_NAME={bucket}
 RECONCILE_INTERVAL_SECONDS={reconcile_interval}
+WEBHOOK_RETRIES=12
+WEBHOOK_RETRY_SECONDS=5
 DSIMAGING_STORE_CONTROLLER_IMAGE={controller_image}
 DSIMAGING_STORE_MINIO_IMAGE={minio_image}
 DSIMAGING_STORE_MC_IMAGE={mc_image}
@@ -283,6 +287,8 @@ MINIO_ENDPOINT="${MINIO_ENDPOINT:-http://minio:9000}"
 MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
 MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minioadmin123}"
 BUCKET_NAME="${BUCKET_NAME:-imaging-data}"
+WEBHOOK_RETRIES="${WEBHOOK_RETRIES:-12}"
+WEBHOOK_RETRY_SECONDS="${WEBHOOK_RETRY_SECONDS:-5}"
 
 mc alias set local "${MINIO_ENDPOINT}" "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}" --api s3v4
 
@@ -297,10 +303,33 @@ mc version enable "local/${BUCKET_NAME}"
 mkdir -p /tmp/dsimaging-init
 : > /tmp/dsimaging-init/.keep
 mc cp /tmp/dsimaging-init/.keep "local/${BUCKET_NAME}/datasets/.keep" 2>/dev/null || true
-mc event add "local/${BUCKET_NAME}" arn:minio:sqs::DSIMAGING:webhook \
-  --event put,delete \
-  --prefix "datasets/" \
-  2>/dev/null || echo "[init] Webhook notification already configured or controller not ready"
+configure_webhook() {
+  attempt=1
+  while [ "${attempt}" -le "${WEBHOOK_RETRIES}" ]; do
+    if output=$(mc event add "local/${BUCKET_NAME}" arn:minio:sqs::DSIMAGING:webhook \
+      --event put,delete \
+      --prefix "datasets/" 2>&1); then
+      echo "[init] Webhook notification configured"
+      return 0
+    fi
+
+    if echo "${output}" | grep -qi "already"; then
+      echo "[init] Webhook notification already configured"
+      return 0
+    fi
+
+    echo "[init] Webhook notification attempt ${attempt}/${WEBHOOK_RETRIES} failed: ${output}"
+    attempt=$((attempt + 1))
+    if [ "${attempt}" -le "${WEBHOOK_RETRIES}" ]; then
+      sleep "${WEBHOOK_RETRY_SECONDS}"
+    fi
+  done
+
+  echo "[init] ERROR: Unable to configure webhook notification after ${WEBHOOK_RETRIES} attempts" >&2
+  return 1
+}
+
+configure_webhook
 
 echo "[init] MinIO initialization complete"
 """

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/tests/test_admin_features.py

@@ -104,7 +104,7 @@ class AdminFeatureTests(unittest.TestCase):
 
     def test_verify_dataset_accepts_matching_single_file(self):
         bucket = "imaging-data"
-        prefix = "datasets/lung"
+        prefix = "datasets/study_ct_v1"
         payload = b"nifti"
         content_hash = hashlib.sha256(payload).hexdigest()
         sample = {
@@ -131,7 +131,7 @@ class AdminFeatureTests(unittest.TestCase):
             },
         })
 
-        result = verify_dataset(s3, bucket, "lung")
+        result = verify_dataset(s3, bucket, "study_ct_v1")
 
         self.assertTrue(result.ok)
         self.assertEqual(result.checked, 1)
@@ -139,7 +139,7 @@ class AdminFeatureTests(unittest.TestCase):
 
     def test_verify_dataset_reports_mismatch(self):
         bucket = "imaging-data"
-        prefix = "datasets/lung"
+        prefix = "datasets/study_ct_v1"
         sample = {
             "sample_id": "case001",
             "source_kind": "single_file",
@@ -162,14 +162,14 @@ class AdminFeatureTests(unittest.TestCase):
             },
         })
 
-        result = verify_dataset(s3, bucket, "lung")
+        result = verify_dataset(s3, bucket, "study_ct_v1")
 
         self.assertFalse(result.ok)
         self.assertEqual(result.mismatched, 1)
 
     def test_verify_quick_accepts_unchanged_etag(self):
         bucket = "imaging-data"
-        prefix = "datasets/lung"
+        prefix = "datasets/study_ct_v1"
         sample = {
             "sample_id": "case001",
             "source_kind": "single_file",
@@ -194,7 +194,7 @@ class AdminFeatureTests(unittest.TestCase):
             },
         })
 
-        result = verify_dataset(s3, bucket, "lung", quick=True)
+        result = verify_dataset(s3, bucket, "study_ct_v1", quick=True)
 
         self.assertTrue(result.ok)
         self.assertEqual(result.quick_ok, 1)

{dsimaging_admin-0.3.2 → dsimaging_admin-0.3.5}/tests/test_manifest.py

@@ -32,7 +32,7 @@ class FakeS3:
 
 class ManifestTests(unittest.TestCase):
     def test_validate_dataset_id_rejects_unsafe_values(self):
-        validate_dataset_id("lung1_site-a.v1")
+        validate_dataset_id("study_ct_v1_site-a.v1")
         with self.assertRaises(ValueError):
             validate_dataset_id("../bad")
         with self.assertRaises(ValueError):
@@ -56,7 +56,7 @@ class ManifestTests(unittest.TestCase):
 
     def test_scan_s3_images_groups_single_files_and_dicom(self):
         bucket = "imaging-data"
-        prefix = "datasets/lung"
+        prefix = "datasets/study_ct_v1"
         payloads = {
             f"{prefix}/source/images/a.nii.gz": b"nifti",
             f"{prefix}/source/images/series1/001.dcm": b"one",
@@ -93,7 +93,7 @@ class ManifestTests(unittest.TestCase):
 
     def test_scan_s3_masks_and_hash_index_use_mask_uris(self):
         bucket = "imaging-data"
-        prefix = "datasets/lung"
+        prefix = "datasets/study_ct_v1"
         payloads = {
             f"{prefix}/source/masks/case001_GTV-1.nii.gz": b"mask",
         }
@@ -111,14 +111,14 @@ class ManifestTests(unittest.TestCase):
         self.assertEqual(masks[0]["sample_id"], "case001")
         self.assertEqual(
             mask_index.to_pydict()["uri"][0],
-            "s3://imaging-data/datasets/lung/source/masks/case001_GTV-1.nii.gz",
+            "s3://imaging-data/datasets/study_ct_v1/source/masks/case001_GTV-1.nii.gz",
         )
 
     def test_generate_manifest_can_declare_mask_asset(self):
         manifest = generate_manifest(
-            "lung",
+            "study_ct_v1",
             "imaging-data",
-            "datasets/lung",
+            "datasets/study_ct_v1",
             modality="ct",
             has_masks=True,
         )
@@ -126,7 +126,7 @@ class ManifestTests(unittest.TestCase):
         self.assertEqual(manifest["assets"]["masks"]["kind"], "mask_root")
         self.assertEqual(
             manifest["assets"]["masks"]["content_hash_index"],
-            "s3://imaging-data/datasets/lung/indexes/masks_content_hash_index.parquet",
+            "s3://imaging-data/datasets/study_ct_v1/indexes/masks_content_hash_index.parquet",
         )
 
     def test_build_tables_use_canonical_uris(self):
@@ -140,12 +140,12 @@ class ManifestTests(unittest.TestCase):
             "size": 5,
         }]
 
-        hash_index = build_hash_index(samples, "imaging-data", "datasets/lung")
+        hash_index = build_hash_index(samples, "imaging-data", "datasets/study_ct_v1")
         manifests = build_sample_manifests(samples)
 
         self.assertEqual(
             hash_index.to_pydict()["uri"][0],
-            "s3://imaging-data/datasets/lung/source/images/a.nii.gz",
+            "s3://imaging-data/datasets/study_ct_v1/source/images/a.nii.gz",
         )
         self.assertEqual(manifests.to_pydict()["n_files"][0], 1)
 

dsimaging_admin-0.3.2/IMPROVEMENTS.md

@@ -1,244 +0,0 @@
-# IMPROVEMENTS
-
-Candidate improvements to make `dsimaging-admin` a complete administration tool
-for `dsimaging-store`, beyond the happy-path coverage shipped in **v0.2.0**.
-
-This is a roadmap, not a backlog. Items are prioritised by how much real
-operator pain they remove on the current `dsImaging` / `dsHPC` deployment.
-
-## Current coverage (v0.2.0)
-
-| Command   | What it does                                                           |
-|-----------|------------------------------------------------------------------------|
-| `init`    | Write `~/.dsimaging.yaml`.                                             |
-| `publish` | Local dir -> S3 (upload + indexes + manifest, optional Opal register). |
-| `rescan`  | Recompute indexes / manifest from current S3 contents.                 |
-| `list`    | Enumerate datasets; `published` if `manifest.yaml` exists.             |
-| `doctor`  | Check S3 connectivity, bucket existence, versioning, dataset count.    |
-
-Two structural observations that motivate most of what follows:
-
-1. The CLI does **not** talk to the `dsimaging-store` controller
-   (`:8080/health`, `:8080/datasets`). All status comes from S3 directly.
-2. `cli.py::_write_dataset_artifacts` and
-   `dsimaging-store/controller/controller.py::write_dataset_artifacts`
-   re-implement the same scan / hash / manifest pipeline in parallel. They
-   already differ subtly (extra-metadata handling). This is a drift hazard.
-
----
-
-## Priority 1 — missing core admin verbs
-
-### 1.1 `delete <dataset-id>`
-
-Hard gap. Today the only way to drop a broken or obsolete dataset is
-`mc rm --recursive`.
-
-```
-dsimaging-admin delete <dataset-id> [--yes]
-                                    [--purge-versions]
-                                    [--opal-url ... --opal-project ... --opal-resource ...]
-```
-
-- Refuse without `--yes`.
-- By default, delete current objects only; prior versions stay (the bucket has
-  versioning on). `--purge-versions` issues `delete_object_versions` in pages
-  of 1000.
-- If `--opal-*` is provided, also remove the Opal resource.
-
-Effort: **low**. Pure `list_objects_v2` + `delete_objects` paginator work.
-
-### 1.2 `verify <dataset-id>`
-
-Validate that the dataset on S3 still matches its
-`content_hash_index.parquet`. The hash index is the whole reason content
-hashes are computed; there is no command to check it.
-
-```
-dsimaging-admin verify <dataset-id> [--sample-fraction 1.0]
-                                    [--output text|json]
-                                    [--quick]
-```
-
-- Reads `indexes/content_hash_index.parquet` (and the masks index if present).
-- Re-streams every object, recomputes sha256, reports
-  `missing` / `extra` / `mismatch` per sample.
-- `--quick` compares S3 ETag against the recorded etag first; only falls back
-  to full hashing on mismatch. Caveat: ETag == sha256 only for single-part
-  uploads, so `--quick` is an heuristic, not a proof.
-- Non-zero exit on drift; suitable for CI.
-
-Effort: **medium**. DICOM series must hash with the same composition rule as
-`controller.scan_s3_images` (sorted per-file sha256 fed back into a parent
-sha256) — another reason to share code (see P4).
-
-### 1.3 `opal {list,register,unregister}` standalone
-
-Today Opal registration is a side-effect of `publish`. Adding a fourth Opal
-site, or replaying registration after a wipe, forces either a full re-upload
-or hand-crafted Opal REST calls.
-
-```
-dsimaging-admin opal list       --opal-url ... [--project IMAGING]
-dsimaging-admin opal register    --dataset-id <id> --opal-url ... [--name ... --replace]
-dsimaging-admin opal unregister  --opal-url ... --project IMAGING --resource <name>
-```
-
-`register` rebuilds the `imaging+dataset://` URL from the dataset's
-manifest on S3, so it does not need access to the original local source.
-
-Implementation: extract `_register_opal_resource` from `cli.py` into an
-`opal.py` module and reuse from the new sub-commands and from `publish`.
-
-Effort: **low**.
-
----
-
-## Priority 2 — observability & drift detection
-
-### 2.1 `doctor` extended for the full module
-
-The current `doctor` is bucket-scoped. Extend it to validate the deployment:
-
-- Ping controller `http://<host>:8080/health`, surface `dirty_datasets` and
-  `last_errors`.
-- Confirm the MinIO webhook is configured for `datasets/` (compare against
-  what the controller advertises).
-- Compare CLI-side `list` against controller `/datasets` and flag any
-  divergence.
-- Validate the init container ran (presence of `datasets/.keep`,
-  versioning enabled).
-
-New flags:
-
-```
---controller-url http://127.0.0.1:8080   (default; from ~/.dsimaging.yaml)
---skip-controller                        (for remote / firewalled stores)
-```
-
-### 2.2 `list` enriched with controller state
-
-When `--controller-url` is reachable, merge `dirty` and `last_error` into the
-table. Same network call as the extended `doctor`; gracefully degrade to
-S3-only when the controller is unreachable.
-
-### 2.3 `status <dataset-id>`
-
-Drill-down for a single dataset:
-
-- Object counts under `source/images/` and `source/masks/`.
-- Manifest `schema_version` + last-modified.
-- Row count of `content_hash_index.parquet` vs object count (cheap drift hint
-  short of a full `verify`).
-- Controller `last_reconcile` / `last_error` if reachable.
-
----
-
-## Priority 3 — robustness of `publish`
-
-### 3.1 Atomic publish via staging prefix
-
-Today, a mid-upload failure leaves a partial set of objects under
-`datasets/<id>/source/`. The controller fires on each PUT and writes a
-partial manifest. Replace with a two-phase flow:
-
-1. Upload to `datasets/<id>/.staging-<uuid>/source/...`.
-2. On full success, server-side `CopyObject` into the canonical prefix and
-   delete the staging copy.
-3. Only then do the webhook events on the final prefix trigger the
-   controller; the controller never sees an in-progress dataset.
-
-Trade-off: intra-bucket copy adds IO but no egress. Acceptable for the
-target scale.
-
-### 3.2 Idempotent upload (skip-if-hash-matches)
-
-For larger public datasets (LUNG1, other TCIA cohorts), re-running `publish`
-should skip objects whose local sha256 already appears in
-`content_hash_index.parquet`. Add `--no-skip` to opt out.
-
-### 3.3 `--dry-run`
-
-Scan + report what would be uploaded, no S3 writes. Useful for sanity-checking
-metadata CSV joins (`--metadata`) before paying the upload cost.
-
----
-
-## Priority 4 — internal refactor (drift killer)
-
-`cli.py::_write_dataset_artifacts` and
-`dsimaging-store/controller/controller.py::write_dataset_artifacts` are
-siblings that must stay in sync. They already diverge:
-
-- The controller has no concept of `extra_metadata` provided from outside;
-  it only preserves whatever was already in `metadata/samples.parquet`.
-- Mask handling code paths are similar but not identical.
-
-Two viable options:
-
-**Option A — shared core library (recommended).**
-Extract `manifest`, `hashing`, `schema`, `s3_layout` into a small package
-(`dsimaging-core` or similar) installed in both the CLI distribution and the
-controller container. CLI keeps `cli` + `opal`; controller keeps the HTTP
-layer + reconcile loop.
-
-**Option B — controller as sole writer.**
-CLI never writes indexes/manifests directly; after upload it issues
-`POST /reconcile/<dataset-id>` and waits. Single source of truth, but the
-CLI now hard-requires the controller stack — that breaks the use-case of
-publishing into a vanilla S3 bucket without the `dsimaging-store` services
-deployed.
-
-Recommendation: **A**. Keep `publish` self-sufficient.
-
----
-
-## Priority 5 — nice-to-haves
-
-- `download <dataset-id> <dest>` — pull a dataset locally for inspection
-  / debugging.
-- `copy <src-id> <dst-id>` — server-side intra-bucket copy for renames or
-  forks.
-- Multi-profile config: `~/.dsimaging.yaml` with `profiles.<name>:` and a
-  `--profile <name>` flag (today only `default` is honoured).
-- `--output json` on `list`, `verify`, `status`, `doctor` for scripting.
-- Structured logging (`--log-format json`).
-- Pre-publish DICOM sanity checks (consistent SeriesInstanceUID per folder,
-  monotonic InstanceNumber, single Modality value).
-
----
-
-## Non-goals (explicit)
-
-- Web UI. The MinIO console plus the CLI is enough for operator needs.
-- Authentication / RBAC inside the CLI. Inherited from S3 and Opal endpoints.
-- Image-level operations (DICOM de-identification, resampling, format
-  conversion). That is `dsImaging` / `dsRadiomics` territory, not admin.
-
----
-
-## Open questions
-
-- Should `delete` cascade to `dsHPC` artifacts that reference the dataset?
-  No back-reference index exists today; probably out of scope for the admin
-  CLI.
-- Should `verify` write a tombstone (`indexes/last_verify.json`) on detected
-  drift so the controller can refuse to republish stale manifests?
-- Minimal `release.yml` for PyPI Trusted Publishing from
-  `isglobal-brge/dsimaging-admin`? Tracked separately; only blocks future
-  cuts.
-
----
-
-## Sequencing suggestion
-
-**One full day of work**:
-
-1. Extract shared core (**P4 Option A**). Pays for everything below.
-2. `delete` + `opal *` standalone (**P1.1** + **P1.3**). Both small, both
-   unblock real operations.
-3. `verify` (**P1.2**). Turns the hash index from a hopeful artifact into a
-   real safety net.
-
-**An afternoon**: skip P4, do P1.1 + P1.3 only. Defer the shared-core
-refactor until the next time the manifest schema changes.