mlarray 0.0.53__tar.gz → 1.0.0__tar.gz

{mlarray-0.0.53 → mlarray-1.0.0}/.github/workflows/workflow.yml

@@ -18,7 +18,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -35,7 +35,13 @@ jobs:
 
       - name: Run tests
         run: |
-          pytest || if [ $? -eq 5 ]; then echo "No tests collected; skipping."; exit 0; else exit $?; fi
+          status=0
+          pytest || status=$?
+          if [ "$status" -eq 5 ]; then
+            echo "No tests collected; skipping."
+            exit 0
+          fi
+          exit "$status"
 
   publish:
     needs: tests
@@ -61,10 +67,8 @@ jobs:
       - name: Check metadata
         run: python -m twine check dist/*
 
-      - name: Publish to PyPI
+      - name: Publish to PyPI (trusted publishing)
         uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          password: ${{ secrets.PYPI_API_TOKEN }}
 
   docs:
     needs: tests

{mlarray-0.0.53 → mlarray-1.0.0}/PKG-INFO

@@ -1,25 +1,29 @@
 Metadata-Version: 2.4
 Name: mlarray
-Version: 0.0.53
+Version: 1.0.0
 Summary: Array format specialized for Machine Learning with Blosc2 backend and standardized metadata.
 Author-email: Karol Gotkowski <karol.gotkowski@dkfz.de>
 License: MIT
 Project-URL: Homepage, https://github.com/MIC-DKFZ/mlarray
 Project-URL: Source, https://github.com/MIC-DKFZ/mlarray
 Project-URL: Issues, https://github.com/MIC-DKFZ/mlarray/issues
-Keywords: copier,template,python
+Keywords: machine learning,medical imaging,blosc2,array,file format
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: blosc2>=3
 Requires-Dist: numpy>=2
+Requires-Dist: msgpack>=1.0
+Requires-Dist: xxhash>=3.0
 Provides-Extra: dev
 Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: jsonschema>=4.18; extra == "dev"
 Requires-Dist: pytest>=7.0; extra == "dev"
 Requires-Dist: twine>=4.0; extra == "dev"
 Requires-Dist: setuptools_scm[toml]>=8.0; extra == "dev"
@@ -29,7 +33,7 @@ Requires-Dist: mkdocs-include-markdown-plugin>=6.0; extra == "docs"
 Requires-Dist: mkdocstrings[python]>=0.25; extra == "docs"
 Requires-Dist: pymdown-extensions>=10.0; extra == "docs"
 Provides-Extra: all
-Requires-Dist: medvol; extra == "all"
+Requires-Dist: medvol>=1.0.0; extra == "all"
 Dynamic: license-file
 
 <p align="center">
@@ -55,7 +59,7 @@ You can install mlarray via [pip](https://pypi.org/project/mlarray/):
 pip install mlarray
 ```
 
-To enable the `mlarray_convert` CLI command, install MLArray with the necessary extra dependencies:
+To enable the `mlarray_convert` CLI command (conversion with MedVol 1.0 or newer), install MLArray with the extra dependencies:
 ```bash
 pip install mlarray[all]
 ```
@@ -101,7 +105,15 @@ image = MLArray.create("sample.mla", shape=array.shape, dtype=array.dtype, mmap_
 image[...] = array  # Modify image in memory and disk
 ```
 
-### Metadata inspection and manipulation
+### Fast reading
+
+`MLArray.open` reads lazily: `image[...]` decompresses only the chunks a patch touches. The first open in a process also turns on blosc2's read-ahead (4 threads), which fetches those chunks in parallel and makes reads of data not in the page cache several times faster, especially on network storage. Change it with `mlarray.set_read_ahead(n)` (`0` turns it off) or the `BLOSC_PREFETCH` environment variable. Files are written with nnU-Net's layout and compression settings; see [ML Optimization](docs/optimization.md) for details.
+
+### Metadata
+
+Metadata is explicit and domain-neutral: what each axis is, where the array sits in physical space, what
+its values mean, optional domain profiles, and your own metadata. Geometry is never implied: spacing,
+origin and direction always refer to a declared space.
 
 ```python
 import numpy as np
@@ -110,63 +122,44 @@ from mlarray import MLArray
 array = np.random.random((64, 128, 128))
 image = MLArray(
     array,
-    spacing=(1.0, 1.0, 1.5),
-    origin=(10.0, 10.0, 30.0),
-    direction=[[1, 0, 0], [0, 1, 0], [0, 0, 1]],
-    meta={"patient_id": "123", "modality": "CT"},  # Any metadata from the original image source (for example raw DICOM metadata)
+    axes=["z", "y", "x"],
+    spacing=(1.5, 1.0, 1.0),          # per array axis, in array order
+    origin=(10.0, 10.0, 30.0),        # centre of index (0, 0, 0)
+    direction=[[0, 0, 1], [0, 1, 0], [1, 0, 0]],  # column c: unit vector of array axis c
+    space="patient",                  # DICOM patient frame (LPS)
+    values={"kind": "measurement", "unit": "hounsfield_unit"},
+    user={"patient_id": "123"},       # any metadata of your own
 )
 
-print(image.spacing)  # [1.0, 1.0, 1.5]
-print(image.origin)  # [10.0, 10.0, 30.0]
-print(image.meta.source)  # {"patient_id": "123", "modality": "CT"}
-
-image.spacing[1] = 5.3
-image.meta.source["study_id"] = "study-001"
+print(image.spacing)         # (1.5, 1.0, 1.0)
+print(image.letter_codes())  # ('S', 'P', 'L')
+image.set_geometry(spacing=(1.5, 5.3, 1.0))
+image.user["study_id"] = "study-001"
 image.save("with-metadata.mla")
 
 # Open memory-mapped
-image = MLArray.open("with-metadata.mla", mmap_mode='r+')  
-image.meta.source["study_id"] = "new-study"  # Modify metadata
+image = MLArray.open("with-metadata.mla", mmap_mode='r+')
+image.user["study_id"] = "new-study"  # Modify metadata
 image.close()  # Close and save metadata, only necessary to save modified metadata
 ```
 
-### Copy metadata with overrides
+Segmentations, ML profiles and derived images:
 
 ```python
 import numpy as np
 from mlarray import MLArray
 
-base = MLArray("sample.mla")
-array = np.random.random(base.shape)
-
-image = MLArray(
-    array,
-    spacing=(0.8, 0.8, 1.0),
-    copy=base,  # Copies all non-explicitly set arguments from base
+base = MLArray("with-metadata.mla")
+seg = MLArray(
+    np.zeros(base.shape, dtype=np.uint8),
+    copy=base,  # geometry, values, profiles and user metadata of base (never its source)
+    values={"kind": "label", "labels": {"0": {"name": "background"}, "1": {"name": "liver"}}},
+    profiles={"ml": {"split": "train"}},
 )
-
-image.save("copied-metadata.mla")
+seg.save("seg.mla")
 ```
 
-### Standardized metadata usage
-
-```python
-import numpy as np
-from mlarray import MLArray, Meta
-
-array = np.random.random((64, 128, 128))
-image = MLArray(
-    array,
-    meta=Meta(source={"patient_id": "123", "modality": "CT"}, is_seg=True),  # Add metadata in a pre-defined format
-)
-
-print(image.meta.source)  # {"patient_id": "123", "modality": "CT"}
-print(image.meta.is_seg)  # True
-
-image.meta.source["study_id"] = "study-001"
-image.meta.is_seg = False
-image.save("with-metadata.mla")
-```
+See [Usage](docs/usage.md) and [Schema](docs/schema.md) for the full metadata model.
 
 ### Patch size variants
 
@@ -228,7 +221,8 @@ image.save("blosc2-auto.mla")
 
 ### mlarray_header
 
-Print the metadata header from a `.mla` file.
+Print the metadata of a `.mla` file as JSON (`--source` and `--user` add the original metadata of
+converted files and the user metadata).
 
 ```bash
 mlarray_header sample.mla
@@ -236,12 +230,21 @@ mlarray_header sample.mla
 
 ### mlarray_convert
 
-Convert a NIfTI or NRRD file to MLArray and copy metadata.
+Convert between MLArray and NIfTI, NRRD, MetaImage, MGH/MGZ and DICOM with
+[MedVol](https://github.com/MIC-DKFZ/MedVol) 1.0 or newer (`pip install "mlarray[all]"`). Nothing is reoriented, rescaled or
+dropped silently: the array keeps its stored order and dtype, the geometry is taken from the file's own fields,
+the original header is kept in `source` so the file can be written back faithfully, and every choice is reported.
 
 ```bash
-mlarray_convert sample.nii.gz output.mla
+mlarray_convert sample.nii.gz sample.mla
+mlarray_convert sample.mla roundtrip.nii.gz
+mlarray_convert dicom_dir/ ct.mla --dicom-tilt exact
+mlarray_convert ct.mla out_dir/ --target-format dicom      # re-emits the original DICOM series
 ```
 
+Ambiguities are resolved by explicit MedVol policies, e.g. `--nifti-form sform` when a NIfTI's qform and sform
+disagree, or `--world-frame ras` to store the patient space in RAS instead of DICOM's LPS.
+
 ## Contributing
 
 Contributions are welcome! Please open a pull request with clear changes and add tests when appropriate.

{mlarray-0.0.53 → mlarray-1.0.0}/README.md

@@ -21,7 +21,7 @@ You can install mlarray via [pip](https://pypi.org/project/mlarray/):
 pip install mlarray
 ```
 
-To enable the `mlarray_convert` CLI command, install MLArray with the necessary extra dependencies:
+To enable the `mlarray_convert` CLI command (conversion with MedVol 1.0 or newer), install MLArray with the extra dependencies:
 ```bash
 pip install mlarray[all]
 ```
@@ -67,7 +67,15 @@ image = MLArray.create("sample.mla", shape=array.shape, dtype=array.dtype, mmap_
 image[...] = array  # Modify image in memory and disk
 ```
 
-### Metadata inspection and manipulation
+### Fast reading
+
+`MLArray.open` reads lazily: `image[...]` decompresses only the chunks a patch touches. The first open in a process also turns on blosc2's read-ahead (4 threads), which fetches those chunks in parallel and makes reads of data not in the page cache several times faster, especially on network storage. Change it with `mlarray.set_read_ahead(n)` (`0` turns it off) or the `BLOSC_PREFETCH` environment variable. Files are written with nnU-Net's layout and compression settings; see [ML Optimization](docs/optimization.md) for details.
+
+### Metadata
+
+Metadata is explicit and domain-neutral: what each axis is, where the array sits in physical space, what
+its values mean, optional domain profiles, and your own metadata. Geometry is never implied: spacing,
+origin and direction always refer to a declared space.
 
 ```python
 import numpy as np
@@ -76,63 +84,44 @@ from mlarray import MLArray
 array = np.random.random((64, 128, 128))
 image = MLArray(
     array,
-    spacing=(1.0, 1.0, 1.5),
-    origin=(10.0, 10.0, 30.0),
-    direction=[[1, 0, 0], [0, 1, 0], [0, 0, 1]],
-    meta={"patient_id": "123", "modality": "CT"},  # Any metadata from the original image source (for example raw DICOM metadata)
+    axes=["z", "y", "x"],
+    spacing=(1.5, 1.0, 1.0),          # per array axis, in array order
+    origin=(10.0, 10.0, 30.0),        # centre of index (0, 0, 0)
+    direction=[[0, 0, 1], [0, 1, 0], [1, 0, 0]],  # column c: unit vector of array axis c
+    space="patient",                  # DICOM patient frame (LPS)
+    values={"kind": "measurement", "unit": "hounsfield_unit"},
+    user={"patient_id": "123"},       # any metadata of your own
 )
 
-print(image.spacing)  # [1.0, 1.0, 1.5]
-print(image.origin)  # [10.0, 10.0, 30.0]
-print(image.meta.source)  # {"patient_id": "123", "modality": "CT"}
-
-image.spacing[1] = 5.3
-image.meta.source["study_id"] = "study-001"
+print(image.spacing)         # (1.5, 1.0, 1.0)
+print(image.letter_codes())  # ('S', 'P', 'L')
+image.set_geometry(spacing=(1.5, 5.3, 1.0))
+image.user["study_id"] = "study-001"
 image.save("with-metadata.mla")
 
 # Open memory-mapped
-image = MLArray.open("with-metadata.mla", mmap_mode='r+')  
-image.meta.source["study_id"] = "new-study"  # Modify metadata
+image = MLArray.open("with-metadata.mla", mmap_mode='r+')
+image.user["study_id"] = "new-study"  # Modify metadata
 image.close()  # Close and save metadata, only necessary to save modified metadata
 ```
 
-### Copy metadata with overrides
+Segmentations, ML profiles and derived images:
 
 ```python
 import numpy as np
 from mlarray import MLArray
 
-base = MLArray("sample.mla")
-array = np.random.random(base.shape)
-
-image = MLArray(
-    array,
-    spacing=(0.8, 0.8, 1.0),
-    copy=base,  # Copies all non-explicitly set arguments from base
+base = MLArray("with-metadata.mla")
+seg = MLArray(
+    np.zeros(base.shape, dtype=np.uint8),
+    copy=base,  # geometry, values, profiles and user metadata of base (never its source)
+    values={"kind": "label", "labels": {"0": {"name": "background"}, "1": {"name": "liver"}}},
+    profiles={"ml": {"split": "train"}},
 )
-
-image.save("copied-metadata.mla")
+seg.save("seg.mla")
 ```
 
-### Standardized metadata usage
-
-```python
-import numpy as np
-from mlarray import MLArray, Meta
-
-array = np.random.random((64, 128, 128))
-image = MLArray(
-    array,
-    meta=Meta(source={"patient_id": "123", "modality": "CT"}, is_seg=True),  # Add metadata in a pre-defined format
-)
-
-print(image.meta.source)  # {"patient_id": "123", "modality": "CT"}
-print(image.meta.is_seg)  # True
-
-image.meta.source["study_id"] = "study-001"
-image.meta.is_seg = False
-image.save("with-metadata.mla")
-```
+See [Usage](docs/usage.md) and [Schema](docs/schema.md) for the full metadata model.
 
 ### Patch size variants
 
@@ -194,7 +183,8 @@ image.save("blosc2-auto.mla")
 
 ### mlarray_header
 
-Print the metadata header from a `.mla` file.
+Print the metadata of a `.mla` file as JSON (`--source` and `--user` add the original metadata of
+converted files and the user metadata).
 
 ```bash
 mlarray_header sample.mla
@@ -202,12 +192,21 @@ mlarray_header sample.mla
 
 ### mlarray_convert
 
-Convert a NIfTI or NRRD file to MLArray and copy metadata.
+Convert between MLArray and NIfTI, NRRD, MetaImage, MGH/MGZ and DICOM with
+[MedVol](https://github.com/MIC-DKFZ/MedVol) 1.0 or newer (`pip install "mlarray[all]"`). Nothing is reoriented, rescaled or
+dropped silently: the array keeps its stored order and dtype, the geometry is taken from the file's own fields,
+the original header is kept in `source` so the file can be written back faithfully, and every choice is reported.
 
 ```bash
-mlarray_convert sample.nii.gz output.mla
+mlarray_convert sample.nii.gz sample.mla
+mlarray_convert sample.mla roundtrip.nii.gz
+mlarray_convert dicom_dir/ ct.mla --dicom-tilt exact
+mlarray_convert ct.mla out_dir/ --target-format dicom      # re-emits the original DICOM series
 ```
 
+Ambiguities are resolved by explicit MedVol policies, e.g. `--nifti-form sform` when a NIfTI's qform and sform
+disagree, or `--world-frame ras` to store the patient space in RAS instead of DICOM's LPS.
+
 ## Contributing
 
 Contributions are welcome! Please open a pull request with clear changes and add tests when appropriate.

mlarray-1.0.0/bench/.gitignore

@@ -0,0 +1,2 @@
+data/
+results/

mlarray-1.0.0/bench/README.md

@@ -0,0 +1,8 @@
+# Benchmark Scripts
+
+- `benchmark_formats.py`: compares MLArray with other image file formats.
+
+The benchmark behind MLArray's storage defaults (layout, codec, read mode and
+read-ahead, measured on a compute cluster with nnU-Net-style loading) lives on
+the `feat/bench-suite` branch under `bench/`; `docs/optimization.md` summarizes
+its results.