esrf-data-compressor 0.1.0__tar.gz

esrf_data_compressor-0.1.0/LICENSE

@@ -0,0 +1,20 @@
+MIT License
+
+**Copyright (c) 2025 European Synchrotron Radiation Facility**
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

esrf_data_compressor-0.1.0/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.4
+Name: esrf-data-compressor
+Version: 0.1.0
+Summary: A library to compress ESRF data and reduce their footprint
+Author-email: ESRF <dau-pydev@esrf.fr>
+License: MIT License
+        
+        **Copyright (c) 2025 European Synchrotron Radiation Facility**
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of
+        this software and associated documentation files (the "Software"), to deal in
+        the Software without restriction, including without limitation the rights to
+        use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+        the Software, and to permit persons to whom the Software is furnished to do so,
+        subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+        FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+        COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+        IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+        CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+        
+Project-URL: Homepage, https://gitlab.esrf.fr/dau/esrf-data-compressor
+Project-URL: Documentation, https://esrf-data-compressor.readthedocs.io/
+Project-URL: Repository, https://gitlab.esrf.fr/dau/esrf-data-compressor
+Project-URL: Issues, https://gitlab.esrf.fr/dau/esrf-data-compressor/issues
+Project-URL: Changelog, https://gitlab.esrf.fr/dau/esrf-data-compressor/-/blob/main/CHANGELOG.md
+Keywords: ESRF,pathlib
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: h5py
+Requires-Dist: hdf5plugin
+Requires-Dist: blosc2-grok
+Requires-Dist: scikit-image
+Requires-Dist: tqdm
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Provides-Extra: dev
+Requires-Dist: esrf-data-compressor[test]; extra == "dev"
+Requires-Dist: black>=22; extra == "dev"
+Requires-Dist: flake8>=4.0; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Provides-Extra: doc
+Requires-Dist: sphinx>=6.0; extra == "doc"
+Requires-Dist: sphinxcontrib-mermaid>=0.7; extra == "doc"
+Requires-Dist: sphinx-autodoc-typehints>=1.16; extra == "doc"
+Requires-Dist: pydata-sphinx-theme; extra == "doc"
+Dynamic: license-file
+
+# ESRF Data Compressor
+
+**ESRF Data Compressor** is a command-line tool and Python library designed to compress large ESRF HDF5 datasets (3D volumes) and verify data consistency via SSIM. The default compression backend uses Blosc2 + Grok (JPEG2000).
+
+---
+
+## Features
+
+* **Discover raw HDF5 dataset files** under an experiment’s `RAW_DATA`
+
+  * Goes through the HDF5 Virtual Datasets to find the data to compress
+  * Allows to filter down scan by scan based on the value of a key
+
+* **Slice-by-slice compression**
+
+  * Uses Blosc2 + Grok (JPEG2000) on every slice of each 3D dataset (axis 0)
+  * User-configurable compression ratio (e.g. `--cratio 10`)
+
+* **Parallel execution**
+
+  * Automatically factors CPU cores into worker processes × per-process threads
+  * By default, each worker runs up to 4 Blosc2 threads (or falls back to 1 thread if < 4 cores)
+
+* **Non-destructive workflow**
+
+  1. `compress` writes a sibling file `<basename>_<compression_method>.h5` next to each original
+  2. `check` computes SSIM (first and last frames) and writes a report
+  3. `overwrite` (optional) swaps out the raw frame file (irreversible)
+
+* **Four simple CLI subcommands**
+
+  * `list`  Show all raw HDF5 files to be processed
+  * `compress` Generate compressed siblings
+  * `check`  Produce a per-dataset SSIM report between raw & compressed
+  * `overwrite` Atomically replace each raw frame file (irreversible)
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install esrf-data-compressor
+```
+
+Once installed, the `compress-hdf5` command will be available in your `PATH`.
+
+### From Source (for development)
+
+```bash
+git clone https://gitlab.esrf.fr/dau/esrf-data-compressor.git
+cd esrf-data-compressor
+
+# (Optional) Create & activate a virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install build dependencies & the package itself
+pip install .
+```
+
+---
+
+## Documentation
+
+Full documentation is available online:
+[ESRF Data Compressor Docs](https://esrf-data-compressor.readthedocs.io/en/latest/index.html)
+
+## Contributing & Development
+
+* **Clone** the repository:
+
+  ```bash
+  git clone https://gitlab.esrf.fr/dau/esrf-data-compressor.git
+  cd esrf-data-compressor
+  ```
+
+* **Install** dependencies (in a virtual environment):
+
+  ```bash
+  python -m venv venv
+  source venv/bin/activate
+  pip install -e "[dev]"
+  ```
+
+* **Run tests** with coverage:
+
+  ```bash
+  pytest -v --cov=esrf_data_compressor --cov-report=term-missing
+  ```
+
+* **Style:**
+
+  * `black .`
+  * `flake8 .`
+  * `ruff .`
+
+* **Build docs** (Sphinx + pydata theme):
+
+  ```bash
+  sphinx-build doc build/html
+  ```
+
+---
+
+## License
+
+This project is licensed under the [MIT License](LICENSE). See `LICENSE` for full text.
+
+---
+
+## Changelog
+
+All noteworthy changes are recorded in [CHANGELOG.md](CHANGELOG.md). Version 0.1.0 marks the first public release with:
+
+* Initial implementation of Blosc2 + Grok (JPEG2000) compression for 3D HDF5 datasets.
+* SSIM-based integrity check (first & last slice).
+* Four-command CLI (`list`, `compress`, `check`, `overwrite`).
+* Parallelism with worker×thread auto-factoring.
+
+For more details, see the full history in [CHANGELOG.md](CHANGELOG.md).

esrf_data_compressor-0.1.0/README.md

@@ -0,0 +1,122 @@
+# ESRF Data Compressor
+
+**ESRF Data Compressor** is a command-line tool and Python library designed to compress large ESRF HDF5 datasets (3D volumes) and verify data consistency via SSIM. The default compression backend uses Blosc2 + Grok (JPEG2000).
+
+---
+
+## Features
+
+* **Discover raw HDF5 dataset files** under an experiment’s `RAW_DATA`
+
+  * Goes through the HDF5 Virtual Datasets to find the data to compress
+  * Allows to filter down scan by scan based on the value of a key
+
+* **Slice-by-slice compression**
+
+  * Uses Blosc2 + Grok (JPEG2000) on every slice of each 3D dataset (axis 0)
+  * User-configurable compression ratio (e.g. `--cratio 10`)
+
+* **Parallel execution**
+
+  * Automatically factors CPU cores into worker processes × per-process threads
+  * By default, each worker runs up to 4 Blosc2 threads (or falls back to 1 thread if < 4 cores)
+
+* **Non-destructive workflow**
+
+  1. `compress` writes a sibling file `<basename>_<compression_method>.h5` next to each original
+  2. `check` computes SSIM (first and last frames) and writes a report
+  3. `overwrite` (optional) swaps out the raw frame file (irreversible)
+
+* **Four simple CLI subcommands**
+
+  * `list`  Show all raw HDF5 files to be processed
+  * `compress` Generate compressed siblings
+  * `check`  Produce a per-dataset SSIM report between raw & compressed
+  * `overwrite` Atomically replace each raw frame file (irreversible)
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install esrf-data-compressor
+```
+
+Once installed, the `compress-hdf5` command will be available in your `PATH`.
+
+### From Source (for development)
+
+```bash
+git clone https://gitlab.esrf.fr/dau/esrf-data-compressor.git
+cd esrf-data-compressor
+
+# (Optional) Create & activate a virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install build dependencies & the package itself
+pip install .
+```
+
+---
+
+## Documentation
+
+Full documentation is available online:
+[ESRF Data Compressor Docs](https://esrf-data-compressor.readthedocs.io/en/latest/index.html)
+
+## Contributing & Development
+
+* **Clone** the repository:
+
+  ```bash
+  git clone https://gitlab.esrf.fr/dau/esrf-data-compressor.git
+  cd esrf-data-compressor
+  ```
+
+* **Install** dependencies (in a virtual environment):
+
+  ```bash
+  python -m venv venv
+  source venv/bin/activate
+  pip install -e "[dev]"
+  ```
+
+* **Run tests** with coverage:
+
+  ```bash
+  pytest -v --cov=esrf_data_compressor --cov-report=term-missing
+  ```
+
+* **Style:**
+
+  * `black .`
+  * `flake8 .`
+  * `ruff .`
+
+* **Build docs** (Sphinx + pydata theme):
+
+  ```bash
+  sphinx-build doc build/html
+  ```
+
+---
+
+## License
+
+This project is licensed under the [MIT License](LICENSE). See `LICENSE` for full text.
+
+---
+
+## Changelog
+
+All noteworthy changes are recorded in [CHANGELOG.md](CHANGELOG.md). Version 0.1.0 marks the first public release with:
+
+* Initial implementation of Blosc2 + Grok (JPEG2000) compression for 3D HDF5 datasets.
+* SSIM-based integrity check (first & last slice).
+* Four-command CLI (`list`, `compress`, `check`, `overwrite`).
+* Parallelism with worker×thread auto-factoring.
+
+For more details, see the full history in [CHANGELOG.md](CHANGELOG.md).

esrf_data_compressor-0.1.0/pyproject.toml

@@ -0,0 +1,72 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "esrf-data-compressor"
+version = "0.1.0"
+authors = [{ name = "ESRF", email = "dau-pydev@esrf.fr" }]
+description = "A library to compress ESRF data and reduce their footprint"
+readme = { file = "README.md", content-type = "text/markdown" }
+license = { file = "LICENSE" }
+keywords = ["ESRF", "pathlib"]
+classifiers = [
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+requires-python = ">=3.10" #because of blosc2
+dependencies = [
+    "h5py",
+    "hdf5plugin",
+    "blosc2-grok",
+    "scikit-image",
+    "tqdm"
+]
+
+[project.urls]
+Homepage       = "https://gitlab.esrf.fr/dau/esrf-data-compressor"
+Documentation  = "https://esrf-data-compressor.readthedocs.io/"
+Repository     = "https://gitlab.esrf.fr/dau/esrf-data-compressor"
+Issues         = "https://gitlab.esrf.fr/dau/esrf-data-compressor/issues"
+Changelog      = "https://gitlab.esrf.fr/dau/esrf-data-compressor/-/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+test = [
+    "pytest >=7.0"
+]
+dev = [
+    "esrf-data-compressor[test]",
+    "black >=22",
+    "flake8 >=4.0",
+    "ruff"
+]
+doc = [
+    "sphinx >=6.0",
+    "sphinxcontrib-mermaid >=0.7",
+    "sphinx-autodoc-typehints >=1.16",
+    "pydata-sphinx-theme"
+]
+
+[project.scripts]
+compress-hdf5 = "esrf_data_compressor.cli:main"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"*" = ["*.json", "*.ipynb"]
+
+[tool.coverage.run]
+omit = ["*/tests/*"]
+
+[tool.isort]
+profile = "black"
+force_single_line = true

esrf_data_compressor-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

esrf_data_compressor-0.1.0/src/esrf_data_compressor/checker/run_check.py

@@ -0,0 +1,76 @@
+import os
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from tqdm import tqdm
+
+from esrf_data_compressor.checker.ssim import compute_ssim_for_file_pair
+
+
+def run_ssim_check(raw_files: list[str], method: str, report_path: str) -> None:
+    """
+    Given a list of raw HDF5 file paths, partitions into:
+      to_check → those with a sibling <stem>_<method>.h5
+      missing  → those without one
+
+    Writes a report to `report_path`:
+      - '=== NOT COMPRESSED FILES ===' listing each missing
+      - then for each to_check pair, computes SSIM in parallel and appends
+        per‐dataset SSIM lines under '=== <stem> ===' with full paths
+    """
+    to_check: list[tuple[str, str]] = []
+    missing: list[str] = []
+
+    # partition
+    for orig in raw_files:
+        dirname, fname = os.path.dirname(orig), os.path.basename(orig)
+        stem, _ = os.path.splitext(fname)
+        comp_path = os.path.join(dirname, f"{stem}_{method}.h5")
+        if os.path.exists(comp_path):
+            to_check.append((orig, comp_path))
+        else:
+            missing.append(orig)
+    print(
+        f"Found {len(to_check)} file pairs to check, {len(missing)} missing compressed files."
+    )
+
+    # write report
+    with open(report_path, "w") as rpt:
+        if missing:
+            rpt.write("=== NOT COMPRESSED FILES ===\n")
+            for orig in missing:
+                rpt.write(f"{orig} :: NO COMPRESSED DATASET FOUND\n")
+            rpt.write("\n")
+
+        if not to_check:
+            rpt.write("No file pairs to check (no compressed siblings found).\n")
+            return
+
+        # run SSIM in parallel
+        n_workers = min(len(to_check), os.cpu_count() or 1)
+        with ProcessPoolExecutor(max_workers=n_workers) as exe:
+            futures = {
+                exe.submit(compute_ssim_for_file_pair, orig, comp): (orig, comp)
+                for orig, comp in to_check
+            }
+
+            for fut in tqdm(
+                as_completed(futures),
+                total=len(futures),
+                desc="Checking SSIM (files)",
+                unit="file",
+            ):
+                orig, comp = futures[fut]
+                fname = os.path.basename(orig)
+                comp_name = os.path.basename(comp)
+                tqdm.write(f"Checking file: {fname} ↔ {comp_name}")
+                try:
+                    # get results
+                    basename, lines = fut.result()
+                    # write section with both file paths
+                    rpt.write(f"=== {basename} ===\n")
+                    rpt.write(f"Uncompressed file: {orig}\n")
+                    rpt.write(f"Compressed file: {comp}\n")
+                    for line in lines:
+                        rpt.write(line + "\n")
+                    rpt.write("\n")
+                except Exception as e:
+                    rpt.write(f"{orig} :: ERROR processing file pair: {e}\n\n")

esrf_data_compressor-0.1.0/src/esrf_data_compressor/checker/ssim.py

@@ -0,0 +1,87 @@
+# src/esrf_data_compressor/checker/ssim.py
+
+import os
+import numpy as np
+import h5py
+from skimage.metrics import structural_similarity as ssim
+
+
+def _select_win_size(H: int, W: int) -> int:
+    """
+    Choose an odd, valid window size for SSIM given slice dimensions H×W.
+    win_size = min(H, W, 7), made odd, at least 3.
+    """
+    win = min(H, W, 7)
+    if win % 2 == 0:
+        win -= 1
+    return max(win, 3)
+
+
+def compute_ssim_for_dataset_pair(
+    orig_path: str, comp_path: str, dataset_relpath: str
+) -> tuple[float, float]:
+    """
+    Given two HDF5 files and the relative 3D dataset path (e.g., 'entry_0000/ESRF-ID11/marana/data'),
+    compute SSIM on the first (z=0) and last (z=Z-1) slices.
+    Returns (ssim_first, ssim_last). If a slice is constant, SSIM = 1.0.
+    """
+    with h5py.File(orig_path, "r") as fo, h5py.File(comp_path, "r") as fc:
+        ds_o = fo[dataset_relpath]
+        ds_c = fc[dataset_relpath]
+
+        # Ensure both datasets are 3D
+        if ds_o.ndim != 3 or ds_c.ndim != 3:
+            raise IndexError(
+                f"Dataset '{dataset_relpath}' is not 3D (orig: {ds_o.ndim}D, comp: {ds_c.ndim}D)"
+            )
+
+        first_o = ds_o[0].astype(np.float64)
+        last_o = ds_o[-1].astype(np.float64)
+        first_c = ds_c[0].astype(np.float64)
+        last_c = ds_c[-1].astype(np.float64)
+
+        H, W = first_o.shape
+        win = _select_win_size(H, W)
+
+        def _slice_ssim(a: np.ndarray, b: np.ndarray) -> float:
+            amin, amax = a.min(), a.max()
+            if amax == amin:
+                return 1.0
+            dr = amax - amin
+            return ssim(a, b, data_range=dr, win_size=win)
+
+        s0 = _slice_ssim(first_o, first_c)
+        s1 = _slice_ssim(last_o, last_c)
+        return s0, s1
+
+
+def compute_ssim_for_file_pair(orig_path: str, comp_path: str) -> tuple[str, list[str]]:
+    """
+    Compute SSIM for every 3D dataset under `orig_path` vs. `comp_path`.
+    Returns (basename, [report_lines…]), where each line is either:
+    "<dataset_relpath>: SSIM_first=… SSIM_last=…" or an error message.
+    """
+    basename = os.path.basename(orig_path)
+    report_lines: list[str] = []
+
+    with h5py.File(orig_path, "r") as fo:
+        ds_paths: list[str] = []
+
+        def visitor(name, obj):
+            if isinstance(obj, h5py.Dataset) and obj.ndim == 3:
+                ds_paths.append(name)
+
+        fo.visititems(visitor)
+
+    if not ds_paths:
+        report_lines.append(f"No 3D datasets found in {basename}")
+        return basename, report_lines
+
+    for ds in ds_paths:
+        try:
+            s0, s1 = compute_ssim_for_dataset_pair(orig_path, comp_path, ds)
+            report_lines.append(f"{ds}: SSIM_first={s0:.4f}  SSIM_last={s1:.4f}")
+        except Exception as e:
+            report_lines.append(f"{ds}: ERROR computing SSIM: {e}")
+
+    return basename, report_lines