midas-transforms 0.1.0__tar.gz

midas_transforms-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: midas-transforms
+Version: 0.1.0
+Summary: Pure-Python/PyTorch FF-HEDM transforms (drop-in for MergeOverlappingPeaksAllZarr / CalcRadiusAllZarr / FitSetupZarr / SaveBinData)
+Author-email: Hemant Sharma <hsharma@anl.gov>
+License-Expression: BSD-3-Clause
+Project-URL: Homepage, https://github.com/marinerhemant/MIDAS
+Project-URL: Documentation, https://github.com/marinerhemant/MIDAS/tree/master/packages/midas_transforms
+Project-URL: Issues, https://github.com/marinerhemant/MIDAS/issues
+Keywords: MIDAS,HEDM,differentiable,PyTorch,far-field,diffraction,crystallography,polycrystal,grain,geometry-correction
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Physics
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.22
+Requires-Dist: torch>=2.0
+Requires-Dist: zarr<3.0,>=2.18
+Requires-Dist: numcodecs>=0.12
+Requires-Dist: midas-calibrate>=0.2.1
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-benchmark; extra == "dev"
+
+# midas-transforms
+
+Pure-Python / PyTorch FF-HEDM intermediate transforms — the four stages
+between peak-fitting and indexing in the MIDAS workflow.
+
+Drop-in replacement for these C binaries:
+
+| C binary                          | Python entry point         |
+|-----------------------------------|----------------------------|
+| `MergeOverlappingPeaksAllZarr`    | `midas-merge-peaks`        |
+| `CalcRadiusAllZarr`               | `midas-calc-radius`        |
+| `FitSetupZarr`                    | `midas-fit-setup`          |
+| `SaveBinData`                     | `midas-bin-data`           |
+
+Plus an end-to-end `midas-transforms pipeline <zarr>` that runs all four on
+GPU with no CSV / binary disk round-trips between stages.
+
+## Why
+
+- **Speed.** Vectorised PyTorch kernels target ≥ 1× C on CPU and 5–50× on
+  GPU at production scale. Binning and merge are the workflow's longest
+  tails; both are now broadcast + scatter operations.
+- **Bit-matching with C.** Every deterministic stage targets byte-exact
+  output at float64. The geometry refine (`DoFit==1`) targets
+  physics-meaningful tolerance via `midas-calibrate.refine_geometry`
+  (LM with ADAM fallback — no NLopt, no Nelder-Mead).
+- **Differentiable.** All geometry parameters that flow through detector
+  projection (`Lsd`, `BC_y`, `BC_z`, `tx/ty/tz`, `p0..p14`, `wedge`,
+  `dLsd`, `dP2`, residual-correction map values) carry full autograd
+  through `apply_tilt_distortion` — useful for joint calibration with
+  downstream grain mapping.
+- **CPU/GPU portable.** Single `device=` switch (cpu / cuda / mps). No
+  separate `.cu` codebase. No C extensions, no `cibuildwheel`.
+
+## Install
+
+```bash
+pip install -e packages/midas_transforms
+```
+
+(Until uploaded to PyPI; once released, `pip install midas-transforms`.)
+
+## Quick start
+
+### Per-stage CLI (drop-in for the C binaries)
+
+```bash
+midas-merge-peaks scan.zip
+midas-calc-radius scan.zip
+midas-fit-setup scan.zip
+midas-bin-data --result-folder .
+```
+
+### End-to-end Pipeline (intermediates stay on GPU)
+
+```bash
+midas-transforms pipeline scan.zip --device cuda --out-dir /scratch/run42
+```
+
+### Library
+
+```python
+from midas_transforms import Pipeline
+
+pipe = Pipeline.from_zarr("scan.zip", device="cuda")
+result = pipe.run()
+result.merge        # (N, 17) on GPU
+result.radius       # (N, 24) on GPU
+result.fit_setup    # InputAll/Extra tensors on GPU
+result.bins         # Spots/ExtraInfo/Data/nData tensors on GPU
+pipe.dump("/scratch/run42")
+```
+
+Or use stages individually:
+
+```python
+from midas_transforms import merge_overlapping_peaks, calc_radius, fit_setup, bin_data
+
+merge_overlapping_peaks(zarr_path="scan.zip", device="cuda")
+calc_radius(result_folder=".", zarr_params=..., device="cuda")
+fit_setup(result_folder=".", zarr_params=..., device="cuda")
+bin_data(result_folder=".", device="cuda")
+```
+
+## Wiring into `ff_MIDAS.py`
+
+`ff_MIDAS.py` accepts a `--useTorchTransforms 1` flag (mirrors the
+existing `--useTorchIndexer` / `--peakFitGPU` flags). When set, the
+workflow's `merge_overlaps`, `calc_radius`, `data_transform`, and
+`binning` stages dispatch to the Python entry points instead of the C
+binaries. Outputs are byte-compatible (or numerically equivalent for
+the `DoFit` geometry refine).
+
+## Limits in v0.1.0.dev0
+
+- **Pixel-overlap merge mode** (`UsePixelOverlap=1`) is not yet supported
+  — use the C `MergeOverlappingPeaksAllZarr` for that path.
+- **Scanning workflows** (`SaveBinDataScanning`, `MergeMultipleScans`) are
+  out of scope here; they belong with the broader scanning pipeline.
+- **Byte-level regression vs C** is in progress. Unit tests cover kernel
+  correctness; full end-to-end byte-equality against C goldens lands in
+  v0.1.0 proper.
+
+## See also
+
+- `dev/implementation_plan.md` — full design, scope, and risk register.
+- `midas-peakfit` — upstream peak-fitting (writes the consolidated HDF5).
+- `midas-index` — downstream indexer (consumes `Spots.bin`/`Data.bin`).
+- `midas-calibrate` — provides `geometry_torch.pixel_to_REta_torch` and
+  `refine_geometry` (LM solver) used by `fit_setup`.
+- `midas-hkls` — upstream `hkls.csv` generation.

midas_transforms-0.1.0/README.md

@@ -0,0 +1,111 @@
+# midas-transforms
+
+Pure-Python / PyTorch FF-HEDM intermediate transforms — the four stages
+between peak-fitting and indexing in the MIDAS workflow.
+
+Drop-in replacement for these C binaries:
+
+| C binary                          | Python entry point         |
+|-----------------------------------|----------------------------|
+| `MergeOverlappingPeaksAllZarr`    | `midas-merge-peaks`        |
+| `CalcRadiusAllZarr`               | `midas-calc-radius`        |
+| `FitSetupZarr`                    | `midas-fit-setup`          |
+| `SaveBinData`                     | `midas-bin-data`           |
+
+Plus an end-to-end `midas-transforms pipeline <zarr>` that runs all four on
+GPU with no CSV / binary disk round-trips between stages.
+
+## Why
+
+- **Speed.** Vectorised PyTorch kernels target ≥ 1× C on CPU and 5–50× on
+  GPU at production scale. Binning and merge are the workflow's longest
+  tails; both are now broadcast + scatter operations.
+- **Bit-matching with C.** Every deterministic stage targets byte-exact
+  output at float64. The geometry refine (`DoFit==1`) targets
+  physics-meaningful tolerance via `midas-calibrate.refine_geometry`
+  (LM with ADAM fallback — no NLopt, no Nelder-Mead).
+- **Differentiable.** All geometry parameters that flow through detector
+  projection (`Lsd`, `BC_y`, `BC_z`, `tx/ty/tz`, `p0..p14`, `wedge`,
+  `dLsd`, `dP2`, residual-correction map values) carry full autograd
+  through `apply_tilt_distortion` — useful for joint calibration with
+  downstream grain mapping.
+- **CPU/GPU portable.** Single `device=` switch (cpu / cuda / mps). No
+  separate `.cu` codebase. No C extensions, no `cibuildwheel`.
+
+## Install
+
+```bash
+pip install -e packages/midas_transforms
+```
+
+(Until uploaded to PyPI; once released, `pip install midas-transforms`.)
+
+## Quick start
+
+### Per-stage CLI (drop-in for the C binaries)
+
+```bash
+midas-merge-peaks scan.zip
+midas-calc-radius scan.zip
+midas-fit-setup scan.zip
+midas-bin-data --result-folder .
+```
+
+### End-to-end Pipeline (intermediates stay on GPU)
+
+```bash
+midas-transforms pipeline scan.zip --device cuda --out-dir /scratch/run42
+```
+
+### Library
+
+```python
+from midas_transforms import Pipeline
+
+pipe = Pipeline.from_zarr("scan.zip", device="cuda")
+result = pipe.run()
+result.merge        # (N, 17) on GPU
+result.radius       # (N, 24) on GPU
+result.fit_setup    # InputAll/Extra tensors on GPU
+result.bins         # Spots/ExtraInfo/Data/nData tensors on GPU
+pipe.dump("/scratch/run42")
+```
+
+Or use stages individually:
+
+```python
+from midas_transforms import merge_overlapping_peaks, calc_radius, fit_setup, bin_data
+
+merge_overlapping_peaks(zarr_path="scan.zip", device="cuda")
+calc_radius(result_folder=".", zarr_params=..., device="cuda")
+fit_setup(result_folder=".", zarr_params=..., device="cuda")
+bin_data(result_folder=".", device="cuda")
+```
+
+## Wiring into `ff_MIDAS.py`
+
+`ff_MIDAS.py` accepts a `--useTorchTransforms 1` flag (mirrors the
+existing `--useTorchIndexer` / `--peakFitGPU` flags). When set, the
+workflow's `merge_overlaps`, `calc_radius`, `data_transform`, and
+`binning` stages dispatch to the Python entry points instead of the C
+binaries. Outputs are byte-compatible (or numerically equivalent for
+the `DoFit` geometry refine).
+
+## Limits in v0.1.0.dev0
+
+- **Pixel-overlap merge mode** (`UsePixelOverlap=1`) is not yet supported
+  — use the C `MergeOverlappingPeaksAllZarr` for that path.
+- **Scanning workflows** (`SaveBinDataScanning`, `MergeMultipleScans`) are
+  out of scope here; they belong with the broader scanning pipeline.
+- **Byte-level regression vs C** is in progress. Unit tests cover kernel
+  correctness; full end-to-end byte-equality against C goldens lands in
+  v0.1.0 proper.
+
+## See also
+
+- `dev/implementation_plan.md` — full design, scope, and risk register.
+- `midas-peakfit` — upstream peak-fitting (writes the consolidated HDF5).
+- `midas-index` — downstream indexer (consumes `Spots.bin`/`Data.bin`).
+- `midas-calibrate` — provides `geometry_torch.pixel_to_REta_torch` and
+  `refine_geometry` (LM solver) used by `fit_setup`.
+- `midas-hkls` — upstream `hkls.csv` generation.

midas_transforms-0.1.0/midas_transforms/__init__.py

@@ -0,0 +1,46 @@
+"""midas-transforms: Pure-Python/PyTorch FF-HEDM transforms.
+
+Drop-in replacement for the four C binaries that sit between peak-fitting
+(``midas-peakfit``) and indexing (``midas-index``):
+
+- ``MergeOverlappingPeaksAllZarr`` -> ``midas_transforms.merge_overlapping_peaks``
+- ``CalcRadiusAllZarr``            -> ``midas_transforms.calc_radius``
+- ``FitSetupZarr``                 -> ``midas_transforms.fit_setup``
+- ``SaveBinData``                  -> ``midas_transforms.bin_data``
+
+Two equally-supported usage modes:
+
+**Mode 1 - per-stage (round-trips through disk, like the C binaries):**
+
+    from midas_transforms import merge_overlapping_peaks, calc_radius, fit_setup, bin_data
+    merge_overlapping_peaks(zarr_path="...", result_folder="...", device="cuda")
+    calc_radius(result_folder="...", device="cuda")
+    fit_setup(result_folder="...", device="cuda")
+    bin_data(result_folder="...", device="cuda")
+
+**Mode 2 - chained Pipeline (intermediates stay on GPU, only final outputs written):**
+
+    from midas_transforms import Pipeline
+    pipe = Pipeline.from_zarr(zarr_path, device="cuda")
+    result = pipe.run()
+    pipe.dump(out_dir)
+
+See ``dev/implementation_plan.md`` for design and roadmap.
+"""
+
+__version__ = "0.1.0"
+
+from .merge import merge_overlapping_peaks
+from .radius import calc_radius
+from .fit_setup import fit_setup
+from .bin_data import bin_data
+from .pipeline import Pipeline
+
+__all__ = [
+    "merge_overlapping_peaks",
+    "calc_radius",
+    "fit_setup",
+    "bin_data",
+    "Pipeline",
+    "__version__",
+]

midas_transforms-0.1.0/midas_transforms/__main__.py

@@ -0,0 +1,6 @@
+"""Allow ``python -m midas_transforms <stage> [args]``."""
+
+from .cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

midas_transforms-0.1.0/midas_transforms/bin_data/__init__.py

@@ -0,0 +1,10 @@
+"""bin_data — replaces the C ``SaveBinData`` binary.
+
+Reads ``InputAll.csv``, ``InputAllExtraInfoFittingAll.csv``, ``paramstest.txt``;
+writes ``Spots.bin``, ``ExtraInfo.bin``, and (unless ``NoSaveAll==1``)
+``Data.bin`` + ``nData.bin``.
+"""
+
+from .core import bin_data, BinDataResult
+
+__all__ = ["bin_data", "BinDataResult"]