midas-process-grains 0.1.0__tar.gz

midas_process_grains-0.1.0/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: midas-process-grains
+Version: 0.1.0
+Summary: Pure-Python/PyTorch FF-HEDM grain-determination + strain pipeline (drop-in replacement for ProcessGrains)
+Author-email: Hemant Sharma <hsharma@anl.gov>
+License-Expression: BSD-3-Clause
+Project-URL: Homepage, https://github.com/marinerhemant/MIDAS
+Project-URL: Documentation, https://github.com/marinerhemant/MIDAS/tree/master/packages/midas_process_grains
+Project-URL: Issues, https://github.com/marinerhemant/MIDAS/issues
+Keywords: MIDAS,HEDM,grain-determination,strain,PyTorch,far-field,diffraction,crystallography,polycrystal,ProcessGrains
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Physics
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.22
+Requires-Dist: scipy>=1.9
+Requires-Dist: torch>=2.0
+Requires-Dist: h5py>=3.7
+Requires-Dist: midas-stress<1.0,>=0.5.0
+Requires-Dist: midas-hkls>=0.1.0
+Requires-Dist: midas-diffract>=0.1.0
+Requires-Dist: midas-transforms>=0.1.0
+Requires-Dist: midas-index>=0.3.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-benchmark; extra == "dev"
+
+# midas-process-grains
+
+Pure-Python (PyTorch) replacement for `FF_HEDM/src/ProcessGrains.c`. Reads the
+binary outputs of the upstream MIDAS pipeline (`OrientPosFit.bin`, `Key.bin`,
+`ProcessKey.bin`, `IndexBestFull.bin`, `FitBest.bin`) and emits the canonical
+`Grains.csv` / `SpotMatrix.csv` / `GrainIDsKey.csv` files.
+
+## Scope: bit-level parity with the C reference
+
+The package has one shippable mode: **`c_parity`**, which mirrors
+`FF_HEDM/src/ProcessGrains.c` exactly. The Stage 1 cluster-build, the Pass A
+position+orientation dedup, the confidence cut, and the 47-column
+`Grains.csv` / 12-column `SpotMatrix.csv` / `GrainIDsKey.csv` writers all
+follow the C source line-for-line.
+
+On the `peakfit_hard` reference dataset (357 k seeds → 22 k grains), the
+Python output is **bit-identical** to the C output for every column except
+the Kenesei strain tensor — see "Parity verdict" below.
+
+Earlier experimental modes (`legacy`, `paper_claim`, `spot_aware`) shipped in
+v0.1 internal builds are still present in the source tree but are not
+exposed through the supported public surface. They will be removed in a
+future cleanup.
+
+## CLI
+
+```bash
+midas-process-grains paramstest.txt 8 --mode c_parity --device cuda
+```
+
+The CLI reads `paramstest.txt` from the run directory, writes the three
+output files into `--out-dir` (defaulting to the run directory), and exits.
+Use `--device cpu` if you do not have a CUDA-capable GPU.
+
+```bash
+midas-process-grains paramstest.txt 8 \
+    --mode c_parity \
+    --device cuda \
+    --min-nr-spots 1 \
+    --out-dir ./output
+```
+
+`--min-nr-spots` matches the `MinNrSpots` parameter in `paramstest.txt`
+(Stage 1 cluster-size cutoff). Default is `1`, which keeps every cluster.
+
+## Library
+
+```python
+from midas_process_grains.compute.c_parity_run import (
+    run_c_parity_pipeline_from_disk,
+)
+
+run_c_parity_pipeline_from_disk(
+    run_dir="/scratch/.../LayerNr_1",
+    out_dir="/scratch/.../LayerNr_1",
+    device="cuda",          # or "cpu"
+    min_nr_spots=1,
+)
+```
+
+For lower-level access (run only Stage 1, only Pass A, only the writers,
+etc.) see `midas_process_grains.compute.c_parity` and
+`midas_process_grains.compute.c_parity_emit`.
+
+## Parity verdict (peakfit_hard, 22 k grains)
+
+| Column | Python vs C max abs diff |
+|---|---|
+| `GrainID`, OM (9), `X`, `Y`, `Z`, lattice (6), `DiffPos`, `DiffOme`, `DiffAngle`, `GrainRadius`, `Confidence`, **Fable strain** (9), `Eul0`, `Eul1`, `Eul2` | **0** (bit-identical) |
+| **Kenesei strain** (9 components) | ≤ 35 µε (NLOPT vs SciPy `lsq_linear` solver convergence) |
+| `RMSErrorStrain` | ≤ 0.085 µε |
+
+Cluster identity: 21,504 of 22,003 grains share the same `rep_pos` between
+the C and Python runs. The remaining ~2 % is OMP `atomic_test_and_set`
+non-determinism in the C source — running C on the same input twice produces
+two outputs that disagree on **846 grains** (3.8 %). Python and a current C
+rerun agree at **99.58 %** — closer than C agrees with itself across runs.
+
+## Performance
+
+Wall time on a single peakfit_hard run (8-thread alleppey, NVIDIA H100 NVL):
+
+| Pipeline | Wall | CPU time |
+|---|---:|---:|
+| C ProcessGrains, 8-thread OMP | 50 min | 396 min |
+| Python `c_parity`, CPU 8-thread torch | 119 s | 676 s |
+| Python `c_parity`, CUDA H100 | **113 s** | **125 s** |
+
+Roughly **27× faster** wall-clock and **190× less CPU** on GPU. The biggest
+wins are (a) Pass A's `O(N)` spatial-hash replacing C's `O(N²)` all-pairs,
+(b) precomputing the misorientation graph for all spot-overlap candidates in
+one batched torch call, and (c) batching all per-grain Kenesei solves into a
+single `torch.linalg.solve` over a `(B, 6, 6)` stack.
+
+## Inputs
+
+The pipeline reads the standard MIDAS run-directory layout:
+
+```
+<run_dir>/
+  paramstest.txt
+  hkls.csv
+  IDsHash.csv
+  SpotsToIndex.csv
+  InputAllExtraInfoFittingAll.csv
+  Output/
+    IndexBestFull.bin
+    FitBest.bin
+  Results/
+    OrientPosFit.bin
+    Key.bin
+    ProcessKey.bin
+```
+
+## Outputs
+
+```
+<out_dir>/
+  Grains.csv            # 47 columns, C ProcessGrains layout
+  GrainIDsKey.csv       # one line per kept grain
+  SpotMatrix.csv        # 12 columns, C ProcessGrains layout
+```
+
+## Implementation notes
+
+* Stage 1 (`FindInternalAngles` equivalent) does a recursive DFS over the
+  `ProcessKey`-defined spot-overlap candidate graph, filtered by misori
+  < `0.4°`. The misorientation for every candidate edge is precomputed in
+  one batched torch call before the DFS.
+* Pass A (`misori < 0.1° AND |Δpos| < 5 µm` dedup) uses a 5 µm spatial hash
+  on rep positions to limit pairs to those within the position threshold,
+  then vectorised misori on the surviving pairs. Greedy outer-serial dedup
+  matches C's order.
+* Confidence filter `OPF[26] >= 0.05` (matches C `OPs[ri][22] < 0.05` cut).
+* Strain — Fable-Beaudoin from refined lattice (closed form), Kenesei from
+  per-spot lstsq (`scipy.optimize.lsq_linear` with the same ±0.01 bounds C
+  uses with NLOPT Nelder-Mead). Kenesei is solved in batch over all grains
+  in a single `torch.linalg.solve(GTG + λI, GTb)` call when running on GPU.
+* Euler angles use C's exact `OrientMat2Euler` algorithm with the
+  `sin_cos_to_angle(s, c) = acos(c) if s ≥ 0 else 2π − acos(c)` helper.
+  Output is in **radians**, matching C.
+
+See the docstrings in `compute/c_parity.py` and `compute/c_parity_emit.py`
+for the full algorithm spec, with line-number references back to the C
+source.

midas_process_grains-0.1.0/README.md

@@ -0,0 +1,145 @@
+# midas-process-grains
+
+Pure-Python (PyTorch) replacement for `FF_HEDM/src/ProcessGrains.c`. Reads the
+binary outputs of the upstream MIDAS pipeline (`OrientPosFit.bin`, `Key.bin`,
+`ProcessKey.bin`, `IndexBestFull.bin`, `FitBest.bin`) and emits the canonical
+`Grains.csv` / `SpotMatrix.csv` / `GrainIDsKey.csv` files.
+
+## Scope: bit-level parity with the C reference
+
+The package has one shippable mode: **`c_parity`**, which mirrors
+`FF_HEDM/src/ProcessGrains.c` exactly. The Stage 1 cluster-build, the Pass A
+position+orientation dedup, the confidence cut, and the 47-column
+`Grains.csv` / 12-column `SpotMatrix.csv` / `GrainIDsKey.csv` writers all
+follow the C source line-for-line.
+
+On the `peakfit_hard` reference dataset (357 k seeds → 22 k grains), the
+Python output is **bit-identical** to the C output for every column except
+the Kenesei strain tensor — see "Parity verdict" below.
+
+Earlier experimental modes (`legacy`, `paper_claim`, `spot_aware`) shipped in
+v0.1 internal builds are still present in the source tree but are not
+exposed through the supported public surface. They will be removed in a
+future cleanup.
+
+## CLI
+
+```bash
+midas-process-grains paramstest.txt 8 --mode c_parity --device cuda
+```
+
+The CLI reads `paramstest.txt` from the run directory, writes the three
+output files into `--out-dir` (defaulting to the run directory), and exits.
+Use `--device cpu` if you do not have a CUDA-capable GPU.
+
+```bash
+midas-process-grains paramstest.txt 8 \
+    --mode c_parity \
+    --device cuda \
+    --min-nr-spots 1 \
+    --out-dir ./output
+```
+
+`--min-nr-spots` matches the `MinNrSpots` parameter in `paramstest.txt`
+(Stage 1 cluster-size cutoff). Default is `1`, which keeps every cluster.
+
+## Library
+
+```python
+from midas_process_grains.compute.c_parity_run import (
+    run_c_parity_pipeline_from_disk,
+)
+
+run_c_parity_pipeline_from_disk(
+    run_dir="/scratch/.../LayerNr_1",
+    out_dir="/scratch/.../LayerNr_1",
+    device="cuda",          # or "cpu"
+    min_nr_spots=1,
+)
+```
+
+For lower-level access (run only Stage 1, only Pass A, only the writers,
+etc.) see `midas_process_grains.compute.c_parity` and
+`midas_process_grains.compute.c_parity_emit`.
+
+## Parity verdict (peakfit_hard, 22 k grains)
+
+| Column | Python vs C max abs diff |
+|---|---|
+| `GrainID`, OM (9), `X`, `Y`, `Z`, lattice (6), `DiffPos`, `DiffOme`, `DiffAngle`, `GrainRadius`, `Confidence`, **Fable strain** (9), `Eul0`, `Eul1`, `Eul2` | **0** (bit-identical) |
+| **Kenesei strain** (9 components) | ≤ 35 µε (NLOPT vs SciPy `lsq_linear` solver convergence) |
+| `RMSErrorStrain` | ≤ 0.085 µε |
+
+Cluster identity: 21,504 of 22,003 grains share the same `rep_pos` between
+the C and Python runs. The remaining ~2 % is OMP `atomic_test_and_set`
+non-determinism in the C source — running C on the same input twice produces
+two outputs that disagree on **846 grains** (3.8 %). Python and a current C
+rerun agree at **99.58 %** — closer than C agrees with itself across runs.
+
+## Performance
+
+Wall time on a single peakfit_hard run (8-thread alleppey, NVIDIA H100 NVL):
+
+| Pipeline | Wall | CPU time |
+|---|---:|---:|
+| C ProcessGrains, 8-thread OMP | 50 min | 396 min |
+| Python `c_parity`, CPU 8-thread torch | 119 s | 676 s |
+| Python `c_parity`, CUDA H100 | **113 s** | **125 s** |
+
+Roughly **27× faster** wall-clock and **190× less CPU** on GPU. The biggest
+wins are (a) Pass A's `O(N)` spatial-hash replacing C's `O(N²)` all-pairs,
+(b) precomputing the misorientation graph for all spot-overlap candidates in
+one batched torch call, and (c) batching all per-grain Kenesei solves into a
+single `torch.linalg.solve` over a `(B, 6, 6)` stack.
+
+## Inputs
+
+The pipeline reads the standard MIDAS run-directory layout:
+
+```
+<run_dir>/
+  paramstest.txt
+  hkls.csv
+  IDsHash.csv
+  SpotsToIndex.csv
+  InputAllExtraInfoFittingAll.csv
+  Output/
+    IndexBestFull.bin
+    FitBest.bin
+  Results/
+    OrientPosFit.bin
+    Key.bin
+    ProcessKey.bin
+```
+
+## Outputs
+
+```
+<out_dir>/
+  Grains.csv            # 47 columns, C ProcessGrains layout
+  GrainIDsKey.csv       # one line per kept grain
+  SpotMatrix.csv        # 12 columns, C ProcessGrains layout
+```
+
+## Implementation notes
+
+* Stage 1 (`FindInternalAngles` equivalent) does a recursive DFS over the
+  `ProcessKey`-defined spot-overlap candidate graph, filtered by misori
+  < `0.4°`. The misorientation for every candidate edge is precomputed in
+  one batched torch call before the DFS.
+* Pass A (`misori < 0.1° AND |Δpos| < 5 µm` dedup) uses a 5 µm spatial hash
+  on rep positions to limit pairs to those within the position threshold,
+  then vectorised misori on the surviving pairs. Greedy outer-serial dedup
+  matches C's order.
+* Confidence filter `OPF[26] >= 0.05` (matches C `OPs[ri][22] < 0.05` cut).
+* Strain — Fable-Beaudoin from refined lattice (closed form), Kenesei from
+  per-spot lstsq (`scipy.optimize.lsq_linear` with the same ±0.01 bounds C
+  uses with NLOPT Nelder-Mead). Kenesei is solved in batch over all grains
+  in a single `torch.linalg.solve(GTG + λI, GTb)` call when running on GPU.
+* Euler angles use C's exact `OrientMat2Euler` algorithm with the
+  `sin_cos_to_angle(s, c) = acos(c) if s ≥ 0 else 2π − acos(c)` helper.
+  Output is in **radians**, matching C.
+
+See the docstrings in `compute/c_parity.py` and `compute/c_parity_emit.py`
+for the full algorithm spec, with line-number references back to the C
+source.

midas_process_grains-0.1.0/midas_process_grains/__init__.py

@@ -0,0 +1,41 @@
+"""midas-process-grains: pure-Python FF-HEDM grain-determination + strain.
+
+Drop-in replacement for ``FF_HEDM/src/ProcessGrains.c``. Reads the binary
+outputs of the upstream pipeline (``IndexBest{,Full}.bin``, ``FitBest.bin``,
+``Key.bin``, ``OrientPosFit.bin``, ``ProcessKey.bin``) and emits the canonical
+MIDAS grain artefacts (``Grains.csv``, ``SpotMatrix.csv``, ``GrainIDsKey.csv``).
+
+Three operating modes (`mode=` kwarg):
+
+  * ``"legacy"``      — bit-for-bit reproduce the current C ProcessGrains
+                        output (used for regression tests during migration).
+  * ``"paper_claim"`` — the §3.6 spec from the MIDAS methodology paper that
+                        the current C code does not actually enforce
+                        (90% shared peaks, 0.01° misorientation, 15 µm pos).
+  * ``"spot_aware"``  — DEFAULT. Symmetry-aware row-aligned per-hkl SpotID
+                        consistency, Jaccard pre-screen, union-of-cluster
+                        emission, lstsq strain. No position gate.
+"""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from .params import ProcessGrainsParams, read_paramstest_pg
+
+__all__ = [
+    "__version__",
+    "ProcessGrainsParams",
+    "read_paramstest_pg",
+]
+
+
+def __getattr__(name):
+    """Lazy import of pipeline-level symbols (avoid module cycles during build-up)."""
+    if name == "ProcessGrains":
+        from .pipeline import ProcessGrains
+        return ProcessGrains
+    if name == "ProcessGrainsResult":
+        from .result import ProcessGrainsResult
+        return ProcessGrainsResult
+    raise AttributeError(f"module 'midas_process_grains' has no attribute {name!r}")

midas_process_grains-0.1.0/midas_process_grains/__main__.py

@@ -0,0 +1,9 @@
+"""``python -m midas_process_grains`` shim."""
+from __future__ import annotations
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

midas_process_grains-0.1.0/midas_process_grains/cli.py

@@ -0,0 +1,145 @@
+"""CLI: ``midas-process-grains`` (and ``python -m midas_process_grains``).
+
+Mirrors the C ``ProcessGrains`` invocation pattern (single positional arg:
+the parameter file path) with optional flags to override mode, device,
+dtype, and a couple of merge knobs.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+from typing import List, Optional
+
+from . import __version__
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="midas-process-grains",
+        description=(
+            "Pure-Python FF-HEDM grain-determination + strain pipeline "
+            "(drop-in for ProcessGrains)."
+        ),
+    )
+    p.add_argument(
+        "param_file",
+        type=Path,
+        help="Path to paramstest.txt (the same file IndexerOMP/FitPosOrStrains "
+             "consumed for this run).",
+    )
+    p.add_argument(
+        "num_procs", type=int, nargs="?", default=1,
+        help="CPU thread count (used only on cpu device). Default 1.",
+    )
+    p.add_argument(
+        "--mode", choices=("legacy", "paper_claim", "spot_aware", "c_parity"),
+        default="spot_aware",
+        help="Pipeline mode. Use 'c_parity' for a bit-level replica of the "
+             "C ProcessGrains pipeline (writes Grains.csv, GrainIDsKey.csv, "
+             "SpotMatrix.csv in C's exact format).",
+    )
+    p.add_argument(
+        "--min-nr-spots", type=int, default=None,
+        help="MinNrSpots threshold (Stage 1 cluster-size cutoff). C ProcessGrains "
+             "default is 1; the original peakfit_hard run used 3.",
+    )
+    p.add_argument("--device", choices=("cpu", "cuda", "mps"), default=None)
+    p.add_argument("--dtype", choices=("float32", "float64"), default=None)
+    p.add_argument("--misori-tol", type=float, default=None,
+                   help="Override the Phase 1 misorientation tolerance (degrees).")
+    p.add_argument(
+        "--strain-method",
+        choices=(
+            "kenesei", "kenesei_unbounded", "fable_beaudoin", "both",
+            # backwards-compat aliases (resolved in params.validated())
+            "lstsq", "lattice",
+        ),
+        default=None,
+        help="Per-grain strain solver. Default: kenesei (bounded ±0.01, "
+             "matches C reference). Use fable_beaudoin for the lattice-"
+             "parameter route, or both to emit each.",
+    )
+    p.add_argument("--material", default=None,
+                   help="Material name for stiffness lookup (e.g. Cu, Ni, Fe).")
+    p.add_argument("--stiffness-file", type=Path, default=None,
+                   help="Path to a 6×6 stiffness matrix (CSV/TXT/NPY).")
+    p.add_argument("--out-dir", type=Path, default=None,
+                   help="Where to write outputs. Default: param-file directory.")
+    p.add_argument("--no-h5", action="store_true",
+                   help="Skip writing data_consolidated.h5.")
+    p.add_argument("--no-diagnostics-h5", action="store_true",
+                   help="Skip writing processgrains_diagnostics.h5.")
+    p.add_argument("--max-seeds", type=int, default=None,
+                   help="Process only the first N alive seeds (smoke / dev).")
+    p.add_argument("--version", action="version",
+                   version=f"midas-process-grains {__version__}")
+    return p
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    """CLI entry point. Returns process exit code."""
+    args = _build_parser().parse_args(argv)
+
+    from .device import apply_cpu_threads, resolve_device, resolve_dtype
+    from .pipeline import ProcessGrains
+
+    # ── c_parity mode: dispatch to the C-replica pipeline and return ────────
+    if args.mode == "c_parity":
+        from .compute.c_parity_run import run_c_parity_pipeline_from_disk
+        run_dir = args.param_file.parent
+        out_dir = args.out_dir if args.out_dir is not None else run_dir
+        device = resolve_device(args.device)
+        # torch device strings: "cpu" / "cuda" / "cuda:0" / "mps"
+        device_str = str(device) if not hasattr(device, "type") else (
+            device.type if device.index is None else f"{device.type}:{device.index}"
+        )
+        apply_cpu_threads(args.num_procs, device)
+        run_c_parity_pipeline_from_disk(
+            run_dir=run_dir,
+            out_dir=out_dir,
+            min_nr_spots=(args.min_nr_spots
+                          if args.min_nr_spots is not None else 1),
+            device=device_str,
+        )
+        return 0
+
+    pg = ProcessGrains.from_param_file(
+        args.param_file,
+        device=args.device,
+        dtype=args.dtype,
+    )
+    apply_cpu_threads(args.num_procs, pg.device)
+
+    # CLI overrides on top of paramstest.
+    if args.misori_tol is not None:
+        pg.params.MisoriTol = float(args.misori_tol)
+    if args.strain_method is not None:
+        pg.params.StrainMethod = args.strain_method
+    if args.material is not None:
+        pg.params.MaterialName = args.material
+    if args.stiffness_file is not None:
+        pg.params.StiffnessFile = str(args.stiffness_file)
+    pg.params = pg.params.validated()
+
+    if args.max_seeds is not None:
+        pg.params.raw["__max_seeds__"] = [str(args.max_seeds)]
+
+    result = pg.run(mode=args.mode)
+    out_dir = args.out_dir if args.out_dir is not None else pg.run_dir
+    result.write(
+        out_dir,
+        h5=not args.no_h5,
+        diagnostics_h5=not args.no_diagnostics_h5,
+    )
+    print(
+        f"midas-process-grains {__version__}: "
+        f"{result.n_grains} grains written to {out_dir}",
+        file=sys.stderr,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

midas_process_grains-0.1.0/midas_process_grains/compute/__init__.py

@@ -0,0 +1,35 @@
+"""Compute submodule.
+
+Pure-tensor implementations of:
+  - symmetry table builders (24-op cubic / hexagonal / etc.)
+  - hkl-row permutation under each symmetry op
+  - cluster-level orientation canonicalisation
+  - misorientation graph + connected components (Phase 1)
+  - spot-aware sub-clustering (Phase 2)
+  - per-hkl SpotID conflict resolution (Phase 3)
+  - lstsq strain solver (Phase 4)
+  - Hooke's-law stress (Phase 5)
+  - twin post-processor
+
+Design rule: every public function takes a ``device`` / ``dtype`` argument or
+honours the caller's tensors' device + dtype, mirroring the conventions of
+``midas_index`` and ``midas_transforms``.
+"""
+
+from .symmetry import (
+    SymmetryTable,
+    build_symmetry_table,
+    apply_sym_to_hkl_int,
+)
+from .canonicalize import (
+    pick_best_sym_op,
+    align_member_to_rep,
+)
+
+__all__ = [
+    "SymmetryTable",
+    "build_symmetry_table",
+    "apply_sym_to_hkl_int",
+    "pick_best_sym_op",
+    "align_member_to_rep",
+]