midas-pipeline 0.1.0__tar.gz

midas_pipeline-0.1.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: midas-pipeline
+Version: 0.1.0
+Summary: Unified MIDAS HEDM orchestrator. FF is the single-scan degeneracy of PF; one orchestrator, --scan-mode {ff,pf}.
+Author-email: Hemant Sharma <hsharma@anl.gov>
+License-Expression: BSD-3-Clause
+Project-URL: Homepage, https://github.com/marinerhemant/MIDAS
+Project-URL: Documentation, https://github.com/marinerhemant/MIDAS
+Project-URL: Issues, https://github.com/marinerhemant/MIDAS/issues
+Keywords: MIDAS,HEDM,FF-HEDM,PF-HEDM,scanning,diffraction,pipeline,PyTorch,multi-detector,crystallography,polycrystal,tomography
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Physics
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.22
+Requires-Dist: torch>=2.0
+Requires-Dist: zarr<3,>=2.14
+Requires-Dist: h5py>=3.7
+Requires-Dist: tifffile
+Requires-Dist: pandas>=1.5
+Requires-Dist: midas-hkls>=0.2.0
+Requires-Dist: midas-peakfit>=0.2.0
+Requires-Dist: midas-transforms>=0.1.1
+Requires-Dist: midas-index>=0.3.0
+Requires-Dist: midas-fit-grain>=0.1.0a0
+Requires-Dist: midas-process-grains>=0.1.0
+Requires-Dist: midas-diffract>=0.1.0
+Requires-Dist: midas-parsl-configs>=0.1.0
+Requires-Dist: midas-stress>=0.6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: ipykernel; extra == "dev"
+Provides-Extra: notebook
+Requires-Dist: matplotlib; extra == "notebook"
+Requires-Dist: ipykernel; extra == "notebook"
+Requires-Dist: jupyter; extra == "notebook"
+Provides-Extra: fast
+Requires-Dist: numba>=0.56; extra == "fast"
+
+# midas-pipeline
+
+End-to-end MIDAS HEDM orchestrator. **FF is the single-scan degeneracy of PF.** One package, one CLI, two scan modes.
+
+## Status
+
+**Alpha (0.1.0a0) — end-to-end PF and FF paths live.** The scanning indexer matches the C `IndexerScanningOMP` reference on its 1-voxel C-parity gate (seed identity, solution counts, voxel-center positions exact; orientation matrices within mrad-scale, the refiner closes the gap downstream). Cross-implementation perf optimization (~130s/voxel single-threaded today) is open work — see [project_midas_index_scanning_perf](https://github.com/marinerhemant/MIDAS/blob/master/packages/midas_pipeline/dev/perf-notes.md) tracking.
+
+Stages call in-process Python kernels via `midas-index` / `midas-fit-grain` / `midas-transforms` / `midas-stress`. FF mode shells out to `python -m midas_index` and `python -m midas_fit_grain` (same kernels, subprocess for the FF parity-preserving pattern). No CUDA C; GPU is torch-only.
+
+## Install
+
+```bash
+pip install -e packages/midas_pipeline
+```
+
+## CLI
+
+```bash
+midas-pipeline run --scan-mode {ff,pf,auto} --params Parameters.txt --result rundir/
+midas-pipeline status rundir/
+midas-pipeline resume rundir/ --from <stage>
+midas-pipeline reprocess rundir/
+midas-pipeline inspect rundir/LayerNr_1/
+midas-pipeline simulate --out simdir/ --params Parameters.txt
+midas-pipeline seed --params ... --output UniqueOrientations.csv
+```
+
+When `--scan-mode` is omitted (default `auto`), the CLI sniffs the parameter file: `nScans > 1` or presence of `BeamSize` / scanning keys → `pf`, otherwise `ff`. For PF mode, `--n-scans`, `--scan-step`, `--beam-size`, and `--scan-pos-tol` default to values in the params file (CLI flags override).
+
+## Coexistence with `midas-ff-pipeline`
+
+The legacy `midas-ff-pipeline` console-script is preserved as an independent FF orchestrator (its own kernels, its own CLI). It is **not** deprecated by `midas-pipeline run --scan-mode ff` — both paths invoke the same `midas-index` / `midas-fit-grain` kernels under the hood, and both stay green on the FF parity gate. Pick whichever is more convenient for your workflow.
+
+## Architecture
+
+See [`../../.claude/plans/for-pf-we-don-t-lovely-locket.md`](../../.claude/plans/for-pf-we-don-t-lovely-locket.md) for the long-form plan. Quick summary:
+
+- **One orchestrator** with a mode-dependent `STAGE_ORDER`.
+- **Shared kernel packages** (`midas-index`, `midas-fit-grain`, `midas-transforms`, etc.) extended in place; FF behavior preserved by parity gates.
+- **PF-only modules** live inside `midas_pipeline` (`find_grains/`, `sinogen`, `recon/`, `fuse`, `potts`, `em_refine`, `seeding/`).
+- **Differentiability + multi-device** mandatory on every new compute path (CPU / CUDA / MPS via torch).
+
+## Constraints
+
+- No CUDA C; GPU support is torch-only.
+- No deletions of legacy code in this effort.
+- `midas-process-grains` is FF-only; PF consolidation is fresh pure-Python.
+- `utils/calcMiso.py` is not imported by this package; all orientation math comes from `midas-stress`.
+- `TOMO/midas_tomo_python.py` is imported in place, not relocated.

midas_pipeline-0.1.0/README.md

@@ -0,0 +1,50 @@
+# midas-pipeline
+
+End-to-end MIDAS HEDM orchestrator. **FF is the single-scan degeneracy of PF.** One package, one CLI, two scan modes.
+
+## Status
+
+**Alpha (0.1.0a0) — end-to-end PF and FF paths live.** The scanning indexer matches the C `IndexerScanningOMP` reference on its 1-voxel C-parity gate (seed identity, solution counts, voxel-center positions exact; orientation matrices within mrad-scale, the refiner closes the gap downstream). Cross-implementation perf optimization (~130s/voxel single-threaded today) is open work — see [project_midas_index_scanning_perf](https://github.com/marinerhemant/MIDAS/blob/master/packages/midas_pipeline/dev/perf-notes.md) tracking.
+
+Stages call in-process Python kernels via `midas-index` / `midas-fit-grain` / `midas-transforms` / `midas-stress`. FF mode shells out to `python -m midas_index` and `python -m midas_fit_grain` (same kernels, subprocess for the FF parity-preserving pattern). No CUDA C; GPU is torch-only.
+
+## Install
+
+```bash
+pip install -e packages/midas_pipeline
+```
+
+## CLI
+
+```bash
+midas-pipeline run --scan-mode {ff,pf,auto} --params Parameters.txt --result rundir/
+midas-pipeline status rundir/
+midas-pipeline resume rundir/ --from <stage>
+midas-pipeline reprocess rundir/
+midas-pipeline inspect rundir/LayerNr_1/
+midas-pipeline simulate --out simdir/ --params Parameters.txt
+midas-pipeline seed --params ... --output UniqueOrientations.csv
+```
+
+When `--scan-mode` is omitted (default `auto`), the CLI sniffs the parameter file: `nScans > 1` or presence of `BeamSize` / scanning keys → `pf`, otherwise `ff`. For PF mode, `--n-scans`, `--scan-step`, `--beam-size`, and `--scan-pos-tol` default to values in the params file (CLI flags override).
+
+## Coexistence with `midas-ff-pipeline`
+
+The legacy `midas-ff-pipeline` console-script is preserved as an independent FF orchestrator (its own kernels, its own CLI). It is **not** deprecated by `midas-pipeline run --scan-mode ff` — both paths invoke the same `midas-index` / `midas-fit-grain` kernels under the hood, and both stay green on the FF parity gate. Pick whichever is more convenient for your workflow.
+
+## Architecture
+
+See [`../../.claude/plans/for-pf-we-don-t-lovely-locket.md`](../../.claude/plans/for-pf-we-don-t-lovely-locket.md) for the long-form plan. Quick summary:
+
+- **One orchestrator** with a mode-dependent `STAGE_ORDER`.
+- **Shared kernel packages** (`midas-index`, `midas-fit-grain`, `midas-transforms`, etc.) extended in place; FF behavior preserved by parity gates.
+- **PF-only modules** live inside `midas_pipeline` (`find_grains/`, `sinogen`, `recon/`, `fuse`, `potts`, `em_refine`, `seeding/`).
+- **Differentiability + multi-device** mandatory on every new compute path (CPU / CUDA / MPS via torch).
+
+## Constraints
+
+- No CUDA C; GPU support is torch-only.
+- No deletions of legacy code in this effort.
+- `midas-process-grains` is FF-only; PF consolidation is fresh pure-Python.
+- `utils/calcMiso.py` is not imported by this package; all orientation math comes from `midas-stress`.
+- `TOMO/midas_tomo_python.py` is imported in place, not relocated.

midas_pipeline-0.1.0/midas_pipeline/__init__.py

@@ -0,0 +1,82 @@
+"""midas-pipeline: Unified MIDAS HEDM orchestrator (FF + PF, single source).
+
+FF is the single-scan degeneracy of PF: ``ScanGeometry.ff()`` produces
+``scan_mode='ff'`` with ``n_scans=1``; everything else is a regular PF
+run with ``n_scans > 1``. One orchestrator, one CLI, two scan modes.
+"""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from .config import (
+    AlignMethod,
+    Device,
+    Dtype,
+    EMConfig,
+    FusionConfig,
+    LayerSelection,
+    MachineConfig,
+    PipelineConfig,
+    ProcessGrainsMode,
+    ReconConfig,
+    ReconMethod,
+    RefineLoss,
+    RefineMode,
+    RefinePositionMode,
+    RefineSolver,
+    RefinementConfig,
+    ResumeMode,
+    ScanGeometry,
+    ScanMode,
+    SeedingConfig,
+    SeedingMode,
+    SinoSource,
+    SinoType,
+    sniff_scan_mode_from_paramfile,
+)
+from .pipeline import Pipeline, all_stage_names, stage_order_for
+from .results import (
+    LayerResult,
+    StageResult,
+    BinningResult,
+    CalcRadiusResult,
+    ConsolidationResult,
+    CrossDetMergeResult,
+    EMRefineResult,
+    FindGrainsResult,
+    FuseResult,
+    HKLResult,
+    IndexResult,
+    MergeOverlapsResult,
+    MergeScansResult,
+    PeakFitResult,
+    PottsResult,
+    ProcessGrainsResult,
+    ReconResult,
+    RefineResult,
+    SinogenResult,
+    TransformsResult,
+)
+
+__all__ = [
+    "__version__",
+    # config
+    "AlignMethod", "Device", "Dtype",
+    "EMConfig", "FusionConfig", "LayerSelection", "MachineConfig",
+    "PipelineConfig", "ProcessGrainsMode", "ReconConfig", "ReconMethod",
+    "RefineLoss", "RefineMode", "RefinePositionMode", "RefineSolver",
+    "RefinementConfig", "ResumeMode", "ScanGeometry", "ScanMode",
+    "SeedingConfig", "SeedingMode", "SinoSource", "SinoType",
+    "sniff_scan_mode_from_paramfile",
+    # pipeline
+    "Pipeline", "all_stage_names", "stage_order_for",
+    # results
+    "LayerResult", "StageResult",
+    "BinningResult", "CalcRadiusResult", "ConsolidationResult",
+    "CrossDetMergeResult", "EMRefineResult", "FindGrainsResult",
+    "FuseResult", "HKLResult", "IndexResult", "MergeOverlapsResult",
+    "MergeScansResult", "PeakFitResult", "PottsResult",
+    "ProcessGrainsResult", "ReconResult", "RefineResult",
+    "SinogenResult", "TransformsResult",
+]

midas_pipeline-0.1.0/midas_pipeline/__main__.py

@@ -0,0 +1,11 @@
+"""``python -m midas_pipeline`` entry point."""
+
+from __future__ import annotations
+
+import sys
+
+from .cli import main
+
+
+if __name__ == "__main__":
+    sys.exit(main())

midas_pipeline-0.1.0/midas_pipeline/_logging.py

@@ -0,0 +1,47 @@
+"""Pipeline-wide logger + stage timer.
+
+Single logger named ``midas_pipeline`` to keep child stage logs
+co-located with the orchestrator's own.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+from contextlib import contextmanager
+from typing import Iterator
+
+LOG = logging.getLogger("midas_pipeline")
+
+
+def configure_logging(level: int = logging.INFO) -> None:
+    """Idempotent root-logger setup. Safe to call from CLI or notebook."""
+    root = logging.getLogger("midas_pipeline")
+    if root.handlers:
+        root.setLevel(level)
+        return
+    handler = logging.StreamHandler()
+    handler.setFormatter(logging.Formatter(
+        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%H:%M:%S",
+    ))
+    root.addHandler(handler)
+    root.setLevel(level)
+    root.propagate = False
+
+
+@contextmanager
+def stage_timer(stage_name: str) -> Iterator[dict]:
+    """Time a stage and yield a dict that gets populated with start/end/duration."""
+    info: dict[str, float] = {"started_at": time.time()}
+    LOG.info("→ %s", stage_name)
+    try:
+        yield info
+    except Exception:
+        info["finished_at"] = time.time()
+        info["duration_s"] = info["finished_at"] - info["started_at"]
+        LOG.exception("✗ %s failed after %.2fs", stage_name, info["duration_s"])
+        raise
+    info["finished_at"] = time.time()
+    info["duration_s"] = info["finished_at"] - info["started_at"]
+    LOG.info("✓ %s — %.2fs", stage_name, info["duration_s"])