euler-eval 2.16.0__tar.gz → 2.17.0__tar.gz

{euler_eval-2.16.0 → euler_eval-2.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.16.0
+Version: 2.17.0
 Summary: Depth map evaluation toolkit with comprehensive metrics
 Author: Depth Eval Contributors
 License: MIT
@@ -221,6 +221,77 @@ The interval bounds are `near=[MIN, near_max)`, `mid=[near_max, mid_max)`, and
 | `mid` | `[9.290856529, 35.954189863)` |
 | `far` | `[35.954189863, 80.0]` |
 
+## Programmatic use (in-training validation)
+
+Besides the file-based CLI, `euler_eval.validation` exposes the same depth
+metric semantics for **in-memory predictions** — e.g. scoring a training
+run's validation pass against dense *or sparse* GT without writing
+predictions to disk:
+
+```python
+from euler_eval import (
+    DepthValidationAggregator,
+    build_validation_gt_dataset,
+    evaluate_dense_depth_sample,
+    evaluate_sparse_depth_sample,
+    get_sample_intrinsics,
+    get_sample_pointcloud_to_camera_extrinsics,
+    summarize_reduced_state,
+)
+
+# GT-only dataset from euler-loading compatible paths (.ds_crawler indices,
+# inline :split#scope= selectors supported).
+dataset = build_validation_gt_dataset(
+    sparse_depth_path="/data/muses.zip:val#scope=lidar",
+    rgb_path="/data/muses.zip:val#scope=rgb",
+    intrinsics_path="/data/muses.zip:val#scope=intrinsics",
+    camera_extrinsics_path="/data/muses.zip:val#scope=extrinsics",
+)
+
+aggregator = DepthValidationAggregator()
+for i in range(len(dataset)):
+    sample = dataset[i]
+    depth_pred = my_model(sample["rgb"])          # (H, W) metres, planar z
+    K = get_sample_intrinsics(sample)
+    lidar2cam, _ = get_sample_pointcloud_to_camera_extrinsics(sample)
+    aggregator.update(
+        evaluate_sparse_depth_sample(
+            depth_pred,
+            sample["sparse_depth"],               # (N, C>=3) lidar points
+            K,
+            lidar2cam,
+            alignment="none",                     # "affine" for relative depth
+        )
+    )
+
+summary = aggregator.summary()
+print(summary["standard"]["image_mean"]["absrel"])
+```
+
+Key pieces:
+
+- `evaluate_dense_depth_sample(pred, gt, valid_mask=None, alignment=...,
+  min_depth=..., max_depth=...)` — one dense prediction vs a dense GT map.
+  GT at another resolution is aligned to the prediction plane; returns the
+  standard metric set (`absrel`, `sqrel`, `mae`, `rmse`, `rmse_log`, `log10`,
+  `silog`, `delta1-3`) plus pooled pixel statistics, or `None` when too few
+  valid pixels remain.
+- `evaluate_sparse_depth_sample(pred, point_cloud, intrinsics,
+  camera_extrinsics, lidar_extrinsics=None, pred_is_radial=False, ...)` —
+  projects the sparse GT cloud into the prediction plane (radial depth,
+  nearest-z occlusion handling), converts a planar prediction to radial with
+  the same intrinsics, and scores only the projected pixels — matching the
+  CLI's `gt.sparse_depth` pipeline. Pass the intrinsics of the *actual
+  prediction plane* (i.e. adjusted for any crop/resize of the model input).
+- `alignment="none"` scores metric predictions as-is; `"affine"` fits
+  least-squares scale+shift first (relative/affine depth models).
+- `DepthValidationAggregator` accumulates per-sample results into the CLI's
+  `image_mean` / `image_median` / `pixel_pool` reducers. For multi-process
+  validation, sum `aggregator.reduced_state()` vectors across ranks (fixed
+  key order via `DepthValidationAggregator.state_keys()`) and rebuild
+  mean-based summaries with `summarize_reduced_state(...)`.
+- Inputs accept torch tensors or numpy arrays; all math runs on CPU numpy.
+
 ## Configuration
 
 ### `config.json`

{euler_eval-2.16.0 → euler_eval-2.17.0}/README.md

@@ -183,6 +183,77 @@ The interval bounds are `near=[MIN, near_max)`, `mid=[near_max, mid_max)`, and
 | `mid` | `[9.290856529, 35.954189863)` |
 | `far` | `[35.954189863, 80.0]` |
 
+## Programmatic use (in-training validation)
+
+Besides the file-based CLI, `euler_eval.validation` exposes the same depth
+metric semantics for **in-memory predictions** — e.g. scoring a training
+run's validation pass against dense *or sparse* GT without writing
+predictions to disk:
+
+```python
+from euler_eval import (
+    DepthValidationAggregator,
+    build_validation_gt_dataset,
+    evaluate_dense_depth_sample,
+    evaluate_sparse_depth_sample,
+    get_sample_intrinsics,
+    get_sample_pointcloud_to_camera_extrinsics,
+    summarize_reduced_state,
+)
+
+# GT-only dataset from euler-loading compatible paths (.ds_crawler indices,
+# inline :split#scope= selectors supported).
+dataset = build_validation_gt_dataset(
+    sparse_depth_path="/data/muses.zip:val#scope=lidar",
+    rgb_path="/data/muses.zip:val#scope=rgb",
+    intrinsics_path="/data/muses.zip:val#scope=intrinsics",
+    camera_extrinsics_path="/data/muses.zip:val#scope=extrinsics",
+)
+
+aggregator = DepthValidationAggregator()
+for i in range(len(dataset)):
+    sample = dataset[i]
+    depth_pred = my_model(sample["rgb"])          # (H, W) metres, planar z
+    K = get_sample_intrinsics(sample)
+    lidar2cam, _ = get_sample_pointcloud_to_camera_extrinsics(sample)
+    aggregator.update(
+        evaluate_sparse_depth_sample(
+            depth_pred,
+            sample["sparse_depth"],               # (N, C>=3) lidar points
+            K,
+            lidar2cam,
+            alignment="none",                     # "affine" for relative depth
+        )
+    )
+
+summary = aggregator.summary()
+print(summary["standard"]["image_mean"]["absrel"])
+```
+
+Key pieces:
+
+- `evaluate_dense_depth_sample(pred, gt, valid_mask=None, alignment=...,
+  min_depth=..., max_depth=...)` — one dense prediction vs a dense GT map.
+  GT at another resolution is aligned to the prediction plane; returns the
+  standard metric set (`absrel`, `sqrel`, `mae`, `rmse`, `rmse_log`, `log10`,
+  `silog`, `delta1-3`) plus pooled pixel statistics, or `None` when too few
+  valid pixels remain.
+- `evaluate_sparse_depth_sample(pred, point_cloud, intrinsics,
+  camera_extrinsics, lidar_extrinsics=None, pred_is_radial=False, ...)` —
+  projects the sparse GT cloud into the prediction plane (radial depth,
+  nearest-z occlusion handling), converts a planar prediction to radial with
+  the same intrinsics, and scores only the projected pixels — matching the
+  CLI's `gt.sparse_depth` pipeline. Pass the intrinsics of the *actual
+  prediction plane* (i.e. adjusted for any crop/resize of the model input).
+- `alignment="none"` scores metric predictions as-is; `"affine"` fits
+  least-squares scale+shift first (relative/affine depth models).
+- `DepthValidationAggregator` accumulates per-sample results into the CLI's
+  `image_mean` / `image_median` / `pixel_pool` reducers. For multi-process
+  validation, sum `aggregator.reduced_state()` vectors across ranks (fixed
+  key order via `DepthValidationAggregator.state_keys()`) and rebuild
+  mean-based summaries with `summarize_reduced_state(...)`.
+- Inputs accept torch tensors or numpy arrays; all math runs on CPU numpy.
+
 ## Configuration
 
 ### `config.json`

euler_eval-2.17.0/euler_eval/__init__.py

@@ -0,0 +1,65 @@
+"""Depth evaluation package.
+
+Public programmatic API (lazily imported so that ``import euler_eval`` stays
+cheap; heavy metric backends only load on first use):
+
+* :mod:`euler_eval.validation` — in-training/programmatic depth validation
+  (``evaluate_dense_depth_sample``, ``evaluate_sparse_depth_sample``,
+  ``DepthValidationAggregator``, ``build_validation_gt_dataset``).
+* :mod:`euler_eval.calibration` — calibration extraction from euler-loading
+  samples.
+* :mod:`euler_eval.metrics` — pure-array metric functions.
+* :mod:`euler_eval.data` — dataset builders, converters, projection, and
+  alignment helpers.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+_VALIDATION_EXPORTS = (
+    "DepthSampleEvaluation",
+    "DepthValidationAggregator",
+    "VALIDATION_ALIGNMENT_MODES",
+    "build_validation_gt_dataset",
+    "evaluate_dense_depth_sample",
+    "evaluate_sparse_depth_sample",
+    "summarize_reduced_state",
+)
+_CALIBRATION_EXPORTS = (
+    "get_sample_intrinsics",
+    "get_sample_pointcloud_to_camera_extrinsics",
+)
+
+__all__ = [*_VALIDATION_EXPORTS, *_CALIBRATION_EXPORTS]
+
+if TYPE_CHECKING:  # pragma: no cover - static import surface for type checkers
+    from .calibration import (  # noqa: F401
+        get_sample_intrinsics,
+        get_sample_pointcloud_to_camera_extrinsics,
+    )
+    from .validation import (  # noqa: F401
+        VALIDATION_ALIGNMENT_MODES,
+        DepthSampleEvaluation,
+        DepthValidationAggregator,
+        build_validation_gt_dataset,
+        evaluate_dense_depth_sample,
+        evaluate_sparse_depth_sample,
+        summarize_reduced_state,
+    )
+
+
+def __getattr__(name: str):
+    if name in _VALIDATION_EXPORTS:
+        from . import validation
+
+        return getattr(validation, name)
+    if name in _CALIBRATION_EXPORTS:
+        from . import calibration
+
+        return getattr(calibration, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+def __dir__() -> list[str]:
+    return sorted(set(globals()) | set(__all__))

euler_eval-2.17.0/euler_eval/calibration.py

@@ -0,0 +1,258 @@
+"""Camera-calibration extraction from euler-loading samples.
+
+MultiModalDataset samples carry calibration either directly (a ``(3, 3)`` /
+``(4, 4)`` tensor per sample) or wrapped in hierarchical-modality dicts and
+nested calibration payloads (e.g. a MUSES ``calib.json`` with named
+``lidar2rgb`` transforms).  This module owns the tolerant extraction logic
+shared by the CLI evaluators (:mod:`euler_eval.evaluate`) and the
+programmatic validation interface (:mod:`euler_eval.validation`).
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Optional
+
+import numpy as np
+
+from .data import (
+    compose_sensor_to_camera_extrinsics,
+    to_numpy_extrinsics,
+    to_numpy_intrinsics,
+)
+
+__all__ = [
+    "get_first_hierarchical_value",
+    "iter_hierarchical_values",
+    "get_sample_intrinsics",
+    "get_sample_pointcloud_to_camera_extrinsics",
+    "extract_wrapped_extrinsics",
+    "extract_named_extrinsics",
+    "extract_direct_source_to_camera",
+    "DIRECT_SOURCE_TO_CAMERA_KEYS",
+    "SOURCE_SENSOR_POSE_KEYS",
+    "CAMERA_SENSOR_POSE_KEYS",
+    "MATRIX_WRAPPER_KEYS",
+]
+
+
+def get_first_hierarchical_value(sample: dict, key: str):
+    """Return ``sample[key]``, unwrapping hierarchical ``{file_id: value}`` dicts."""
+    data = sample.get(key)
+    if data is None:
+        return None
+    if isinstance(data, dict):
+        if not data:
+            return None
+        return next(iter(data.values()))
+    return data
+
+
+def iter_hierarchical_values(sample: dict, key: str) -> list:
+    """Return candidate values for ``sample[key]`` (dict itself plus entries)."""
+    data = sample.get(key)
+    if data is None:
+        return []
+    if isinstance(data, dict):
+        return [data, *data.values()]
+    return [data]
+
+
+def get_sample_intrinsics(sample: dict) -> Optional[np.ndarray]:
+    """Extract a ``(3, 3)`` intrinsics matrix from sample calibration data."""
+    K_data = get_first_hierarchical_value(sample, "intrinsics")
+    if K_data is None:
+        K_data = get_first_hierarchical_value(sample, "calibration")
+    if K_data is None:
+        return None
+    return to_numpy_intrinsics(K_data)
+
+
+DIRECT_SOURCE_TO_CAMERA_KEYS = (
+    "lidar2rgb",
+    "lidar2camera",
+    "lidar2cam",
+    "lidar_to_rgb",
+    "lidar_to_camera",
+    "lidar_to_cam",
+    "source2rgb",
+    "source2camera",
+    "source_to_rgb",
+    "source_to_camera",
+    "sensor2rgb",
+    "sensor2camera",
+    "sensor_to_rgb",
+    "sensor_to_camera",
+)
+SOURCE_SENSOR_POSE_KEYS = (
+    "lidar",
+    "lidar_pose",
+    "lidar_extrinsics",
+    "source",
+    "source_sensor",
+    "source_extrinsics",
+    "sensor",
+    "sensor_extrinsics",
+    "lidar2gnss",
+    "lidar2ego",
+    "lidar2vehicle",
+    "lidar2world",
+    "lidar_to_gnss",
+    "lidar_to_ego",
+    "lidar_to_vehicle",
+    "lidar_to_world",
+)
+CAMERA_SENSOR_POSE_KEYS = (
+    "rgb",
+    "camera",
+    "cam",
+    "frame_camera",
+    "rgb_pose",
+    "camera_pose",
+    "camera_extrinsics",
+    "target",
+    "target_sensor",
+    "target_extrinsics",
+    "rgb2gnss",
+    "camera2gnss",
+    "cam2gnss",
+    "frame_camera2gnss",
+    "rgb2ego",
+    "camera2ego",
+    "cam2ego",
+    "frame_camera2ego",
+    "rgb2vehicle",
+    "camera2vehicle",
+    "cam2vehicle",
+    "frame_camera2vehicle",
+    "rgb2world",
+    "camera2world",
+    "cam2world",
+    "frame_camera2world",
+)
+MATRIX_WRAPPER_KEYS = ("T", "transform", "matrix", "extrinsics", "pose")
+
+
+def _as_extrinsics_or_none(data) -> Optional[np.ndarray]:
+    try:
+        return to_numpy_extrinsics(data)
+    except (TypeError, ValueError):
+        return None
+
+
+def extract_wrapped_extrinsics(data) -> Optional[np.ndarray]:
+    """Extract a ``(4, 4)`` transform from a value or common wrapper mappings."""
+    matrix = _as_extrinsics_or_none(data)
+    if matrix is not None:
+        return matrix
+
+    if not isinstance(data, Mapping):
+        return None
+
+    for key in MATRIX_WRAPPER_KEYS:
+        if key in data:
+            matrix = extract_wrapped_extrinsics(data[key])
+            if matrix is not None:
+                return matrix
+
+    if len(data) == 1:
+        return extract_wrapped_extrinsics(next(iter(data.values())))
+
+    return None
+
+
+def extract_named_extrinsics(data, names: tuple[str, ...]) -> Optional[np.ndarray]:
+    """Extract a transform stored under one of ``names`` in a calibration payload."""
+    if not isinstance(data, Mapping):
+        return extract_wrapped_extrinsics(data)
+
+    for name in names:
+        if name in data:
+            matrix = extract_wrapped_extrinsics(data[name])
+            if matrix is not None:
+                return matrix
+
+    for key in ("extrinsics", "sensors", "calibration", "poses"):
+        if key in data and isinstance(data[key], Mapping):
+            matrix = extract_named_extrinsics(data[key], names)
+            if matrix is not None:
+                return matrix
+
+    if len(data) == 1:
+        return extract_named_extrinsics(next(iter(data.values())), names)
+
+    return None
+
+
+def extract_direct_source_to_camera(data) -> Optional[np.ndarray]:
+    """Extract a direct source→camera transform (e.g. MUSES ``lidar2rgb``)."""
+    matrix = _as_extrinsics_or_none(data)
+    if matrix is not None:
+        return matrix
+
+    if not isinstance(data, Mapping):
+        return None
+
+    for key in DIRECT_SOURCE_TO_CAMERA_KEYS:
+        if key in data:
+            matrix = extract_wrapped_extrinsics(data[key])
+            if matrix is not None:
+                return matrix
+
+    for key in ("extrinsics", "calibration"):
+        if key in data and isinstance(data[key], Mapping):
+            matrix = extract_direct_source_to_camera(data[key])
+            if matrix is not None:
+                return matrix
+
+    if len(data) == 1:
+        return extract_direct_source_to_camera(next(iter(data.values())))
+
+    return None
+
+
+def get_sample_pointcloud_to_camera_extrinsics(
+    sample: dict,
+) -> tuple[Optional[np.ndarray], Optional[str]]:
+    """Extract or compose the transform used for sparse point projection.
+
+    The common MUSES path supplies a direct ``lidar2rgb`` transform via
+    ``camera_extrinsics``.  Some datasets instead expose separate lidar and
+    camera sensor poses in a shared frame; in that case we compose them to the
+    same source-to-camera transform expected by the projector.
+    """
+    lidar_pose = None
+    for value in iter_hierarchical_values(sample, "lidar_extrinsics"):
+        lidar_pose = extract_named_extrinsics(value, SOURCE_SENSOR_POSE_KEYS)
+        if lidar_pose is not None:
+            break
+
+    camera_values = iter_hierarchical_values(sample, "camera_extrinsics")
+    if not camera_values:
+        camera_values = iter_hierarchical_values(sample, "extrinsics")
+
+    if lidar_pose is not None:
+        for value in camera_values:
+            camera_pose = extract_named_extrinsics(value, CAMERA_SENSOR_POSE_KEYS)
+            if camera_pose is not None:
+                return (
+                    compose_sensor_to_camera_extrinsics(lidar_pose, camera_pose),
+                    "composed_lidar_and_camera_sensor_poses",
+                )
+        return None, None
+
+    for value in camera_values:
+        direct = extract_direct_source_to_camera(value)
+        if direct is not None:
+            return direct, "direct_source_to_camera"
+
+    for value in camera_values:
+        lidar_pose = extract_named_extrinsics(value, SOURCE_SENSOR_POSE_KEYS)
+        camera_pose = extract_named_extrinsics(value, CAMERA_SENSOR_POSE_KEYS)
+        if lidar_pose is not None and camera_pose is not None:
+            return (
+                compose_sensor_to_camera_extrinsics(lidar_pose, camera_pose),
+                "composed_lidar_and_camera_sensor_poses",
+            )
+
+    return None, None