euler-eval 2.8.0__tar.gz → 2.10.0__tar.gz

{euler_eval-2.8.0 → euler_eval-2.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.8.0
+Version: 2.10.0
 Summary: Depth map evaluation toolkit with comprehensive metrics
 Author: Depth Eval Contributors
 License: MIT
@@ -45,7 +45,7 @@ A comprehensive evaluation toolkit for comparing predicted depth maps, RGB image
 - **Depth metrics**: PSNR, SSIM, LPIPS, FID, KID, AbsRel, RMSE, Scale-Invariant Log Error, Normal Consistency, Depth Edge F1
 - **RGB metrics**: PSNR, SSIM, LPIPS, FID, SCE (Structural Chromatic Error), Edge F1, Tail Errors (p95/p99), High-Frequency Energy Ratio, Depth-Binned Photometric Error
 - **Rays metrics**: ρ_A (AUC of angular accuracy curve), Angular Error statistics and threshold percentages
-- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into log-scaled near/mid/far bins
+- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into square-root-scaled near/mid/far bins
 - **Sanity checking**: Automatic validation of metric results against configurable thresholds, with detailed warning reports
 - **Sky masking**: Optional exclusion of sky regions from metrics using GT segmentation
 - **Flexible dataset loading**: Automatic loader resolution via euler_loading and ds-crawler index metadata
@@ -141,7 +141,7 @@ This pre-downloads:
 | `--metrics-config` | `str` | auto-detect | Path to `metrics_config.json` for sanity checking |
 | `--depth-alignment` | `{none,auto_affine,affine}` | `auto_affine` | Depth calibration mode; outputs are emitted in semantic `native`/`metric` spaces and `depth` aliases the canonical branch |
 | `--rgb-fid-backend` | `{builtin,clean-fid}` | `builtin` | RGB FID backend; `clean-fid` requires optional dependency |
-| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into log-scaled near/mid/far bins (additive to regular metrics) |
+| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into square-root-scaled near/mid/far bins (additive to regular metrics) |
 
 ### Examples
 
@@ -168,7 +168,7 @@ depth-eval config.json --depth-alignment affine
 depth-eval config.json --rgb-fid-backend clean-fid
 
 # Benchmark depth and RGB metrics within a depth range (near/mid/far bins)
-depth-eval config.json --benchmark-depth-range 0.5 80.0
+depth-eval config.json --benchmark-depth-range 0.01 80.0
 
 # Evaluate dense depth predictions against sparse pointcloud GT
 depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
@@ -177,6 +177,30 @@ depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
 depth-eval config.json --skip-rays
 ```
 
+### Benchmark Depth Bins
+
+`--benchmark-depth-range MIN MAX` first filters valid GT depth pixels to
+`MIN <= depth <= MAX`. The `near`, `mid`, and `far` bins are then computed as
+three equal-width intervals in square-root depth space:
+
+```text
+sqrt_min = sqrt(MIN)
+sqrt_max = sqrt(MAX)
+step = (sqrt_max - sqrt_min) / 3
+near_max = (sqrt_min + step)^2
+mid_max = (sqrt_min + 2 * step)^2
+```
+
+The interval bounds are `near=[MIN, near_max)`, `mid=[near_max, mid_max)`, and
+`far=[mid_max, MAX]`. For `--benchmark-depth-range 0.01 80.0`, this gives:
+
+| Bin | Depth interval |
+|---|---|
+| `all` | `[0.01, 80.0]` |
+| `near` | `[0.01, 9.290856529)` |
+| `mid` | `[9.290856529, 35.954189863)` |
+| `far` | `[35.954189863, 80.0]` |
+
 ## Configuration
 
 ### `config.json`
@@ -215,7 +239,7 @@ Defines GT modalities, prediction datasets to evaluate, and optional euler_train
 }
 ```
 
-Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`).
+Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`). euler-loading inline selectors are also accepted in `path`, such as `/data/muses.zip:test` or `/data/muses.zip:test#scope=rgb`.
 
 For sparse pointcloud depth evaluation, use `gt.sparse_depth` instead of `gt.depth`. The prediction still uses `datasets[].depth` because the model output is a dense depth map. The evaluator projects the sparse GT point cloud into the prediction image plane using `gt.intrinsics` and `gt.camera_extrinsics`, then computes pointwise depth metrics only at projected valid pixels.
 
@@ -269,7 +293,7 @@ Dataset metadata (e.g. `radial_depth`, `rgb_range`, sparse point columns, and co
 
 ### Dataset Metadata
 
-Each dataset directory must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
+Each dataset directory or archive must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
 
 ```text
 dataset-root/
@@ -279,6 +303,8 @@ dataset-root/
     index.json
 ```
 
+When one physical root or archive contains several logical modalities, the artifacts may instead be scoped under `.ds_crawler/<modality>/` with a `.ds_crawler/scopes.json` manifest. `euler-eval` passes modality scopes to euler-loading so paths such as `/data/muses.zip:test` can load `.ds_crawler/rgb/index.json`, `.ds_crawler/depth/index.json`, and related scoped artifacts from the same archive.
+
 GT and prediction datasets are matched by hierarchy path and file ID through `MultiModalDataset`.
 
 ### `metrics_config.json`

{euler_eval-2.8.0 → euler_eval-2.10.0}/README.md

@@ -7,7 +7,7 @@ A comprehensive evaluation toolkit for comparing predicted depth maps, RGB image
 - **Depth metrics**: PSNR, SSIM, LPIPS, FID, KID, AbsRel, RMSE, Scale-Invariant Log Error, Normal Consistency, Depth Edge F1
 - **RGB metrics**: PSNR, SSIM, LPIPS, FID, SCE (Structural Chromatic Error), Edge F1, Tail Errors (p95/p99), High-Frequency Energy Ratio, Depth-Binned Photometric Error
 - **Rays metrics**: ρ_A (AUC of angular accuracy curve), Angular Error statistics and threshold percentages
-- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into log-scaled near/mid/far bins
+- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into square-root-scaled near/mid/far bins
 - **Sanity checking**: Automatic validation of metric results against configurable thresholds, with detailed warning reports
 - **Sky masking**: Optional exclusion of sky regions from metrics using GT segmentation
 - **Flexible dataset loading**: Automatic loader resolution via euler_loading and ds-crawler index metadata
@@ -103,7 +103,7 @@ This pre-downloads:
 | `--metrics-config` | `str` | auto-detect | Path to `metrics_config.json` for sanity checking |
 | `--depth-alignment` | `{none,auto_affine,affine}` | `auto_affine` | Depth calibration mode; outputs are emitted in semantic `native`/`metric` spaces and `depth` aliases the canonical branch |
 | `--rgb-fid-backend` | `{builtin,clean-fid}` | `builtin` | RGB FID backend; `clean-fid` requires optional dependency |
-| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into log-scaled near/mid/far bins (additive to regular metrics) |
+| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into square-root-scaled near/mid/far bins (additive to regular metrics) |
 
 ### Examples
 
@@ -130,7 +130,7 @@ depth-eval config.json --depth-alignment affine
 depth-eval config.json --rgb-fid-backend clean-fid
 
 # Benchmark depth and RGB metrics within a depth range (near/mid/far bins)
-depth-eval config.json --benchmark-depth-range 0.5 80.0
+depth-eval config.json --benchmark-depth-range 0.01 80.0
 
 # Evaluate dense depth predictions against sparse pointcloud GT
 depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
@@ -139,6 +139,30 @@ depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
 depth-eval config.json --skip-rays
 ```
 
+### Benchmark Depth Bins
+
+`--benchmark-depth-range MIN MAX` first filters valid GT depth pixels to
+`MIN <= depth <= MAX`. The `near`, `mid`, and `far` bins are then computed as
+three equal-width intervals in square-root depth space:
+
+```text
+sqrt_min = sqrt(MIN)
+sqrt_max = sqrt(MAX)
+step = (sqrt_max - sqrt_min) / 3
+near_max = (sqrt_min + step)^2
+mid_max = (sqrt_min + 2 * step)^2
+```
+
+The interval bounds are `near=[MIN, near_max)`, `mid=[near_max, mid_max)`, and
+`far=[mid_max, MAX]`. For `--benchmark-depth-range 0.01 80.0`, this gives:
+
+| Bin | Depth interval |
+|---|---|
+| `all` | `[0.01, 80.0]` |
+| `near` | `[0.01, 9.290856529)` |
+| `mid` | `[9.290856529, 35.954189863)` |
+| `far` | `[35.954189863, 80.0]` |
+
 ## Configuration
 
 ### `config.json`
@@ -177,7 +201,7 @@ Defines GT modalities, prediction datasets to evaluate, and optional euler_train
 }
 ```
 
-Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`).
+Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`). euler-loading inline selectors are also accepted in `path`, such as `/data/muses.zip:test` or `/data/muses.zip:test#scope=rgb`.
 
 For sparse pointcloud depth evaluation, use `gt.sparse_depth` instead of `gt.depth`. The prediction still uses `datasets[].depth` because the model output is a dense depth map. The evaluator projects the sparse GT point cloud into the prediction image plane using `gt.intrinsics` and `gt.camera_extrinsics`, then computes pointwise depth metrics only at projected valid pixels.
 
@@ -231,7 +255,7 @@ Dataset metadata (e.g. `radial_depth`, `rgb_range`, sparse point columns, and co
 
 ### Dataset Metadata
 
-Each dataset directory must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
+Each dataset directory or archive must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
 
 ```text
 dataset-root/
@@ -241,6 +265,8 @@ dataset-root/
     index.json
 ```
 
+When one physical root or archive contains several logical modalities, the artifacts may instead be scoped under `.ds_crawler/<modality>/` with a `.ds_crawler/scopes.json` manifest. `euler-eval` passes modality scopes to euler-loading so paths such as `/data/muses.zip:test` can load `.ds_crawler/rgb/index.json`, `.ds_crawler/depth/index.json`, and related scoped artifacts from the same archive.
+
 GT and prediction datasets are matched by hierarchy path and file ID through `MultiModalDataset`.
 
 ### `metrics_config.json`

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval/cli.py

@@ -13,6 +13,7 @@ import zipfile
 from pathlib import Path
 
 import torch
+from euler_loading import Modality
 
 from .data import (
     build_depth_eval_dataset,
@@ -40,6 +41,22 @@ except ImportError:
 from euler_metric_naming import AxisDeclaration, MetricDescription, MetricNamespace
 
 
+def _normalize_modality_path(
+    path: str | Path,
+    *,
+    modality_key: str,
+    split: str | None = None,
+) -> Path:
+    """Return the filesystem/archive root for an euler-loading modality path."""
+    modality = Modality(
+        str(path),
+        modality_type=modality_key,
+        metadata_scope=modality_key,
+        split=split,
+    )
+    return Path(modality.path)
+
+
 # ── Eval namespace ──────────────────────────────────────────────────────────
 # MetricNamespace subclass for eval context with custom axis declarations,
 # following the Normed Metric Namespacing convention (§3.6, §4, §5.2).
@@ -292,6 +309,10 @@ _SPARSE_DEPTH_EVAL_DESCRIPTIONS = {
         "silog.p90",
     }
 }
+_SPARSE_DEPTH_EVAL_DESCRIPTIONS["valid_pixel_count"] = MetricDescription(
+    unit="pixels",
+    display_name="Valid Pixel Count",
+)
 
 _RGB_EVAL_DESCRIPTIONS = {
     "psnr": MetricDescription(is_higher_better=True, unit="dB", display_name="PSNR"),
@@ -389,7 +410,11 @@ def _sparse_depth_metric_set_envelope(
     return envelope
 
 
-def _clean_metric_tree(tree: dict) -> dict:
+def _clean_metric_tree(
+    tree: dict,
+    *,
+    preserve_empty_dict_keys: set[str] | None = None,
+) -> dict:
     """Recursively sanitize a metric dict for JSON schema compliance.
 
     - Removes entries where the value is ``None``
@@ -397,6 +422,7 @@ def _clean_metric_tree(tree: dict) -> dict:
     - Recursively processes nested dicts and list items
     - Prunes empty dicts after cleaning
     """
+    preserve_empty_dict_keys = preserve_empty_dict_keys or set()
     cleaned = {}
     for key, value in tree.items():
         if value is None:
@@ -404,12 +430,20 @@ def _clean_metric_tree(tree: dict) -> dict:
         if isinstance(value, float) and not math.isfinite(value):
             continue
         if isinstance(value, dict):
-            sub = _clean_metric_tree(value)
-            if sub:
+            sub = _clean_metric_tree(
+                value,
+                preserve_empty_dict_keys=preserve_empty_dict_keys,
+            )
+            if sub or key in preserve_empty_dict_keys:
                 cleaned[key] = sub
         elif isinstance(value, list):
             cleaned[key] = [
-                _clean_metric_tree(item) if isinstance(item, dict) else item
+                _clean_metric_tree(
+                    item,
+                    preserve_empty_dict_keys=preserve_empty_dict_keys,
+                )
+                if isinstance(item, dict)
+                else item
                 for item in value
             ]
         else:
@@ -417,6 +451,71 @@ def _clean_metric_tree(tree: dict) -> dict:
     return cleaned
 
 
+def _empty_per_file_metric_ids(
+    node: dict,
+    hierarchy: tuple[str, ...] = (),
+) -> list[str]:
+    """Return file identifiers whose cleaned per-file metrics are empty."""
+    ids = []
+    for file_entry in node.get("files", []):
+        if not isinstance(file_entry, dict):
+            continue
+        if file_entry.get("metrics") == {}:
+            file_id = str(file_entry.get("id", "<missing-id>"))
+            ids.append("/".join((*hierarchy, file_id)))
+
+    children = node.get("children", {})
+    if isinstance(children, dict):
+        for child_name, child_node in children.items():
+            if isinstance(child_node, dict):
+                ids.extend(
+                    _empty_per_file_metric_ids(
+                        child_node,
+                        (*hierarchy, str(child_name)),
+                    )
+                )
+    return ids
+
+
+def _clean_per_file_metrics(tree: dict, *, label: str = "per_file_metrics") -> dict:
+    """Clean a per-file metric tree while preserving file-entry shape.
+
+    Some per-file metric entries can legitimately clean down to no scalar
+    metrics, for example when every metric for a file is ``None``/NaN after a
+    failed computation or an invalid sample.  The generic cleaner prunes empty
+    dicts, but per-file entries are schema-shaped as ``{"id", "metrics"}``;
+    dropping the empty ``metrics`` object leaves an id-only file entry.
+    """
+    cleaned = _clean_metric_tree(tree, preserve_empty_dict_keys={"metrics"})
+    empty_ids = _empty_per_file_metric_ids(cleaned)
+    if empty_ids:
+        sample = ", ".join(empty_ids[:10])
+        suffix = "" if len(empty_ids) <= 10 else f", ... +{len(empty_ids) - 10} more"
+        print(
+            f"Warning: {label} has {len(empty_ids)} file entries with no finite "
+            f"metrics after cleaning: {sample}{suffix}"
+        )
+    return cleaned
+
+
+def _wrap_depth_space_pfm_metrics(
+    metrics: dict,
+    *,
+    metric_root: str,
+    canonical_key: str,
+    canonical_space: str | None = "metric",
+) -> dict:
+    """Wrap per-file depth-like metrics under the serialized namespace path."""
+    space_metrics = {
+        space: metrics[f"{canonical_key}_{space}"]
+        for space in ("native", "metric")
+        if f"{canonical_key}_{space}" in metrics
+    }
+    if not space_metrics and canonical_key in metrics:
+        space_metrics[canonical_space or "metric"] = metrics[canonical_key]
+    return {metric_root: {"eval": space_metrics}}
+
+
 def _wrap_pfm_metrics(pfm: dict, wrapper_fn) -> dict:
     """Walk a ``perFileHierarchyNode`` tree, applying *wrapper_fn* to each
     file entry's ``metrics`` dict.  This restructures per-file metric keys
@@ -517,7 +616,11 @@ def validate_gt_config(gt: dict) -> None:
         "camera_extrinsics",
     ):
         if modality in gt and "path" in gt[modality]:
-            p = Path(gt[modality]["path"])
+            p = _normalize_modality_path(
+                gt[modality]["path"],
+                modality_key=modality,
+                split=gt[modality].get("split"),
+            )
             if not p.exists():
                 raise ValueError(f"gt.{modality}.path does not exist: {p}")
 
@@ -542,7 +645,11 @@ def validate_dataset_entry(entry: dict, index: int) -> None:
 
     for modality in ("rgb", "depth", "rays"):
         if modality in entry and "path" in entry[modality]:
-            p = Path(entry[modality]["path"])
+            p = _normalize_modality_path(
+                entry[modality]["path"],
+                modality_key=modality,
+                split=entry[modality].get("split"),
+            )
             if not p.exists():
                 raise ValueError(f"{label}.{modality}.path does not exist: {p}")
 
@@ -667,12 +774,26 @@ def save_results(
             and modality in dataset_config
             and "path" in dataset_config[modality]
         ):
-            output_file = Path(dataset_config[modality]["path"]) / "eval.json"
+            output_file = (
+                _normalize_modality_path(
+                    dataset_config[modality]["path"],
+                    modality_key=modality,
+                    split=dataset_config[modality].get("split"),
+                )
+                / "eval.json"
+            )
         else:
             # Default: save alongside first available modality path
             for mod in ("depth", "rgb", "rays"):
                 if mod in dataset_config and "path" in dataset_config[mod]:
-                    output_file = Path(dataset_config[mod]["path"]) / "eval.json"
+                    output_file = (
+                        _normalize_modality_path(
+                            dataset_config[mod]["path"],
+                            modality_key=mod,
+                            split=dataset_config[mod].get("split"),
+                        )
+                        / "eval.json"
+                    )
                     break
         if output_file is None:
             output_file = Path("eval.json")
@@ -1065,21 +1186,18 @@ def main():
                     all_results[depth_key] = depth_results[depth_key]
             depth_pfm = depth_results.get("per_file_metrics", {})
             if depth_pfm:
-                depth_save["per_file_metrics"] = _clean_metric_tree(
+                canonical_space = space_info.get("canonical_space", "metric")
+                depth_save["per_file_metrics"] = _clean_per_file_metrics(
                     _wrap_pfm_metrics(
                         depth_pfm,
-                        lambda m: (
-                            {
-                                "depth": {
-                                    "eval": {
-                                        space: m[f"depth_{space}"]
-                                        for space in ("native", "metric")
-                                        if f"depth_{space}" in m
-                                    },
-                                },
-                            }
+                        lambda m: _wrap_depth_space_pfm_metrics(
+                            m,
+                            metric_root="depth",
+                            canonical_key="depth",
+                            canonical_space=canonical_space,
                         ),
-                    )
+                    ),
+                    label="depth per_file_metrics",
                 )
             all_results.setdefault("per_file_metrics", {}).update(depth_pfm)
 
@@ -1258,21 +1376,18 @@ def main():
                     all_results[depth_key] = sparse_depth_results[depth_key]
             sparse_depth_pfm = sparse_depth_results.get("per_file_metrics", {})
             if sparse_depth_pfm:
-                depth_save["per_file_metrics"] = _clean_metric_tree(
+                canonical_space = space_info.get("canonical_space", "metric")
+                depth_save["per_file_metrics"] = _clean_per_file_metrics(
                     _wrap_pfm_metrics(
                         sparse_depth_pfm,
-                        lambda m: (
-                            {
-                                _SPARSE_DEPTH_METRIC_ROOT: {
-                                    "eval": {
-                                        space: m[f"sparse_depth_{space}"]
-                                        for space in ("native", "metric")
-                                        if f"sparse_depth_{space}" in m
-                                    },
-                                },
-                            }
+                        lambda m: _wrap_depth_space_pfm_metrics(
+                            m,
+                            metric_root=_SPARSE_DEPTH_METRIC_ROOT,
+                            canonical_key="sparse_depth",
+                            canonical_space=canonical_space,
                         ),
-                    )
+                    ),
+                    label="sparse_depth per_file_metrics",
                 )
             all_results.setdefault("per_file_metrics", {}).update(sparse_depth_pfm)
 
@@ -1417,11 +1532,12 @@ def main():
                 all_results["rgb_benchmark"] = rgb_benchmark
             rgb_pfm = rgb_results.get("per_file_metrics", {})
             if rgb_pfm:
-                rgb_save["per_file_metrics"] = _clean_metric_tree(
+                rgb_save["per_file_metrics"] = _clean_per_file_metrics(
                     _wrap_pfm_metrics(
                         rgb_pfm,
                         lambda m: {"rgb": {"eval": m.get("rgb", {})}},
-                    )
+                    ),
+                    label="rgb per_file_metrics",
                 )
             all_results.setdefault("per_file_metrics", {}).update(rgb_pfm)
 
@@ -1519,11 +1635,12 @@ def main():
                 all_results["rays"] = rays_metrics
             rays_pfm = rays_results.get("per_file_metrics", {})
             if rays_pfm:
-                rays_save["per_file_metrics"] = _clean_metric_tree(
+                rays_save["per_file_metrics"] = _clean_per_file_metrics(
                     _wrap_pfm_metrics(
                         rays_pfm,
                         lambda m: {"rays": {"eval": m.get("rays", {})}},
-                    )
+                    ),
+                    label="rays per_file_metrics",
                 )
             all_results.setdefault("per_file_metrics", {}).update(rays_pfm)
 

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval/data.py

@@ -525,13 +525,14 @@ def _resolve_sky_mask_loader(segmentation_path: str) -> Callable[..., Any]:
     its ``sky_mask`` function so that raw class-segmentation data is
     converted to a boolean sky mask at load time.
     """
+    modality = _modality(path=segmentation_path, modality_key="segmentation")
     try:
         index = index_dataset_from_path(
-            segmentation_path,
-            metadata_scope="segmentation",
+            modality.path,
+            metadata_scope=modality.metadata_scope,
         )
     except FileNotFoundError:
-        index = index_dataset_from_path(segmentation_path)
+        index = index_dataset_from_path(modality.path)
     # Resolve euler_loading metadata the same way euler-loading does:
     # try the contract addon API first (handles addons.euler_loading),
     # then fall back to top-level euler_loading for legacy indices.
@@ -546,14 +547,14 @@ def _resolve_sky_mask_loader(segmentation_path: str) -> Callable[..., Any]:
     loader_name = euler_meta.get("loader")
     if loader_name is None:
         raise ValueError(
-            f"Segmentation dataset at {segmentation_path!r} does not declare "
+            f"Segmentation dataset at {modality.path!r} does not declare "
             f"an 'euler_loading.loader' in its ds-crawler index."
         )
     module = resolve_loader_module(loader_name)
     sky_fn = getattr(module, "sky_mask", None)
     if sky_fn is None or not callable(sky_fn):
         raise ValueError(
-            f"Loader module {loader_name!r} (resolved from {segmentation_path!r}) "
+            f"Loader module {loader_name!r} (resolved from {modality.path!r}) "
             f"does not expose a 'sky_mask' function."
         )
     return sky_fn

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval/evaluate.py

@@ -1613,6 +1613,7 @@ def evaluate_sparse_depth_samples(
                 for key, value in metrics["standard_metrics"].items()
             },
             "depth_metrics": {
+                "valid_pixel_count": metrics["valid_pixel_count"],
                 "absrel": float(np.mean(absrel_arr)) if len(absrel_arr) > 0 else None,
                 "rmse": (
                     float(np.sqrt(np.mean(rmse_arr))) if len(rmse_arr) > 0 else None

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval/metrics/utils.py

@@ -145,6 +145,7 @@ def get_benchmark_depth_bins(
     well-behaved as range_min approaches zero (unlike log).
 
     For example:
+        [0.01, 80] -> near=[0.01, 9.29), mid=[9.29, 35.95), far=[35.95, 80].
         [0.1, 80] -> near=[0.1, 10.19), mid=[10.19, 36.83), far=[36.83, 80].
         [1, 80]   -> near=[1, 13.31),   mid=[13.31, 39.64), far=[39.64, 80].
 

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.8.0
+Version: 2.10.0
 Summary: Depth map evaluation toolkit with comprehensive metrics
 Author: Depth Eval Contributors
 License: MIT
@@ -45,7 +45,7 @@ A comprehensive evaluation toolkit for comparing predicted depth maps, RGB image
 - **Depth metrics**: PSNR, SSIM, LPIPS, FID, KID, AbsRel, RMSE, Scale-Invariant Log Error, Normal Consistency, Depth Edge F1
 - **RGB metrics**: PSNR, SSIM, LPIPS, FID, SCE (Structural Chromatic Error), Edge F1, Tail Errors (p95/p99), High-Frequency Energy Ratio, Depth-Binned Photometric Error
 - **Rays metrics**: ρ_A (AUC of angular accuracy curve), Angular Error statistics and threshold percentages
-- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into log-scaled near/mid/far bins
+- **Benchmark binning**: Optional depth-range benchmark that subdivides metrics into square-root-scaled near/mid/far bins
 - **Sanity checking**: Automatic validation of metric results against configurable thresholds, with detailed warning reports
 - **Sky masking**: Optional exclusion of sky regions from metrics using GT segmentation
 - **Flexible dataset loading**: Automatic loader resolution via euler_loading and ds-crawler index metadata
@@ -141,7 +141,7 @@ This pre-downloads:
 | `--metrics-config` | `str` | auto-detect | Path to `metrics_config.json` for sanity checking |
 | `--depth-alignment` | `{none,auto_affine,affine}` | `auto_affine` | Depth calibration mode; outputs are emitted in semantic `native`/`metric` spaces and `depth` aliases the canonical branch |
 | `--rgb-fid-backend` | `{builtin,clean-fid}` | `builtin` | RGB FID backend; `clean-fid` requires optional dependency |
-| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into log-scaled near/mid/far bins (additive to regular metrics) |
+| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` in meters for benchmark evaluation; computes depth and RGB metrics for pixels within this range, subdivided into square-root-scaled near/mid/far bins (additive to regular metrics) |
 
 ### Examples
 
@@ -168,7 +168,7 @@ depth-eval config.json --depth-alignment affine
 depth-eval config.json --rgb-fid-backend clean-fid
 
 # Benchmark depth and RGB metrics within a depth range (near/mid/far bins)
-depth-eval config.json --benchmark-depth-range 0.5 80.0
+depth-eval config.json --benchmark-depth-range 0.01 80.0
 
 # Evaluate dense depth predictions against sparse pointcloud GT
 depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
@@ -177,6 +177,30 @@ depth-eval example_sparse_depth_config.json --skip-rgb --skip-rays
 depth-eval config.json --skip-rays
 ```
 
+### Benchmark Depth Bins
+
+`--benchmark-depth-range MIN MAX` first filters valid GT depth pixels to
+`MIN <= depth <= MAX`. The `near`, `mid`, and `far` bins are then computed as
+three equal-width intervals in square-root depth space:
+
+```text
+sqrt_min = sqrt(MIN)
+sqrt_max = sqrt(MAX)
+step = (sqrt_max - sqrt_min) / 3
+near_max = (sqrt_min + step)^2
+mid_max = (sqrt_min + 2 * step)^2
+```
+
+The interval bounds are `near=[MIN, near_max)`, `mid=[near_max, mid_max)`, and
+`far=[mid_max, MAX]`. For `--benchmark-depth-range 0.01 80.0`, this gives:
+
+| Bin | Depth interval |
+|---|---|
+| `all` | `[0.01, 80.0]` |
+| `near` | `[0.01, 9.290856529)` |
+| `mid` | `[9.290856529, 35.954189863)` |
+| `far` | `[35.954189863, 80.0]` |
+
 ## Configuration
 
 ### `config.json`
@@ -215,7 +239,7 @@ Defines GT modalities, prediction datasets to evaluate, and optional euler_train
 }
 ```
 
-Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`).
+Each modality entry can optionally include a `split` field to select a specific split from the dataset (e.g. `{ "path": "/data/gt/depth", "split": "test" }`). euler-loading inline selectors are also accepted in `path`, such as `/data/muses.zip:test` or `/data/muses.zip:test#scope=rgb`.
 
 For sparse pointcloud depth evaluation, use `gt.sparse_depth` instead of `gt.depth`. The prediction still uses `datasets[].depth` because the model output is a dense depth map. The evaluator projects the sparse GT point cloud into the prediction image plane using `gt.intrinsics` and `gt.camera_extrinsics`, then computes pointwise depth metrics only at projected valid pixels.
 
@@ -269,7 +293,7 @@ Dataset metadata (e.g. `radial_depth`, `rgb_range`, sparse point columns, and co
 
 ### Dataset Metadata
 
-Each dataset directory must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
+Each dataset directory or archive must contain ds-crawler metadata artifacts, typically generated under `.ds_crawler/`:
 
 ```text
 dataset-root/
@@ -279,6 +303,8 @@ dataset-root/
     index.json
 ```
 
+When one physical root or archive contains several logical modalities, the artifacts may instead be scoped under `.ds_crawler/<modality>/` with a `.ds_crawler/scopes.json` manifest. `euler-eval` passes modality scopes to euler-loading so paths such as `/data/muses.zip:test` can load `.ds_crawler/rgb/index.json`, `.ds_crawler/depth/index.json`, and related scoped artifacts from the same archive.
+
 GT and prediction datasets are matched by hierarchy path and file ID through `MultiModalDataset`.
 
 ### `metrics_config.json`

{euler_eval-2.8.0 → euler_eval-2.10.0}/euler_eval.egg-info/SOURCES.txt

@@ -46,6 +46,7 @@ tests/test_evaluate_helpers.py
 tests/test_init_cache.py
 tests/test_integration.py
 tests/test_meta_output.py
+tests/test_metrics_utils.py
 tests/test_rgb_fid_output.py
 tests/test_rho_a.py
 tests/test_save_results.py

{euler_eval-2.8.0 → euler_eval-2.10.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "euler-eval"
-version = "2.8.0"
+version = "2.10.0"
 description = "Depth map evaluation toolkit with comprehensive metrics"
 readme = "README.md"
 requires-python = ">=3.9"

{euler_eval-2.8.0 → euler_eval-2.10.0}/tests/test_config.py

@@ -1,6 +1,7 @@
 """Tests for main.py config validation."""
 
 import json
+import zipfile
 import pytest
 
 from euler_eval.cli import load_config, validate_dataset_entry, validate_gt_config
@@ -50,6 +51,20 @@ class TestValidateGtConfig:
         gt = {"rgb": {"path": str(rgb_path)}}
         validate_gt_config(gt)
 
+    def test_inline_zip_split_gt_path_is_valid(self, tmp_path):
+        rgb_path = tmp_path / "reference.zip"
+        with zipfile.ZipFile(rgb_path, "w") as zf:
+            zf.writestr("dummy.txt", "x")
+        gt = {"rgb": {"path": f"{rgb_path}:fog_day"}}
+        validate_gt_config(gt)
+
+    def test_inline_scope_gt_path_is_valid(self, tmp_path):
+        rgb_path = tmp_path / "reference.zip"
+        with zipfile.ZipFile(rgb_path, "w") as zf:
+            zf.writestr("dummy.txt", "x")
+        gt = {"rgb": {"path": f"{rgb_path}#scope=rgb"}}
+        validate_gt_config(gt)
+
     def test_sparse_depth_requires_projection_modalities(self, tmp_path):
         sparse_path = tmp_path / "sparse_depth"
         sparse_path.mkdir()
@@ -132,6 +147,16 @@ class TestValidateDatasetEntry:
         with pytest.raises(ValueError, match="does not exist"):
             validate_dataset_entry(entry, 0)
 
+    def test_inline_zip_split_dataset_path_is_valid(self, tmp_path):
+        rgb_path = tmp_path / "prediction.zip"
+        with zipfile.ZipFile(rgb_path, "w") as zf:
+            zf.writestr("dummy.txt", "x")
+        entry = {
+            "name": "model_a",
+            "rgb": {"path": f"{rgb_path}:fog_day"},
+        }
+        validate_dataset_entry(entry, 0)
+
     def test_index_in_error_message(self, tmp_path):
         entry = {"name": "test"}
         with pytest.raises(ValueError, match=r"datasets\[3\]"):

{euler_eval-2.8.0 → euler_eval-2.10.0}/tests/test_data_builders.py

@@ -169,3 +169,26 @@ def test_sky_mask_loader_resolution_uses_segmentation_scope(monkeypatch):
     assert calls == [
         ("/datasets/shared", {"metadata_scope": "segmentation"}),
     ]
+
+
+def test_sky_mask_loader_resolution_strips_inline_split(monkeypatch):
+    calls = []
+
+    def fake_index_dataset_from_path(path, **kwargs):
+        calls.append((path, kwargs))
+        return {"euler_loading": {"loader": "vkitti2"}}
+
+    def sky_mask(path, meta=None):
+        return None
+
+    monkeypatch.setattr(data, "index_dataset_from_path", fake_index_dataset_from_path)
+    monkeypatch.setattr(
+        data,
+        "resolve_loader_module",
+        lambda name: SimpleNamespace(sky_mask=sky_mask),
+    )
+
+    assert data._resolve_sky_mask_loader("/datasets/shared.zip:fog_day") is sky_mask
+    assert calls == [
+        ("/datasets/shared.zip", {"metadata_scope": "segmentation"}),
+    ]

{euler_eval-2.8.0 → euler_eval-2.10.0}/tests/test_meta_output.py

@@ -4,6 +4,7 @@ import re
 
 from euler_eval.cli import (
     _clean_metric_tree,
+    _clean_per_file_metrics,
     _depth_eval_axes,
     _DEPTH_EVAL_DESCRIPTIONS,
     _sparse_depth_eval_axes,
@@ -15,6 +16,7 @@ from euler_eval.cli import (
     _RGB_EVAL_DESCRIPTIONS,
     _RAYS_EVAL_AXES,
     _RAYS_EVAL_DESCRIPTIONS,
+    _wrap_depth_space_pfm_metrics,
 )
 
 # Build base (non-benchmark) axis dicts for testing
@@ -97,6 +99,105 @@ class TestMetaBlockStructure:
         cleaned = _clean_metric_tree(meta)
         assert "eval_params" not in cleaned
 
+    def test_per_file_metrics_object_preserved_when_metrics_clean_empty(self):
+        """Per-file entries keep a metrics object even if all values are pruned."""
+        per_file = {
+            "files": [
+                {
+                    "id": "frame_0001",
+                    "metrics": {
+                        "rgb": {
+                            "eval": {
+                                "image_quality": {
+                                    "psnr": None,
+                                    "ssim": float("nan"),
+                                },
+                            },
+                        },
+                    },
+                },
+            ],
+        }
+
+        assert _clean_metric_tree(per_file) == {"files": [{"id": "frame_0001"}]}
+        assert _clean_per_file_metrics(per_file) == {
+            "files": [{"id": "frame_0001", "metrics": {}}]
+        }
+
+    def test_depth_per_file_wrapper_falls_back_to_canonical_metrics(self):
+        """Canonical-only per-file depth metrics still serialize under a space."""
+        metrics = {
+            "depth": {
+                "standard": {
+                    "absrel": 0.1,
+                },
+            },
+        }
+
+        assert _wrap_depth_space_pfm_metrics(
+            metrics,
+            metric_root="depth",
+            canonical_key="depth",
+            canonical_space="native",
+        ) == {
+            "depth": {
+                "eval": {
+                    "native": metrics["depth"],
+                },
+            },
+        }
+
+    def test_sparse_depth_zero_valid_count_survives_cleaning(self):
+        """Zero-support sparse entries keep a finite support metric in eval.json."""
+        metrics = {
+            "sparse_depth": {
+                "standard": {
+                    "absrel": None,
+                },
+                "depth_metrics": {
+                    "valid_pixel_count": 0,
+                    "absrel": None,
+                    "rmse": None,
+                },
+            },
+        }
+
+        wrapped = _wrap_depth_space_pfm_metrics(
+            metrics,
+            metric_root=_SPARSE_DEPTH_METRIC_ROOT,
+            canonical_key="sparse_depth",
+            canonical_space="metric",
+        )
+        cleaned = _clean_per_file_metrics(
+            {
+                "files": [
+                    {
+                        "id": "frame",
+                        "metrics": wrapped,
+                    },
+                ],
+            }
+        )
+
+        assert cleaned == {
+            "files": [
+                {
+                    "id": "frame",
+                    "metrics": {
+                        _SPARSE_DEPTH_METRIC_ROOT: {
+                            "eval": {
+                                "metric": {
+                                    "depth_metrics": {
+                                        "valid_pixel_count": 0,
+                                    },
+                                },
+                            },
+                        },
+                    },
+                },
+            ],
+        }
+
     def test_meta_in_full_save_dict(self):
         """meta sits alongside metricSet and dataset_info without conflict."""
         save_dict = {

euler_eval-2.10.0/tests/test_metrics_utils.py

@@ -0,0 +1,57 @@
+"""Tests for shared metric utility helpers."""
+
+import numpy as np
+
+from euler_eval.metrics.utils import get_benchmark_depth_bins
+
+
+def test_benchmark_depth_bins_for_001_to_80m_range():
+    range_min = 0.01
+    range_max = 80.0
+    sqrt_min = np.sqrt(range_min)
+    sqrt_max = np.sqrt(range_max)
+    step = (sqrt_max - sqrt_min) / 3.0
+    near_max = float((sqrt_min + step) ** 2)
+    mid_max = float((sqrt_min + 2 * step) ** 2)
+
+    depth = np.array(
+        [
+            0.0,
+            range_min,
+            np.nextafter(near_max, range_min),
+            near_max,
+            np.nextafter(mid_max, near_max),
+            mid_max,
+            range_max,
+            np.nextafter(range_max, np.inf),
+        ],
+        dtype=np.float64,
+    )
+
+    bins = get_benchmark_depth_bins(depth, range_min, range_max)
+
+    assert bins["boundaries"] == {
+        "range": [range_min, range_max],
+        "near": [range_min, near_max],
+        "mid": [near_max, mid_max],
+        "far": [mid_max, range_max],
+    }
+    assert np.isclose(near_max, 9.290856529333297)
+    assert np.isclose(mid_max, 35.95418986266663)
+
+    np.testing.assert_array_equal(
+        bins["all"],
+        np.array([False, True, True, True, True, True, True, False]),
+    )
+    np.testing.assert_array_equal(
+        bins["near"],
+        np.array([False, True, True, False, False, False, False, False]),
+    )
+    np.testing.assert_array_equal(
+        bins["mid"],
+        np.array([False, False, False, True, True, False, False, False]),
+    )
+    np.testing.assert_array_equal(
+        bins["far"],
+        np.array([False, False, False, False, False, True, True, False]),
+    )

{euler_eval-2.8.0 → euler_eval-2.10.0}/tests/test_save_results.py

@@ -214,6 +214,21 @@ class TestSaveResultsZip:
             stored = json.loads(zf.read("eval.json"))
             assert stored == results
 
+    def test_inline_zip_split_writes_eval_json_into_zip(self, tmp_path):
+        zp = tmp_path / "dataset.zip"
+        with zipfile.ZipFile(zp, "w") as zf:
+            zf.writestr("image_001.png", b"\x89PNG")
+
+        results = {"depth": {"rmse": 0.42}}
+        config = {"name": "model", "depth": {"path": f"{zp}:fog_day"}}
+        out = save_results(results, config)
+
+        assert out == zp / "eval.json"
+        with zipfile.ZipFile(zp, "r") as zf:
+            assert "eval.json" in zf.namelist()
+            stored = json.loads(zf.read("eval.json"))
+            assert stored == results
+
     def test_preserves_existing_zip_contents(self, tmp_path):
         zp = tmp_path / "dataset.zip"
         with zipfile.ZipFile(zp, "w") as zf:

{euler_eval-2.8.0 → euler_eval-2.10.0}/tests/test_sparse_depth.py

@@ -93,6 +93,12 @@ class _OneSampleSparseDataset:
         return {"gt": {"path": "/gt"}, "pred": {"path": "/pred"}}
 
 
+class _ZeroValidSparseDataset(_OneSampleSparseDataset):
+    def __init__(self):
+        super().__init__()
+        self.pred = np.zeros((3, 3), dtype=np.float32)
+
+
 def test_sparse_depth_eval_reports_only_pointwise_depth_metrics():
     dataset = _OneSampleSparseDataset()
 
@@ -111,3 +117,20 @@ def test_sparse_depth_eval_reports_only_pointwise_depth_metrics():
     np.testing.assert_allclose(metrics["standard"]["pixel_pool"]["absrel"], 0.0)
     np.testing.assert_allclose(metrics["standard"]["pixel_pool"]["delta1"], 1.0)
     np.testing.assert_allclose(metrics["depth_metrics"]["rmse"]["median"], 0.0)
+
+
+def test_sparse_depth_per_file_reports_zero_valid_support():
+    result = evaluate_sparse_depth_samples(
+        _ZeroValidSparseDataset(),
+        pred_is_radial=True,
+        num_workers=0,
+        alignment_mode="none",
+    )
+
+    assert result["dataset_info"]["projected_pixels"] == 2
+    assert result["dataset_info"]["evaluated_pixels"] == 0
+
+    per_file = result["per_file_metrics"]["files"][0]["metrics"]["sparse_depth"]
+    assert per_file["depth_metrics"]["valid_pixel_count"] == 0
+    assert per_file["depth_metrics"]["absrel"] is None
+    assert per_file["depth_metrics"]["rmse"] is None