euler-eval 2.31.0__tar.gz → 2.32.0__tar.gz

{euler_eval-2.31.0/euler_eval.egg-info → euler_eval-2.32.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.31.0
+Version: 2.32.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT
@@ -133,8 +133,10 @@ euler-eval config.json --batch-size 32
 
 Ground truth and prediction files are matched by ID rather than by directory
 order, so an incomplete prediction set scores the frames it does have instead of
-silently comparing the wrong pairs. Results are written next to each prediction
-modality:
+silently comparing the wrong pairs. Results are written into each prediction
+dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
+`/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
+dataset ([details](docs/output.md#where-results-land)):
 
 ```json
 {

{euler_eval-2.31.0 → euler_eval-2.32.0}/README.md

@@ -83,8 +83,10 @@ euler-eval config.json --batch-size 32
 
 Ground truth and prediction files are matched by ID rather than by directory
 order, so an incomplete prediction set scores the frames it does have instead of
-silently comparing the wrong pairs. Results are written next to each prediction
-modality:
+silently comparing the wrong pairs. Results are written into each prediction
+dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
+`/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
+dataset ([details](docs/output.md#where-results-land)):
 
 ```json
 {

{euler_eval-2.31.0 → euler_eval-2.32.0}/docs/README.md

@@ -7,7 +7,7 @@
 | [Spaces & alignment](alignment.md) | Why `native` and `metric` exist, depth affine fitting, points-3d gauge alignment, benchmark depth bins |
 | [Metrics](metrics.md) | The full metric inventory per modality, with the key each one is written under |
 | [Error distributions](distributions.md) | Optional per-sample and pooled RMSE histograms, bin settings, axes and JSON layout |
-| [Results & output](output.md) | Where files land, the anatomy of `eval.json`, per-file metrics, the sanity-check report |
+| [Results & output](output.md) | Where files land, the anatomy of `eval.json`, comparing evaluations, per-file metrics, the sanity-check report |
 | [In-training validation](validation.md) | The programmatic API for scoring in-memory predictions during training |
 
 ## Proposed features

{euler_eval-2.31.0 → euler_eval-2.32.0}/docs/cli.md

@@ -37,6 +37,7 @@ validation exits with status `1` before any dataset is opened.
 | `--distribution-bins` | `int` | `50` | Total histogram bins, including tails; minimum 3 |
 | `--distribution-scale` | `{log,linear}` | `log` | Spacing of finite histogram bins |
 | `--distribution-range` | `float float` | `0.001 100` | Finite error range `[MIN, MAX)` in evaluated space units; zero/small-error and overflow bins retain the tails |
+| `--eval-name` | `str` | `eval` | Name this evaluation: results are written as `<name>.json` in each prediction dataset's `.eval/` folder, so differently named runs sit side by side; a dataset's `output_file` still wins ([details](output.md#where-results-land)) |
 
 Histogram settings can also be specified in the config's `distributions`
 section. A bin, scale or range flag enables histograms and overrides that
@@ -69,6 +70,9 @@ euler-eval config.json --sky-iou --sky-depth 100
 # Only depth evaluation
 euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
 
+# Keep a sky-masked run beside the default results: .eval/[<scope>/]no_sky.json
+euler-eval config.json --mask-sky --eval-name no_sky
+
 # Force affine scale+shift alignment on all depth predictions
 euler-eval config.json --depth-alignment affine
 

{euler_eval-2.31.0 → euler_eval-2.32.0}/docs/configuration.md

@@ -127,7 +127,7 @@ would be ambiguous.
 | `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth`, `--mask-sky-prediction` and `--sky-iou` |
 | `rays.path` | no\* | Predicted ray direction map dataset |
 | `points_3d.path` | no\* | Predicted per-pixel 3D point map `(H,W,3)` |
-| `output_file` | no | Custom results path (default: `eval.json` in the first modality path) |
+| `output_file` | no | Custom results path for every modality of this entry, replacing the `.eval/` layout and `--eval-name` (default: `.eval/[<scope>/][split_<split>/]eval.json` in each prediction modality's dataset root; see [where results land](output.md#where-results-land)) |
 
 \* At least one prediction modality must be present.
 
@@ -214,10 +214,11 @@ and evaluated as `points_3d`.
 
 An explicit `datasets[].points_3d` prediction can be evaluated separately
 against the same sparse cloud. That evaluation uses the points-3D similarity
-gauge, writes an atomic `points3d.eval` metric set beside the point-map
-prediction, and is controlled by `--skip-points-3d`. If a dataset provides both
-depth and `points_3d`, each prediction modality gets its own `eval.json` at its
-own path.
+gauge, writes an atomic `points3d.eval` metric set to the point-map
+prediction's results file, and is controlled by `--skip-points-3d`. If a dataset
+provides both depth and `points_3d`, each prediction modality gets its own
+results file, also when both live in one archive under separate scopes (see
+[where results land](output.md#where-results-land)).
 
 Sparse depth does not require segmentation GT. `gt.segmentation` is loaded only
 when `--mask-sky` or `--sky-iou` is set. `--mask-sky` excludes sky pixels from

{euler_eval-2.31.0 → euler_eval-2.32.0}/docs/output.md

@@ -2,20 +2,56 @@
 
 ## Where results land
 
-One `eval.json` is written **per modality, per prediction dataset**, next to the
-prediction it scores — `/data/model_a/depth/eval.json` for the depth results of
-the dataset whose depth path is `/data/model_a/depth`, and so on. If that path
-passes through a `.zip`, the JSON is written *into* the archive rather than
-beside it, so zip-backed datasets stay self-contained.
+Results are written **per modality, per prediction dataset**, into a `.eval/`
+folder at the prediction dataset's root, beside its `.ds_crawler/` metadata:
 
-`output_file` in a dataset entry overrides the destination. It applies to the
-dataset as a whole, so when several modalities are evaluated for the same
-dataset the per-modality default locations are usually what you want.
+```text
+/data/model_a/                     # a datasets[].<modality>.path
+  .ds_crawler/
+    depth/  rgb/                   # ds-crawler metadata scopes
+  .eval/
+    depth/
+      eval.json                    # depth results
+      no_sky.json                  # depth results of --eval-name no_sky
+      split_test/eval.json         # depth results for "/data/model_a:test"
+    rgb/
+      eval.json                    # rgb results
+```
+
+The folder below `.eval/` mirrors how the prediction was loaded:
+
+| Prediction | Results file |
+|---|---|
+| Metadata scope with its own `.ds_crawler/<scope>/` | `.eval/<scope>/<name>.json` |
+| Root-level `.ds_crawler/` metadata (no scope) | `.eval/<name>.json` |
+| A split is selected (`:test` or `split`) | `split_<split>/` is added before the file name |
+
+The scope is the path's `#scope=` selector, otherwise the config key (`depth`,
+`relative_depth`, `rgb`, `points_3d`, …). As in euler-loading, a scope without
+metadata of its own falls back to the root-level `.ds_crawler/`, and so do its
+results. This way modalities and splits that share one dataset root or archive
+each get their own file. If the root is a `.zip`, `.eval/` is written *into* the
+archive (inside its top-level folder when the archive wraps everything in one
+named after it), so zip-backed datasets stay self-contained.
+
+`<name>` is `eval` unless the run is named with `--eval-name`:
+`--mask-sky --eval-name no_sky` writes `no_sky.json` next to an earlier
+`eval.json` instead of replacing it. Names use letters, digits, `.`, `-` and
+`_`; `.json` is appended.
+
+`output_file` in a dataset entry overrides all of this. It is one path for the
+whole dataset entry: every modality evaluated for that entry is written to it,
+and the last one written replaces the others, so use it for single-modality
+entries.
+
+Earlier releases wrote `eval.json` directly at the dataset root (inside the
+archive for zips). Readers such as euler-view look in `.eval/<scope>/`, then
+`.eval/`, then fall back to that legacy location.
 
 Each `eval.json` is atomic and declares one metric namespace. With
 `gt.sparse_depth`, a depth-family prediction writes only `sparsedepth.eval`; an
-explicit `points_3d` prediction writes only `points3d.eval` beside the point
-map. A depth prediction does not implicitly produce a second points-3D metric
+explicit `points_3d` prediction writes only `points3d.eval` to the point
+map's own results file. A depth prediction does not implicitly produce a second points-3D metric
 set.
 
 ## Anatomy of `eval.json`
@@ -37,6 +73,10 @@ set.
     "axes": { "space": { "…": "…" }, "category": { "…": "…" } },
     "metricDescriptions": { "absrel": { "…": "…" } }
   },
+  "protocol": {
+    "schemaVersion": "1.0",
+    "parameters": { "reference.gt.dataset": "muses", "method.depth_alignment": "auto_affine", "…": "…" }
+  },
   "dataset_info": { "num_pairs": 500, "gt_name": "GT", "pred_name": "model_a" },
   "meta": {
     "version": "2.25.0",
@@ -56,6 +96,7 @@ set.
 | Block | What it is for |
 |---|---|
 | `metricSet` | The euler-metric-naming envelope: producer, namespace, axis declarations and per-metric descriptions (unit, direction, bounds), plus how the spaces were resolved |
+| `protocol` | The comparison keys: every setting that changes metric values, as a flat map ([Comparing evaluations](#comparing-evaluations)) |
 | `dataset_info` | How many pairs were actually matched, and the display names of both sides |
 | `meta` | Full provenance: package version, device, both source paths and their resolved dimensions, modality metadata, and the effective evaluation flags |
 | *metric tree* | The metrics themselves, under the modality root (`depth`, `rgb`, `rays`, `sparsedepth`, `points3d`) |
@@ -137,6 +178,88 @@ because these keys have no depth space. `metricDescriptions` gains `iou`,
 `precision` and `recall` (higher is better, bounds `[0, 1]`). Every other key
 decomposes exactly as before.
 
+## Comparing evaluations
+
+`protocol.parameters` says how an evaluation was set up, as one flat map of
+dotted keys. Its keys are the comparison keys: two evaluations of the same
+modality were set up the same way when their maps are equal, and the keys whose
+values differ say where. Nothing else in the file needs to be compared for that;
+`meta` stays the full record of the run, paths and device included.
+
+```json
+"protocol": {
+  "schemaVersion": "1.0",
+  "parameters": {
+    "reference.gt.modality": "sparse_depth",
+    "reference.gt.dataset": "seeingthroughfog",
+    "reference.gt.scope": "sparse_depth",
+    "reference.gt.split": "test",
+    "reference.intrinsics.dataset": "seeingthroughfog",
+    "reference.intrinsics.scope": "intrinsics",
+    "reference.intrinsics.split": "test",
+    "reference.camera_extrinsics.dataset": "seeingthroughfog",
+    "reference.camera_extrinsics.scope": "camera_extrinsics",
+    "reference.camera_extrinsics.split": "test",
+    "reference.lidar_extrinsics.dataset": null,
+    "reference.lidar_extrinsics.scope": null,
+    "reference.lidar_extrinsics.split": null,
+    "reference.segmentation.dataset": null,
+    "reference.segmentation.scope": null,
+    "reference.segmentation.split": null,
+    "population.samples": 943,
+    "population.gt_sky_masking": false,
+    "population.pred_sky_masking": false,
+    "population.evaluated_resolution": [1024, 1920],
+    "population.benchmark_depth_range_m": [0.01, 80.0],
+    "method.metrics_revision": 1,
+    "method.depth_alignment": "auto_affine",
+    "method.sky_depth_m": null
+  }
+}
+```
+
+The first segment of a key is its section:
+
+| Section | Holds |
+|---|---|
+| `reference` | The GT the prediction is scored against (`gt`, with its config key as `modality`) and every other GT input the modality reads, each identified by `dataset` (the `dataset.id` of its `dataset-head.json`), metadata `scope` and `split` — not by file path, so the same data on another machine matches |
+| `population` | Which samples and pixels were scored: the matched `samples`, sky masking, the grid the metrics were computed on and benchmark depth bins |
+| `method` | How they were scored: alignment modes, the sky-depth cap, how GT and prediction resolutions were reconciled, the FID backend and `metrics_revision` |
+
+What is recorded, per modality:
+
+| Modality | `reference` inputs besides `gt` | `population` | `method` |
+|---|---|---|---|
+| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `spatial_alignment` |
+| `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m` |
+| `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
+| `rays` | `calibration` | `samples`, `evaluated_resolution` | `spatial_alignment` |
+| `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
+| `points3d` (sparse GT) | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment` |
+
+Every modality also has `method.metrics_revision`.
+
+Rules that keep comparisons a plain map comparison:
+
+- Only settings that change the value of a metric both files contain are
+  parameters. `--domain`, `--sky-iou` and histogram settings only add metrics
+  (histograms describe their bins in the distribution registry); device, batch
+  size, workers, paths and `--eval-name` do not change values.
+- Every key of a modality is always written, `null` or `false` when off, so a
+  key missing from one side means "not recorded", never "off". Compare the union
+  of both key sets.
+- Values are scalars or lists of scalars, never objects.
+- `method.metrics_revision` changes only when a release changes metric values
+  for identical inputs and settings, which the [changelog](../CHANGELOG.md)
+  then says. Different revisions are not comparable even when every other
+  key matches.
+- Settings are recorded, not what they caused. With `--mask-sky-prediction`,
+  `population.pred_sky_masking` is `true` for every prediction, while
+  `meta.pred_sky_masking` records how much each model's own sky removed;
+  `metricSet.metadata` records whether an alignment was actually applied.
+- Renaming or removing a key bumps the major `schemaVersion`; adding one bumps
+  the minor version.
+
 ## Spaces and canonical aliases
 
 Depth-like and points-3d results are nested by space:

{euler_eval-2.31.0 → euler_eval-2.32.0}/euler_eval/cli.py

@@ -11,16 +11,19 @@ import importlib.metadata
 import json
 import math
 import platform
+import re
 import sys
 import zipfile
 from dataclasses import replace
 from pathlib import Path
 
 import torch
+from ds_crawler.zip_utils import get_zip_root_prefix
 
 from .config_paths import (
     GT_SEGMENTATION_KEYS,
     normalize_modality_path,
+    resolve_dataset_location,
     resolve_gt_segmentation_path,
 )
 from .data import (
@@ -53,6 +56,7 @@ from .metric_sets import (
     resolve_metric_sets,
     selected_domain_categories,
 )
+from .protocol import build_protocol, evaluated_resolution, reference_identities
 from .sanity_checker import SanityChecker
 
 try:
@@ -1151,10 +1155,82 @@ def _save_json_to_zip(zip_path: Path, internal_name: str, data: dict) -> None:
             zf.writestr(internal_name, json_bytes)
 
 
+EVAL_DIR = ".eval"
+DEFAULT_EVAL_NAME = "eval"
+_EVAL_NAME_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]*$")
+
+
+def validate_eval_name(value: str) -> str:
+    """Validate an evaluation name and return it without a ``.json`` suffix.
+
+    The name becomes the results file name (``<name>.json``) below each
+    prediction dataset's ``.eval/`` folder, so it is one portable path
+    segment, like a ds-crawler split or scope name.
+
+    Raises:
+        ValueError: When *value* is not a plain name.
+    """
+    name = value.strip()
+    if name.endswith(".json"):
+        name = name[: -len(".json")]
+    if not _EVAL_NAME_PATTERN.match(name):
+        raise ValueError(
+            f"{value!r} is not a valid evaluation name: use letters, digits, "
+            "'.', '-' or '_', starting with a letter or digit (e.g. 'no_sky'); "
+            "'.json' is appended."
+        )
+    return name
+
+
+def default_eval_path(
+    modality_path: str | Path,
+    *,
+    modality_key: str,
+    split: str | None = None,
+    eval_name: str = DEFAULT_EVAL_NAME,
+) -> Path:
+    """Return where a prediction modality's results land by default.
+
+    Results go to a dataset-level ``.eval/`` folder beside ``.ds_crawler/``:
+    ``.eval/<scope>/`` when the modality's metadata scope (the ``#scope=``
+    selector, else the config key) has its own ``.ds_crawler/<scope>/``
+    metadata, otherwise ``.eval/`` itself, matching how euler-loading falls
+    back to root-level metadata. A selected split adds ``split_<split>/``.
+    """
+    root, scope, split = resolve_dataset_location(
+        modality_path, modality_key=modality_key, split=split
+    )
+    eval_dir = root / EVAL_DIR
+    if scope is not None:
+        eval_dir = eval_dir / scope
+    if split is not None:
+        eval_dir = eval_dir / f"split_{split}"
+    return eval_dir / f"{eval_name}.json"
+
+
+def _zip_internal_name(zip_path: Path, internal_name: str) -> str:
+    """Place *internal_name* inside a zip's wrapper folder, if it has one.
+
+    ds-crawler treats a single top-level folder named after the archive
+    (``model.zip`` -> ``model/``) as the dataset root. Writing beside that
+    folder would end the single-root layout and hide ``.ds_crawler/``.
+    """
+    prefix = get_zip_root_prefix(zip_path)
+    if (
+        prefix
+        and prefix.rstrip("/") == zip_path.stem
+        and not internal_name.startswith(prefix)
+    ):
+        return prefix + internal_name
+    return internal_name
+
+
 def save_results(
     results: dict,
     dataset_config: dict,
     modality: str | None = None,
+    *,
+    eval_name: str = DEFAULT_EVAL_NAME,
 ) -> Path:
     """Save results to output file.
 
@@ -1168,6 +1244,10 @@ def save_results(
         modality: When set, save to this specific modality's path
             (e.g. ``"depth"`` or ``"rgb"``).  Falls back to the first
             available modality path when *None*.
+        eval_name: Evaluation name (``--eval-name``); results are written
+            as ``<eval_name>.json`` in the dataset's ``.eval/`` layout, see
+            :func:`default_eval_path`.  A dataset's explicit ``output_file``
+            takes precedence over the whole layout.
 
     Returns:
         Path where results were saved.
@@ -1177,38 +1257,38 @@ def save_results(
         selected_modality = modality
         if selected_modality == "depth" and "depth" not in dataset_config:
             selected_modality, _ = _prediction_depth_entry(dataset_config)
-        if (
+        if not (
             selected_modality is not None
             and selected_modality in dataset_config
             and "path" in dataset_config[selected_modality]
         ):
-            output_file = (
-                normalize_modality_path(
-                    dataset_config[selected_modality]["path"],
-                    split=dataset_config[selected_modality].get("split"),
-                )
-                / "eval.json"
+            # Default: save alongside first available modality path
+            selected_modality = next(
+                (
+                    mod
+                    for mod in (*_PRED_DEPTH_KEYS, "rgb", "rays")
+                    if mod in dataset_config and "path" in dataset_config[mod]
+                ),
+                None,
+            )
+        if selected_modality is not None:
+            entry = dataset_config[selected_modality]
+            output_file = default_eval_path(
+                entry["path"],
+                modality_key=selected_modality,
+                split=entry.get("split"),
+                eval_name=eval_name,
             )
         else:
-            # Default: save alongside first available modality path
-            for mod in (*_PRED_DEPTH_KEYS, "rgb", "rays"):
-                if mod in dataset_config and "path" in dataset_config[mod]:
-                    output_file = (
-                        normalize_modality_path(
-                            dataset_config[mod]["path"],
-                            split=dataset_config[mod].get("split"),
-                        )
-                        / "eval.json"
-                    )
-                    break
-        if output_file is None:
-            output_file = Path("eval.json")
+            output_file = Path(f"{eval_name}.json")
     else:
         output_file = Path(output_file)
 
     zip_path, internal_name = _find_zip_ancestor(output_file)
     if zip_path is not None and internal_name:
-        _save_json_to_zip(zip_path, internal_name, results)
+        _save_json_to_zip(
+            zip_path, _zip_internal_name(zip_path, internal_name), results
+        )
     else:
         output_file.parent.mkdir(parents=True, exist_ok=True)
         with open(output_file, "w") as f:
@@ -1513,12 +1593,27 @@ def main():
         "--distribution-range", type=float, nargs=2, default=None, metavar=("MIN", "MAX"),
         help="Finite histogram range in evaluated space units (default: 0.001 100); tails are included",
     )
+    parser.add_argument(
+        "--eval-name",
+        default=DEFAULT_EVAL_NAME,
+        metavar="NAME",
+        help=(
+            "Name this evaluation: results are written as NAME.json in each "
+            "prediction dataset's .eval/ folder, so differently named runs "
+            "sit side by side (default: %(default)s). A dataset's "
+            "output_file still takes precedence."
+        ),
+    )
 
     args = parser.parse_args()
     try:
         validate_sky_depth(args.sky_depth)
     except ValueError as e:
         parser.error(f"--sky-depth: {e}")
+    try:
+        args.eval_name = validate_eval_name(args.eval_name)
+    except ValueError as e:
+        parser.error(f"--eval-name: {e}")
 
     selected_metric_sets = resolve_metric_sets(args.domains)
 
@@ -1629,6 +1724,10 @@ def main():
     print("-" * 60)
 
     gt = config["gt"]
+    protocol_identities = reference_identities(gt)
+    protocol_benchmark_range = (
+        list(args.benchmark_depth_range) if args.benchmark_depth_range else None
+    )
     gt_depth_path = gt.get("depth", {}).get("path")
     gt_sparse_depth_path = gt.get("sparse_depth", {}).get("path")
     gt_rgb_path = gt.get("rgb", {}).get("path")
@@ -1825,6 +1924,23 @@ def main():
                         "canonical_space": space_info.get("canonical_space", "metric"),
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="depth",
+                    inputs=("calibration", "segmentation"),
+                    population={
+                        "samples": depth_dataset_info.get("num_pairs"),
+                        "gt_sky_masking": bool(args.mask_sky),
+                        "pred_sky_masking": bool(args.mask_sky_prediction),
+                        "evaluated_resolution": evaluated_resolution(depth_spatial),
+                        "benchmark_depth_range_m": protocol_benchmark_range,
+                    },
+                    method={
+                        "depth_alignment": args.depth_alignment,
+                        "sky_depth_m": args.sky_depth,
+                        "spatial_alignment": depth_spatial.get("method", "none"),
+                    },
+                ),
                 "dataset_info": depth_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2063,6 +2179,29 @@ def main():
                         "canonical_space": space_info.get("canonical_space", "metric"),
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="sparse_depth",
+                    inputs=(
+                        "intrinsics",
+                        "camera_extrinsics",
+                        "lidar_extrinsics",
+                        "segmentation",
+                    ),
+                    population={
+                        "samples": sparse_depth_dataset_info.get("num_pairs"),
+                        "gt_sky_masking": bool(args.mask_sky),
+                        "pred_sky_masking": bool(args.mask_sky_prediction),
+                        "evaluated_resolution": evaluated_resolution(
+                            sparse_depth_spatial
+                        ),
+                        "benchmark_depth_range_m": protocol_benchmark_range,
+                    },
+                    method={
+                        "depth_alignment": args.depth_alignment,
+                        "sky_depth_m": args.sky_depth,
+                    },
+                ),
                 "dataset_info": sparse_depth_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2294,6 +2433,21 @@ def main():
                         ]
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="rgb",
+                    inputs=("depth", "calibration", "segmentation"),
+                    population={
+                        "samples": rgb_dataset_info.get("num_pairs"),
+                        "gt_sky_masking": bool(args.mask_sky),
+                        "evaluated_resolution": evaluated_resolution(rgb_spatial),
+                        "benchmark_depth_range_m": protocol_benchmark_range,
+                    },
+                    method={
+                        "fid_backend": args.rgb_fid_backend,
+                        "spatial_alignment": rgb_spatial.get("method", "none"),
+                    },
+                ),
                 "dataset_info": rgb_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2427,6 +2581,18 @@ def main():
                         "threshold_deg": rays_dataset_info.get("threshold_deg"),
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="rays",
+                    inputs=("calibration",),
+                    population={
+                        "samples": rays_dataset_info.get("num_pairs"),
+                        "evaluated_resolution": evaluated_resolution(rays_spatial),
+                    },
+                    method={
+                        "spatial_alignment": rays_spatial.get("method", "none"),
+                    },
+                ),
                 "dataset_info": rays_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2588,6 +2754,20 @@ def main():
                         "threshold_deg": p3_dataset_info.get("threshold_deg"),
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="depth" if p3d_gt_is_depth else "points_3d",
+                    inputs=("intrinsics", "calibration", "segmentation"),
+                    population={
+                        "samples": p3_dataset_info.get("num_pairs"),
+                        "gt_sky_masking": bool(args.mask_sky),
+                        "evaluated_resolution": evaluated_resolution(p3_spatial),
+                    },
+                    method={
+                        "points_3d_alignment": args.points_3d_alignment,
+                        "spatial_alignment": p3_spatial.get("method", "none"),
+                    },
+                ),
                 "dataset_info": p3_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2767,6 +2947,24 @@ def main():
                         "threshold_deg": p3s_dataset_info.get("threshold_deg"),
                     },
                 ),
+                "protocol": build_protocol(
+                    protocol_identities,
+                    gt="sparse_depth",
+                    inputs=(
+                        "intrinsics",
+                        "camera_extrinsics",
+                        "lidar_extrinsics",
+                        "segmentation",
+                    ),
+                    population={
+                        "samples": p3s_dataset_info.get("num_pairs"),
+                        "gt_sky_masking": bool(args.mask_sky),
+                        "evaluated_resolution": evaluated_resolution(p3s_spatial),
+                    },
+                    method={
+                        "points_3d_alignment": p3s_alignment_mode,
+                    },
+                ),
                 "dataset_info": p3s_dataset_info,
                 "meta": _clean_metric_tree({
                     "version": _get_version(),
@@ -2845,6 +3043,7 @@ def main():
                 depth_save,
                 dataset_config,
                 modality=pred_depth_key or "depth",
+                eval_name=args.eval_name,
             )
             depth_label = (
                 "Sparse depth"
@@ -2853,14 +3052,27 @@ def main():
             )
             print(f"\n  {depth_label} results saved to: {depth_out}")
         if rgb_save:
-            rgb_out = save_results(rgb_save, dataset_config, modality="rgb")
+            rgb_out = save_results(
+                rgb_save,
+                dataset_config,
+                modality="rgb",
+                eval_name=args.eval_name,
+            )
             print(f"\n  RGB results saved to: {rgb_out}")
         if rays_save:
-            rays_out = save_results(rays_save, dataset_config, modality="rays")
+            rays_out = save_results(
+                rays_save,
+                dataset_config,
+                modality="rays",
+                eval_name=args.eval_name,
+            )
             print(f"\n  Rays results saved to: {rays_out}")
         if points_3d_save:
             points_3d_out = save_results(
-                points_3d_save, dataset_config, modality="points_3d"
+                points_3d_save,
+                dataset_config,
+                modality="points_3d",
+                eval_name=args.eval_name,
             )
             print(f"\n  Points-3D results saved to: {points_3d_out}")
         if points_3d_sparse_save:
@@ -2868,6 +3080,7 @@ def main():
                 points_3d_sparse_save,
                 dataset_config,
                 modality="points_3d",
+                eval_name=args.eval_name,
             )
             print(
                 f"\n  Points-3D (sparse GT) results saved to: {points_3d_sparse_out}"