euler-eval 2.35.0__tar.gz → 2.36.0__tar.gz

{euler_eval-2.35.0/euler_eval.egg-info → euler_eval-2.36.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.35.0
+Version: 2.36.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT

{euler_eval-2.35.0 → euler_eval-2.36.0}/docs/cli.md

@@ -23,6 +23,7 @@ validation exits with status `1` before any dataset is opened.
 | `--skip-rays` | flag | off | Skip rays (spherical direction map) evaluation |
 | `--skip-points-3d` | flag | off | Skip points_3d (per-pixel 3D point map) evaluation |
 | `--mask-sky` | flag | off | Mask sky regions from metrics using GT segmentation |
+| `--missing-segmentation` | `{drop,unmasked}` | `drop` | Frames without their own GT segmentation: `drop` leaves them out; `unmasked` evaluates them without GT sky masking and leaves them out of the sky IoU ([details](configuration.md#gt-segmentation)) |
 | `--mask-sky-prediction` | flag | off | Exclude pixels in a prediction's `datasets[].sky_mask` from dense and sparse depth metrics and the alignment fit; predictions without a `sky_mask` are scored in full ([details](configuration.md#masking-predicted-sky)) |
 | `--sky-depth` | `float` | none | Cap GT and predicted depths in meters and fill predicted `sky_mask` pixels with this value; must be finite and positive |
 | `--sky-iou` | flag | off | Score predicted sky against GT segmentation sky: IoU, precision and recall for the `explicit` (`datasets[].sky_mask`) and `implicit` (`depth >= --sky-depth`, metric predictions only) variants. Requires `gt.segmentation`; depth metrics are unchanged ([details](metrics.md#sky-segmentation)) |
@@ -58,6 +59,9 @@ euler-eval config.json --batch-size 32
 # Sky masking (requires gt.segmentation)
 euler-eval config.json --mask-sky -v
 
+# Keep frames that have no GT segmentation, evaluated without sky masking
+euler-eval config.json --mask-sky --missing-segmentation unmasked
+
 # Exclude each prediction's own sky (datasets[].sky_mask) from depth metrics
 euler-eval config.json --mask-sky-prediction
 

{euler_eval-2.35.0 → euler_eval-2.36.0}/docs/configuration.md

@@ -103,6 +103,17 @@ warns and the sky IoU is skipped. A segmentation whose IDs match no frame
 stops the run with euler-loading's "No common IDs" error and per-modality file
 counts.
 
+When a split has segmentation for only some frames (MUSES lacks some), pass
+`--missing-segmentation unmasked` to keep the other frames: they are evaluated
+without GT sky masking and left out of the sky IoU, while frames with
+segmentation are scored exactly as before. The metrics then mix masked and
+unmasked frames, so `meta.gt_sky_masking` records how many frames had no
+segmentation, and `protocol.parameters` records the policy
+(`population.missing_segmentation`). Unmasked frames score their sky wherever
+the GT has valid depth there: little for a lidar GT, which has few returns in
+the sky, but possibly a lot for a dense GT with sky depth. The default,
+`drop`, leaves those frames out.
+
 The config key is also the default metadata scope. If the path has no
 `#scope=` selector, no root-level metadata, and no scope of that name, the
 scope is chosen among the archive's scopes: the one whose name or

{euler_eval-2.35.0 → euler_eval-2.36.0}/docs/output.md

@@ -74,7 +74,7 @@ set.
     "metricDescriptions": { "absrel": { "…": "…" } }
   },
   "protocol": {
-    "schemaVersion": "1.1",
+    "schemaVersion": "1.2",
     "parameters": { "reference.gt.dataset": "muses", "method.depth_alignment": "auto_affine", "…": "…" }
   },
   "dataset_info": { "num_pairs": 500, "gt_name": "GT", "pred_name": "model_a" },
@@ -171,6 +171,19 @@ sky removed, since that differs from model to model:
 depth) that would be scored without the flag, after `--mask-sky` if set;
 `excluded_gt_pixels` is how many of them fell in predicted sky.
 
+### GT sky masking coverage
+
+With `--mask-sky`, every modality it applies to records how many frames had
+their own GT segmentation:
+
+```json
+"gt_sky_masking": { "samples": 500, "samples_without_segmentation": 100, "samples_with_sky": 396 }
+```
+
+`samples_without_segmentation` is `0` unless `--missing-segmentation unmasked`
+kept frames without one; those frames were scored without sky masking.
+`samples_with_sky` counts the frames whose mask removed any pixel.
+
 When sky keys are written, the metric-set axes also declare them:
 `sky_segmentation` joins the `category` values, an optional `variant` axis
 (`explicit`, `implicit`) follows `category`, and `space` becomes optional
@@ -188,7 +201,7 @@ values differ say where. Nothing else in the file needs to be compared for that;
 
 ```json
 "protocol": {
-  "schemaVersion": "1.1",
+  "schemaVersion": "1.2",
   "parameters": {
     "reference.gt.modality": "sparse_depth",
     "reference.gt.dataset": "seeingthroughfog",
@@ -208,6 +221,7 @@ values differ say where. Nothing else in the file needs to be compared for that;
     "reference.segmentation.split": null,
     "population.samples": 943,
     "population.gt_sky_masking": false,
+    "population.missing_segmentation": null,
     "population.pred_sky_masking": false,
     "population.evaluated_resolution": [1024, 1920],
     "population.benchmark_depth_range_m": [0.01, 80.0],
@@ -231,12 +245,12 @@ What is recorded, per modality:
 
 | Modality | `reference` inputs besides `gt` | `population` | `method` |
 |---|---|---|---|
-| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask`, `spatial_alignment` |
+| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `missing_segmentation`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask`, `spatial_alignment` |
 | `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask` |
-| `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
+| `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `missing_segmentation`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
 | `rays` | `calibration` | `samples`, `evaluated_resolution` | `spatial_alignment` |
-| `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
-| `points3d` (sparse GT) | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment` |
+| `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `missing_segmentation`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
+| `points3d` (sparse GT) | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | `samples`, `gt_sky_masking`, `missing_segmentation`, `evaluated_resolution` | `points_3d_alignment` |
 
 Every modality also has `method.metrics_revision`.
 

{euler_eval-2.35.0 → euler_eval-2.36.0}/euler_eval/cli.py

@@ -29,6 +29,7 @@ from .config_paths import (
     resolve_gt_segmentation_path,
 )
 from .data import (
+    PartialInputDataset,
     build_depth_eval_dataset,
     build_points_3d_eval_dataset,
     build_points_3d_sparse_eval_dataset,
@@ -1478,7 +1479,12 @@ _PAIRED_OPTIONAL_INPUTS = {
 
 
 def _build_eval_dataset(
-    build, kwargs: dict, *, ds_name: str, iou_only: tuple[str, ...] = ()
+    build,
+    kwargs: dict,
+    *,
+    ds_name: str,
+    iou_only: tuple[str, ...] = (),
+    missing_segmentation: str = "drop",
 ) -> tuple:
     """Build an eval dataset and account for the frames its optional inputs drop.
 
@@ -1488,13 +1494,24 @@ def _build_eval_dataset(
     only for ``--sky-iou``, which must not change the evaluated pairs: if it
     lacks any, the dataset is rebuilt without it.
 
+    With *missing_segmentation* ``"unmasked"``, frames without their own GT
+    segmentation are kept instead: they are read without it, so ``--mask-sky``
+    leaves them unmasked and ``--sky-iou`` leaves them out of the IoU.
+
     Returns ``(dataset, skipped)`` with ``{input: reason}`` for every
     *iou_only* input that was left out.
     """
     optional = tuple(_PAIRED_OPTIONAL_INPUTS)
     dataset = build(**kwargs)
+    partial_segmentation = (
+        missing_segmentation == "unmasked"
+        and kwargs.get("segmentation_path") is not None
+        and bool(count_ids_dropped_by(dataset, "segmentation", optional=optional))
+    )
     skipped = {}
     for name in iou_only:
+        if name == "segmentation" and partial_segmentation:
+            continue
         dropped = count_ids_dropped_by(dataset, name, optional=optional)
         if not dropped:
             continue
@@ -1510,11 +1527,23 @@ def _build_eval_dataset(
         dataset = build(**kwargs)
     for name, (_, label) in _PAIRED_OPTIONAL_INPUTS.items():
         dropped = count_ids_dropped_by(dataset, name, optional=optional)
-        if dropped:
+        if not dropped:
+            continue
+        if name == "segmentation" and partial_segmentation:
+            print(
+                f"Note: the {label} of '{ds_name}' lacks {dropped} pairs that the "
+                "other inputs share; with --missing-segmentation unmasked they "
+                "are evaluated without GT sky masking and left out of the sky IoU."
+            )
+        else:
             print(
                 f"Warning: the {label} of '{ds_name}' lacks {dropped} pairs that "
                 "the other inputs share; those pairs are not evaluated."
             )
+    if partial_segmentation:
+        keys, _ = _PAIRED_OPTIONAL_INPUTS["segmentation"]
+        complete = build(**{**kwargs, **dict.fromkeys(keys)})
+        dataset = PartialInputDataset(complete, dataset)
     return dataset, skipped
 
 
@@ -1828,6 +1857,16 @@ def _build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="Mask sky regions from metrics using GT segmentation",
     )
+    parser.add_argument(
+        "--missing-segmentation",
+        choices=["drop", "unmasked"],
+        default="drop",
+        help=(
+            "Frames without their own GT segmentation: 'drop' leaves them out "
+            "(default); 'unmasked' evaluates them without GT sky masking and "
+            "leaves them out of the sky IoU"
+        ),
+    )
     parser.add_argument(
         "--mask-sky-prediction",
         action="store_true",
@@ -2172,6 +2211,7 @@ def _evaluate(
                 ),
                 ds_name=ds_name,
                 iou_only=iou_only_inputs,
+                missing_segmentation=args.missing_segmentation,
             )
             et_eval_datasets["depth"] = depth_dataset
 
@@ -2259,6 +2299,9 @@ def _evaluate(
                     population={
                         "samples": depth_dataset_info.get("num_pairs"),
                         "gt_sky_masking": bool(args.mask_sky),
+                        "missing_segmentation": (
+                            args.missing_segmentation if args.mask_sky else None
+                        ),
                         "pred_sky_masking": bool(args.mask_sky_prediction),
                         "evaluated_resolution": evaluated_resolution(depth_spatial),
                         "benchmark_depth_range_m": protocol_benchmark_range,
@@ -2302,6 +2345,7 @@ def _evaluate(
                         ),
                     },
                     "modality_params": depth_meta,
+                    "gt_sky_masking": depth_results.get("gt_sky_masking"),
                     "sky_segmentation": _sky_segmentation_meta(
                         depth_results, segmentation_key, skipped_inputs
                     ),
@@ -2438,6 +2482,7 @@ def _evaluate(
                 ),
                 ds_name=ds_name,
                 iou_only=iou_only_inputs,
+                missing_segmentation=args.missing_segmentation,
             )
             et_eval_datasets["sparse_depth"] = sparse_depth_dataset
 
@@ -2523,6 +2568,9 @@ def _evaluate(
                     population={
                         "samples": sparse_depth_dataset_info.get("num_pairs"),
                         "gt_sky_masking": bool(args.mask_sky),
+                        "missing_segmentation": (
+                            args.missing_segmentation if args.mask_sky else None
+                        ),
                         "pred_sky_masking": bool(args.mask_sky_prediction),
                         "evaluated_resolution": evaluated_resolution(
                             sparse_depth_spatial
@@ -2577,6 +2625,7 @@ def _evaluate(
                         ),
                     },
                     "modality_params": sparse_depth_meta,
+                    "gt_sky_masking": sparse_depth_results.get("gt_sky_masking"),
                     "sky_segmentation": _sky_segmentation_meta(
                         sparse_depth_results, segmentation_key, skipped_inputs
                     ),
@@ -2707,6 +2756,7 @@ def _evaluate(
                     segmentation_modality_key=segmentation_key or "segmentation",
                 ),
                 ds_name=ds_name,
+                missing_segmentation=args.missing_segmentation,
             )
             et_eval_datasets["rgb"] = rgb_dataset
 
@@ -2776,6 +2826,9 @@ def _evaluate(
                     population={
                         "samples": rgb_dataset_info.get("num_pairs"),
                         "gt_sky_masking": bool(args.mask_sky),
+                        "missing_segmentation": (
+                            args.missing_segmentation if args.mask_sky else None
+                        ),
                         "evaluated_resolution": evaluated_resolution(rgb_spatial),
                         "benchmark_depth_range_m": protocol_benchmark_range,
                     },
@@ -2807,6 +2860,7 @@ def _evaluate(
                         ),
                     },
                     "modality_params": rgb_meta,
+                    "gt_sky_masking": rgb_results.get("gt_sky_masking"),
                     "eval_params": {
                         "sky_masking": args.mask_sky,
                         "fid_backend": args.rgb_fid_backend,
@@ -3028,6 +3082,7 @@ def _evaluate(
                     segmentation_modality_key=segmentation_key or "segmentation",
                 ),
                 ds_name=ds_name,
+                missing_segmentation=args.missing_segmentation,
             )
             et_eval_datasets["points_3d"] = points_3d_dataset
 
@@ -3099,6 +3154,9 @@ def _evaluate(
                     population={
                         "samples": p3_dataset_info.get("num_pairs"),
                         "gt_sky_masking": bool(args.mask_sky),
+                        "missing_segmentation": (
+                            args.missing_segmentation if args.mask_sky else None
+                        ),
                         "evaluated_resolution": evaluated_resolution(p3_spatial),
                     },
                     method={
@@ -3130,6 +3188,7 @@ def _evaluate(
                         ),
                     },
                     "modality_params": points_3d_meta,
+                    "gt_sky_masking": points_3d_results.get("gt_sky_masking"),
                     "eval_params": {
                         "sky_masking": args.mask_sky,
                         "points_3d_alignment_mode": args.points_3d_alignment,
@@ -3223,6 +3282,7 @@ def _evaluate(
                     segmentation_split=segmentation_split,
                 ),
                 ds_name=ds_name,
+                missing_segmentation=args.missing_segmentation,
             )
             p3ds_meta = get_points_3d_metadata(points_3d_sparse_dataset)
             p3s_alignment_mode = args.points_3d_alignment
@@ -3298,6 +3358,9 @@ def _evaluate(
                     population={
                         "samples": p3s_dataset_info.get("num_pairs"),
                         "gt_sky_masking": bool(args.mask_sky),
+                        "missing_segmentation": (
+                            args.missing_segmentation if args.mask_sky else None
+                        ),
                         "evaluated_resolution": evaluated_resolution(p3s_spatial),
                     },
                     method={
@@ -3337,6 +3400,7 @@ def _evaluate(
                         ),
                     },
                     "modality_params": p3ds_meta,
+                    "gt_sky_masking": points_3d_sparse_results.get("gt_sky_masking"),
                     "eval_params": {
                         "sky_masking": args.mask_sky,
                         "alignment_mode": p3s_alignment_mode,

{euler_eval-2.35.0 → euler_eval-2.36.0}/euler_eval/data.py

@@ -1117,6 +1117,60 @@ def build_rgb_eval_dataset(
     )
 
 
+class PartialInputDataset:
+    """Every frame of *complete*, with an optional input where a frame has it.
+
+    euler-loading intersects every modality by ID, so an input such as GT
+    segmentation that covers only some frames removes the others from the
+    pairing. This dataset keeps them: *complete* is built without the input
+    and *with_input* with it, from the same other inputs, so its frames are a
+    subset of *complete*'s. A frame of *with_input* is read from it, any
+    other frame from *complete*, whose sample then has no entry for the
+    input. Frames keep *complete*'s order and identities.
+
+    Public attributes such as ``get_modality_metadata`` and ``modality_paths``
+    come from *with_input*, which describes every input.
+    """
+
+    def __init__(self, complete: Any, with_input: Any) -> None:
+        self._complete = complete
+        self._with_input = with_input
+        index = {
+            with_input.sample_full_id(j): j for j in range(len(with_input))
+        }
+        self._source = [
+            index.pop(complete.sample_full_id(i), None) for i in range(len(complete))
+        ]
+        if index:
+            raise ValueError(
+                f"{len(index)} frames with the optional input are not among the "
+                "frames without it; both datasets must share their other inputs"
+            )
+
+    @property
+    def frames_without_input(self) -> int:
+        return sum(source is None for source in self._source)
+
+    def __len__(self) -> int:
+        return len(self._source)
+
+    def __getitem__(self, index: int) -> dict[str, Any]:
+        source = self._source[index]
+        if source is None:
+            return self._complete[index]
+        return self._with_input[source]
+
+    def sample_full_id(self, index: int, modality_name: Optional[str] = None) -> str:
+        return self._complete.sample_full_id(index, modality_name)
+
+    def __getattr__(self, name: str) -> Any:
+        # Private and dunder lookups (pickling, ``_lookups``) are not forwarded:
+        # they would describe *with_input*'s frames, not this dataset's.
+        if name.startswith("_"):
+            raise AttributeError(name)
+        return getattr(self._with_input, name)
+
+
 def count_ids_dropped_by(
     dataset: Any, modality: str, *, optional: tuple[str, ...] = ()
 ) -> Optional[int]:

{euler_eval-2.35.0 → euler_eval-2.36.0}/euler_eval/evaluate.py

@@ -588,6 +588,14 @@ class _SkyMaskCoverage:
         elif not np.all(sky_valid):
             self.with_sky += 1
 
+    def info(self) -> dict:
+        """Return provenance for ``meta.gt_sky_masking`` in ``eval.json``."""
+        return {
+            "samples": self.samples,
+            "samples_without_segmentation": self.missing,
+            "samples_with_sky": self.with_sky,
+        }
+
     def warn_if_empty(self, modality: str) -> None:
         if self.missing:
             print(
@@ -2042,6 +2050,8 @@ def evaluate_depth_samples(
                 result["sky_segmentation_info"] = sky_iou_scores.info()
             if pred_sky_masking is not None:
                 result["pred_sky_masking"] = pred_sky_masking.info()
+            if sky_mask_enabled:
+                result["gt_sky_masking"] = sky_coverage.info()
             return _finalize_depth_distributions(
                 result, distribution_config, "depth", benchmark_depth_range, sky_depth
             )
@@ -2676,6 +2686,8 @@ def evaluate_sparse_depth_samples(
                 result["sky_segmentation_info"] = sky_iou_scores.info()
             if pred_sky_masking is not None:
                 result["pred_sky_masking"] = pred_sky_masking.info()
+            if sky_mask_enabled:
+                result["gt_sky_masking"] = sky_coverage.info()
             return _finalize_depth_distributions(
                 result, distribution_config, "sparse_depth",
                 benchmark_depth_range, sky_depth,
@@ -3246,6 +3258,7 @@ def evaluate_rgb_samples(
             return {
                 "rgb": rgb_results,
                 "rgb_benchmark": rgb_benchmark,
+                "gt_sky_masking": sky_coverage.info() if sky_mask_enabled else None,
                 "per_file_metrics": per_file_metrics,
                 "dataset_info": {
                     "num_pairs": num_samples,
@@ -4083,6 +4096,8 @@ def evaluate_points_3d_samples(
                     else None,
                 },
             }
+            if sky_mask_enabled:
+                result["gt_sky_masking"] = sky_coverage.info()
             return finalize_distributions(result, distribution_config, "rmse3d", "points_3d")
         finally:
             for st in stores.values():
@@ -4819,6 +4834,8 @@ def evaluate_points_3d_sparse_samples(
                     else None,
                 },
             }
+            if sky_mask_enabled:
+                result["gt_sky_masking"] = sky_coverage.info()
             return finalize_distributions(result, distribution_config, "rmse3d", "points_3d")
         finally:
             for st in stores.values():

{euler_eval-2.35.0 → euler_eval-2.36.0}/euler_eval/protocol.py

@@ -30,7 +30,7 @@ from .config_paths import (
 
 #: Version of the ``protocol`` block. Renaming or removing a key is a major
 #: bump; adding one is a minor bump.
-SCHEMA_VERSION = "1.1"
+SCHEMA_VERSION = "1.2"
 
 #: Bump whenever a release changes metric values for identical inputs and
 #: settings, and say so in the CHANGELOG. Evaluations with different revisions

{euler_eval-2.35.0 → euler_eval-2.36.0/euler_eval.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.35.0
+Version: 2.36.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT

{euler_eval-2.35.0 → euler_eval-2.36.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "euler-eval"
-version = "2.35.0"
+version = "2.36.0"
 description = "Evaluation toolkit for depth, RGB, ray and 3D point-map predictions"
 readme = "README.md"
 requires-python = ">=3.9"

{euler_eval-2.35.0 → euler_eval-2.36.0}/tests/test_mask_sky_pairing.py

@@ -195,3 +195,130 @@ def test_every_builder_pairs_segmentation_by_id(monkeypatch, build, kwargs):
     dataset = build(**kwargs, segmentation_path="/seg")
     assert dataset.modalities["segmentation"].loader == "sky"
     assert "segmentation" not in dataset.hierarchical_modalities
+
+
+# ---------------------------------------------------------------------------
+# --missing-segmentation unmasked
+# ---------------------------------------------------------------------------
+
+
+def test_missing_segmentation_unmasked_evaluates_those_frames_unmasked(
+    monkeypatch, capsys, flat_frames
+):
+    _patch_depth_metrics(monkeypatch)
+    config_path, paths = flat_frames(segmentation_frames=("f0", "f1"))
+
+    saved = _run(
+        monkeypatch, config_path, paths["pred"],
+        "--mask-sky", "--missing-segmentation", "unmasked",
+    )
+
+    assert saved["dataset_info"]["num_pairs"] == 3
+    # f0 and f1 are masked with their own sky; f2 keeps its sky row: 900 m
+    # error on 4 of 16 pixels.
+    assert _per_file_mae(saved) == {"f0": 0.0, "f1": 0.0, "f2": 225.0}
+    assert saved["meta"]["gt_sky_masking"] == {
+        "samples": 3, "samples_without_segmentation": 1, "samples_with_sky": 2,
+    }
+    parameters = saved["protocol"]["parameters"]
+    assert parameters["population.gt_sky_masking"] is True
+    assert parameters["population.missing_segmentation"] == "unmasked"
+    out = capsys.readouterr().out
+    assert (
+        "the GT segmentation of 'model' lacks 1 pairs that the other inputs "
+        "share; with --missing-segmentation unmasked they are evaluated without "
+        "GT sky masking"
+    ) in out
+    assert "found no GT segmentation of their own frame for 1 of the 3" in out
+
+
+def test_missing_segmentation_unmasked_matches_drop_on_covered_frames(
+    monkeypatch, flat_frames
+):
+    _patch_depth_metrics(monkeypatch)
+    config_path, paths = flat_frames(segmentation_frames=("f0", "f1"))
+
+    dropped = _run(monkeypatch, config_path, paths["pred"], "--mask-sky")
+    kept = _run(
+        monkeypatch, config_path, paths["pred"],
+        "--mask-sky", "--missing-segmentation", "unmasked",
+    )
+
+    covered = {k: v for k, v in _per_file_mae(kept).items() if k != "f2"}
+    assert covered == _per_file_mae(dropped)
+    assert dropped["protocol"]["parameters"]["population.missing_segmentation"] == "drop"
+
+
+def test_missing_segmentation_unmasked_scores_sky_iou_on_covered_frames(
+    monkeypatch, flat_frames
+):
+    _patch_depth_metrics(monkeypatch)
+    config_path, paths = flat_frames(segmentation_frames=("f0", "f1"))
+
+    saved = _run(
+        monkeypatch, config_path, paths["pred"],
+        "--sky-iou", "--sky-depth", "50", "--missing-segmentation", "unmasked",
+    )
+
+    assert saved["dataset_info"]["num_pairs"] == 3
+    meta = saved["meta"]["sky_segmentation"]
+    assert meta["implicit"]["available"] is True
+    assert meta["samples"] == 3
+    assert meta["samples_without_gt"] == 1
+    # Without --mask-sky, the policy is not a comparison parameter.
+    assert saved["protocol"]["parameters"]["population.missing_segmentation"] is None
+
+
+def test_missing_segmentation_unmasked_with_prefetch_workers(monkeypatch, flat_frames):
+    """Worker processes receive the combined dataset intact."""
+    _patch_depth_metrics(monkeypatch)
+    config_path, paths = flat_frames(segmentation_frames=("f0", "f1"))
+
+    saved = _run(
+        monkeypatch, config_path, paths["pred"],
+        "--mask-sky", "--missing-segmentation", "unmasked", "--num-workers", "2",
+    )
+
+    assert _per_file_mae(saved) == {"f0": 0.0, "f1": 0.0, "f2": 225.0}
+
+
+class _Frames:
+    def __init__(self, ids, tag):
+        self.ids, self.tag = list(ids), tag
+        self.public = f"{tag} attribute"
+
+    def __len__(self):
+        return len(self.ids)
+
+    def __getitem__(self, index):
+        return {"id": self.ids[index], "source": self.tag}
+
+    def sample_full_id(self, index, modality_name=None):
+        return f"/{self.ids[index]}"
+
+
+def test_partial_input_dataset_keeps_every_frame_in_order():
+    import pickle
+
+    dataset = data.PartialInputDataset(
+        _Frames(["a", "b", "c"], "complete"), _Frames(["a", "c"], "with")
+    )
+
+    assert len(dataset) == 3
+    assert [dataset[i] for i in range(3)] == [
+        {"id": "a", "source": "with"},
+        {"id": "b", "source": "complete"},
+        {"id": "c", "source": "with"},
+    ]
+    assert dataset.frames_without_input == 1
+    assert dataset.sample_full_id(1) == "/b"
+    assert dataset.public == "with attribute"
+    with pytest.raises(AttributeError):
+        dataset._lookups
+    restored = pickle.loads(pickle.dumps(dataset))
+    assert [restored[i]["source"] for i in range(3)] == ["with", "complete", "with"]
+
+
+def test_partial_input_dataset_rejects_frames_missing_from_the_complete_set():
+    with pytest.raises(ValueError, match="1 frames with the optional input"):
+        data.PartialInputDataset(_Frames(["a"], "complete"), _Frames(["a", "z"], "with"))

{euler_eval-2.35.0 → euler_eval-2.36.0}/tests/test_protocol.py

@@ -37,6 +37,7 @@ EXPECTED_KEYS = {
     "depth": _reference_keys("calibration", "segmentation") | {
         "population.samples",
         "population.gt_sky_masking",
+        "population.missing_segmentation",
         "population.pred_sky_masking",
         "population.evaluated_resolution",
         "population.benchmark_depth_range_m",
@@ -49,6 +50,7 @@ EXPECTED_KEYS = {
     "sparsedepth": _reference_keys(*SPARSE_INPUTS) | {
         "population.samples",
         "population.gt_sky_masking",
+        "population.missing_segmentation",
         "population.pred_sky_masking",
         "population.evaluated_resolution",
         "population.benchmark_depth_range_m",
@@ -60,6 +62,7 @@ EXPECTED_KEYS = {
     "rgb": _reference_keys("depth", "calibration", "segmentation") | {
         "population.samples",
         "population.gt_sky_masking",
+        "population.missing_segmentation",
         "population.evaluated_resolution",
         "population.benchmark_depth_range_m",
         "method.metrics_revision",
@@ -75,6 +78,7 @@ EXPECTED_KEYS = {
     "points3d": _reference_keys("intrinsics", "calibration", "segmentation") | {
         "population.samples",
         "population.gt_sky_masking",
+        "population.missing_segmentation",
         "population.evaluated_resolution",
         "method.metrics_revision",
         "method.points_3d_alignment",
@@ -83,6 +87,7 @@ EXPECTED_KEYS = {
     "points3d_sparse": _reference_keys(*SPARSE_INPUTS) | {
         "population.samples",
         "population.gt_sky_masking",
+        "population.missing_segmentation",
         "population.evaluated_resolution",
         "method.metrics_revision",
         "method.points_3d_alignment",
@@ -250,6 +255,7 @@ def test_depth_parameters(monkeypatch, tmp_path, depth_paths):
     assert parameters["reference.calibration.dataset"] is None
     assert parameters["population.samples"] == 2
     assert parameters["population.gt_sky_masking"] is False
+    assert parameters["population.missing_segmentation"] is None
     assert parameters["population.evaluated_resolution"] == [2, 2]
     assert parameters["population.benchmark_depth_range_m"] is None
     assert parameters["method.metrics_revision"] == METRICS_REVISION
@@ -267,7 +273,12 @@ def test_depth_parameters(monkeypatch, tmp_path, depth_paths):
         (("--depth-alignment", "affine"), {"method.depth_alignment"}),
         # The config has a sky_mask, which both settings then score with.
         (("--sky-depth", "50"), {"method.sky_depth_m", "method.pred_sky_mask"}),
-        (("--mask-sky",), {"population.gt_sky_masking"}),
+        (
+            ("--mask-sky",),
+            {"population.gt_sky_masking", "population.missing_segmentation"},
+        ),
+        # Without --mask-sky, what happens to unsegmented frames changes nothing.
+        (("--missing-segmentation", "unmasked"), set()),
         (
             ("--mask-sky-prediction",),
             {"population.pred_sky_masking", "method.pred_sky_mask"},