euler-eval 2.29.0__tar.gz → 2.31.0__tar.gz

{euler_eval-2.29.0/euler_eval.egg-info → euler_eval-2.31.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.29.0
+Version: 2.31.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT
@@ -33,7 +33,7 @@ Requires-Dist: opencv-python-headless>=4.5.0
 Requires-Dist: lpips>=0.1.4
 Requires-Dist: torchmetrics>=1.0
 Requires-Dist: tqdm>=4.62.0
-Requires-Dist: euler-loading>=2.15
+Requires-Dist: euler-loading>=2.29
 Requires-Dist: ds-crawler>=2.10
 Requires-Dist: euler-metric-naming>=0.2
 Provides-Extra: logging
@@ -176,7 +176,8 @@ metric is written under, is in [Metrics](docs/metrics.md).
 | **Metadata-driven decoding** | Radial vs planar depth, value ranges, point-cloud column layout and loader choice all come from the dataset index — no per-dataset flags. |
 | **Honest spaces** | Relative models are scored `native` *and* `metric` (after scale/shift or a similarity gauge), so a scale failure never reads as a geometry failure. |
 | **Benchmark depth bins** | `--benchmark-depth-range MIN MAX` adds near/mid/far bins in square-root depth space, additive to the regular metrics. |
-| **Sky masking** | `--mask-sky` drops sky pixels using GT segmentation — from the metrics *and* from the alignment fit. |
+| **Sky masking** | `--mask-sky` drops sky pixels using GT segmentation — from the metrics *and* from the alignment fit. `--mask-sky-prediction` does the same with each prediction's own `sky_mask`. |
+| **Sky IoU** | `--sky-iou` scores predicted sky — a `sky_mask`, or depth reaching a fixed cap — against GT segmentation, without changing any depth metric. |
 | **Per-file + aggregate** | Dataset-level numbers plus per-image metrics in dataset-hierarchy order, in the same file. |
 | **Sanity checks** | Results are validated against configurable thresholds; implausible ranges, degenerate inputs and scale mismatches are reported instead of quietly scored. |
 | **Structured metric names** | Each result carries a `metricSet` envelope declaring its namespace, axes, units and metric directions. |

{euler_eval-2.29.0 → euler_eval-2.31.0}/README.md

@@ -126,7 +126,8 @@ metric is written under, is in [Metrics](docs/metrics.md).
 | **Metadata-driven decoding** | Radial vs planar depth, value ranges, point-cloud column layout and loader choice all come from the dataset index — no per-dataset flags. |
 | **Honest spaces** | Relative models are scored `native` *and* `metric` (after scale/shift or a similarity gauge), so a scale failure never reads as a geometry failure. |
 | **Benchmark depth bins** | `--benchmark-depth-range MIN MAX` adds near/mid/far bins in square-root depth space, additive to the regular metrics. |
-| **Sky masking** | `--mask-sky` drops sky pixels using GT segmentation — from the metrics *and* from the alignment fit. |
+| **Sky masking** | `--mask-sky` drops sky pixels using GT segmentation — from the metrics *and* from the alignment fit. `--mask-sky-prediction` does the same with each prediction's own `sky_mask`. |
+| **Sky IoU** | `--sky-iou` scores predicted sky — a `sky_mask`, or depth reaching a fixed cap — against GT segmentation, without changing any depth metric. |
 | **Per-file + aggregate** | Dataset-level numbers plus per-image metrics in dataset-hierarchy order, in the same file. |
 | **Sanity checks** | Results are validated against configurable thresholds; implausible ranges, degenerate inputs and scale mismatches are reported instead of quietly scored. |
 | **Structured metric names** | Each result carries a `metricSet` envelope declaring its namespace, axes, units and metric directions. |

{euler_eval-2.29.0 → euler_eval-2.31.0}/docs/README.md

@@ -10,6 +10,12 @@
 | [Results & output](output.md) | Where files land, the anatomy of `eval.json`, per-file metrics, the sanity-check report |
 | [In-training validation](validation.md) | The programmatic API for scoring in-memory predictions during training |
 
+## Proposed features
+
+- [Unified evaluation framework](feature-request-unified-evaluation-framework.md):
+  a shared evaluation lifecycle with explicit modality-specific preparation,
+  metric dependencies, populations, aggregation and output compatibility.
+
 ## Concepts in one page
 
 **A dataset is an indexed root, not a folder of files.** Every path in the

{euler_eval-2.29.0 → euler_eval-2.31.0}/docs/alignment.md

@@ -32,7 +32,8 @@ map `d_aligned = s · d_pred + t` by least squares over valid pixels.
 
 The fit uses pixels where GT and prediction are finite and GT > 0. With
 `--mask-sky`, sky pixels are excluded and the fit is trimmed to `gt ≤ P95(gt)`
-so residual sky outliers do not dominate.
+so residual sky outliers do not dominate. Pixels in a prediction's `sky_mask`
+are excluded from the fit with `--sky-depth` or `--mask-sky-prediction`.
 
 Declaring the prediction with the `relative_depth` or `affine_depth` config key
 (instead of `depth`) tells the evaluator the model is not metric, so

{euler_eval-2.29.0 → euler_eval-2.31.0}/docs/cli.md

@@ -23,7 +23,9 @@ validation exits with status `1` before any dataset is opened.
 | `--skip-rays` | flag | off | Skip rays (spherical direction map) evaluation |
 | `--skip-points-3d` | flag | off | Skip points_3d (per-pixel 3D point map) evaluation |
 | `--mask-sky` | flag | off | Mask sky regions from metrics using GT segmentation |
+| `--mask-sky-prediction` | flag | off | Exclude pixels in a prediction's `datasets[].sky_mask` from dense and sparse depth metrics and the alignment fit; predictions without a `sky_mask` are scored in full ([details](configuration.md#masking-predicted-sky)) |
 | `--sky-depth` | `float` | none | Cap GT and predicted depths in meters and fill predicted `sky_mask` pixels with this value; must be finite and positive |
+| `--sky-iou` | flag | off | Score predicted sky against GT segmentation sky: IoU, precision and recall for the `explicit` (`datasets[].sky_mask`) and `implicit` (`depth >= --sky-depth`, metric predictions only) variants. Requires `gt.segmentation`; depth metrics are unchanged ([details](metrics.md#sky-segmentation)) |
 | `--no-sanity-check` | flag | off | Disable sanity checking of metric configurations |
 | `--metrics-config` | `str` | auto-detect | Path to `metrics_config.json` for sanity checking |
 | `--depth-alignment` | `{none,auto_affine,affine}` | `auto_affine` | Depth calibration mode ([details](alignment.md#depth-alignment)) |
@@ -54,9 +56,16 @@ euler-eval config.json --batch-size 32
 # Sky masking (requires gt.segmentation)
 euler-eval config.json --mask-sky -v
 
+# Exclude each prediction's own sky (datasets[].sky_mask) from depth metrics
+euler-eval config.json --mask-sky-prediction
+
 # Cap depth at 100 m and fill predicted sky at 100 m (optional datasets[].sky_mask)
 euler-eval config.json --sky-depth 100
 
+# Sky IoU: explicit from datasets[].sky_mask, implicit at depth >= 100 m
+# (requires gt.segmentation; the threshold is also the depth cap)
+euler-eval config.json --sky-iou --sky-depth 100
+
 # Only depth evaluation
 euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
 
@@ -109,6 +118,13 @@ per prediction dataset and modality with the GT and prediction paths, the
 resolved modality metadata (such as `radial_depth`), and the number of matched
 pairs.
 
+With `--sky-iou`, each depth block ends with one line holding the pooled
+(`pixel_pool`) sky IoU, precision and recall of every computed variant, for
+example `Sky IoU, pixel_pool, 'model': explicit 0.9120 (precision 0.9500,
+recall 0.9580); implicit (depth >= 100 m) 0.4410 (precision 0.9800, recall
+0.4450)`. Skipped variants, GT segmentation without sky and an empty implicit
+sky are reported as warnings above it.
+
 A matched-pair count that is much lower than expected is the first sign that GT
 and predictions are not indexed consistently — see
 [Loader resolution & dataset metadata](configuration.md#loader-resolution--dataset-metadata).

{euler_eval-2.29.0 → euler_eval-2.31.0}/docs/configuration.md

@@ -71,7 +71,8 @@ required.
 | `gt.sparse_depth.path` | no\* | Sparse pointcloud GT, `(N,C)` points whose first columns are `x,y,z` in metres |
 | `gt.rays.path` | no\* | GT ray direction map dataset |
 | `gt.points_3d.path` | no\* | GT per-pixel 3D point map `(H,W,3)` in camera-frame metres |
-| `gt.segmentation.path` | no | GT segmentation (needed for `--mask-sky`) |
+| `gt.segmentation.path` | no | GT segmentation (needed for `--mask-sky` and `--sky-iou`); see [GT segmentation](#gt-segmentation) |
+| `gt.semantic_segmentation.path` | no | Alias of `gt.segmentation`; configure only one of the two |
 | `gt.calibration.path` | no | Calibration data (camera intrinsics); enables true camera-frame [surface normals](metrics.md#surface-normals-and-calibration) |
 | `gt.intrinsics.path` | with `gt.sparse_depth` | Camera intrinsics for pointcloud projection |
 | `gt.camera_extrinsics.path` | with `gt.sparse_depth` | Source-to-camera extrinsics (e.g. MUSES `lidar2rgb`) |
@@ -80,6 +81,35 @@ required.
 
 \* At least one of the five modality entries must be present.
 
+### GT segmentation
+
+`gt.segmentation` is the canonical key. `gt.semantic_segmentation` is accepted
+as an alias, for archives whose segmentation scope carries that name (MUSES,
+for example); configuring both keys is an error. The segmentation is read only
+when `--mask-sky` or `--sky-iou` needs it. `--mask-sky` without it warns and
+disables itself, while `--sky-iou` without it exits with an error.
+
+The dataset head's loader module must expose a `sky_mask` function, and the
+head must declare the sky label as `skyclass` (euler-loading 2.29 or later).
+euler-eval contains no label IDs.
+
+Segmentation is per frame, so it is paired with the evaluated frames by
+hierarchy path and file ID, like depth: its index must use the same folder
+structure and IDs as the frames it masks. A frame without its own segmentation
+file is not evaluated, and the run prints how many pairs the segmentation
+dropped. When segmentation is read only for `--sky-iou`, it must not change
+the evaluated pairs, so a segmentation that lacks frames is not loaded; the run
+warns and the sky IoU is skipped. A segmentation whose IDs match no frame
+stops the run with euler-loading's "No common IDs" error and per-modality file
+counts.
+
+The config key is also the default metadata scope. If the path has no
+`#scope=` selector, no root-level metadata, and no scope of that name, the
+scope is chosen among the archive's scopes: the one whose name or
+`dataset-head.json` modality key is `segmentation` or `semantic_segmentation`
+(both names are candidates under either key), or the only scope. If several
+scopes match, or none among several, the run stops and asks for `#scope=`.
+
 ## `datasets` section
 
 Each entry needs a `name` and at least one prediction modality. Use only **one**
@@ -94,7 +124,7 @@ would be ambiguous.
 | `depth.path` | no\* | Predicted dense metric depth; also used against sparse pointcloud GT |
 | `relative_depth.path` | no\* | Predicted dense relative depth (scale/shift alignment supported) |
 | `affine_depth.path` | no\* | Predicted dense affine depth (scale/shift alignment supported) |
-| `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth` |
+| `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth`, `--mask-sky-prediction` and `--sky-iou` |
 | `rays.path` | no\* | Predicted ray direction map dataset |
 | `points_3d.path` | no\* | Predicted per-pixel 3D point map `(H,W,3)` |
 | `output_file` | no | Custom results path (default: `eval.json` in the first modality path) |
@@ -123,8 +153,8 @@ The sky mask is paired with GT and predicted depth by sample ID and hierarchy.
 It supports the same archive, split, and `#scope=` selectors as depth, defaulting
 to the `sky_mask` metadata scope. Its loader must return a boolean mask with
 **True meaning sky**. For example, `generic_dense_depth.sky_mask` uses the RGB
-sky color in the mask dataset's `meta.sky_mask` (`[R, G, B]`) and returns a
-`(1, H, W)` tensor. Masks at another resolution follow the depth grid's crop
+sky color in the mask dataset's `meta.skyclass` (`[R, G, B]`; the older
+`meta.sky_mask` key is deprecated) and returns a `(1, H, W)` tensor. Masks at another resolution follow the depth grid's crop
 rule or use nearest-neighbor resizing.
 
 `--sky-depth` applies to dense and sparse depth evaluation. GT and both native
@@ -139,9 +169,37 @@ contributes only at projected points. Predicted sky stays in the scores where
 GT is valid; `--mask-sky` separately excludes pixels using GT segmentation and
 can be combined with `--sky-depth`.
 
-Without `--sky-depth`, the predicted mask is not loaded and depth evaluation
-uses the usual uncapped values. The effective cap and mask source are recorded
-in `eval.json` under `meta.eval_params.sky_depth` and `meta.pred.sky_mask`.
+Without `--sky-depth`, `--mask-sky-prediction` or `--sky-iou`, the predicted
+mask is not loaded and depth evaluation uses the usual uncapped values.
+`--sky-iou` loads the mask
+even without `--sky-depth`, but only to score it against GT sky
+([Sky segmentation](metrics.md#sky-segmentation)); the depth fill and the
+affine-fit exclusion stay tied to `--sky-depth` (and `--mask-sky-prediction`),
+so depth values do not change.
+Because the mask is paired by sample ID, a mask that lacks some depth frames
+also drops them from the depth metrics; the run prints how many. When the mask
+is loaded only for `--sky-iou`, such a mask is not loaded at all: the run
+warns, keeps every depth pair and skips the explicit variant. The effective cap and mask
+source are recorded in `eval.json` under `meta.eval_params.sky_depth` and
+`meta.pred.sky_mask`.
+
+### Masking predicted sky
+
+`--mask-sky-prediction` excludes the pixels of a prediction's `sky_mask` from
+dense and sparse depth evaluation, as `--mask-sky` does for GT sky: from every
+metric, benchmark bin and histogram, and from the scale/shift fit. For sparse
+depth, lidar points that project into predicted sky are not scored. A dataset
+without a `sky_mask` is scored in full and the run warns. The flag loads the
+mask without `--sky-depth`, combines with `--mask-sky` (the union of both skies
+is excluded) and with `--sky-depth` (the cap still applies to the remaining
+pixels). It does not change `--sky-iou`, which scores the masks themselves.
+
+The excluded pixels come from the evaluated model, so two models are scored on
+different pixel sets, and a model can lower its error by calling hard pixels
+sky. Each result records how much it excluded under `meta.pred_sky_masking`
+([Results & output](output.md#predicted-sky-masking)); compare models with
+this flag only alongside that fraction, or prefer `--mask-sky`, whose GT sky is
+the same for every model.
 
 ## Sparse depth (pointcloud GT)
 
@@ -162,8 +220,9 @@ depth and `points_3d`, each prediction modality gets its own `eval.json` at its
 own path.
 
 Sparse depth does not require segmentation GT. `gt.segmentation` is loaded only
-when `--mask-sky` is set, and then excludes sky pixels from projected-point
-metrics and from scale/shift fitting.
+when `--mask-sky` or `--sky-iou` is set. `--mask-sky` excludes sky pixels from
+projected-point metrics and from scale/shift fitting; `--sky-iou` scores the
+dense prediction's sky against it without changing any depth metric.
 
 ```json
 {
@@ -239,11 +298,15 @@ The scope defaults to the configuration key name, so the common case is compact:
 
 Use `#scope=<scope>` when an artifact scope is not named exactly like the config
 key, e.g. `"sparse_depth": { "path": "/data/capture.zip:test#scope=lidar" }`.
-
-Regular modalities (`rgb`, `depth`, `rays`, `points_3d`, `sparse_depth`)
-participate in file-ID intersection; calibration and pose data (`intrinsics`,
-`calibration`, `camera_extrinsics`, `lidar_extrinsics`, segmentation) are loaded
-hierarchically and do not.
+GT segmentation is the exception: under either of its keys it also finds a
+scope named after the other key, as described in [GT segmentation](#gt-segmentation).
+
+Regular modalities (`rgb`, `depth`, `rays`, `points_3d`, `sparse_depth`, GT
+segmentation and a predicted `sky_mask`) participate in file-ID intersection;
+calibration and pose data (`intrinsics`, `calibration`, `camera_extrinsics`,
+`lidar_extrinsics`) are loaded hierarchically and do not. A hierarchical file
+applies to every frame at or below its folder, which suits one calibration per
+scene but not per-frame data.
 
 ## `metrics_config.json`
 

euler_eval-2.31.0/docs/feature-request-unified-evaluation-framework.md

@@ -0,0 +1,381 @@
+# Feature request: a unified evaluation framework
+
+Status: proposed for a future release. This document specifies an architectural
+change; it does not introduce a new API or change evaluation behavior.
+
+## Recommendation
+
+Use a **modality-agnostic evaluation lifecycle with modality-aware preparation,
+observations and metric definitions**. Represent each evaluation as an explicit
+plan of typed steps and dependencies. The same plan should drive computation,
+aggregation, explanations and output metadata.
+
+Depth, RGB, rays and point clouds can share this lifecycle without sharing one
+preprocessing function, validity mask or mathematical reduction. A metric may
+consume several modalities, a whole image, two point sets, a group of frames or
+dataset-wide features. Those requirements belong in its definition.
+
+Start with ordinary Python data classes, protocols and a small dependency
+executor. Keep the current CLI and library entry points as compatibility
+wrappers while moving their computation into the shared framework.
+
+## Problem and motivation
+
+Today, adding a core metric requires wiring computation and storage in
+[`evaluate.py`](../euler_eval/evaluate.py), descriptions and paths in
+[`cli.py`](../euler_eval/cli.py), and sometimes separate per-file and dataset
+serialization logic. Deferred CPU/GPU batch results also patch mutable output
+dictionaries. The distribution contract is attached through a later tree walk
+in [`distribution_contract.py`](../euler_eval/distribution_contract.py).
+
+This makes several questions difficult to answer from one place:
+
+- Which transformed prediction and GT does a metric consume?
+- Which observations are included, and when is that selection applied?
+- Is a dataset result a mean of image scores, a pooled observation statistic,
+  or a computation over the dataset as a whole?
+- Do a scalar and its associated distribution describe the same population?
+- When is a deferred result complete, and which accumulators receive it?
+
+The recent depth benchmark-range fix is a concrete example: the default
+distribution needed its selection explicitly connected to the benchmark
+population after sky-depth processing. Shared population definitions should
+make that relationship part of the plan.
+
+As a metric author, I should be able to declare inputs, computation, supported
+reductions and result semantics once. As a user, I should be able to inspect
+how any emitted result was produced. As a consumer, euler-view should receive
+stable output with enough provenance to distinguish incompatible evaluations.
+
+## Unified flow
+
+```mermaid
+flowchart TD
+    C[Configuration and selected metric sets] --> P[Build and validate evaluation plan]
+    M[Input metadata and available capabilities] --> P
+    P --> S[Resolve required inputs and sample or group identities]
+    S --> A[Prepare typed artifacts through declared adapter steps]
+    A --> V[Resolve views, populations and correspondences]
+    V --> K[Compute declared observations, scores or features]
+    K --> I[Completed item contributions]
+    I --> R[Reducers or retained dataset inputs]
+    R --> F[Finalize dataset or group computations]
+    K --> N[Optional dependent stages]
+    F --> N
+    N --> O[Typed result collection]
+    I --> O
+    F --> O
+    O --> E[Compatibility output adapter]
+    P -.-> X[Plan explanation and execution provenance]
+    O -.-> X
+```
+
+This is a lifecycle overview. An individual plan is a directed acyclic graph:
+preparation may itself require correspondence construction before alignment,
+and later metrics may consume artifacts or results from earlier stages. The
+diagram does not mandate that every metric executes every box.
+
+For example, RGB plus GT depth can produce depth-binned photometric scores;
+depth plus calibration can produce a point map; a future reconstruction stage
+can consume several frames and produce a cloud for geometric evaluation.
+Each dependency must be explicit, including its coordinate system and scope.
+
+## Responsibilities and contracts
+
+The initial implementation should formalize these concepts. Names below are
+design suggestions, not a promised public Python API.
+
+| Concept | Required responsibility |
+|---|---|
+| `EvaluationPlan` and step definitions | Requested outputs, typed dependencies, execution scope, effective policies and implementation versions. Validate unsupported combinations and cycles before expensive work. |
+| Artifact specification and reference | Payload kind, GT/prediction role, source identity, units, coordinate frame, spatial layout, transformation lineage and ownership. A reference may point to memory or managed storage. |
+| View and population definitions | Evaluated representation, eligibility predicates, selection order, correspondence policy and observation unit. Record distinct source and target populations where needed. |
+| `MetricSpec` | Required artifacts or observations, computation, validity requirements, parameters, supported scopes/reducers and semantic descriptions. Metric sets select these definitions. |
+| `ReducerSpec` | Accepted contribution type, state, update/finalize operations, merge capability, weighting, precision and retention requirements. |
+| Contribution and result records | Stable request/item identity, payload, status, counts and provenance references. Results are completed values; pending work belongs to execution state. |
+
+The shared core owns planning, dependency resolution, resource lifetime,
+batch completion, aggregation and result collection. It dispatches by declared
+capabilities and contracts. Modality-specific adapters and metric kernels own
+geometry, image preprocessing, feature extraction and domain-specific math.
+
+Metric definitions should supply descriptions, units and allowed reductions to
+the exporter. Adding a metric using existing capabilities should not require
+new modality conditionals in the executor or manual CLI output-tree assembly.
+
+### Inputs and execution scope
+
+Treat the required inputs as named roles, rather than assuming every operation
+takes a `(prediction, GT)` array pair. Support declarations for:
+
+- One item: paired images, predicted image alone, ray maps, or two point sets.
+- A group: a scene, sequence, temporal window or another explicitly identified
+  collection, including any ordering requirement.
+- A dataset: feature collections, accumulated statistics or other retained
+  inputs needed after the item stream completes.
+
+An execution batch is a performance choice, not a statistical grouping. A
+batch size must not silently define a sample, population or dataset reducer.
+
+Each request identifies its matching/join policy and required modalities. A
+missing optional modality should affect the requests that need it; it should
+not silently shrink unrelated evaluation populations. Preserve existing
+matching behavior during migration, and make any change to that behavior an
+explicit, separately tested semantic change.
+
+NIQE/FADE kernels require predictions only, although today's RGB evaluation
+workflow pairs GT and predictions. Express that kernel requirement now;
+enabling a standalone prediction-only CLI workflow can be a separate feature.
+
+### Artifacts, views and populations
+
+An artifact must carry enough information to reject invalid combinations:
+metres versus unspecified relative values, incompatible coordinate frames,
+unaligned image grids, or sparse samples supplied to dense-neighbourhood
+metrics. Calibration assumptions and inferred input spaces must be recorded.
+
+A population is a named selection over a declared observation domain. It need
+not be a dense Boolean mask: it may be projected correspondences, valid
+directions, selected images, or a directed set of nearest-neighbour queries.
+Two populations with the same display name are not automatically compatible.
+
+Separate fitting and scoring populations. For depth, the current intended
+order must remain explicit:
+
+1. Spatial preparation, calibration handling and planar/radial conversion.
+2. Alignment fitting on its declared fitting population, where requested.
+3. GT/prediction sky-depth capping and predicted-sky filling under the existing
+   policy, with GT sky exclusion recorded separately.
+4. Scoring validity and benchmark membership on evaluated GT. The benchmark
+   `all` interval includes both endpoints.
+5. Scalar and distribution computations for the selected space/population.
+
+An unrestricted scalar and a benchmark-filtered histogram may intentionally
+coexist, as they currently do. Their internal identities must expose that
+difference. Scalar and distribution requests for the *same* population must
+consume the same selected observations.
+
+Additional requirements such as positive inputs for logarithms or a valid
+neighbourhood for normals belong to the relevant computation. Explicitly name
+the resulting population or support policy. Likewise, RGB sky filling and
+RGB pixel exclusion are different operations; the framework must preserve
+that distinction and the current unfilled inputs used for NIQE/FADE.
+
+## Mapping current modalities into the framework
+
+| Evaluation family | Preparation and dependencies | Computation and scope |
+|---|---|---|
+| Dense depth | Spatial alignment, intrinsics, radial/planar conversion, optional affine fitting, sky policy and benchmark populations | Corresponding-pixel errors and standard reducers; dense geometry; image scores; dataset feature metrics |
+| Sparse depth | Point-cloud transforms, projection, visibility/collision policy, depth preparation and alignment | Errors at evaluated projected correspondences; dense-neighbourhood metrics remain unavailable |
+| RGB | Image layout/range handling, spatial alignment and metric-specific sky policy | Pixel errors, local/image scores, prediction-only quality kernels and dataset FID |
+| RGB with depth | RGB inputs plus explicitly registered GT depth and calibration | Photometric observations selected by GT depth; demonstrates an existing metric with multiple modalities |
+| Rays | Direction preparation, spatial alignment, validity and resolved FoV policy | Angular observations, thresholds and accuracy curves; declared exact/approximate summaries |
+| Dense points-3D | Point maps or unprojected depth, coordinate-frame handling and optional gauge fitting | Correspondence errors, radial/lateral decomposition, dense geometry and separate cloud comparisons |
+| Sparse points-3D | Sparse cloud visibility and projection plus predicted points or the existing programmatic depth conversion | Correspondence errors and cloud queries with explicitly declared directions |
+
+Preserve current entry-point eligibility: representing an existing library
+conversion in the plan does not automatically enable an additional CLI result
+family. For point-map metrics, also preserve computations intentionally made in
+the native frame even when aligned results are available.
+
+Correspondence error and cloud distance require different observations.
+Nearest-neighbour cloud queries must declare their source, target, direction
+and subsampling policy. Current sparse fixed-threshold diagnostics are
+GT-to-prediction completeness/recall, while optional F-score AUC also needs the
+reverse direction. Keep those requirements separate. Do not pool points from
+unrelated frames before nearest-neighbour matching; a scene-level comparison
+would require an explicit scene artifact and common coordinate frame.
+
+The ray evaluator already uses a histogram approximation for its pooled
+median. Migrating it must preserve and disclose that approximation instead of
+implicitly promising an exact quantile.
+
+## Observations and aggregation
+
+Share computations only when their full input, view, population and numerical
+policies match. Reuse references or chunks where practical; the framework
+must not require retaining all pixel errors in memory.
+
+A scalar, histogram and curve are different outputs derived from declared
+observations. A depth histogram associated with `rmse` bins absolute residuals;
+the points-3D counterpart bins Euclidean correspondence errors. Neither is a
+distribution of repeated estimates of dataset RMSE.
+
+| Requested result | Contribution/state | Final computation |
+|---|---|---|
+| Pixel-pool depth RMSE | Squared-error sum and valid count | `sqrt(sum_sq / count)` |
+| Mean per-image RMSE | Valid image RMSE sum and image count | Divide; preserve equal image weighting |
+| Error-magnitude histogram | Fixed-bin counts | Elementwise sum |
+| Error sum by GT-depth bin | Fixed-bin error sums | Elementwise sum |
+| RMSE by GT-depth bin | Squared-error sums and counts for matching bins | Square root of each bin's mean squared error |
+| Exact quantile | Retained values or another exact supported state | Quantile over the declared population |
+| LPIPS dataset mean | Completed per-image scores and validity | Declared image-score reducer |
+| FID | Compatible GT/prediction feature collections, or sufficient feature moments if supported by that backend | One dataset-level calculation after ingestion |
+| KID | Feature collections plus declared subset selection/seed policy | Dataset-level estimator with its required retained state |
+
+All reducers have `update` and `finalize`; only reducers declaring a compatible
+merge operation may combine partial states. State may be sufficient statistics,
+retained values or ordered/global inputs. Do not force all reducers into sums
+or average finalized shard scores. In particular, per-shard FID values cannot
+be averaged to obtain dataset FID. Backend-specific feature preprocessing,
+model weights and covariance conventions are part of FID compatibility.
+
+Define merge compatibility from the complete resolved semantics: observation,
+units, view, population policy, reducer, bin edges/intervals, implementation
+and estimator settings. Registry IDs and matching output paths alone are
+insufficient. Check partition identity separately so overlapping inputs are
+not double counted. Integer counts must remain exact; floating accumulation
+and backend equivalence require declared tolerances.
+
+For matching populations and fixed bins, dataset histogram counts must equal
+the sum of item contributions. Benchmark subsets and canonical aliases must
+not be accumulated again as additional observations. Exact RMSE continues to
+use sufficient statistics, independently of histogram resolution.
+
+## Execution and later stages
+
+Compile selected requests into explicit dependencies. Resolve metadata-based
+choices early; leave data-dependent choices marked unresolved until the
+required input is available, then record the decision and its policy. Reject
+missing required capabilities and incompatible artifacts with useful messages.
+
+CPU kernels, GPU batches and dataset computations should emit typed completed
+contributions into the same coordinator. Completion is tracked per logical
+request and item/group; each contribution reaches its reducer once. A failed
+or retried batch must not duplicate counts. Exported dictionaries must not be
+used as pending-result storage.
+
+Define `ok`, `no_observations`, `skipped`, `unavailable` and `failed` outcomes
+with reasons and relevant counts. Empty or failed measurements must not enter
+means as zero. The plan declares whether a failure stops the run or leaves a
+result unavailable, preserving existing failure behavior during migration.
+
+Use bounded queues and explicit artifact ownership. Release intermediates
+after their consumers finish; declare temporary-file/feature retention and
+clean it up on completion or failure. Expensive optional metrics should only
+load models and compute features when selected. Shared preparation should not
+force an entire dataset or all modalities into memory.
+
+Allow a later stage to consume an earlier artifact or typed result through a
+declared dependency. Examples include frame fusion followed by cloud scoring,
+temporal metrics over ordered frame groups, and summaries derived from
+completed dataset results. Specify group membership, ordering, frame transforms
+and missing-input behavior. These examples are extension requirements, not
+requests to implement reconstruction or temporal metrics in this refactor.
+
+Initially, a local dependency executor is sufficient. A distributed scheduler,
+plugin marketplace and user-authored workflow language are outside this
+feature's scope. Merge contracts prepare suitable computations for later
+parallel execution without making distribution a prerequisite.
+
+## Transparency and output compatibility
+
+Provide a proposed `--explain` mode and a programmatic plan description. They
+should identify inputs, transforms, fitting/scoring populations, computation
+dependencies, scope, reducer, approximation and requested output paths.
+An explanation that does not evaluate data must mark inferred settings as
+pending. Runtime provenance records resolved choices and contribution counts;
+it should be possible to follow an output back to its plan definition without
+dumping arrays or masks.
+
+Use a structured internal result identity containing evaluation family, view,
+population, measurement, reducer, scope/group and output kind, referencing the
+resolved semantic definitions. A path is an export representation of that
+identity. Canonical aliases reference existing results and perform no work.
+
+Keep the current `eval.json` envelope, per-modality files, metric paths, aliases,
+library return shapes, and distribution v1 leaves/registry during migration.
+One serialization layer should translate typed results to those contracts.
+Generate descriptions from metric definitions rather than rediscovering
+semantics through dictionary names. Keep transport validation, including
+JavaScript-safe integer counts and nonfinite-value handling, at that boundary.
+
+Preserve existing `distributionPopulation` metadata. A later, separately
+versioned provenance extension can describe per-result views and populations
+for all output kinds, including scalar results. Distinguish evaluation
+semantics versions from wire-schema versions: an internal refactor must not
+silently change metric meaning, and a new meaning does not justify pretending
+older results used the new policy.
+
+euler-view should be able to ingest compatibility output without changes.
+Future provenance can support richer explanations and comparison checks, but
+this feature request contains no euler-view implementation work.
+
+## Delivery plan
+
+1. **Characterize existing behavior.** Inventory current metric definitions,
+   requirements, populations, reduction rules, aliases and exceptions. Capture
+   representative existing outputs and numerical behavior, including known
+   policy differences between CLI and training validation.
+2. **Introduce the contracts through depth.** Extract prepared observations,
+   explicit populations and standard/distribution reducers. Reuse the
+   sufficient-statistic foundation in
+   [`depth_standard.py`](../euler_eval/metrics/depth_standard.py) and the typed
+   sample/aggregator approach in [`validation.py`](../euler_eval/validation.py).
+   Move dense and sparse depth plus training validation onto the same core
+   where semantics match. Keep heavyweight optional metrics outside the
+   lightweight validation API.
+3. **Validate the design against different computations.** Migrate RGB image
+   and prediction-only kernels, LPIPS completion, dataset FID/KID paths where
+   supported, and RGB-with-depth metrics. This milestone must demonstrate
+   image, dataset and multiple-modality dependencies before treating the
+   abstraction as stable.
+4. **Migrate rays and points.** Preserve frame choices, directional cloud
+   comparisons, sparse eligibility and approximation policies. Share declared
+   preparation where compatible and retain specialized kernels.
+5. **Complete explanations and remove duplicate wiring.** Generate result
+   descriptions and compatibility output from definitions. Document how to
+   add a metric, adapter and reducer. Stabilize extension contracts only after
+   the current modalities exercise them.
+
+Each milestone must leave the existing entry points usable. Numerical or
+selection-policy corrections discovered during migration should be reviewed,
+tested and release-noted separately from structural changes.
+
+## Acceptance criteria
+
+- All current evaluation families use the same planning, contribution and
+  result lifecycle while preserving modality-specific preparation.
+- A metric using existing artifact and reducer types can be added through its
+  kernel and declaration without editing the executor or CLI tree-building
+  logic. Its descriptions and supported output forms come from that definition.
+- Plans cover a pixel metric, an image metric, a prediction-only kernel, a
+  dataset feature metric and a metric requiring RGB plus depth. A small test
+  stage can consume an upstream artifact/result without adding a scheduler
+  branch for that modality.
+- Depth parity cases cover dense/sparse inputs, native/aligned spaces,
+  planar/radial conversion, benchmark boundaries, sky capping/filling,
+  excluded/invalid inputs and empty populations. Fitting and scoring
+  populations are independently inspectable.
+- Scalars and distributions requesting the same population agree on selected
+  observations. Fixed-bin dataset counts equal summed item counts, and
+  aliases/subset summaries do not double count.
+- Reducer tests distinguish pooled RMSE from mean image RMSE, verify compatible
+  partition merging, reject incompatible state merges and preserve exact or
+  approximate quantile policies. Dataset metrics are finalized at their
+  declared scope instead of averaging arbitrary batch/shard scores.
+- Cloud tests preserve coordinate frames, correspondence/nearest-neighbour
+  distinctions, sparse directions and subsampling. RGB tests preserve filled
+  versus unfilled inputs and dependencies on optional depth.
+- CPU and batched implementations agree within metric-specific tolerances;
+  integer histograms agree exactly. Completion-order tests detect lost or
+  duplicated contributions, including failures and empty results. Stochastic
+  estimator checks use declared seed/subset policies.
+- Public CLI/library compatibility fixtures retain result paths, aliases,
+  values within the existing tolerances, metadata and distribution v1 schema
+  validity. Changes in eligibility, weights or semantics are explicit changes.
+- Explanation output identifies effective policies and unresolved choices;
+  result provenance includes resolved dependencies, counts and outcome reasons.
+- Streaming checks demonstrate bounded intermediate memory for streaming
+  reducers and cleanup of retained resources. Existing tests remain usable
+  without datasets, network access or a GPU; backend stubs can exercise
+  scheduling and feature contracts.
+
+## Design boundary
+
+The framework should make the computation inspectable and extensible through
+shared contracts. It should preserve specialized mathematical kernels and
+make their differences visible. Its success is measured by fewer independent
+places to wire a metric, demonstrable scalar/distribution population agreement,
+and the ability to express the current non-pixel and dataset computations
+within the same lifecycle.