euler-eval 2.28.0__tar.gz → 2.30.0__tar.gz

euler_eval-2.30.0/MANIFEST.in

@@ -0,0 +1,3 @@
+recursive-include docs *.md
+recursive-include tests *.py *.json *.md
+include scripts/generate_distribution_fixtures.py

{euler_eval-2.28.0 → euler_eval-2.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.28.0
+Version: 2.30.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT
@@ -42,6 +42,7 @@ Provides-Extra: fid
 Requires-Dist: clean-fid>=0.1.35; extra == "fid"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: jsonschema>=4.18; extra == "dev"
 Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
 Requires-Dist: black>=22.0.0; extra == "dev"
 Requires-Dist: ruff>=0.2; extra == "dev"

euler_eval-2.30.0/docs/README.md

@@ -0,0 +1,58 @@
+# euler-eval documentation
+
+| Guide | Covers |
+|---|---|
+| [Configuration](configuration.md) | The config file, `gt` and `datasets` sections, paths and selectors, sparse-depth and points-3d ground truth, euler-train logging |
+| [CLI reference](cli.md) | Every flag, worked examples, device selection, offline cache warmup |
+| [Spaces & alignment](alignment.md) | Why `native` and `metric` exist, depth affine fitting, points-3d gauge alignment, benchmark depth bins |
+| [Metrics](metrics.md) | The full metric inventory per modality, with the key each one is written under |
+| [Error distributions](distributions.md) | Optional per-sample and pooled RMSE histograms, bin settings, axes and JSON layout |
+| [Results & output](output.md) | Where files land, the anatomy of `eval.json`, per-file metrics, the sanity-check report |
+| [In-training validation](validation.md) | The programmatic API for scoring in-memory predictions during training |
+
+## Proposed features
+
+- [Unified evaluation framework](feature-request-unified-evaluation-framework.md):
+  a shared evaluation lifecycle with explicit modality-specific preparation,
+  metric dependencies, populations, aggregation and output compatibility.
+
+## Concepts in one page
+
+**A dataset is an indexed root, not a folder of files.** Every path in the
+config points at a root — a directory or a `.zip` — carrying
+[ds-crawler](https://github.com/d-rothen/ds-crawler) metadata under
+`.ds_crawler/`. That metadata is what makes a path self-describing: it names the
+loader to use, whether depth is radial or planar, what the value range is, and
+which columns of a point cloud are `x,y,z`.
+
+**Ground truth and predictions are paired by ID, not by order.**
+[euler-loading](https://github.com/d-rothen/euler-loading) intersects file IDs
+across modalities, so sample *i* holds the GT depth, the predicted depth and the
+RGB frame that genuinely belong together. Files present on one side only are
+left out of the pairing rather than silently misaligned. Calibration and pose
+data are matched hierarchically instead — one intrinsics file can serve every
+sample beneath it in the tree.
+
+**Predictions are scored in a space, and there can be two.** A metric model is
+compared to GT directly. A relative or free-frame model first has its unknown
+degrees of freedom fitted away — scale and shift for depth, a similarity
+transform for point maps. euler-eval reports the raw comparison as `native` and
+the fitted one as `metric`, so a scale-recovery failure never masquerades as a
+geometry failure. See [Spaces & alignment](alignment.md).
+
+**Per-image scores are reduced three ways.** `image_mean` and `image_median`
+average per-image values across the dataset; `pixel_pool` pools every valid
+pixel first and reduces once. They answer different questions — a few huge
+images dominate `pixel_pool` but not `image_mean` — so all three are reported.
+
+**Metric names are structured, not free-form.** Keys are built with
+[euler-metric-naming](https://github.com/d-rothen/euler-metric-naming), so each
+`eval.json` carries a `metricSet` envelope declaring its namespace and axes, and
+consumers such as euler-train and euler-view can decompose a key like
+`depth.eval.metric.standard.image_mean.absrel` structurally rather than by
+string-splitting.
+
+**Nothing is computed twice, and nothing is required.** Each modality is
+evaluated only when both GT and prediction sides are configured (and not
+skipped), and results are written per modality — so a depth-only prediction is a
+perfectly valid run.

euler_eval-2.30.0/docs/alignment.md

@@ -0,0 +1,91 @@
+# Spaces & alignment
+
+Not every prediction is directly comparable to metric ground truth. A monocular
+depth model may output values that are only correct up to a scale and a shift; a
+point-map model may reconstruct the scene in its own frame entirely. Scoring
+such a prediction as-is measures the missing calibration, not the geometry.
+
+euler-eval handles this by fitting away exactly the unknown degrees of freedom
+and reporting **both** results:
+
+| Space | Meaning |
+|---|---|
+| `native` | The prediction as the model emitted it, compared to GT directly |
+| `metric` | The prediction after the unknown gauge has been fitted to GT |
+
+`native` diagnoses whether the model is metric at all; `metric` isolates
+geometry quality. A canonical alias (`depth`, `points_3d`) points at `metric`
+when it exists, else `native`, so downstream tooling can read one key without
+knowing how the run was configured — see [Results & output](output.md).
+
+## Depth alignment
+
+Relative depth predictions are aligned to metric GT by fitting a global affine
+map `d_aligned = s · d_pred + t` by least squares over valid pixels.
+`--depth-alignment` controls when the fit runs:
+
+| Mode | Behaviour |
+|---|---|
+| `none` | Never fit; score the prediction as-is (already-metric models) |
+| `auto_affine` *(default)* | Fit only when the first prediction looks normalized (roughly `[-1, 1]`) |
+| `affine` | Always fit |
+
+The fit uses pixels where GT and prediction are finite and GT > 0. With
+`--mask-sky`, sky pixels are excluded and the fit is trimmed to `gt ≤ P95(gt)`
+so residual sky outliers do not dominate.
+
+Declaring the prediction with the `relative_depth` or `affine_depth` config key
+(instead of `depth`) tells the evaluator the model is not metric, so
+`auto_affine` fits even when the values happen to fall outside the normalized
+range — see [Configuration](configuration.md#datasets-section).
+
+## Points-3D gauge alignment
+
+Point maps from different model families live in different frames, so
+`points_3d` resolves an unknown **similarity gauge** — the 3D generalization of
+depth's scale-and-shift — before the `metric`-space comparison.
+`--points-3d-alignment` selects it:
+
+| Mode | Fits | Typical model |
+|---|---|---|
+| `none` | nothing | Metric, predict-your-own-camera models |
+| `scale` | a single global scalar | Scale-ambiguous but frame-correct predictions |
+| `similarity` | a 7-DoF Umeyama transform `s·R·p + t` over known per-pixel correspondences | Relative / free-frame models |
+| `auto` *(default)* | `similarity` for declared-relative predictions, else nothing | Mixed comparisons |
+
+The angular `error_decomposition` ray metrics and `rho_a` are computed on the
+**native** point-map directions — the camera-faithful frame — because rotating
+the cloud into GT would erase exactly the camera error they measure.
+Radial/lateral and Euclidean metrics are reported per space.
+
+## Benchmark depth bins
+
+`--benchmark-depth-range MIN MAX` filters valid GT depth pixels to `[MIN, MAX]`,
+then splits them into three equal-width intervals in **square-root** depth
+space, which keeps the near field from being compressed into a single bin:
+
+```text
+sqrt_min = sqrt(MIN);  sqrt_max = sqrt(MAX);  step = (sqrt_max - sqrt_min) / 3
+near = [MIN, (sqrt_min + step)^2)
+mid  = [(sqrt_min + step)^2, (sqrt_min + 2·step)^2)
+far  = [(sqrt_min + 2·step)^2, MAX]
+```
+
+For `--benchmark-depth-range 0.01 80.0` this yields `near=[0.01, 9.29)`,
+`mid=[9.29, 35.95)`, `far=[35.95, 80.0]`, plus an `all` bin over the full range.
+
+Benchmark bins are **additive**: the regular metrics are still computed and
+written exactly as before, with the binned values added under a `bin` axis
+(`…standard.image_mean.near.absrel`) and the resolved boundaries recorded in the
+`metricSet` metadata. RGB metrics are binned too, using GT depth to decide which
+pixels fall in which bin.
+
+The same binning is available programmatically through
+`benchmark_depth_range=(MIN, MAX)` — see
+[In-training validation](validation.md).
+
+With `--distributions`, default depth dataset/per-file histograms use the
+benchmark `all` population, with optional sky-depth capping applied before GT
+range selection. Their counts match the explicit `all` histogram, while the
+unbinned scalar summaries retain their existing population. See
+[Error distributions](distributions.md#benchmark-depth-range-and-sky-depth).

euler_eval-2.30.0/docs/cli.md

@@ -0,0 +1,150 @@
+# CLI reference
+
+```bash
+euler-eval <config> [options]
+```
+
+The config is the only positional argument — a JSON file describing the ground
+truth and the prediction datasets to score against it, documented in
+[Configuration](configuration.md). Invalid JSON or a config that fails
+validation exits with status `1` before any dataset is opened.
+
+## Options
+
+| Flag | Type | Default | Description |
+|---|---|---|---|
+| `--device` | `{auto,cuda,cpu}` | `auto` | Compute device (`auto` prefers CUDA when available) |
+| `--batch-size` | `int` | `16` | Batch size for metrics that support batching |
+| `--num-workers` | `int` | `4` | Number of data-loading workers |
+| `--verbose`, `-v` | flag | off | Enable verbose output |
+| `--domain` | `{dehazing}` | none | Add a domain-specific metric set to the core metrics; repeatable |
+| `--skip-depth` | flag | off | Skip depth evaluation |
+| `--skip-rgb` | flag | off | Skip RGB evaluation |
+| `--skip-rays` | flag | off | Skip rays (spherical direction map) evaluation |
+| `--skip-points-3d` | flag | off | Skip points_3d (per-pixel 3D point map) evaluation |
+| `--mask-sky` | flag | off | Mask sky regions from metrics using GT segmentation |
+| `--sky-depth` | `float` | none | Cap GT and predicted depths in meters and fill predicted `sky_mask` pixels with this value; must be finite and positive |
+| `--no-sanity-check` | flag | off | Disable sanity checking of metric configurations |
+| `--metrics-config` | `str` | auto-detect | Path to `metrics_config.json` for sanity checking |
+| `--depth-alignment` | `{none,auto_affine,affine}` | `auto_affine` | Depth calibration mode ([details](alignment.md#depth-alignment)) |
+| `--points-3d-alignment` | `{none,scale,similarity,auto}` | `auto` | points_3d gauge alignment ([details](alignment.md#points-3d-gauge-alignment)) |
+| `--rgb-fid-backend` | `{builtin,clean-fid}` | `builtin` | RGB FID backend; `clean-fid` requires the `[fid]` extra |
+| `--benchmark-depth-range` | `float float` | none | Depth range `[MIN, MAX]` (m) for near/mid/far benchmark bins ([details](alignment.md#benchmark-depth-bins)) |
+| `--distributions` | flag | off | Enable per-sample and pixel-pool RMSE error histograms for depth and point maps ([details](distributions.md)) |
+| `--no-distributions` | flag | off | Disable histograms, overriding config settings |
+| `--distribution-bins` | `int` | `50` | Total histogram bins, including tails; minimum 3 |
+| `--distribution-scale` | `{log,linear}` | `log` | Spacing of finite histogram bins |
+| `--distribution-range` | `float float` | `0.001 100` | Finite error range `[MIN, MAX)` in evaluated space units; zero/small-error and overflow bins retain the tails |
+
+Histogram settings can also be specified in the config's `distributions`
+section. A bin, scale or range flag enables histograms and overrides that
+config field; `--no-distributions` disables them regardless.
+
+For depth, `--benchmark-depth-range` also restricts the default dataset/per-file
+histograms to the benchmark's inclusive GT range. `--sky-depth` caps GT and
+predictions before this selection and error computation. See the
+[population rules](distributions.md#benchmark-depth-range-and-sky-depth).
+
+## Examples
+
+```bash
+# Default settings (auto-selects CUDA when available)
+euler-eval config.json --batch-size 32
+
+# Sky masking (requires gt.segmentation)
+euler-eval config.json --mask-sky -v
+
+# Cap depth at 100 m and fill predicted sky at 100 m (optional datasets[].sky_mask)
+euler-eval config.json --sky-depth 100
+
+# Only depth evaluation
+euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
+
+# Force affine scale+shift alignment on all depth predictions
+euler-eval config.json --depth-alignment affine
+
+# Use clean-fid for RGB FID
+euler-eval config.json --rgb-fid-backend clean-fid
+
+# Keep all core metrics and add no-reference dehazing metrics (NIQE + FADE)
+euler-eval config.json --domain dehazing
+
+# Benchmark depth/RGB metrics within a depth range (near/mid/far bins)
+euler-eval config.json --benchmark-depth-range 0.01 80.0
+
+# Per-image and pooled RMSE error histograms with shared logarithmic bins
+euler-eval config.json --distributions --distribution-bins 64
+
+# Dense depth against sparse pointcloud GT (pointwise sparse-depth metrics)
+euler-eval example_sparse_depth_config.json --skip-rgb --skip-rays
+
+# Per-pixel 3D point maps (predict-your-own-camera models)
+euler-eval example_points_3d_config.json --skip-depth --skip-rgb --skip-rays
+
+# Align relative point maps to GT with a similarity (Umeyama) gauge
+euler-eval example_points_3d_config.json --points-3d-alignment similarity
+```
+
+## Metric domains
+
+The default metric set is named `core` and is always enabled. `--domain` adds
+metrics suited to a particular evaluation domain without removing or changing
+the core results. The flag is repeatable so future domains can be composed in a
+single run.
+
+`--domain dehazing` adds NIQE and FADE under `rgb.eval.dehazing`. Both are
+no-reference metrics: they score each predicted RGB image, do not use the paired
+GT image, and are averaged across valid images for the dataset result. They use
+the full prediction even with `--mask-sky`, because filling masked pixels would
+change the natural-scene statistics being measured. See [Metrics](metrics.md#dehazing-domain).
+
+## What a run prints
+
+Before any dataset is opened, the run prints its own provenance — package
+version, Python, torch and CUDA versions — which is also what gets recorded in
+`meta` and, when configured, in the euler-train run. It then echoes the resolved
+device, alignment mode, selected metric sets, FID backend and benchmark range,
+followed by one block
+per prediction dataset and modality with the GT and prediction paths, the
+resolved modality metadata (such as `radial_depth`), and the number of matched
+pairs.
+
+A matched-pair count that is much lower than expected is the first sign that GT
+and predictions are not indexed consistently — see
+[Loader resolution & dataset metadata](configuration.md#loader-resolution--dataset-metadata).
+
+## Device selection
+
+`--device auto` resolves to CUDA when `torch.cuda.is_available()`, otherwise
+CPU. Requesting `--device cuda` on a machine without CUDA prints a warning and
+falls back to CPU rather than failing.
+
+On CUDA, the run enables cuDNN benchmarking and TF32 matmuls — throughput knobs
+that do not change which metrics are computed.
+
+`--batch-size` applies to the batched image-quality metrics (PSNR, SSIM, LPIPS,
+FID/KID); `--num-workers` is passed to the underlying dataloaders. Sparse-depth
+and points-3d evaluation is CPU/NumPy work, so `--num-workers` matters more than
+`--batch-size` there.
+
+## Offline cache warmup
+
+Compute nodes without network access need the model weights pre-fetched. Run the
+cache warmup helper on a machine that does have network access, pointing the
+cache environment variables at a shared location:
+
+```bash
+HF_HOME=/shared/cache/hf \
+TORCH_HOME=/shared/cache/hf/torch \
+CLEANFID_CACHE_DIR=/shared/cache/clean-fid \
+euler-eval.init
+```
+
+This pre-downloads the torchvision AlexNet and Inception v3 weights (used by
+LPIPS and the builtin FID/KID), the LPIPS AlexNet weights, and the clean-fid
+Inception checkpoint if `clean-fid` is installed. `TORCH_HOME` is derived from
+`HF_HOME` when it is not set explicitly, so pointing `HF_HOME` at the shared
+cache is usually enough.
+
+Set the same variables for the evaluation run itself so the cached weights are
+found.

euler_eval-2.30.0/docs/configuration.md

@@ -0,0 +1,269 @@
+# Configuration
+
+A run is described by one JSON file: the ground truth, the prediction datasets
+to score against it, and optional experiment logging.
+
+```json
+{
+  "euler_train": { "dir": "runs/my_project" },
+  "gt": {
+    "rgb":          { "path": "/data/gt/rgb" },
+    "depth":        { "path": "/data/gt/depth" },
+    "rays":         { "path": "/data/gt/rays" },
+    "segmentation": { "path": "/data/gt/segmentation" },
+    "calibration":  { "path": "/data/gt/calibration" }
+  },
+  "datasets": [
+    {
+      "name": "model_a",
+      "rgb":   { "path": "/data/model_a/rgb" },
+      "depth": { "path": "/data/model_a/depth" },
+      "rays":  { "path": "/data/model_a/rays" },
+      "output_file": "/path/to/output/model_a_eval.json"
+    },
+    { "name": "model_b_depth_only", "depth": { "path": "/data/model_b/depth" } }
+  ]
+}
+```
+
+Runnable starting points live in the repository root:
+[`example_config.json`](../example_config.json),
+[`example_sparse_depth_config.json`](../example_sparse_depth_config.json), and
+[`example_points_3d_config.json`](../example_points_3d_config.json).
+
+The config is validated before any work starts: a missing `gt` section, an empty
+`datasets` array, a dataset without a `name` or without a single prediction
+modality, and any path that does not exist all fail immediately with a message
+naming the offending field.
+
+The optional top-level [`distributions`](distributions.md#bin-settings) section
+enables per-sample and pooled RMSE histograms. Use `true` for defaults or an
+object with `n_bins`, `scale`, `min_error` and `max_error`; CLI histogram settings
+override the corresponding fields. Omitted or `false` leaves histograms disabled.
+
+## Paths and splits
+
+Every modality entry is an object with a `path`, and optionally a `split`:
+
+```json
+{ "path": "/data/gt/depth", "split": "test" }
+```
+
+euler-loading inline selectors work directly in `path` as well, which is often
+shorter:
+
+| Form | Meaning |
+|---|---|
+| `/data/muses` | Directory root, canonical index |
+| `/data/muses.zip` | Zip archive, read without extraction |
+| `/data/muses.zip:test` | The `test` split of that archive |
+| `/data/muses.zip:test#scope=rgb` | The `test` split, reading `.ds_crawler/rgb/` metadata |
+
+## `gt` section
+
+At least one of `rgb`, `depth`, `sparse_depth`, `rays`, or `points_3d` is
+required.
+
+| Field | Required | Description |
+|---|---|---|
+| `gt.rgb.path` | no\* | GT RGB dataset |
+| `gt.depth.path` | no\* | GT dense depth dataset |
+| `gt.sparse_depth.path` | no\* | Sparse pointcloud GT, `(N,C)` points whose first columns are `x,y,z` in metres |
+| `gt.rays.path` | no\* | GT ray direction map dataset |
+| `gt.points_3d.path` | no\* | GT per-pixel 3D point map `(H,W,3)` in camera-frame metres |
+| `gt.segmentation.path` | no | GT segmentation (needed for `--mask-sky`) |
+| `gt.calibration.path` | no | Calibration data (camera intrinsics); enables true camera-frame [surface normals](metrics.md#surface-normals-and-calibration) |
+| `gt.intrinsics.path` | with `gt.sparse_depth` | Camera intrinsics for pointcloud projection |
+| `gt.camera_extrinsics.path` | with `gt.sparse_depth` | Source-to-camera extrinsics (e.g. MUSES `lidar2rgb`) |
+| `gt.lidar_extrinsics.path` | no | Optional lidar pose in a shared frame; composed with `gt.camera_extrinsics` |
+| `gt.name` | no | Display name for ground truth (default `"GT"`) |
+
+\* At least one of the five modality entries must be present.
+
+## `datasets` section
+
+Each entry needs a `name` and at least one prediction modality. Use only **one**
+dense depth-like entry (`depth`, `relative_depth`, or `affine_depth`) per
+dataset — configuring two is an error, because the space the prediction lives in
+would be ambiguous.
+
+| Field | Required | Description |
+|---|---|---|
+| `name` | yes | Display name for this prediction dataset |
+| `rgb.path` | no\* | Predicted RGB dataset |
+| `depth.path` | no\* | Predicted dense metric depth; also used against sparse pointcloud GT |
+| `relative_depth.path` | no\* | Predicted dense relative depth (scale/shift alignment supported) |
+| `affine_depth.path` | no\* | Predicted dense affine depth (scale/shift alignment supported) |
+| `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth` |
+| `rays.path` | no\* | Predicted ray direction map dataset |
+| `points_3d.path` | no\* | Predicted per-pixel 3D point map `(H,W,3)` |
+| `output_file` | no | Custom results path (default: `eval.json` in the first modality path) |
+
+\* At least one prediction modality must be present.
+
+Which entry you use is also a *declaration*: `relative_depth` and `affine_depth`
+tell the evaluator the prediction is not metric, which is what
+`--depth-alignment auto_affine` and `--points-3d-alignment auto` key off. See
+[Spaces & alignment](alignment.md).
+
+### Predicted sky depth
+
+Add a `sky_mask` entry alongside `depth`, `relative_depth`, or `affine_depth`
+and run with `--sky-depth 100` to evaluate predicted sky at 100 meters:
+
+```json
+{
+  "name": "model_with_sky",
+  "depth": { "path": "/data/model/depth" },
+  "sky_mask": { "path": "/data/model/sky_mask", "split": "test" }
+}
+```
+
+The sky mask is paired with GT and predicted depth by sample ID and hierarchy.
+It supports the same archive, split, and `#scope=` selectors as depth, defaulting
+to the `sky_mask` metadata scope. Its loader must return a boolean mask with
+**True meaning sky**. For example, `generic_dense_depth.sky_mask` uses the RGB
+sky color in the mask dataset's `meta.sky_mask` (`[R, G, B]`) and returns a
+`(1, H, W)` tensor. Masks at another resolution follow the depth grid's crop
+rule or use nearest-neighbor resizing.
+
+`--sky-depth` applies to dense and sparse depth evaluation. GT and both native
+and aligned prediction branches are capped after depth conversion and affine
+fitting; predicted sky pixels are then set to the cap. Predicted sky pixels are
+excluded from the affine fit. Benchmark bins use the capped GT depth.
+
+The cap also works without a predicted mask. Positive infinity is capped;
+NaNs and nonpositive depths remain invalid, except that masked prediction
+pixels are filled even if their original depth is invalid. Sparse GT still
+contributes only at projected points. Predicted sky stays in the scores where
+GT is valid; `--mask-sky` separately excludes pixels using GT segmentation and
+can be combined with `--sky-depth`.
+
+Without `--sky-depth`, the predicted mask is not loaded and depth evaluation
+uses the usual uncapped values. The effective cap and mask source are recorded
+in `eval.json` under `meta.eval_params.sky_depth` and `meta.pred.sky_mask`.
+
+## Sparse depth (pointcloud GT)
+
+Use `gt.sparse_depth` instead of `gt.depth` to evaluate a dense depth prediction
+against a sparse pointcloud, e.g. raw LiDAR returns.
+
+The evaluator projects the sparse GT cloud into the prediction plane using
+`gt.intrinsics` and `gt.camera_extrinsics`, then computes pointwise depth
+metrics only at projected valid pixels. The resulting depth `eval.json` is an
+atomic `sparsedepth.eval` metric set; a depth prediction is not also unprojected
+and evaluated as `points_3d`.
+
+An explicit `datasets[].points_3d` prediction can be evaluated separately
+against the same sparse cloud. That evaluation uses the points-3D similarity
+gauge, writes an atomic `points3d.eval` metric set beside the point-map
+prediction, and is controlled by `--skip-points-3d`. If a dataset provides both
+depth and `points_3d`, each prediction modality gets its own `eval.json` at its
+own path.
+
+Sparse depth does not require segmentation GT. `gt.segmentation` is loaded only
+when `--mask-sky` is set, and then excludes sky pixels from projected-point
+metrics and from scale/shift fitting.
+
+```json
+{
+  "gt": {
+    "name": "MUSES sparse GT",
+    "sparse_depth":      { "path": "/data/muses.zip:test#scope=lidar" },
+    "intrinsics":        { "path": "/data/muses.zip:test#scope=intrinsics" },
+    "camera_extrinsics": { "path": "/data/muses.zip:test#scope=extrinsics" }
+  },
+  "datasets": [
+    { "name": "model_a", "depth": { "path": "/data/model_a/dense_depth:test" } }
+  ]
+}
+```
+
+## Points-3D ground truth sources
+
+A `points_3d` prediction is compared against a GT point map from one of two
+sources:
+
+1. **Explicit** — `gt.points_3d.path` (a stored `(H,W,3)` map).
+2. **Synthesized from depth** — if `gt.points_3d` is absent but `gt.depth.path`
+   and intrinsics (`gt.intrinsics` or `gt.calibration`) are present, GT depth is
+   unprojected on the fly. Whether the GT depth is radial or planar is read from
+   its metadata (`radial_depth`).
+
+## `euler_train` section (optional)
+
+When present, results are logged to an
+[euler-train](https://github.com/d-rothen/euler-train) run, which requires the
+`[logging]` extra. `euler_train.dir` is either a project directory (a new run is
+created and finished on completion) or a full path to an existing run directory
+(the run is resumed and detached afterwards, leaving it active). `dir` is the
+only supported key.
+
+Each prediction dataset is registered as an evaluation named after its config
+`name`, first as `running` and then with its metrics attached, alongside package
+provenance (euler-eval version, Python, torch, CUDA).
+
+## Loader resolution & dataset metadata
+
+Loaders are resolved automatically by euler-loading from each dataset's
+ds-crawler index metadata — no manual loader selection is needed. Each dataset
+directory declares its loader via the `euler_loading.loader` and
+`euler_loading.function` fields of its index. Modality metadata
+(`radial_depth`, `rgb_range`, sparse point columns, coordinate units) is read
+automatically; depth and point-cloud coordinates are assumed to already be in
+metres.
+
+Each dataset root or archive must contain ds-crawler metadata under
+`.ds_crawler/` (`dataset-head.json`, `ds-crawler.json`, `index.json`). When one
+physical root holds several logical modalities, the artifacts may be scoped
+under `.ds_crawler/<modality>/` with a `.ds_crawler/scopes.json` manifest,
+letting a path like `/data/muses.zip:test` load `.ds_crawler/rgb/index.json`,
+`.ds_crawler/depth/index.json`, and so on from the same archive. GT and
+prediction datasets are matched by hierarchy path and file ID.
+
+### Multiple modalities from one archive
+
+Give every modality the same archive root and let the scope select the metadata.
+The scope defaults to the configuration key name, so the common case is compact:
+
+```json
+{
+  "gt": {
+    "rgb":               { "path": "/data/capture.zip:test" },
+    "depth":             { "path": "/data/capture.zip:test" },
+    "intrinsics":        { "path": "/data/capture.zip:test" },
+    "camera_extrinsics": { "path": "/data/capture.zip:test" }
+  }
+}
+```
+
+Use `#scope=<scope>` when an artifact scope is not named exactly like the config
+key, e.g. `"sparse_depth": { "path": "/data/capture.zip:test#scope=lidar" }`.
+
+Regular modalities (`rgb`, `depth`, `rays`, `points_3d`, `sparse_depth`)
+participate in file-ID intersection; calibration and pose data (`intrinsics`,
+`calibration`, `camera_extrinsics`, `lidar_extrinsics`, segmentation) are loaded
+hierarchically and do not.
+
+## `metrics_config.json`
+
+Controls the thresholds used by sanity checking. When `--metrics-config` is not
+given, the tool auto-detects `metrics_config.json` at the project root and falls
+back to built-in defaults if there is none.
+
+The file is grouped by modality, and each metric block carries its thresholds
+plus a `description` explaining what the check is for:
+
+```json
+{
+  "depth": {
+    "expected_range": { "min": 0.0, "max": 100.0 },
+    "absrel": { "warn_if_median_exceeds": 1.0, "warn_if_p90_exceeds": 2.0 },
+    "silog":  { "warn_if_exceeds": 0.5 }
+  }
+}
+```
+
+See [`metrics_config.json`](../metrics_config.json) for the full set of options,
+and [Results & output](output.md#sanity-check-report) for what the checks
+produce.