euler-eval 2.32.0__tar.gz → 2.34.0__tar.gz

{euler_eval-2.32.0/euler_eval.egg-info → euler_eval-2.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-eval
-Version: 2.32.0
+Version: 2.34.0
 Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License: MIT

{euler_eval-2.32.0 → euler_eval-2.34.0}/docs/cli.md

@@ -38,6 +38,7 @@ validation exits with status `1` before any dataset is opened.
 | `--distribution-scale` | `{log,linear}` | `log` | Spacing of finite histogram bins |
 | `--distribution-range` | `float float` | `0.001 100` | Finite error range `[MIN, MAX)` in evaluated space units; zero/small-error and overflow bins retain the tails |
 | `--eval-name` | `str` | `eval` | Name this evaluation: results are written as `<name>.json` in each prediction dataset's `.eval/` folder, so differently named runs sit side by side; a dataset's `output_file` still wins ([details](output.md#where-results-land)) |
+| `--experiment` | `str` | all | Run only this experiment from the config's [`experiments`](configuration.md#experiments-section-optional) section; repeatable |
 
 Histogram settings can also be specified in the config's `distributions`
 section. A bin, scale or range flag enables histograms and overrides that
@@ -73,6 +74,10 @@ euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
 # Keep a sky-masked run beside the default results: .eval/[<scope>/]no_sky.json
 euler-eval config.json --mask-sky --eval-name no_sky
 
+# Run every experiment in the config's "experiments" section, or just one
+euler-eval config.json --device cuda --batch-size 32
+euler-eval config.json --experiment no_sky
+
 # Force affine scale+shift alignment on all depth predictions
 euler-eval config.json --depth-alignment affine
 

{euler_eval-2.32.0 → euler_eval-2.34.0}/docs/configuration.md

@@ -250,6 +250,66 @@ sources:
    unprojected on the fly. Whether the GT depth is radial or planar is read from
    its metadata (`radial_depth`).
 
+## `experiments` section (optional)
+
+Runs the same config several times in one invocation, each time with its own
+command-line flags and under its own name:
+
+```json
+{
+  "gt": { "depth": { "path": "/data/gt/depth" } },
+  "datasets": [{ "name": "model_a", "depth": { "path": "/data/model_a/depth" } }],
+  "experiments": [
+    { "name": "eval" },
+    { "name": "no_sky", "args": ["--mask-sky", "--benchmark-depth-range", 0.01, 80] },
+    { "name": "affine", "args": ["--depth-alignment", "affine"] },
+    { "name": "capped_no_mask", "args": ["--sky-depth", 80], "modalities": ["depth"] }
+  ]
+}
+```
+
+| Field | Required | Description |
+|---|---|---|
+| `name` | yes | Evaluation name: results are written as `<name>.json` in each prediction dataset's `.eval/` folder ([where results land](output.md#where-results-land)). Same rules as `--eval-name`; unique |
+| `args` | no | Command-line tokens this experiment adds, exactly as you would type them; numbers may be written as JSON numbers |
+| `modalities` | no | The prediction modalities (`datasets[]` entries) this experiment may load (default: all): `depth` (also covers `relative_depth` and `affine_depth`), `rgb`, `rays`, `points_3d`, `sky_mask`. The `gt` section is always used as configured |
+
+- **The command line is the base.** Each experiment runs with the flags given on
+  the command line followed by its own `args`, so for a flag that takes a
+  value, the experiment's value wins. Switches such as `--mask-sky` can only be
+  turned on that way, so put the ones that vary between experiments in
+  `args`, not on the command line.
+- **Checked before any work.** Every experiment's `args` are parsed and
+  validated before the first one runs. `--eval-name` and `--experiment` are not
+  allowed in `args`, and `--eval-name` cannot be combined with an `experiments`
+  section, because `name` names the results.
+- **Modalities.** An experiment with `modalities` runs as if the unlisted
+  prediction entries were not in `datasets[]`; a `sky_mask` goes with its depth
+  prediction. GT inputs are not selected this way: they stay as configured,
+  and flags decide which ones an evaluation reads (`--mask-sky` the
+  segmentation, for example), so GT names such as `segmentation` or
+  `sparse_depth` are rejected. A listed modality is still read only when the
+  evaluation needs it, e.g. `sky_mask` only with `--sky-depth`,
+  `--mask-sky-prediction` or `--sky-iou`; with just `--mask-sky`, listing
+  `sky_mask` changes nothing. This is how to compare runs where a modality's
+  presence changes the output: above, `capped_no_mask` caps depth at 80 m
+  without filling the predicted sky. A dataset left without a prediction is
+  not evaluated in that experiment. Listing a modality no dataset has only
+  prints a warning, so one experiment list can serve several generated
+  configs.
+- **Selecting experiments.** By default all experiments run, in config order.
+  `--experiment NAME` (repeatable) runs only the named ones, e.g. one per task
+  of a job array.
+- **Failures.** An experiment that fails is reported and the rest still run;
+  the exit status is then `1`.
+
+Each experiment loads and scores the data again, and its results carry the
+settings it ran with in [`protocol.parameters`](output.md#comparing-evaluations)
+and its name in `meta.eval_name`. With euler-train logging, each prediction
+dataset is logged once per experiment, as `<name>/<experiment>`; the
+sanity-check report is written per experiment as
+`sanity_check_report_<experiment>.json`.
+
 ## `euler_train` section (optional)
 
 When present, results are logged to an
@@ -260,7 +320,8 @@ created and finished on completion) or a full path to an existing run directory
 only supported key.
 
 Each prediction dataset is registered as an evaluation named after its config
-`name`, first as `running` and then with its metrics attached, alongside package
+`name` (`<name>/<experiment>` with [`experiments`](#experiments-section-optional)),
+first as `running` and then with its metrics attached, alongside package
 provenance (euler-eval version, Python, torch, CUDA).
 
 ## Loader resolution & dataset metadata

{euler_eval-2.32.0 → euler_eval-2.34.0}/docs/output.md

@@ -74,7 +74,7 @@ set.
     "metricDescriptions": { "absrel": { "…": "…" } }
   },
   "protocol": {
-    "schemaVersion": "1.0",
+    "schemaVersion": "1.1",
     "parameters": { "reference.gt.dataset": "muses", "method.depth_alignment": "auto_affine", "…": "…" }
   },
   "dataset_info": { "num_pairs": 500, "gt_name": "GT", "pred_name": "model_a" },
@@ -188,7 +188,7 @@ values differ say where. Nothing else in the file needs to be compared for that;
 
 ```json
 "protocol": {
-  "schemaVersion": "1.0",
+  "schemaVersion": "1.1",
   "parameters": {
     "reference.gt.modality": "sparse_depth",
     "reference.gt.dataset": "seeingthroughfog",
@@ -213,7 +213,8 @@ values differ say where. Nothing else in the file needs to be compared for that;
     "population.benchmark_depth_range_m": [0.01, 80.0],
     "method.metrics_revision": 1,
     "method.depth_alignment": "auto_affine",
-    "method.sky_depth_m": null
+    "method.sky_depth_m": null,
+    "method.pred_sky_mask": false
   }
 }
 ```
@@ -224,14 +225,14 @@ The first segment of a key is its section:
 |---|---|
 | `reference` | The GT the prediction is scored against (`gt`, with its config key as `modality`) and every other GT input the modality reads, each identified by `dataset` (the `dataset.id` of its `dataset-head.json`), metadata `scope` and `split` — not by file path, so the same data on another machine matches |
 | `population` | Which samples and pixels were scored: the matched `samples`, sky masking, the grid the metrics were computed on and benchmark depth bins |
-| `method` | How they were scored: alignment modes, the sky-depth cap, how GT and prediction resolutions were reconciled, the FID backend and `metrics_revision` |
+| `method` | How they were scored: alignment modes, the sky-depth cap and whether the prediction's `sky_mask` was used, how GT and prediction resolutions were reconciled, the FID backend and `metrics_revision` |
 
 What is recorded, per modality:
 
 | Modality | `reference` inputs besides `gt` | `population` | `method` |
 |---|---|---|---|
-| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `spatial_alignment` |
-| `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m` |
+| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask`, `spatial_alignment` |
+| `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask` |
 | `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
 | `rays` | `calibration` | `samples`, `evaluated_resolution` | `spatial_alignment` |
 | `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
@@ -253,10 +254,14 @@ Rules that keep comparisons a plain map comparison:
   for identical inputs and settings, which the [changelog](../CHANGELOG.md)
   then says. Different revisions are not comparable even when every other
   key matches.
-- Settings are recorded, not what they caused. With `--mask-sky-prediction`,
-  `population.pred_sky_masking` is `true` for every prediction, while
-  `meta.pred_sky_masking` records how much each model's own sky removed;
-  `metricSet.metadata` records whether an alignment was actually applied.
+- Settings are recorded, not what they caused. `population.pred_sky_masking`
+  is the `--mask-sky-prediction` flag, and `method.pred_sky_mask` says whether
+  the prediction's `sky_mask` was loaded and scored with (filled at
+  `--sky-depth`, or excluded by `--mask-sky-prediction`). It is `false` for a
+  prediction without one, or an [experiment](configuration.md#experiments-section-optional)
+  that leaves it out. How much each model's own sky removed is in
+  `meta.pred_sky_masking`, and `metricSet.metadata` records whether an
+  alignment was actually applied.
 - Renaming or removing a key bumps the major `schemaVersion`; adding one bumps
   the minor version.