euler-eval 2.31.0__tar.gz → 2.33.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (115) hide show
  1. {euler_eval-2.31.0/euler_eval.egg-info → euler_eval-2.33.0}/PKG-INFO +5 -3
  2. {euler_eval-2.31.0 → euler_eval-2.33.0}/README.md +4 -2
  3. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/README.md +1 -1
  4. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/cli.md +9 -0
  5. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/configuration.md +65 -6
  6. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/output.md +138 -10
  7. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/cli.py +678 -74
  8. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/config_paths.py +55 -0
  9. euler_eval-2.33.0/euler_eval/protocol.py +145 -0
  10. {euler_eval-2.31.0 → euler_eval-2.33.0/euler_eval.egg-info}/PKG-INFO +5 -3
  11. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/SOURCES.txt +4 -0
  12. {euler_eval-2.31.0 → euler_eval-2.33.0}/pyproject.toml +1 -1
  13. euler_eval-2.33.0/tests/eval_output.py +16 -0
  14. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_cli_sparse_outputs.py +20 -7
  15. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_depth_alignment_output.py +2 -1
  16. euler_eval-2.33.0/tests/test_experiments.py +409 -0
  17. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_mask_sky_pairing.py +2 -2
  18. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_mask_sky_prediction.py +2 -2
  19. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_points_3d_sparse.py +8 -8
  20. euler_eval-2.33.0/tests/test_protocol.py +405 -0
  21. euler_eval-2.33.0/tests/test_save_results.py +513 -0
  22. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sky_iou_cli.py +3 -2
  23. euler_eval-2.31.0/tests/test_save_results.py +0 -301
  24. {euler_eval-2.31.0 → euler_eval-2.33.0}/LICENSE +0 -0
  25. {euler_eval-2.31.0 → euler_eval-2.33.0}/MANIFEST.in +0 -0
  26. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/alignment.md +0 -0
  27. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/distributions.md +0 -0
  28. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/feature-request-unified-evaluation-framework.md +0 -0
  29. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/metrics.md +0 -0
  30. {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/validation.md +0 -0
  31. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/THIRD_PARTY_NOTICES.md +0 -0
  32. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/__init__.py +0 -0
  33. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/calibration.py +0 -0
  34. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/data.py +0 -0
  35. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/distribution_contract.py +0 -0
  36. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/distributions.py +0 -0
  37. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/evaluate.py +0 -0
  38. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metric_sets.py +0 -0
  39. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/__init__.py +0 -0
  40. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/_natural_scene_models.py +0 -0
  41. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/absrel.py +0 -0
  42. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/daniel_error.py +0 -0
  43. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_binned_error.py +0 -0
  44. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_edge_f1.py +0 -0
  45. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_standard.py +0 -0
  46. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/fade.py +0 -0
  47. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/fid_kid.py +0 -0
  48. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/gpu_depth_batch.py +0 -0
  49. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/gpu_image_batch.py +0 -0
  50. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/high_freq_energy.py +0 -0
  51. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/lpips_metric.py +0 -0
  52. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/niqe.py +0 -0
  53. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/normal_consistency.py +0 -0
  54. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_cloud.py +0 -0
  55. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_decomposition.py +0 -0
  56. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_distance.py +0 -0
  57. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_geometry.py +0 -0
  58. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/psnr.py +0 -0
  59. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_edge_f1.py +0 -0
  60. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_lpips.py +0 -0
  61. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_psnr_ssim.py +0 -0
  62. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rho_a.py +0 -0
  63. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rmse.py +0 -0
  64. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/scale_invariant_log.py +0 -0
  65. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/sky_iou.py +0 -0
  66. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/ssim.py +0 -0
  67. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/tail_errors.py +0 -0
  68. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/utils.py +0 -0
  69. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/sanity_checker.py +0 -0
  70. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/schemas/distribution-v1.schema.json +0 -0
  71. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/utils/hierarchy_parser.py +0 -0
  72. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/validation.py +0 -0
  73. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/dependency_links.txt +0 -0
  74. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/entry_points.txt +0 -0
  75. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/requires.txt +0 -0
  76. {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/top_level.txt +0 -0
  77. {euler_eval-2.31.0 → euler_eval-2.33.0}/init_cache.py +0 -0
  78. {euler_eval-2.31.0 → euler_eval-2.33.0}/scripts/generate_distribution_fixtures.py +0 -0
  79. {euler_eval-2.31.0 → euler_eval-2.33.0}/setup.cfg +0 -0
  80. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/__init__.py +0 -0
  81. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/conftest.py +0 -0
  82. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/distribution_contract_utils.py +0 -0
  83. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/README.md +0 -0
  84. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-depth.eval.json +0 -0
  85. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-points3d.eval.json +0 -0
  86. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-sparsedepth.eval.json +0 -0
  87. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/future-sum-by-depth.eval.json +0 -0
  88. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/histogram-only.eval.json +0 -0
  89. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/per-file-only.eval.json +0 -0
  90. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/scalar-only.eval.json +0 -0
  91. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/calibration_output.json +0 -0
  92. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/depth_output.json +0 -0
  93. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/rgb_output.json +0 -0
  94. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/segmentation_output.json +0 -0
  95. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_alignment.py +0 -0
  96. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_cli_device.py +0 -0
  97. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_config.py +0 -0
  98. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_data.py +0 -0
  99. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_data_builders.py +0 -0
  100. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_depth_standard.py +0 -0
  101. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_distribution_contract.py +0 -0
  102. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_distributions.py +0 -0
  103. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_domain_metrics.py +0 -0
  104. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_evaluate_helpers.py +0 -0
  105. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_init_cache.py +0 -0
  106. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_integration.py +0 -0
  107. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_meta_output.py +0 -0
  108. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_metrics_utils.py +0 -0
  109. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_normal_consistency.py +0 -0
  110. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_points_3d.py +0 -0
  111. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_rgb_fid_output.py +0 -0
  112. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_rho_a.py +0 -0
  113. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sky_iou.py +0 -0
  114. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sparse_depth.py +0 -0
  115. {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_validation.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: euler-eval
3
- Version: 2.31.0
3
+ Version: 2.33.0
4
4
  Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
5
5
  Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
6
6
  License: MIT
@@ -133,8 +133,10 @@ euler-eval config.json --batch-size 32
133
133
 
134
134
  Ground truth and prediction files are matched by ID rather than by directory
135
135
  order, so an incomplete prediction set scores the frames it does have instead of
136
- silently comparing the wrong pairs. Results are written next to each prediction
137
- modality:
136
+ silently comparing the wrong pairs. Results are written into each prediction
137
+ dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
138
+ `/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
139
+ dataset ([details](docs/output.md#where-results-land)):
138
140
 
139
141
  ```json
140
142
  {
@@ -83,8 +83,10 @@ euler-eval config.json --batch-size 32
83
83
 
84
84
  Ground truth and prediction files are matched by ID rather than by directory
85
85
  order, so an incomplete prediction set scores the frames it does have instead of
86
- silently comparing the wrong pairs. Results are written next to each prediction
87
- modality:
86
+ silently comparing the wrong pairs. Results are written into each prediction
87
+ dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
88
+ `/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
89
+ dataset ([details](docs/output.md#where-results-land)):
88
90
 
89
91
  ```json
90
92
  {
@@ -7,7 +7,7 @@
7
7
  | [Spaces & alignment](alignment.md) | Why `native` and `metric` exist, depth affine fitting, points-3d gauge alignment, benchmark depth bins |
8
8
  | [Metrics](metrics.md) | The full metric inventory per modality, with the key each one is written under |
9
9
  | [Error distributions](distributions.md) | Optional per-sample and pooled RMSE histograms, bin settings, axes and JSON layout |
10
- | [Results & output](output.md) | Where files land, the anatomy of `eval.json`, per-file metrics, the sanity-check report |
10
+ | [Results & output](output.md) | Where files land, the anatomy of `eval.json`, comparing evaluations, per-file metrics, the sanity-check report |
11
11
  | [In-training validation](validation.md) | The programmatic API for scoring in-memory predictions during training |
12
12
 
13
13
  ## Proposed features
@@ -37,6 +37,8 @@ validation exits with status `1` before any dataset is opened.
37
37
  | `--distribution-bins` | `int` | `50` | Total histogram bins, including tails; minimum 3 |
38
38
  | `--distribution-scale` | `{log,linear}` | `log` | Spacing of finite histogram bins |
39
39
  | `--distribution-range` | `float float` | `0.001 100` | Finite error range `[MIN, MAX)` in evaluated space units; zero/small-error and overflow bins retain the tails |
40
+ | `--eval-name` | `str` | `eval` | Name this evaluation: results are written as `<name>.json` in each prediction dataset's `.eval/` folder, so differently named runs sit side by side; a dataset's `output_file` still wins ([details](output.md#where-results-land)) |
41
+ | `--experiment` | `str` | all | Run only this experiment from the config's [`experiments`](configuration.md#experiments-section-optional) section; repeatable |
40
42
 
41
43
  Histogram settings can also be specified in the config's `distributions`
42
44
  section. A bin, scale or range flag enables histograms and overrides that
@@ -69,6 +71,13 @@ euler-eval config.json --sky-iou --sky-depth 100
69
71
  # Only depth evaluation
70
72
  euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
71
73
 
74
+ # Keep a sky-masked run beside the default results: .eval/[<scope>/]no_sky.json
75
+ euler-eval config.json --mask-sky --eval-name no_sky
76
+
77
+ # Run every experiment in the config's "experiments" section, or just one
78
+ euler-eval config.json --device cuda --batch-size 32
79
+ euler-eval config.json --experiment no_sky
80
+
72
81
  # Force affine scale+shift alignment on all depth predictions
73
82
  euler-eval config.json --depth-alignment affine
74
83
 
@@ -127,7 +127,7 @@ would be ambiguous.
127
127
  | `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth`, `--mask-sky-prediction` and `--sky-iou` |
128
128
  | `rays.path` | no\* | Predicted ray direction map dataset |
129
129
  | `points_3d.path` | no\* | Predicted per-pixel 3D point map `(H,W,3)` |
130
- | `output_file` | no | Custom results path (default: `eval.json` in the first modality path) |
130
+ | `output_file` | no | Custom results path for every modality of this entry, replacing the `.eval/` layout and `--eval-name` (default: `.eval/[<scope>/][split_<split>/]eval.json` in each prediction modality's dataset root; see [where results land](output.md#where-results-land)) |
131
131
 
132
132
  \* At least one prediction modality must be present.
133
133
 
@@ -214,10 +214,11 @@ and evaluated as `points_3d`.
214
214
 
215
215
  An explicit `datasets[].points_3d` prediction can be evaluated separately
216
216
  against the same sparse cloud. That evaluation uses the points-3D similarity
217
- gauge, writes an atomic `points3d.eval` metric set beside the point-map
218
- prediction, and is controlled by `--skip-points-3d`. If a dataset provides both
219
- depth and `points_3d`, each prediction modality gets its own `eval.json` at its
220
- own path.
217
+ gauge, writes an atomic `points3d.eval` metric set to the point-map
218
+ prediction's results file, and is controlled by `--skip-points-3d`. If a dataset
219
+ provides both depth and `points_3d`, each prediction modality gets its own
220
+ results file, also when both live in one archive under separate scopes (see
221
+ [where results land](output.md#where-results-land)).
221
222
 
222
223
  Sparse depth does not require segmentation GT. `gt.segmentation` is loaded only
223
224
  when `--mask-sky` or `--sky-iou` is set. `--mask-sky` excludes sky pixels from
@@ -249,6 +250,63 @@ sources:
249
250
  unprojected on the fly. Whether the GT depth is radial or planar is read from
250
251
  its metadata (`radial_depth`).
251
252
 
253
+ ## `experiments` section (optional)
254
+
255
+ Runs the same config several times in one invocation, each time with its own
256
+ command-line flags and under its own name:
257
+
258
+ ```json
259
+ {
260
+ "gt": { "depth": { "path": "/data/gt/depth" } },
261
+ "datasets": [{ "name": "model_a", "depth": { "path": "/data/model_a/depth" } }],
262
+ "experiments": [
263
+ { "name": "eval" },
264
+ { "name": "no_sky", "args": ["--mask-sky", "--benchmark-depth-range", 0.01, 80] },
265
+ { "name": "affine", "args": ["--depth-alignment", "affine"] },
266
+ { "name": "capped_no_mask", "args": ["--sky-depth", 80], "modalities": ["depth"] }
267
+ ]
268
+ }
269
+ ```
270
+
271
+ | Field | Required | Description |
272
+ |---|---|---|
273
+ | `name` | yes | Evaluation name: results are written as `<name>.json` in each prediction dataset's `.eval/` folder ([where results land](output.md#where-results-land)). Same rules as `--eval-name`; unique |
274
+ | `args` | no | Command-line tokens this experiment adds, exactly as you would type them; numbers may be written as JSON numbers |
275
+ | `modalities` | no | The configured modalities this experiment may load (default: all). Names: `depth` (also covers `relative_depth` and `affine_depth` predictions), `sparse_depth`, `rgb`, `rays`, `points_3d`, `sky_mask`, `segmentation` (also covers `semantic_segmentation`), `calibration`, `intrinsics`, `camera_extrinsics`, `lidar_extrinsics` |
276
+
277
+ - **The command line is the base.** Each experiment runs with the flags given on
278
+ the command line followed by its own `args`, so for a flag that takes a
279
+ value, the experiment's value wins. Switches such as `--mask-sky` can only be
280
+ turned on that way, so put the ones that vary between experiments in
281
+ `args`, not on the command line.
282
+ - **Checked before any work.** Every experiment's `args` are parsed and
283
+ validated before the first one runs. `--eval-name` and `--experiment` are not
284
+ allowed in `args`, and `--eval-name` cannot be combined with an `experiments`
285
+ section, because `name` names the results.
286
+ - **Modalities.** An experiment with `modalities` runs as if the unlisted GT
287
+ inputs and prediction entries were not in the config; a name covers both
288
+ sides. A listed modality is still read only when the evaluation needs it,
289
+ e.g. `sky_mask` only with `--sky-depth`, `--mask-sky-prediction` or
290
+ `--sky-iou`. This is how to compare runs where a modality's presence changes
291
+ the output: above, `capped_no_mask` caps depth at 80 m without filling the
292
+ predicted sky. A dataset left without a prediction is not evaluated in that
293
+ experiment, and the reduced config must still be valid (e.g. `sparse_depth`
294
+ needs `intrinsics` and `camera_extrinsics`). Listing a modality the config
295
+ does not have only prints a warning, so one experiment list can serve several
296
+ generated configs.
297
+ - **Selecting experiments.** By default all experiments run, in config order.
298
+ `--experiment NAME` (repeatable) runs only the named ones, e.g. one per task
299
+ of a job array.
300
+ - **Failures.** An experiment that fails is reported and the rest still run;
301
+ the exit status is then `1`.
302
+
303
+ Each experiment loads and scores the data again, and its results carry the
304
+ settings it ran with in [`protocol.parameters`](output.md#comparing-evaluations)
305
+ and its name in `meta.eval_name`. With euler-train logging, each prediction
306
+ dataset is logged once per experiment, as `<name>/<experiment>`; the
307
+ sanity-check report is written per experiment as
308
+ `sanity_check_report_<experiment>.json`.
309
+
252
310
  ## `euler_train` section (optional)
253
311
 
254
312
  When present, results are logged to an
@@ -259,7 +317,8 @@ created and finished on completion) or a full path to an existing run directory
259
317
  only supported key.
260
318
 
261
319
  Each prediction dataset is registered as an evaluation named after its config
262
- `name`, first as `running` and then with its metrics attached, alongside package
320
+ `name` (`<name>/<experiment>` with [`experiments`](#experiments-section-optional)),
321
+ first as `running` and then with its metrics attached, alongside package
263
322
  provenance (euler-eval version, Python, torch, CUDA).
264
323
 
265
324
  ## Loader resolution & dataset metadata
@@ -2,20 +2,56 @@
2
2
 
3
3
  ## Where results land
4
4
 
5
- One `eval.json` is written **per modality, per prediction dataset**, next to the
6
- prediction it scores — `/data/model_a/depth/eval.json` for the depth results of
7
- the dataset whose depth path is `/data/model_a/depth`, and so on. If that path
8
- passes through a `.zip`, the JSON is written *into* the archive rather than
9
- beside it, so zip-backed datasets stay self-contained.
5
+ Results are written **per modality, per prediction dataset**, into a `.eval/`
6
+ folder at the prediction dataset's root, beside its `.ds_crawler/` metadata:
10
7
 
11
- `output_file` in a dataset entry overrides the destination. It applies to the
12
- dataset as a whole, so when several modalities are evaluated for the same
13
- dataset the per-modality default locations are usually what you want.
8
+ ```text
9
+ /data/model_a/ # a datasets[].<modality>.path
10
+ .ds_crawler/
11
+ depth/ rgb/ # ds-crawler metadata scopes
12
+ .eval/
13
+ depth/
14
+ eval.json # depth results
15
+ no_sky.json # depth results of --eval-name no_sky
16
+ split_test/eval.json # depth results for "/data/model_a:test"
17
+ rgb/
18
+ eval.json # rgb results
19
+ ```
20
+
21
+ The folder below `.eval/` mirrors how the prediction was loaded:
22
+
23
+ | Prediction | Results file |
24
+ |---|---|
25
+ | Metadata scope with its own `.ds_crawler/<scope>/` | `.eval/<scope>/<name>.json` |
26
+ | Root-level `.ds_crawler/` metadata (no scope) | `.eval/<name>.json` |
27
+ | A split is selected (`:test` or `split`) | `split_<split>/` is added before the file name |
28
+
29
+ The scope is the path's `#scope=` selector, otherwise the config key (`depth`,
30
+ `relative_depth`, `rgb`, `points_3d`, …). As in euler-loading, a scope without
31
+ metadata of its own falls back to the root-level `.ds_crawler/`, and so do its
32
+ results. This way modalities and splits that share one dataset root or archive
33
+ each get their own file. If the root is a `.zip`, `.eval/` is written *into* the
34
+ archive (inside its top-level folder when the archive wraps everything in one
35
+ named after it), so zip-backed datasets stay self-contained.
36
+
37
+ `<name>` is `eval` unless the run is named with `--eval-name`:
38
+ `--mask-sky --eval-name no_sky` writes `no_sky.json` next to an earlier
39
+ `eval.json` instead of replacing it. Names use letters, digits, `.`, `-` and
40
+ `_`; `.json` is appended.
41
+
42
+ `output_file` in a dataset entry overrides all of this. It is one path for the
43
+ whole dataset entry: every modality evaluated for that entry is written to it,
44
+ and the last one written replaces the others, so use it for single-modality
45
+ entries.
46
+
47
+ Earlier releases wrote `eval.json` directly at the dataset root (inside the
48
+ archive for zips). Readers such as euler-view look in `.eval/<scope>/`, then
49
+ `.eval/`, then fall back to that legacy location.
14
50
 
15
51
  Each `eval.json` is atomic and declares one metric namespace. With
16
52
  `gt.sparse_depth`, a depth-family prediction writes only `sparsedepth.eval`; an
17
- explicit `points_3d` prediction writes only `points3d.eval` beside the point
18
- map. A depth prediction does not implicitly produce a second points-3D metric
53
+ explicit `points_3d` prediction writes only `points3d.eval` to the point
54
+ map's own results file. A depth prediction does not implicitly produce a second points-3D metric
19
55
  set.
20
56
 
21
57
  ## Anatomy of `eval.json`
@@ -37,6 +73,10 @@ set.
37
73
  "axes": { "space": { "…": "…" }, "category": { "…": "…" } },
38
74
  "metricDescriptions": { "absrel": { "…": "…" } }
39
75
  },
76
+ "protocol": {
77
+ "schemaVersion": "1.1",
78
+ "parameters": { "reference.gt.dataset": "muses", "method.depth_alignment": "auto_affine", "…": "…" }
79
+ },
40
80
  "dataset_info": { "num_pairs": 500, "gt_name": "GT", "pred_name": "model_a" },
41
81
  "meta": {
42
82
  "version": "2.25.0",
@@ -56,6 +96,7 @@ set.
56
96
  | Block | What it is for |
57
97
  |---|---|
58
98
  | `metricSet` | The euler-metric-naming envelope: producer, namespace, axis declarations and per-metric descriptions (unit, direction, bounds), plus how the spaces were resolved |
99
+ | `protocol` | The comparison keys: every setting that changes metric values, as a flat map ([Comparing evaluations](#comparing-evaluations)) |
59
100
  | `dataset_info` | How many pairs were actually matched, and the display names of both sides |
60
101
  | `meta` | Full provenance: package version, device, both source paths and their resolved dimensions, modality metadata, and the effective evaluation flags |
61
102
  | *metric tree* | The metrics themselves, under the modality root (`depth`, `rgb`, `rays`, `sparsedepth`, `points3d`) |
@@ -137,6 +178,93 @@ because these keys have no depth space. `metricDescriptions` gains `iou`,
137
178
  `precision` and `recall` (higher is better, bounds `[0, 1]`). Every other key
138
179
  decomposes exactly as before.
139
180
 
181
+ ## Comparing evaluations
182
+
183
+ `protocol.parameters` says how an evaluation was set up, as one flat map of
184
+ dotted keys. Its keys are the comparison keys: two evaluations of the same
185
+ modality were set up the same way when their maps are equal, and the keys whose
186
+ values differ say where. Nothing else in the file needs to be compared for that;
187
+ `meta` stays the full record of the run, paths and device included.
188
+
189
+ ```json
190
+ "protocol": {
191
+ "schemaVersion": "1.1",
192
+ "parameters": {
193
+ "reference.gt.modality": "sparse_depth",
194
+ "reference.gt.dataset": "seeingthroughfog",
195
+ "reference.gt.scope": "sparse_depth",
196
+ "reference.gt.split": "test",
197
+ "reference.intrinsics.dataset": "seeingthroughfog",
198
+ "reference.intrinsics.scope": "intrinsics",
199
+ "reference.intrinsics.split": "test",
200
+ "reference.camera_extrinsics.dataset": "seeingthroughfog",
201
+ "reference.camera_extrinsics.scope": "camera_extrinsics",
202
+ "reference.camera_extrinsics.split": "test",
203
+ "reference.lidar_extrinsics.dataset": null,
204
+ "reference.lidar_extrinsics.scope": null,
205
+ "reference.lidar_extrinsics.split": null,
206
+ "reference.segmentation.dataset": null,
207
+ "reference.segmentation.scope": null,
208
+ "reference.segmentation.split": null,
209
+ "population.samples": 943,
210
+ "population.gt_sky_masking": false,
211
+ "population.pred_sky_masking": false,
212
+ "population.evaluated_resolution": [1024, 1920],
213
+ "population.benchmark_depth_range_m": [0.01, 80.0],
214
+ "method.metrics_revision": 1,
215
+ "method.depth_alignment": "auto_affine",
216
+ "method.sky_depth_m": null,
217
+ "method.pred_sky_mask": false
218
+ }
219
+ }
220
+ ```
221
+
222
+ The first segment of a key is its section:
223
+
224
+ | Section | Holds |
225
+ |---|---|
226
+ | `reference` | The GT the prediction is scored against (`gt`, with its config key as `modality`) and every other GT input the modality reads, each identified by `dataset` (the `dataset.id` of its `dataset-head.json`), metadata `scope` and `split` — not by file path, so the same data on another machine matches |
227
+ | `population` | Which samples and pixels were scored: the matched `samples`, sky masking, the grid the metrics were computed on and benchmark depth bins |
228
+ | `method` | How they were scored: alignment modes, the sky-depth cap and whether the prediction's `sky_mask` was used, how GT and prediction resolutions were reconciled, the FID backend and `metrics_revision` |
229
+
230
+ What is recorded, per modality:
231
+
232
+ | Modality | `reference` inputs besides `gt` | `population` | `method` |
233
+ |---|---|---|---|
234
+ | `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask`, `spatial_alignment` |
235
+ | `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask` |
236
+ | `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
237
+ | `rays` | `calibration` | `samples`, `evaluated_resolution` | `spatial_alignment` |
238
+ | `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
239
+ | `points3d` (sparse GT) | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment` |
240
+
241
+ Every modality also has `method.metrics_revision`.
242
+
243
+ Rules that keep comparisons a plain map comparison:
244
+
245
+ - Only settings that change the value of a metric both files contain are
246
+ parameters. `--domain`, `--sky-iou` and histogram settings only add metrics
247
+ (histograms describe their bins in the distribution registry); device, batch
248
+ size, workers, paths and `--eval-name` do not change values.
249
+ - Every key of a modality is always written, `null` or `false` when off, so a
250
+ key missing from one side means "not recorded", never "off". Compare the union
251
+ of both key sets.
252
+ - Values are scalars or lists of scalars, never objects.
253
+ - `method.metrics_revision` changes only when a release changes metric values
254
+ for identical inputs and settings, which the [changelog](../CHANGELOG.md)
255
+ then says. Different revisions are not comparable even when every other
256
+ key matches.
257
+ - Settings are recorded, not what they caused. `population.pred_sky_masking`
258
+ is the `--mask-sky-prediction` flag, and `method.pred_sky_mask` says whether
259
+ the prediction's `sky_mask` was loaded and scored with (filled at
260
+ `--sky-depth`, or excluded by `--mask-sky-prediction`). It is `false` for a
261
+ prediction without one, or an [experiment](configuration.md#experiments-section-optional)
262
+ that leaves it out. How much each model's own sky removed is in
263
+ `meta.pred_sky_masking`, and `metricSet.metadata` records whether an
264
+ alignment was actually applied.
265
+ - Renaming or removing a key bumps the major `schemaVersion`; adding one bumps
266
+ the minor version.
267
+
140
268
  ## Spaces and canonical aliases
141
269
 
142
270
  Depth-like and points-3d results are nested by space: