euler-eval 2.31.0__tar.gz → 2.33.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {euler_eval-2.31.0/euler_eval.egg-info → euler_eval-2.33.0}/PKG-INFO +5 -3
- {euler_eval-2.31.0 → euler_eval-2.33.0}/README.md +4 -2
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/README.md +1 -1
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/cli.md +9 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/configuration.md +65 -6
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/output.md +138 -10
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/cli.py +678 -74
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/config_paths.py +55 -0
- euler_eval-2.33.0/euler_eval/protocol.py +145 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0/euler_eval.egg-info}/PKG-INFO +5 -3
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/SOURCES.txt +4 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/pyproject.toml +1 -1
- euler_eval-2.33.0/tests/eval_output.py +16 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_cli_sparse_outputs.py +20 -7
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_depth_alignment_output.py +2 -1
- euler_eval-2.33.0/tests/test_experiments.py +409 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_mask_sky_pairing.py +2 -2
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_mask_sky_prediction.py +2 -2
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_points_3d_sparse.py +8 -8
- euler_eval-2.33.0/tests/test_protocol.py +405 -0
- euler_eval-2.33.0/tests/test_save_results.py +513 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sky_iou_cli.py +3 -2
- euler_eval-2.31.0/tests/test_save_results.py +0 -301
- {euler_eval-2.31.0 → euler_eval-2.33.0}/LICENSE +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/MANIFEST.in +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/alignment.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/distributions.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/feature-request-unified-evaluation-framework.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/metrics.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/docs/validation.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/THIRD_PARTY_NOTICES.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/__init__.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/calibration.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/data.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/distribution_contract.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/distributions.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/evaluate.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metric_sets.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/__init__.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/_natural_scene_models.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/absrel.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/daniel_error.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_binned_error.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_edge_f1.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/depth_standard.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/fade.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/fid_kid.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/gpu_depth_batch.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/gpu_image_batch.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/high_freq_energy.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/lpips_metric.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/niqe.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/normal_consistency.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_cloud.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_decomposition.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_distance.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/points3d_geometry.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/psnr.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_edge_f1.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_lpips.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rgb_psnr_ssim.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rho_a.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/rmse.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/scale_invariant_log.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/sky_iou.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/ssim.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/tail_errors.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/metrics/utils.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/sanity_checker.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/schemas/distribution-v1.schema.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/utils/hierarchy_parser.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval/validation.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/dependency_links.txt +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/entry_points.txt +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/requires.txt +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/euler_eval.egg-info/top_level.txt +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/init_cache.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/scripts/generate_distribution_fixtures.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/setup.cfg +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/__init__.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/conftest.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/distribution_contract_utils.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/README.md +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-depth.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-points3d.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/counts-sparsedepth.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/future-sum-by-depth.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/histogram-only.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/per-file-only.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/fixtures/distributions/scalar-only.eval.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/calibration_output.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/depth_output.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/rgb_output.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/mock_files/segmentation_output.json +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_alignment.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_cli_device.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_config.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_data.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_data_builders.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_depth_standard.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_distribution_contract.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_distributions.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_domain_metrics.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_evaluate_helpers.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_init_cache.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_integration.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_meta_output.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_metrics_utils.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_normal_consistency.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_points_3d.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_rgb_fid_output.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_rho_a.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sky_iou.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_sparse_depth.py +0 -0
- {euler_eval-2.31.0 → euler_eval-2.33.0}/tests/test_validation.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: euler-eval
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.33.0
|
|
4
4
|
Summary: Evaluation toolkit for depth, RGB, ray and 3D point-map predictions
|
|
5
5
|
Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
|
|
6
6
|
License: MIT
|
|
@@ -133,8 +133,10 @@ euler-eval config.json --batch-size 32
|
|
|
133
133
|
|
|
134
134
|
Ground truth and prediction files are matched by ID rather than by directory
|
|
135
135
|
order, so an incomplete prediction set scores the frames it does have instead of
|
|
136
|
-
silently comparing the wrong pairs. Results are written
|
|
137
|
-
|
|
136
|
+
silently comparing the wrong pairs. Results are written into each prediction
|
|
137
|
+
dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
|
|
138
|
+
`/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
|
|
139
|
+
dataset ([details](docs/output.md#where-results-land)):
|
|
138
140
|
|
|
139
141
|
```json
|
|
140
142
|
{
|
|
@@ -83,8 +83,10 @@ euler-eval config.json --batch-size 32
|
|
|
83
83
|
|
|
84
84
|
Ground truth and prediction files are matched by ID rather than by directory
|
|
85
85
|
order, so an incomplete prediction set scores the frames it does have instead of
|
|
86
|
-
silently comparing the wrong pairs. Results are written
|
|
87
|
-
|
|
86
|
+
silently comparing the wrong pairs. Results are written into each prediction
|
|
87
|
+
dataset, under `.eval/` beside its `.ds_crawler/` metadata — e.g.
|
|
88
|
+
`/data/model_a/depth/.eval/eval.json`, or `.eval/<scope>/eval.json` for a scoped
|
|
89
|
+
dataset ([details](docs/output.md#where-results-land)):
|
|
88
90
|
|
|
89
91
|
```json
|
|
90
92
|
{
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
| [Spaces & alignment](alignment.md) | Why `native` and `metric` exist, depth affine fitting, points-3d gauge alignment, benchmark depth bins |
|
|
8
8
|
| [Metrics](metrics.md) | The full metric inventory per modality, with the key each one is written under |
|
|
9
9
|
| [Error distributions](distributions.md) | Optional per-sample and pooled RMSE histograms, bin settings, axes and JSON layout |
|
|
10
|
-
| [Results & output](output.md) | Where files land, the anatomy of `eval.json`, per-file metrics, the sanity-check report |
|
|
10
|
+
| [Results & output](output.md) | Where files land, the anatomy of `eval.json`, comparing evaluations, per-file metrics, the sanity-check report |
|
|
11
11
|
| [In-training validation](validation.md) | The programmatic API for scoring in-memory predictions during training |
|
|
12
12
|
|
|
13
13
|
## Proposed features
|
|
@@ -37,6 +37,8 @@ validation exits with status `1` before any dataset is opened.
|
|
|
37
37
|
| `--distribution-bins` | `int` | `50` | Total histogram bins, including tails; minimum 3 |
|
|
38
38
|
| `--distribution-scale` | `{log,linear}` | `log` | Spacing of finite histogram bins |
|
|
39
39
|
| `--distribution-range` | `float float` | `0.001 100` | Finite error range `[MIN, MAX)` in evaluated space units; zero/small-error and overflow bins retain the tails |
|
|
40
|
+
| `--eval-name` | `str` | `eval` | Name this evaluation: results are written as `<name>.json` in each prediction dataset's `.eval/` folder, so differently named runs sit side by side; a dataset's `output_file` still wins ([details](output.md#where-results-land)) |
|
|
41
|
+
| `--experiment` | `str` | all | Run only this experiment from the config's [`experiments`](configuration.md#experiments-section-optional) section; repeatable |
|
|
40
42
|
|
|
41
43
|
Histogram settings can also be specified in the config's `distributions`
|
|
42
44
|
section. A bin, scale or range flag enables histograms and overrides that
|
|
@@ -69,6 +71,13 @@ euler-eval config.json --sky-iou --sky-depth 100
|
|
|
69
71
|
# Only depth evaluation
|
|
70
72
|
euler-eval config.json --skip-rgb --skip-rays --skip-points-3d
|
|
71
73
|
|
|
74
|
+
# Keep a sky-masked run beside the default results: .eval/[<scope>/]no_sky.json
|
|
75
|
+
euler-eval config.json --mask-sky --eval-name no_sky
|
|
76
|
+
|
|
77
|
+
# Run every experiment in the config's "experiments" section, or just one
|
|
78
|
+
euler-eval config.json --device cuda --batch-size 32
|
|
79
|
+
euler-eval config.json --experiment no_sky
|
|
80
|
+
|
|
72
81
|
# Force affine scale+shift alignment on all depth predictions
|
|
73
82
|
euler-eval config.json --depth-alignment affine
|
|
74
83
|
|
|
@@ -127,7 +127,7 @@ would be ambiguous.
|
|
|
127
127
|
| `sky_mask.path` | no | Auxiliary predicted sky mask for a dense depth entry; used with `--sky-depth`, `--mask-sky-prediction` and `--sky-iou` |
|
|
128
128
|
| `rays.path` | no\* | Predicted ray direction map dataset |
|
|
129
129
|
| `points_3d.path` | no\* | Predicted per-pixel 3D point map `(H,W,3)` |
|
|
130
|
-
| `output_file` | no | Custom results path (default:
|
|
130
|
+
| `output_file` | no | Custom results path for every modality of this entry, replacing the `.eval/` layout and `--eval-name` (default: `.eval/[<scope>/][split_<split>/]eval.json` in each prediction modality's dataset root; see [where results land](output.md#where-results-land)) |
|
|
131
131
|
|
|
132
132
|
\* At least one prediction modality must be present.
|
|
133
133
|
|
|
@@ -214,10 +214,11 @@ and evaluated as `points_3d`.
|
|
|
214
214
|
|
|
215
215
|
An explicit `datasets[].points_3d` prediction can be evaluated separately
|
|
216
216
|
against the same sparse cloud. That evaluation uses the points-3D similarity
|
|
217
|
-
gauge, writes an atomic `points3d.eval` metric set
|
|
218
|
-
prediction, and is controlled by `--skip-points-3d`. If a dataset
|
|
219
|
-
depth and `points_3d`, each prediction modality gets its own
|
|
220
|
-
|
|
217
|
+
gauge, writes an atomic `points3d.eval` metric set to the point-map
|
|
218
|
+
prediction's results file, and is controlled by `--skip-points-3d`. If a dataset
|
|
219
|
+
provides both depth and `points_3d`, each prediction modality gets its own
|
|
220
|
+
results file, also when both live in one archive under separate scopes (see
|
|
221
|
+
[where results land](output.md#where-results-land)).
|
|
221
222
|
|
|
222
223
|
Sparse depth does not require segmentation GT. `gt.segmentation` is loaded only
|
|
223
224
|
when `--mask-sky` or `--sky-iou` is set. `--mask-sky` excludes sky pixels from
|
|
@@ -249,6 +250,63 @@ sources:
|
|
|
249
250
|
unprojected on the fly. Whether the GT depth is radial or planar is read from
|
|
250
251
|
its metadata (`radial_depth`).
|
|
251
252
|
|
|
253
|
+
## `experiments` section (optional)
|
|
254
|
+
|
|
255
|
+
Runs the same config several times in one invocation, each time with its own
|
|
256
|
+
command-line flags and under its own name:
|
|
257
|
+
|
|
258
|
+
```json
|
|
259
|
+
{
|
|
260
|
+
"gt": { "depth": { "path": "/data/gt/depth" } },
|
|
261
|
+
"datasets": [{ "name": "model_a", "depth": { "path": "/data/model_a/depth" } }],
|
|
262
|
+
"experiments": [
|
|
263
|
+
{ "name": "eval" },
|
|
264
|
+
{ "name": "no_sky", "args": ["--mask-sky", "--benchmark-depth-range", 0.01, 80] },
|
|
265
|
+
{ "name": "affine", "args": ["--depth-alignment", "affine"] },
|
|
266
|
+
{ "name": "capped_no_mask", "args": ["--sky-depth", 80], "modalities": ["depth"] }
|
|
267
|
+
]
|
|
268
|
+
}
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
| Field | Required | Description |
|
|
272
|
+
|---|---|---|
|
|
273
|
+
| `name` | yes | Evaluation name: results are written as `<name>.json` in each prediction dataset's `.eval/` folder ([where results land](output.md#where-results-land)). Same rules as `--eval-name`; unique |
|
|
274
|
+
| `args` | no | Command-line tokens this experiment adds, exactly as you would type them; numbers may be written as JSON numbers |
|
|
275
|
+
| `modalities` | no | The configured modalities this experiment may load (default: all). Names: `depth` (also covers `relative_depth` and `affine_depth` predictions), `sparse_depth`, `rgb`, `rays`, `points_3d`, `sky_mask`, `segmentation` (also covers `semantic_segmentation`), `calibration`, `intrinsics`, `camera_extrinsics`, `lidar_extrinsics` |
|
|
276
|
+
|
|
277
|
+
- **The command line is the base.** Each experiment runs with the flags given on
|
|
278
|
+
the command line followed by its own `args`, so for a flag that takes a
|
|
279
|
+
value, the experiment's value wins. Switches such as `--mask-sky` can only be
|
|
280
|
+
turned on that way, so put the ones that vary between experiments in
|
|
281
|
+
`args`, not on the command line.
|
|
282
|
+
- **Checked before any work.** Every experiment's `args` are parsed and
|
|
283
|
+
validated before the first one runs. `--eval-name` and `--experiment` are not
|
|
284
|
+
allowed in `args`, and `--eval-name` cannot be combined with an `experiments`
|
|
285
|
+
section, because `name` names the results.
|
|
286
|
+
- **Modalities.** An experiment with `modalities` runs as if the unlisted GT
|
|
287
|
+
inputs and prediction entries were not in the config; a name covers both
|
|
288
|
+
sides. A listed modality is still read only when the evaluation needs it,
|
|
289
|
+
e.g. `sky_mask` only with `--sky-depth`, `--mask-sky-prediction` or
|
|
290
|
+
`--sky-iou`. This is how to compare runs where a modality's presence changes
|
|
291
|
+
the output: above, `capped_no_mask` caps depth at 80 m without filling the
|
|
292
|
+
predicted sky. A dataset left without a prediction is not evaluated in that
|
|
293
|
+
experiment, and the reduced config must still be valid (e.g. `sparse_depth`
|
|
294
|
+
needs `intrinsics` and `camera_extrinsics`). Listing a modality the config
|
|
295
|
+
does not have only prints a warning, so one experiment list can serve several
|
|
296
|
+
generated configs.
|
|
297
|
+
- **Selecting experiments.** By default all experiments run, in config order.
|
|
298
|
+
`--experiment NAME` (repeatable) runs only the named ones, e.g. one per task
|
|
299
|
+
of a job array.
|
|
300
|
+
- **Failures.** An experiment that fails is reported and the rest still run;
|
|
301
|
+
the exit status is then `1`.
|
|
302
|
+
|
|
303
|
+
Each experiment loads and scores the data again, and its results carry the
|
|
304
|
+
settings it ran with in [`protocol.parameters`](output.md#comparing-evaluations)
|
|
305
|
+
and its name in `meta.eval_name`. With euler-train logging, each prediction
|
|
306
|
+
dataset is logged once per experiment, as `<name>/<experiment>`; the
|
|
307
|
+
sanity-check report is written per experiment as
|
|
308
|
+
`sanity_check_report_<experiment>.json`.
|
|
309
|
+
|
|
252
310
|
## `euler_train` section (optional)
|
|
253
311
|
|
|
254
312
|
When present, results are logged to an
|
|
@@ -259,7 +317,8 @@ created and finished on completion) or a full path to an existing run directory
|
|
|
259
317
|
only supported key.
|
|
260
318
|
|
|
261
319
|
Each prediction dataset is registered as an evaluation named after its config
|
|
262
|
-
`name
|
|
320
|
+
`name` (`<name>/<experiment>` with [`experiments`](#experiments-section-optional)),
|
|
321
|
+
first as `running` and then with its metrics attached, alongside package
|
|
263
322
|
provenance (euler-eval version, Python, torch, CUDA).
|
|
264
323
|
|
|
265
324
|
## Loader resolution & dataset metadata
|
|
@@ -2,20 +2,56 @@
|
|
|
2
2
|
|
|
3
3
|
## Where results land
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
the dataset whose depth path is `/data/model_a/depth`, and so on. If that path
|
|
8
|
-
passes through a `.zip`, the JSON is written *into* the archive rather than
|
|
9
|
-
beside it, so zip-backed datasets stay self-contained.
|
|
5
|
+
Results are written **per modality, per prediction dataset**, into a `.eval/`
|
|
6
|
+
folder at the prediction dataset's root, beside its `.ds_crawler/` metadata:
|
|
10
7
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
8
|
+
```text
|
|
9
|
+
/data/model_a/ # a datasets[].<modality>.path
|
|
10
|
+
.ds_crawler/
|
|
11
|
+
depth/ rgb/ # ds-crawler metadata scopes
|
|
12
|
+
.eval/
|
|
13
|
+
depth/
|
|
14
|
+
eval.json # depth results
|
|
15
|
+
no_sky.json # depth results of --eval-name no_sky
|
|
16
|
+
split_test/eval.json # depth results for "/data/model_a:test"
|
|
17
|
+
rgb/
|
|
18
|
+
eval.json # rgb results
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The folder below `.eval/` mirrors how the prediction was loaded:
|
|
22
|
+
|
|
23
|
+
| Prediction | Results file |
|
|
24
|
+
|---|---|
|
|
25
|
+
| Metadata scope with its own `.ds_crawler/<scope>/` | `.eval/<scope>/<name>.json` |
|
|
26
|
+
| Root-level `.ds_crawler/` metadata (no scope) | `.eval/<name>.json` |
|
|
27
|
+
| A split is selected (`:test` or `split`) | `split_<split>/` is added before the file name |
|
|
28
|
+
|
|
29
|
+
The scope is the path's `#scope=` selector, otherwise the config key (`depth`,
|
|
30
|
+
`relative_depth`, `rgb`, `points_3d`, …). As in euler-loading, a scope without
|
|
31
|
+
metadata of its own falls back to the root-level `.ds_crawler/`, and so do its
|
|
32
|
+
results. This way modalities and splits that share one dataset root or archive
|
|
33
|
+
each get their own file. If the root is a `.zip`, `.eval/` is written *into* the
|
|
34
|
+
archive (inside its top-level folder when the archive wraps everything in one
|
|
35
|
+
named after it), so zip-backed datasets stay self-contained.
|
|
36
|
+
|
|
37
|
+
`<name>` is `eval` unless the run is named with `--eval-name`:
|
|
38
|
+
`--mask-sky --eval-name no_sky` writes `no_sky.json` next to an earlier
|
|
39
|
+
`eval.json` instead of replacing it. Names use letters, digits, `.`, `-` and
|
|
40
|
+
`_`; `.json` is appended.
|
|
41
|
+
|
|
42
|
+
`output_file` in a dataset entry overrides all of this. It is one path for the
|
|
43
|
+
whole dataset entry: every modality evaluated for that entry is written to it,
|
|
44
|
+
and the last one written replaces the others, so use it for single-modality
|
|
45
|
+
entries.
|
|
46
|
+
|
|
47
|
+
Earlier releases wrote `eval.json` directly at the dataset root (inside the
|
|
48
|
+
archive for zips). Readers such as euler-view look in `.eval/<scope>/`, then
|
|
49
|
+
`.eval/`, then fall back to that legacy location.
|
|
14
50
|
|
|
15
51
|
Each `eval.json` is atomic and declares one metric namespace. With
|
|
16
52
|
`gt.sparse_depth`, a depth-family prediction writes only `sparsedepth.eval`; an
|
|
17
|
-
explicit `points_3d` prediction writes only `points3d.eval`
|
|
18
|
-
map. A depth prediction does not implicitly produce a second points-3D metric
|
|
53
|
+
explicit `points_3d` prediction writes only `points3d.eval` to the point
|
|
54
|
+
map's own results file. A depth prediction does not implicitly produce a second points-3D metric
|
|
19
55
|
set.
|
|
20
56
|
|
|
21
57
|
## Anatomy of `eval.json`
|
|
@@ -37,6 +73,10 @@ set.
|
|
|
37
73
|
"axes": { "space": { "…": "…" }, "category": { "…": "…" } },
|
|
38
74
|
"metricDescriptions": { "absrel": { "…": "…" } }
|
|
39
75
|
},
|
|
76
|
+
"protocol": {
|
|
77
|
+
"schemaVersion": "1.1",
|
|
78
|
+
"parameters": { "reference.gt.dataset": "muses", "method.depth_alignment": "auto_affine", "…": "…" }
|
|
79
|
+
},
|
|
40
80
|
"dataset_info": { "num_pairs": 500, "gt_name": "GT", "pred_name": "model_a" },
|
|
41
81
|
"meta": {
|
|
42
82
|
"version": "2.25.0",
|
|
@@ -56,6 +96,7 @@ set.
|
|
|
56
96
|
| Block | What it is for |
|
|
57
97
|
|---|---|
|
|
58
98
|
| `metricSet` | The euler-metric-naming envelope: producer, namespace, axis declarations and per-metric descriptions (unit, direction, bounds), plus how the spaces were resolved |
|
|
99
|
+
| `protocol` | The comparison keys: every setting that changes metric values, as a flat map ([Comparing evaluations](#comparing-evaluations)) |
|
|
59
100
|
| `dataset_info` | How many pairs were actually matched, and the display names of both sides |
|
|
60
101
|
| `meta` | Full provenance: package version, device, both source paths and their resolved dimensions, modality metadata, and the effective evaluation flags |
|
|
61
102
|
| *metric tree* | The metrics themselves, under the modality root (`depth`, `rgb`, `rays`, `sparsedepth`, `points3d`) |
|
|
@@ -137,6 +178,93 @@ because these keys have no depth space. `metricDescriptions` gains `iou`,
|
|
|
137
178
|
`precision` and `recall` (higher is better, bounds `[0, 1]`). Every other key
|
|
138
179
|
decomposes exactly as before.
|
|
139
180
|
|
|
181
|
+
## Comparing evaluations
|
|
182
|
+
|
|
183
|
+
`protocol.parameters` says how an evaluation was set up, as one flat map of
|
|
184
|
+
dotted keys. Its keys are the comparison keys: two evaluations of the same
|
|
185
|
+
modality were set up the same way when their maps are equal, and the keys whose
|
|
186
|
+
values differ say where. Nothing else in the file needs to be compared for that;
|
|
187
|
+
`meta` stays the full record of the run, paths and device included.
|
|
188
|
+
|
|
189
|
+
```json
|
|
190
|
+
"protocol": {
|
|
191
|
+
"schemaVersion": "1.1",
|
|
192
|
+
"parameters": {
|
|
193
|
+
"reference.gt.modality": "sparse_depth",
|
|
194
|
+
"reference.gt.dataset": "seeingthroughfog",
|
|
195
|
+
"reference.gt.scope": "sparse_depth",
|
|
196
|
+
"reference.gt.split": "test",
|
|
197
|
+
"reference.intrinsics.dataset": "seeingthroughfog",
|
|
198
|
+
"reference.intrinsics.scope": "intrinsics",
|
|
199
|
+
"reference.intrinsics.split": "test",
|
|
200
|
+
"reference.camera_extrinsics.dataset": "seeingthroughfog",
|
|
201
|
+
"reference.camera_extrinsics.scope": "camera_extrinsics",
|
|
202
|
+
"reference.camera_extrinsics.split": "test",
|
|
203
|
+
"reference.lidar_extrinsics.dataset": null,
|
|
204
|
+
"reference.lidar_extrinsics.scope": null,
|
|
205
|
+
"reference.lidar_extrinsics.split": null,
|
|
206
|
+
"reference.segmentation.dataset": null,
|
|
207
|
+
"reference.segmentation.scope": null,
|
|
208
|
+
"reference.segmentation.split": null,
|
|
209
|
+
"population.samples": 943,
|
|
210
|
+
"population.gt_sky_masking": false,
|
|
211
|
+
"population.pred_sky_masking": false,
|
|
212
|
+
"population.evaluated_resolution": [1024, 1920],
|
|
213
|
+
"population.benchmark_depth_range_m": [0.01, 80.0],
|
|
214
|
+
"method.metrics_revision": 1,
|
|
215
|
+
"method.depth_alignment": "auto_affine",
|
|
216
|
+
"method.sky_depth_m": null,
|
|
217
|
+
"method.pred_sky_mask": false
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
The first segment of a key is its section:
|
|
223
|
+
|
|
224
|
+
| Section | Holds |
|
|
225
|
+
|---|---|
|
|
226
|
+
| `reference` | The GT the prediction is scored against (`gt`, with its config key as `modality`) and every other GT input the modality reads, each identified by `dataset` (the `dataset.id` of its `dataset-head.json`), metadata `scope` and `split` — not by file path, so the same data on another machine matches |
|
|
227
|
+
| `population` | Which samples and pixels were scored: the matched `samples`, sky masking, the grid the metrics were computed on and benchmark depth bins |
|
|
228
|
+
| `method` | How they were scored: alignment modes, the sky-depth cap and whether the prediction's `sky_mask` was used, how GT and prediction resolutions were reconciled, the FID backend and `metrics_revision` |
|
|
229
|
+
|
|
230
|
+
What is recorded, per modality:
|
|
231
|
+
|
|
232
|
+
| Modality | `reference` inputs besides `gt` | `population` | `method` |
|
|
233
|
+
|---|---|---|---|
|
|
234
|
+
| `depth` | `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `pred_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask`, `spatial_alignment` |
|
|
235
|
+
| `sparsedepth` | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | as `depth` | `depth_alignment`, `sky_depth_m`, `pred_sky_mask` |
|
|
236
|
+
| `rgb` | `depth`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution`, `benchmark_depth_range_m` | `fid_backend`, `spatial_alignment` |
|
|
237
|
+
| `rays` | `calibration` | `samples`, `evaluated_resolution` | `spatial_alignment` |
|
|
238
|
+
| `points3d` (dense) | `intrinsics`, `calibration`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment`, `spatial_alignment` |
|
|
239
|
+
| `points3d` (sparse GT) | `intrinsics`, `camera_extrinsics`, `lidar_extrinsics`, `segmentation` | `samples`, `gt_sky_masking`, `evaluated_resolution` | `points_3d_alignment` |
|
|
240
|
+
|
|
241
|
+
Every modality also has `method.metrics_revision`.
|
|
242
|
+
|
|
243
|
+
Rules that keep comparisons a plain map comparison:
|
|
244
|
+
|
|
245
|
+
- Only settings that change the value of a metric both files contain are
|
|
246
|
+
parameters. `--domain`, `--sky-iou` and histogram settings only add metrics
|
|
247
|
+
(histograms describe their bins in the distribution registry); device, batch
|
|
248
|
+
size, workers, paths and `--eval-name` do not change values.
|
|
249
|
+
- Every key of a modality is always written, `null` or `false` when off, so a
|
|
250
|
+
key missing from one side means "not recorded", never "off". Compare the union
|
|
251
|
+
of both key sets.
|
|
252
|
+
- Values are scalars or lists of scalars, never objects.
|
|
253
|
+
- `method.metrics_revision` changes only when a release changes metric values
|
|
254
|
+
for identical inputs and settings, which the [changelog](../CHANGELOG.md)
|
|
255
|
+
then says. Different revisions are not comparable even when every other
|
|
256
|
+
key matches.
|
|
257
|
+
- Settings are recorded, not what they caused. `population.pred_sky_masking`
|
|
258
|
+
is the `--mask-sky-prediction` flag, and `method.pred_sky_mask` says whether
|
|
259
|
+
the prediction's `sky_mask` was loaded and scored with (filled at
|
|
260
|
+
`--sky-depth`, or excluded by `--mask-sky-prediction`). It is `false` for a
|
|
261
|
+
prediction without one, or an [experiment](configuration.md#experiments-section-optional)
|
|
262
|
+
that leaves it out. How much each model's own sky removed is in
|
|
263
|
+
`meta.pred_sky_masking`, and `metricSet.metadata` records whether an
|
|
264
|
+
alignment was actually applied.
|
|
265
|
+
- Renaming or removing a key bumps the major `schemaVersion`; adding one bumps
|
|
266
|
+
the minor version.
|
|
267
|
+
|
|
140
268
|
## Spaces and canonical aliases
|
|
141
269
|
|
|
142
270
|
Depth-like and points-3d results are nested by space:
|