euler-preprocess 3.18.0__tar.gz → 3.19.0__tar.gz

euler_preprocess-3.19.0/CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+## Unreleased
+
+- Add `euler-preprocess lidar-motion`, which compensates scanning-lidar points for
+  the vehicle's motion during the sweep. It follows the MUSES SDK, moving each
+  point to the frame camera's mid-exposure using the `gnss`, `sensor_timing`
+  and `lidar_to_gnss` modalities.
+- Require euler-loading 2.32.0, the first release with `muses.gnss`,
+  `muses.sensor_timing`, `muses.read_lidar_to_gnss` and `muses.write_sparse_depth`.
+- Add `euler-preprocess crop`, which crops RGB, depth, intrinsics and
+  extrinsics coherently. The box is given as fractions or pixels, either removed
+  per edge or kept with an anchor, relative to a reference raster. Rasters are
+  sliced, pinhole intrinsics get `cx -= left`, `cy -= top` merged into a copy of
+  their file, and lidar point clouds and extrinsics are copied unchanged. Each
+  modality's treatment comes from its loader annotation, not its name. Shared
+  calibration files are written once, and samples with different crops cannot
+  share one. The outputs are reopened and checked against the crop.
+  `configs/crop_stf_valid_region_config.json` removes SeeingThroughFog's black
+  rectification border.
+- `crop` phase 3:
+  - `point_clouds: "prune_to_crop"` keeps only the lidar points that project
+    into the output image (`z > 0`, `rint(K'·T·X)` inside), the projection
+    consumers such as dmd3c-euler use;
+  - `resize` (`size` or `scale`) after the crop, with `K' = S·C·K`; colour
+    images are interpolated bilinearly, all other rasters take the nearest
+    pixel;
+  - `"output_layout": "scoped"` writes one root (directory or zip) with one
+    metadata scope per modality, as dmd3c-euler's `--dataset-root` reads;
+  - every encoded raster or pruned cloud is decoded again and must reproduce
+    the values;
+  - `euler-preprocess crop-box` prints the pixel box and the matching
+    `spatial` preprocessing block.
+- `crop` uses the new public euler-loading accessors (`get_loader_meta`,
+  `get_hierarchical_files`) and ds-crawler's `set_modality_meta` / `path_prefix`.
+  Requires euler-loading 2.33.0 and ds-crawler 2.14.0.
+- `extract_intrinsics` (radial, fog) also uses a hierarchical intrinsics file
+  that is not named `intrinsics` when it is the only one, such as SeeingThroughFog's
+  `calib_cam_stereo_left`. Several matching files now raise instead of being
+  ignored.
+
+## 3.14.0 (unreleased)
+
+Consume shared preprocessing config and add the opt-in spatial producer, explicit variants/mappings and captured source-backed outputs. Preserve ordinary pipeline APIs.

{euler_preprocess-3.18.0/euler_preprocess.egg-info → euler_preprocess-3.19.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 3.18.0
+Version: 3.19.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License-Expression: MIT
@@ -9,8 +9,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: Pillow
-Requires-Dist: euler-loading>=2.32.0
-Requires-Dist: ds-crawler>=2.11.0
+Requires-Dist: euler-loading>=2.33.0
+Requires-Dist: ds-crawler>=2.14.0
 Requires-Dist: euler-dataset-contract>=0.5.0
 Provides-Extra: gpu
 Requires-Dist: torch; extra == "gpu"
@@ -65,6 +65,8 @@ qualitative review.
 | `euler-preprocess sky-depth` | Override depth values in sky regions with a constant |
 | `euler-preprocess radial` | Convert planar (z-buffer) depth to radial (Euclidean) depth |
 | `euler-preprocess lidar-motion` | Compensate scanning-lidar points for the vehicle's motion during a sweep |
+| `euler-preprocess crop` | Crop RGB, depth, intrinsics and extrinsics coherently |
+| `euler-preprocess crop-box` | Print the pixel box a crop config gives for an image size |
 
 ## Installation
 
@@ -79,6 +81,7 @@ euler-preprocess fog       -c configs/example_dataset_config.json
 euler-preprocess sky-depth -c configs/sky_depth_dataset_config.json
 euler-preprocess radial    -c configs/radial_dataset_config.json
 euler-preprocess lidar-motion -c configs/lidar_motion_dataset_config.json
+euler-preprocess crop      -c configs/crop_stf_dataset_config.json
 ```
 
 ## Dataset Config
@@ -737,6 +740,110 @@ holds the target time and the speed and rates used. Point
 
 ---
 
+## Coherent Crop
+
+`crop` cuts the same box out of every image-shaped modality and corrects
+everything that refers to the pixel grid, so a cropped bundle projects exactly
+like the original:
+
+| Modality kind | Crop |
+|---|---|
+| Rasters: RGB, dense depth, masks | Sliced; no resampling, so values are unchanged |
+| Pinhole intrinsics `K` (or `P` for rectified images) | `cx -= left`, `cy -= top`; `fx`, `fy`, skew unchanged |
+| Lidar point clouds | Copied byte for byte, or pruned to the points that project into the crop |
+| Extrinsics | Copied byte for byte |
+
+The extrinsics need no change, because `C·K·[R|t]·X = K'·[R|t]·X` with
+`C = [[1,0,-left],[0,1,-top],[0,0,1]]`: the whole correction lives in the
+camera matrix, which is also what the official SeeingThroughFog tools do. An
+optional resize afterwards adds `S` (an `align_corners=False` rescale), so
+`K' = S·C·K`, still with untouched extrinsics.
+Which kind a modality is comes from its euler-loading loader annotation
+(`shape`, `type`), never from its name.
+
+The dataset config lists every modality to crop; whether a modality is
+per-sample or hierarchical (shared calibration) is inferred from its index:
+
+```json
+{
+  "transform_config_path": "crop_stf_valid_region_config.json",
+  "output_path": "/data/stf_crop",
+  "modalities": {
+    "rgb": "/data/stf/cam_stereo_left_lut",
+    "sparse_depth": "/data/stf/lidar_hdl64_strongest",
+    "intrinsics": "/data/stf/calib_cam_stereo_left",
+    "camera_extrinsics": "/data/stf/calib_tf_tree"
+  },
+  "samples": {"start": 0, "count": 100}
+}
+```
+
+Each modality is written to `<output_path>/<name>`, or to its
+`pipeline.output_targets[]` entry (`slot` = the modality name; directory or zip).
+With pipeline targets, every modality needs one and the outputs manifest lists
+them all. `"output_layout": "scoped"` writes one root at `output_path` instead
+(a directory or `.zip`): one ds-crawler metadata scope per modality, named after
+it, with its files below `<name>/`. That is the layout dmd3c-euler's
+`--dataset-root` reads. The dataset config must not contain `preprocessing`.
+
+The crop config (`configs/crop_config.json`) describes the box relative to the
+`reference` raster:
+
+```json
+{
+  "reference": "rgb",
+  "crop": {"remove": {"top": 0.20, "bottom": 0.05, "left": 0.15, "right": 0.19}},
+  "point_clouds": "keep",
+  "verify": "all"
+}
+```
+
+| Key | Meaning |
+|---|---|
+| `crop.remove` | What to cut off each edge (`top`, `bottom`, `left`, `right`) |
+| `crop.keep` + `crop.anchor` | The kept `height` / `width`, placed at `center` (default), `top`, `bottom`, `left`, `right`, `top_left`, `top_right`, `bottom_left` or `bottom_right` |
+| `crop.unit` | `fraction` (default) or `pixel`. Fractions lie in `[0, 1)` (`keep`: `(0, 1]`) and round half up per edge: `floor(f · size + 0.5)` |
+| `crop.multiple_of` | Trim the kept size down to a multiple, e.g. a network stride; the extra rows and columns come off both sides evenly |
+| `resize` | Optional, after the crop: `{"size": [h, w]}` or `{"scale": 0.5}` (or `[sy, sx]`); `interpolation` maps modality names to `nearest`, `bilinear` or `bicubic` |
+| `point_clouds` | `keep` (default): copy every point, consumers clip to the image. `prune_to_crop`: keep only the points with `z > 0` whose pixel `rint(K'·T·X)` lies in the output image, as consumers such as dmd3c-euler project them |
+| `projection` | For `prune_to_crop`: the `intrinsics` and `extrinsics` modalities to project with, when the dataset has more than one candidate |
+| `verify` | `all` (default), `none`, or a sample count: reopen the outputs and check every file against the crop (rasters and pruned clouds as encoded, intrinsics to 1e-3 px, copies byte-identical) |
+| `roles` | Override a modality's kind with `raster` or `copy`, for loaders whose annotation the crop does not know |
+
+Pixel units reproduce the official boxes exactly, e.g. `RGB2Gatedv2` is
+`{"remove": {"top": 202, "bottom": 54, "left": 280, "right": 360}, "unit": "pixel"}`.
+`configs/crop_stf_valid_region_config.json` removes the black border the
+SeeingThroughFog rectification leaves on `_lut` frames (top 34, bottom 46,
+left 16 px), keeping a fully valid 1904×944 image.
+
+Rasters are re-encoded with the dataset's euler-loading writer, so a raster
+modality needs one (SeeingThroughFog's raw `rccb` TIFFs have none). Each
+encoded file is decoded again right away and must give back the cropped values:
+exactly, or for an interpolated resize within half an 8-bit step. A resize
+interpolates colour images (`rgb`, `rccb`) bilinearly and takes the nearest
+pixel for everything else, so depth and label values stay real values;
+sparse-depth rasters are not resized. Pruned point clouds need a writer too
+(`princeton_dense.write_sparse_depth`, `muses.write_sparse_depth`). Intrinsics
+are written by copying the source file and letting its writer move the matrices
+onto the new grid: for a ROS CameraInfo file such as
+`calib_cam_stereo_left.json`, `K` and `P` shift by exact integers, `width` and
+`height` become the crop size, and `D`, `R`, `roi` and the header are kept. A
+calibration file that records another image size than the reference rasters
+is refused. A hierarchical file is written once, so every sample sharing it
+must get the same box; mixed image sizes under one calibration file are an
+error. Every output head records `meta.crop` (reference, spec and boxes), and
+every file its own `attributes.crop` box; pruned clouds also record
+`attributes.point_clouds` (source and kept point counts).
+
+To reproduce a crop with `euler-preprocess spatial`, print its pixel box and
+the matching `preprocessing` block for a reference size:
+
+```bash
+euler-preprocess crop-box -c configs/crop_config.json --size 1024 1920
+```
+
+---
+
 ## License
 
 [MIT](LICENSE) © Daniel Rothenpieler

{euler_preprocess-3.18.0 → euler_preprocess-3.19.0}/README.md

@@ -45,6 +45,8 @@ qualitative review.
 | `euler-preprocess sky-depth` | Override depth values in sky regions with a constant |
 | `euler-preprocess radial` | Convert planar (z-buffer) depth to radial (Euclidean) depth |
 | `euler-preprocess lidar-motion` | Compensate scanning-lidar points for the vehicle's motion during a sweep |
+| `euler-preprocess crop` | Crop RGB, depth, intrinsics and extrinsics coherently |
+| `euler-preprocess crop-box` | Print the pixel box a crop config gives for an image size |
 
 ## Installation
 
@@ -59,6 +61,7 @@ euler-preprocess fog       -c configs/example_dataset_config.json
 euler-preprocess sky-depth -c configs/sky_depth_dataset_config.json
 euler-preprocess radial    -c configs/radial_dataset_config.json
 euler-preprocess lidar-motion -c configs/lidar_motion_dataset_config.json
+euler-preprocess crop      -c configs/crop_stf_dataset_config.json
 ```
 
 ## Dataset Config
@@ -717,6 +720,110 @@ holds the target time and the speed and rates used. Point
 
 ---
 
+## Coherent Crop
+
+`crop` cuts the same box out of every image-shaped modality and corrects
+everything that refers to the pixel grid, so a cropped bundle projects exactly
+like the original:
+
+| Modality kind | Crop |
+|---|---|
+| Rasters: RGB, dense depth, masks | Sliced; no resampling, so values are unchanged |
+| Pinhole intrinsics `K` (or `P` for rectified images) | `cx -= left`, `cy -= top`; `fx`, `fy`, skew unchanged |
+| Lidar point clouds | Copied byte for byte, or pruned to the points that project into the crop |
+| Extrinsics | Copied byte for byte |
+
+The extrinsics need no change, because `C·K·[R|t]·X = K'·[R|t]·X` with
+`C = [[1,0,-left],[0,1,-top],[0,0,1]]`: the whole correction lives in the
+camera matrix, which is also what the official SeeingThroughFog tools do. An
+optional resize afterwards adds `S` (an `align_corners=False` rescale), so
+`K' = S·C·K`, still with untouched extrinsics.
+Which kind a modality is comes from its euler-loading loader annotation
+(`shape`, `type`), never from its name.
+
+The dataset config lists every modality to crop; whether a modality is
+per-sample or hierarchical (shared calibration) is inferred from its index:
+
+```json
+{
+  "transform_config_path": "crop_stf_valid_region_config.json",
+  "output_path": "/data/stf_crop",
+  "modalities": {
+    "rgb": "/data/stf/cam_stereo_left_lut",
+    "sparse_depth": "/data/stf/lidar_hdl64_strongest",
+    "intrinsics": "/data/stf/calib_cam_stereo_left",
+    "camera_extrinsics": "/data/stf/calib_tf_tree"
+  },
+  "samples": {"start": 0, "count": 100}
+}
+```
+
+Each modality is written to `<output_path>/<name>`, or to its
+`pipeline.output_targets[]` entry (`slot` = the modality name; directory or zip).
+With pipeline targets, every modality needs one and the outputs manifest lists
+them all. `"output_layout": "scoped"` writes one root at `output_path` instead
+(a directory or `.zip`): one ds-crawler metadata scope per modality, named after
+it, with its files below `<name>/`. That is the layout dmd3c-euler's
+`--dataset-root` reads. The dataset config must not contain `preprocessing`.
+
+The crop config (`configs/crop_config.json`) describes the box relative to the
+`reference` raster:
+
+```json
+{
+  "reference": "rgb",
+  "crop": {"remove": {"top": 0.20, "bottom": 0.05, "left": 0.15, "right": 0.19}},
+  "point_clouds": "keep",
+  "verify": "all"
+}
+```
+
+| Key | Meaning |
+|---|---|
+| `crop.remove` | What to cut off each edge (`top`, `bottom`, `left`, `right`) |
+| `crop.keep` + `crop.anchor` | The kept `height` / `width`, placed at `center` (default), `top`, `bottom`, `left`, `right`, `top_left`, `top_right`, `bottom_left` or `bottom_right` |
+| `crop.unit` | `fraction` (default) or `pixel`. Fractions lie in `[0, 1)` (`keep`: `(0, 1]`) and round half up per edge: `floor(f · size + 0.5)` |
+| `crop.multiple_of` | Trim the kept size down to a multiple, e.g. a network stride; the extra rows and columns come off both sides evenly |
+| `resize` | Optional, after the crop: `{"size": [h, w]}` or `{"scale": 0.5}` (or `[sy, sx]`); `interpolation` maps modality names to `nearest`, `bilinear` or `bicubic` |
+| `point_clouds` | `keep` (default): copy every point, consumers clip to the image. `prune_to_crop`: keep only the points with `z > 0` whose pixel `rint(K'·T·X)` lies in the output image, as consumers such as dmd3c-euler project them |
+| `projection` | For `prune_to_crop`: the `intrinsics` and `extrinsics` modalities to project with, when the dataset has more than one candidate |
+| `verify` | `all` (default), `none`, or a sample count: reopen the outputs and check every file against the crop (rasters and pruned clouds as encoded, intrinsics to 1e-3 px, copies byte-identical) |
+| `roles` | Override a modality's kind with `raster` or `copy`, for loaders whose annotation the crop does not know |
+
+Pixel units reproduce the official boxes exactly, e.g. `RGB2Gatedv2` is
+`{"remove": {"top": 202, "bottom": 54, "left": 280, "right": 360}, "unit": "pixel"}`.
+`configs/crop_stf_valid_region_config.json` removes the black border the
+SeeingThroughFog rectification leaves on `_lut` frames (top 34, bottom 46,
+left 16 px), keeping a fully valid 1904×944 image.
+
+Rasters are re-encoded with the dataset's euler-loading writer, so a raster
+modality needs one (SeeingThroughFog's raw `rccb` TIFFs have none). Each
+encoded file is decoded again right away and must give back the cropped values:
+exactly, or for an interpolated resize within half an 8-bit step. A resize
+interpolates colour images (`rgb`, `rccb`) bilinearly and takes the nearest
+pixel for everything else, so depth and label values stay real values;
+sparse-depth rasters are not resized. Pruned point clouds need a writer too
+(`princeton_dense.write_sparse_depth`, `muses.write_sparse_depth`). Intrinsics
+are written by copying the source file and letting its writer move the matrices
+onto the new grid: for a ROS CameraInfo file such as
+`calib_cam_stereo_left.json`, `K` and `P` shift by exact integers, `width` and
+`height` become the crop size, and `D`, `R`, `roi` and the header are kept. A
+calibration file that records another image size than the reference rasters
+is refused. A hierarchical file is written once, so every sample sharing it
+must get the same box; mixed image sizes under one calibration file are an
+error. Every output head records `meta.crop` (reference, spec and boxes), and
+every file its own `attributes.crop` box; pruned clouds also record
+`attributes.point_clouds` (source and kept point counts).
+
+To reproduce a crop with `euler-preprocess spatial`, print its pixel box and
+the matching `preprocessing` block for a reference size:
+
+```bash
+euler-preprocess crop-box -c configs/crop_config.json --size 1024 1920
+```
+
+---
+
 ## License
 
 [MIT](LICENSE) © Daniel Rothenpieler

{euler_preprocess-3.18.0 → euler_preprocess-3.19.0}/docs/plans/coherent-cropping.md

@@ -1,8 +1,12 @@
 # Coherent cropping plan
 
-Status: plan only, nothing implemented yet. First target: **SeeingThroughFog**
-(euler-loading's `princeton_dense` loader) with `rgb`, `sparse_depth`,
-`intrinsics` and `camera_extrinsics`. Written 2026-11-06 against
+Status: **phases 1 and 3 implemented** (2026-11-07) as `euler-preprocess crop`
+(plus `crop-box`), for **SeeingThroughFog** (euler-loading's `princeton_dense`
+loader) with `rgb`, `sparse_depth`, `intrinsics` and `camera_extrinsics`; see §5
+for results. Phase 2 (other datasets) is open. The euler-loading side shipped in
+2.31.0 (STF fixes) and 2.32.0 (MUSES inputs); phase 3 uses euler-loading 2.33.0
+and ds-crawler 2.14.0 (both released), which euler-preprocess now requires.
+euler-view launches it with the `euler-preprocess-crop` addon. Written 2026-11-06 against
 euler-preprocess 3.17.0 and euler-loading 2.30.0.
 
 ## 1. Is there a pathway today?
@@ -72,10 +76,10 @@ two different K. SeeingThroughFog keeps them in separate files.
 | `intrinsics` | `calib_cam_stereo_left.json`, one file, matched hierarchically | `read_intrinsics` (3×3) | ROS CameraInfo: `K, D, R, P, width, height, roi, …` | intrinsics | Patched copy via `princeton_dense.write_intrinsics`: K and P `cx`/`cy`, `width`, `height` |
 | `camera_extrinsics` | `calib_tf_tree_full.json`, one file | `read_extrinsics` (4×4, HDL64 → `cam_stereo_left_optical`) | ROS TF list | unchanged | Byte copy |
 
-The byte copies need no writer. `princeton_dense.write_intrinsics` now exists
-in the euler-loading working tree (§3.2), so the intrinsics patch is covered.
+The byte copies need no writer. `princeton_dense.write_intrinsics` (euler-loading
+2.31.0, §3.2) covers the intrinsics patch.
 
-### 3.2 Fixed in euler-loading (not yet committed): `read_intrinsics` returned the wrong matrix for `_lut` images
+### 3.2 Fixed in euler-loading 2.31.0: `read_intrinsics` returned the wrong matrix for `_lut` images
 
 - `princeton_dense.read_intrinsics` returns `K`
   (`euler_loading/loaders/_princeton_dense.py:170`).
@@ -91,7 +95,7 @@ in the euler-loading working tree (§3.2), so the intrinsics patch is covered.
   about ±55 px at the top and bottom.
 - This is independent of cropping, but a crop corrects whichever matrix the
   loader returns, so it had to be settled first.
-- Done in the euler-loading working tree: `read_intrinsics` now returns the
+- Released in euler-loading 2.31.0: `read_intrinsics` now returns the
   rectified `P[:, :3]` by default (a behaviour change, in its CHANGELOG).
   `intrinsics_matrix: "K"` (alias `"raw"`) in per-file attributes or head
   meta selects the raw K.
@@ -123,8 +127,8 @@ in the euler-loading working tree (§3.2), so the intrinsics patch is covered.
 - Raw TIFF datasets (`princeton_stf+rgb`):
   - `princeton_dense.rgb` silently returns an all-white image for them: Pillow
     clips 12-bit data. Their head must name `rccb`.
-  - `rccb` swapped red and blue; it now demosaics as GRBG (fixed in the
-    euler-loading working tree).
+  - `rccb` swapped red and blue; it now demosaics as GRBG (fixed in
+    euler-loading 2.31.0).
   - The raw frames are not pinhole images (k1 = −0.57), so use the `_lut`
     images for depth evaluation.
 - The official crops are pixel boxes, e.g. `RGB2Gatedv2` keeps rows 202–970 and
@@ -136,8 +140,10 @@ in the euler-loading working tree (§3.2), so the intrinsics patch is covered.
   `remove: {top: 34, bottom: 46, left: 16, right: 0}` (pixel units) leaves a
   fully valid 1904×944 image (91.4 % of the frame). That's a sensible default
   crop for depth evaluation, or else mask those pixels. Computed from
-  `calib_cam_stereo_left.json` and the official conversion code; confirm on a
-  real frame.
+  `calib_cam_stereo_left.json` and the official conversion code, and confirmed
+  on two real dense-fog frames: the black pixels reach 33 rows at the top, 45 at
+  the bottom and 15 columns on the left, and none remain inside the box. Shipped
+  as `configs/crop_stf_valid_region_config.json`.
 - Resampling in the official rectification is mild but uneven: about 1.11 raw
   px per output px at the centre, but 0.76–0.81 near the left and right edges
   (≈1.3× upsampled, so softer there). The `_lut` tone mapping also applies
@@ -291,6 +297,9 @@ modality name, which is what causes the hazard in §1.
 - `euler_preprocess/crop/calibration.py`: hierarchical writer (unique-box
   check, copy then merge).
 - `euler_preprocess/crop/verify.py`.
+- `euler_preprocess/crop/files.py` (added while implementing): zip-aware reads
+  of source files, value fingerprints for the verify step, and the
+  recorded-image-size check for JSON calibration files.
 - `euler_preprocess/common/output.py`: byte-copy registration helper, and a
   source-backed backend factory for any modality with head-meta overrides.
 - `euler_preprocess/common/intrinsics.py`: `single_calibration(value)`. A
@@ -300,17 +309,45 @@ modality name, which is what causes the hazard in §1.
 - `euler_preprocess/cli.py`: a `crop` subcommand with its own `_cmd_crop`, like
   `_cmd_spatial`, because the config drives the modality set.
 - `configs/crop_config.json`, `configs/crop_stf_dataset_config.json`, a README
-  section and a CHANGELOG entry. Bump `pyproject.toml` to
-  `euler-loading>=2.31.0`.
+  section and a CHANGELOG entry. The `euler-loading` pin (`>=2.32.0`, raised
+  for `lidar-motion`) already covers 2.31.0.
 - Optional `tools/check_crop_projection.py`: overlay projected lidar on the
   cropped RGB for a few frames, as a visual K-vs-P and crop check.
 
 ## 5. Work plan
 
-### Phase 1: SeeingThroughFog end to end
+### Phase 1: SeeingThroughFog end to end — implemented 2026-11-07
 
-euler-loading (done in the working tree, uncommitted; ships with the next
-release):
+Results:
+- **Tests:** `tests/test_crop.py` (STF-format fixture: directory and zip
+  outputs with the pipeline manifest, sample selection, the hierarchical
+  conflict, a calibration for another resolution, a raster without a writer,
+  unsafe configs), `test_crop_spec.py`, `test_crop_roles.py`,
+  `test_intrinsics.py`. Full suite 273 passed, 3 skipped. The opt-in real-data
+  test (`EULER_PREPROCESS_STF_ROOT`) passes on two real frames.
+- **Real frames** (two dense-fog `_lut` frames, valid-region crop): every §3.4
+  criterion holds. RGB is the bit-exact slice, lidar and the TF tree are
+  byte-identical, and `K`/`P` shift by exact float64 integers with `D`, `R`,
+  `roi` and the header untouched. Projections with the cropped `P` agree with the
+  shifted originals to 1e-10 px.
+- **dmd3c-euler:** its own `build_dataset` loads the cropped roots unchanged, and
+  its `project_sparse_depth` raster equals the crop of the uncropped projection
+  (0 differing pixels). DMD3C inference itself was not run (no checkpoint or GPU
+  available).
+
+Notes from implementing:
+- euler-loading's datasets resolve the GPU (torch, CHW) loader variant, so the
+  crop slices CHW tensors in practice; the role's layout comes from the
+  annotation of that loader.
+- Each output head's `meta.crop` holds `reference`, `spec` and `boxes` (one
+  box per reference size), set at finalize. `meta.dimensions` follows the crop
+  when present.
+- With `pipeline.output_targets`, every modality needs a target (slot = name,
+  `datasetType` must match the source); the manifest lists them all and is
+  written after verification passes.
+- Outputs inside a source dataset are refused.
+
+euler-loading (released in 2.31.0):
 
 1. `princeton_dense.write_intrinsics` (CPU and GPU share one function) merges
    into an existing ROS CameraInfo JSON, as described in §3.2.
@@ -369,9 +406,11 @@ Verified with the real side data:
 - The GNSS timestamp precedes mid-exposure by up to 16.6 ms, so `meta.json`'s
   exposure times are needed.
 
-Implemented (2026-11-07; uncommitted in both repos). The side data became
-ordinary euler-loading modalities, consumed by euler-preprocess. ds-crawler is
-unchanged: it only interprets paths, never file contents.
+Implemented 2026-11-07 and released (euler-loading 2.32.0; euler-preprocess
+`lidar-motion`, plus the euler-view addon `euler-preprocess-lidar-motion`). The
+side data became ordinary euler-loading modalities, consumed by
+euler-preprocess. ds-crawler is unchanged: it only interprets paths, never file
+contents.
 
 | Modality | ds-crawler indexing | euler-loading |
 |---|---|---|
@@ -389,26 +428,60 @@ Results:
 - **Real data:** on the three real frames the CLI output reproduces the
   verified pole straightening.
 
-Registered in euler-dataset-contract (uncommitted) as `kinematics.platform.state`,
-`timing.platform.acquisition` and `geometry.platform.extrinsics`. Still open:
-release euler-loading with these loaders, then raise euler-preprocess's
-`euler-loading` pin.
+Registered in euler-dataset-contract as `kinematics.platform.state`,
+`timing.platform.acquisition` and `geometry.platform.extrinsics`.
+euler-preprocess requires `euler-loading>=2.32.0`, the first release with these
+loaders.
 
 The side data stays with the local dataset: MUSES license §2.3 forbids
 distributing it.
 
-### Phase 3: optional
+### Phase 3 — implemented 2026-11-07
+
+- `point_clouds: "prune_to_crop"`: keeps the points with `z > 0` whose pixel
+  `rint(K'·T·X)` lies in the output image, the projection dmd3c-euler's
+  `project_sparse_depth` uses, with a 1e-6 px margin so a rounding boundary
+  never drops a visible point. Intrinsics and extrinsics are found by role
+  (`projection` names them when ambiguous). Needed a SeeingThroughFog writer:
+  `princeton_dense.write_sparse_depth` (euler-loading). Per-file
+  `attributes.point_clouds` records the source and kept counts.
+- A single scoped output root (`"output_layout": "scoped"`): one metadata scope
+  per modality, its files below `<name>/` (ds-crawler writer `path_prefix`).
+  Zip roots are staged as a directory and packed once, since one zip writer
+  owns an archive. dmd3c's `--dataset-root` reads it.
+- Resize after the crop (`resize: {size | scale}`): `K' = S·C·K` through
+  euler-loading's `crop_intrinsics` and `resize_intrinsics`. Rasters go through
+  the new public `euler_loading.resize_raster`, the same code as the online
+  `Resize`: bilinear for `rgb`/`rccb`, nearest otherwise (depth, masks, labels).
+  Sparse-depth rasters are refused.
+- `euler-preprocess crop-box -c <crop config> --size H W` prints the box and a
+  `spatial` `preprocessing` block (`crop` with `size`/`offset`, then `resize`).
+- Every encoded raster or pruned cloud is decoded right away with the
+  modality's loader and must match (exactly, or within half an 8-bit step after
+  interpolation); the verify step then compares the reloaded datasets with those
+  decoded values.
+
+The two private hooks of phase 1 became public APIs:
+- euler-loading: `MultiModalDataset.get_loader`, `get_loader_meta` and
+  `get_hierarchical_files`, plus `get_modality_meta(loader)`;
+- ds-crawler: `DatasetWriter.set_modality_meta` (and `path_prefix`).
+euler-preprocess requires euler-loading 2.33.0 and ds-crawler 2.14.0.
+
+Results:
+- **Tests:** euler-preprocess 292 passed, 3 skipped; euler-loading 1022 passed;
+  ds-crawler's new writer tests pass.
+- **Real STF frames** (valid-region crop):
+  - pruning keeps 7.4 % of the HDL64 points;
+  - dmd3c-euler's projection of the pruned clouds equals the projection of
+    all points through the output camera, for the crop alone and with a
+    0.5x resize (472×952).
+
+An euler-view addon (`euler-preprocess-crop`) launches the crop for a dataset
+family.
 
-- `point_clouds: "prune_to_crop"`: drop points that project outside the box.
-  This needs sparse-depth writers, is lossy, and must use the same projection as
-  the consumers.
-- A single scoped output root (one `metadata_scope` per modality), matching
-  dmd3c's `--dataset-root`.
-- Resize, with the same maths: `K' = S·C·K`. The official `Sweden2PSMNet` mode
-  halves 1920×1024.
-- A percentage helper for `spatial` that emits `crop: {size, offset}`.
+## 6. Decisions
 
-## 6. Decisions needed
+Phase 1 adopted every recommendation below.
 
 | # | Decision | Recommendation |
 |---|---|---|
@@ -426,7 +499,7 @@ distributing it.
    euler-loading: crop or resize only values whose spatial size matches the
    reference, or classify fields from the loader annotations.
 2. `princeton_dense.read_intrinsics` returned raw `K` for rectified `_lut`
-   images (§3.2). Fixed in the euler-loading working tree.
+   images (§3.2). Fixed in euler-loading 2.31.0.
 3. `generic_dense_depth` PNG depth with `scale_to_meters_override` doesn't
    survive decode → write: 10.0 m reloads as 0.039 m. The loader applies the
    scale (`generic_dense_depth.py:133`), `write_depth` (`:226`) writes
@@ -436,11 +509,13 @@ distributing it.
    (`vkitti2.py:208`), so it raises on the documented header line
    `frame cameraID r1,1 …`. This is also open question 4 in
    `euler-dataset-contract/docs/ecosystem-streamlining-plan.md`.
-5. `extract_intrinsics` only resolves hierarchical intrinsics whose file ID is
-   `"intrinsics"`, so radial and fog miss STF's `calib_cam_stereo_left`.
+5. `extract_intrinsics` only resolved hierarchical intrinsics whose file ID is
+   `"intrinsics"`, so radial and fog missed STF's `calib_cam_stereo_left`.
+   Fixed with phase 1 (`single_calibration`): a single file is used whatever
+   its ID; several files raise.
 6. RGB writers always emit PNG bytes. VKITTI2 `.jpg` sources become PNG data
    under `.jpg` names, and Real Drive Sim RGBA sources lose their alpha channel.
 7. `princeton_dense.rccb` demosaiced as GBRG, swapping red and blue. Fixed in
-   the euler-loading working tree (GRBG).
+   euler-loading 2.31.0 (GRBG).
 8. `princeton_dense.rgb` on a 12-bit TIFF returns an all-white image without
    an error. Not fixed: a guard that raises would be a small follow-up.

{euler_preprocess-3.18.0 → euler_preprocess-3.19.0}/euler_preprocess/cli.py

@@ -1,8 +1,8 @@
 """CLI entry point for euler-preprocess.
 
 Used both by ``python main.py`` and by the installed ``euler-preprocess`` console
-script.  Supports subcommands: ``fog``, ``sky-depth``, ``radial``, ``lidar-motion``
-and ``spatial``.
+script.  Supports subcommands: ``fog``, ``sky-depth``, ``radial``, ``lidar-motion``,
+``crop``, ``crop-box`` and ``spatial``.
 """
 from __future__ import annotations
 
@@ -257,6 +257,29 @@ def _cmd_lidar_motion(args: argparse.Namespace) -> int:
     return _run_transform(args, LidarMotionTransform)
 
 
+def _cmd_crop(args: argparse.Namespace) -> int:
+    from euler_preprocess.crop.run import run_crop_command
+
+    config_path = Path(args.config).resolve()
+    config = json.loads(config_path.read_text(encoding="utf-8"))
+    get_logger().info("Config: %s", config_path)
+    run_crop_command(config, config_path.parent)
+    return 0
+
+
+def _cmd_crop_box(args: argparse.Namespace) -> int:
+    from euler_preprocess.crop.spec import parse_crop_config, spatial_preprocessing
+
+    config = parse_crop_config(json.loads(Path(args.config).read_text(encoding="utf-8")))
+    height, width = args.size
+    box = config.geometry(height, width)
+    print(json.dumps(
+        {"box": box.to_dict(), "preprocessing": spatial_preprocessing(box, reference=config.reference)},
+        indent=2,
+    ))
+    return 0
+
+
 def _cmd_spatial(args: argparse.Namespace) -> int:
     from euler_preprocess.spatial import export_spatial
 
@@ -317,6 +340,25 @@ def parse_args() -> argparse.Namespace:
     )
     lidar_motion_parser.set_defaults(func=_cmd_lidar_motion)
 
+    crop_parser = subparsers.add_parser(
+        "crop", parents=[common],
+        help="Crop RGB, depth and calibration coherently (K' = C K; lidar and extrinsics unchanged).",
+    )
+    crop_parser.set_defaults(func=_cmd_crop)
+
+    crop_box_parser = subparsers.add_parser(
+        "crop-box",
+        help="Print the pixel box a crop config resolves to, and the matching spatial preprocessing block.",
+    )
+    crop_box_parser.add_argument(
+        "--config", "-c", type=str, required=True, help="Path to the crop config JSON file.",
+    )
+    crop_box_parser.add_argument(
+        "--size", type=int, nargs=2, required=True, metavar=("HEIGHT", "WIDTH"),
+        help="Size of the reference raster in pixels.",
+    )
+    crop_box_parser.set_defaults(func=_cmd_crop_box)
+
     spatial_parser = subparsers.add_parser(
         "spatial", parents=[common], help="Export deterministic crop/resize with verified execution receipts.",
     )