euler-preprocess 3.15.0__tar.gz → 3.16.0__tar.gz

{euler_preprocess-3.15.0/euler_preprocess.egg-info → euler_preprocess-3.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 3.15.0
+Version: 3.16.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License-Expression: MIT
@@ -94,9 +94,7 @@ loader.
   "modalities": {
     "rgb": {"path": "/path/to/rgb", "split": "train"},
     "depth": "/path/to/depth",
-    "semantic_segmentation": "/path/to/classSegmentation"
-  },
-  "hierarchical_modalities": {
+    "semantic_segmentation": "/path/to/classSegmentation",
     "intrinsics": {"path": "/path/to/intrinsics"}
   }
 }
@@ -109,17 +107,27 @@ loader.
 | `output_slot` | Optional slot selector when `pipeline.output_targets` has several entries. Defaults to `rgb` for `fog` and `depth` for `sky-depth` / `radial`. |
 | `sample` | Optional 0-based dataset index. Only `dataset[sample]` is transformed — useful for small benchmark slices of large datasets. |
 | `samples` | Optional multi-sample selector: a list of indices (`[0, 10, 20]`) or a slice object (`{"start": 0, "stop": 1000, "step": 2, "count": 100}`). `stop` is exclusive, `count` caps the result. Mutually exclusive with `sample`. |
-| `modalities` | Modalities that participate in sample-ID intersection. Each value is a path string or an object with `path` and optional `split`. |
-| `hierarchical_modalities` | Per-scene data (e.g. intrinsics), same format. Loaded once per scene and cached. |
+| `modalities` | All input modalities. Each value is a path string or an object with `path` and optional `split`. Whether a modality is per-sample or hierarchical is inferred (see below). |
+| `hierarchical_modalities` | Legacy, optional. Same format; entries are pooled with `modalities` and resolved the same way. |
 | `pipeline` | Optional runtime routing block compatible with `euler-inference`. |
 
 **Required modalities per transform:**
 
-| Transform | `modalities` | `hierarchical_modalities` |
+| Transform | Required | Optional |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics`; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
-| `radial` | `depth` | `intrinsics` |
+| `radial` | `depth`, `intrinsics` | — |
+
+**Per-sample vs. hierarchical modalities.** You don't declare a modality's role:
+it is inferred from the ds-crawler indexes. A modality whose qualified IDs
+(hierarchy path plus file ID) match the transform's per-sample inputs is joined per
+sample, e.g. Synscapes' per-image `meta/<id>.json` intrinsics. A modality whose
+files share no sample IDs but sit at or above the samples' hierarchy level is
+matched hierarchically and shared by every sample beneath it, e.g. one intrinsics
+file per scene or camera. A modality that is neither raises an error, and the
+inferred roles are logged. Listing a per-sample modality under
+`hierarchical_modalities` logs a warning and loads it per sample.
 
 When a modality directory contains ds-crawler split files
 (`.ds_crawler/split_<name>.json`), set `split` on that modality to select a subset.
@@ -677,7 +685,7 @@ d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
 ```
 
 The transform config takes no parameters (`{}`); intrinsics are read from the
-`intrinsics` hierarchical modality.
+`intrinsics` modality, stored either per sample or per scene/camera.
 
 CLI runs write a source-backed depth dataset mirroring the input depth modality's layout
 and writer metadata, with `meta.radial_depth` set to `true` in the emitted `index.json`.

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/README.md

@@ -74,9 +74,7 @@ loader.
   "modalities": {
     "rgb": {"path": "/path/to/rgb", "split": "train"},
     "depth": "/path/to/depth",
-    "semantic_segmentation": "/path/to/classSegmentation"
-  },
-  "hierarchical_modalities": {
+    "semantic_segmentation": "/path/to/classSegmentation",
     "intrinsics": {"path": "/path/to/intrinsics"}
   }
 }
@@ -89,17 +87,27 @@ loader.
 | `output_slot` | Optional slot selector when `pipeline.output_targets` has several entries. Defaults to `rgb` for `fog` and `depth` for `sky-depth` / `radial`. |
 | `sample` | Optional 0-based dataset index. Only `dataset[sample]` is transformed — useful for small benchmark slices of large datasets. |
 | `samples` | Optional multi-sample selector: a list of indices (`[0, 10, 20]`) or a slice object (`{"start": 0, "stop": 1000, "step": 2, "count": 100}`). `stop` is exclusive, `count` caps the result. Mutually exclusive with `sample`. |
-| `modalities` | Modalities that participate in sample-ID intersection. Each value is a path string or an object with `path` and optional `split`. |
-| `hierarchical_modalities` | Per-scene data (e.g. intrinsics), same format. Loaded once per scene and cached. |
+| `modalities` | All input modalities. Each value is a path string or an object with `path` and optional `split`. Whether a modality is per-sample or hierarchical is inferred (see below). |
+| `hierarchical_modalities` | Legacy, optional. Same format; entries are pooled with `modalities` and resolved the same way. |
 | `pipeline` | Optional runtime routing block compatible with `euler-inference`. |
 
 **Required modalities per transform:**
 
-| Transform | `modalities` | `hierarchical_modalities` |
+| Transform | Required | Optional |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics`; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
-| `radial` | `depth` | `intrinsics` |
+| `radial` | `depth`, `intrinsics` | — |
+
+**Per-sample vs. hierarchical modalities.** You don't declare a modality's role:
+it is inferred from the ds-crawler indexes. A modality whose qualified IDs
+(hierarchy path plus file ID) match the transform's per-sample inputs is joined per
+sample, e.g. Synscapes' per-image `meta/<id>.json` intrinsics. A modality whose
+files share no sample IDs but sit at or above the samples' hierarchy level is
+matched hierarchically and shared by every sample beneath it, e.g. one intrinsics
+file per scene or camera. A modality that is neither raises an error, and the
+inferred roles are logged. Listing a per-sample modality under
+`hierarchical_modalities` logs a warning and loads it per sample.
 
 When a modality directory contains ds-crawler split files
 (`.ds_crawler/split_<name>.json`), set `split` on that modality to select a subset.
@@ -657,7 +665,7 @@ d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
 ```
 
 The transform config takes no parameters (`{}`); intrinsics are read from the
-`intrinsics` hierarchical modality.
+`intrinsics` modality, stored either per sample or per scene/camera.
 
 CLI runs write a source-backed depth dataset mirroring the input depth modality's layout
 and writer metadata, with `meta.radial_depth` set to `true` in the emitted `index.json`.

euler_preprocess-3.16.0/euler_preprocess/common/dataset.py

@@ -0,0 +1,212 @@
+from __future__ import annotations
+
+from collections import Counter
+
+from euler_preprocess.common.logging import get_logger
+
+
+def _make_modality(entry: str | dict):
+    from euler_loading import Modality
+
+    if isinstance(entry, str):
+        return Modality(entry)
+    return Modality(
+        entry["path"],
+        **{
+            key: entry[key]
+            for key in (
+                "split",
+                "metadata_scope",
+                "collapse_single",
+                "cache",
+                "applies_to",
+                "hierarchy_scope",
+            )
+            if key in entry
+        },
+    )
+
+
+def _load_index_tree(name: str, modality) -> dict:
+    """Load a modality's ds-crawler tree the same way ``MultiModalDataset`` does."""
+    from ds_crawler import index_dataset_from_path
+    from euler_loading._ds_crawler_utils import load_index_output
+    from euler_loading.dataset import _get_index_tree, _resolve_metadata_scope
+
+    modality = _resolve_metadata_scope(name, modality)
+    index = load_index_output(
+        modality.path,
+        split=modality.split,
+        metadata_scope=modality.metadata_scope,
+        index_dataset_from_path_fn=index_dataset_from_path,
+    )
+    return _get_index_tree(index)
+
+
+def _check_attachment(
+    name: str, seed: str, tree: dict, samples_per_level: Counter
+) -> None:
+    """Check that hierarchical modality *name* applies to the samples by tree position."""
+    from euler_loading.indexing import collect_hierarchical_files, match_hierarchical_files
+
+    files_by_level = collect_hierarchical_files(tree)
+    attached = {
+        level: len(match_hierarchical_files(level, files_by_level))
+        for level in samples_per_level
+    }
+    covered = sum(count for level, count in samples_per_level.items() if attached[level])
+    if not covered:
+        raise ValueError(
+            f"Modality '{name}' shares no sample IDs with '{seed}' and none of its "
+            "files sit at or above the hierarchy level of any sample, so it can be "
+            "neither joined per sample nor matched hierarchically. Check that its "
+            f"ds-crawler IDs and hierarchy follow the same conventions as '{seed}'."
+        )
+    logger = get_logger()
+    total = sum(samples_per_level.values())
+    log = logger.info if covered == total else logger.warning
+    log(
+        "Modality '%s' is hierarchical (no sample IDs shared with '%s'); it applies "
+        "to %d/%d samples by hierarchy position.",
+        name,
+        seed,
+        covered,
+        total,
+    )
+    most = max(attached.values())
+    if most > 1:
+        logger.warning(
+            "Hierarchical modality '%s' attaches up to %d files to one sample. If it "
+            "holds per-sample data, its ds-crawler IDs do not match '%s'.",
+            name,
+            most,
+            seed,
+        )
+
+
+def _resolve_roles(
+    modalities: dict, per_sample: set[str]
+) -> tuple[list[str], list[str]]:
+    """Split modality names into regular and hierarchical ones.
+
+    The role follows from the ds-crawler indexes.  A modality is regular
+    (joined per sample) when its qualified file IDs -- hierarchy path plus
+    file ID -- overlap those of the seed modality.  Otherwise it is
+    hierarchical (matched by tree position, like per-scene intrinsics) and
+    must sit at or above the hierarchy level of at least one sample.
+
+    *per_sample* modalities are regular by contract and the first one is the
+    seed.  Without one, the seed is the modality aligned with the most others,
+    then the one with the most files, then the first listed.
+    """
+    from euler_loading.indexing import collect_files
+
+    names = list(modalities)
+    if len(names) < 2 or per_sample.issuperset(names):
+        return names, []
+
+    trees = {name: _load_index_tree(name, modalities[name]) for name in names}
+    # Qualified sample ID -> hierarchy path, keyed like euler-loading's lookups.
+    sample_ids = {}
+    for name, tree in trees.items():
+        sample_ids[name] = {
+            "/".join(r.hierarchy_path + (r.file_entry["id"],)): r.hierarchy_path
+            for r in collect_files(tree)
+        }
+
+    def aligned(name: str, other: str) -> bool:
+        return not sample_ids[name].keys().isdisjoint(sample_ids[other].keys())
+
+    anchors = [name for name in names if name in per_sample]
+    seed = anchors[0] if anchors else max(
+        names,
+        key=lambda name: (
+            sum(aligned(other, name) for other in names),
+            len(sample_ids[name]),
+        ),
+    )
+    regular = [
+        name
+        for name in names
+        if name in per_sample or name == seed or aligned(name, seed)
+    ]
+    hierarchical = [name for name in names if name not in regular]
+    get_logger().info(
+        "Modality roles inferred from ds-crawler indexes: per-sample %s; "
+        "hierarchical %s",
+        ", ".join(regular),
+        ", ".join(hierarchical) or "none",
+    )
+
+    # Without common samples MultiModalDataset raises its own, clearer error.
+    common = set(sample_ids[seed]).intersection(*(sample_ids[name] for name in regular))
+    if common:
+        samples_per_level = Counter(sample_ids[seed][sample_id] for sample_id in common)
+        for name in hierarchical:
+            _check_attachment(name, seed, trees[name], samples_per_level)
+    return regular, hierarchical
+
+
+def build_dataset(
+    config: dict,
+    required_modalities: set[str],
+    required_hierarchical: set[str] | None = None,
+):
+    """Build a ``MultiModalDataset`` from a config dict.
+
+    Each modality entry is either a plain path string or a dict with
+    ``path`` and optional ``split``.  Loader resolution (which function to
+    call, which module to use) is handled by euler-loading via the
+    ds-crawler index at each path — point the config at a path whose
+    index declares the function you want (e.g. a ``sky_mask`` index for
+    boolean sky masks vs. a ``class_segmentation`` index for raw class
+    id maps).
+
+    Every modality can be listed under ``modalities``: whether it is joined
+    per sample or matched hierarchically (e.g. per-scene intrinsics) is
+    inferred from the ds-crawler indexes.  Entries under the legacy
+    ``hierarchical_modalities`` key are pooled with them and resolved the
+    same way.  *required_modalities* are per-sample inputs and anchor that
+    inference; *required_hierarchical* names further required modalities,
+    whose role is inferred like any other.
+    """
+    from euler_loading import MultiModalDataset, SamplePreprocessor
+
+    raw_modalities = config.get("modalities", {})
+    raw_hierarchical = config.get("hierarchical_modalities", {})
+
+    duplicated = raw_modalities.keys() & raw_hierarchical.keys()
+    if duplicated:
+        raise ValueError(
+            "Modalities listed under both 'modalities' and "
+            f"'hierarchical_modalities': {', '.join(sorted(duplicated))}"
+        )
+    configured = {**raw_modalities, **raw_hierarchical}
+
+    required = set(required_modalities) | set(required_hierarchical or ())
+    missing = required - configured.keys()
+    if missing:
+        raise ValueError(
+            f"Missing required modalities in config: {', '.join(sorted(missing))}. "
+            f"'modalities' must contain at least: {', '.join(sorted(required))}"
+        )
+
+    modalities = {name: _make_modality(entry) for name, entry in configured.items()}
+    regular, hierarchical = _resolve_roles(modalities, set(required_modalities))
+    logger = get_logger()
+    for name in regular:
+        if name in raw_hierarchical:
+            logger.warning(
+                "Modality '%s' is listed under 'hierarchical_modalities' but holds "
+                "per-sample data; loading it per sample.",
+                name,
+            )
+
+    return MultiModalDataset(
+        modalities={name: modalities[name] for name in regular},
+        hierarchical_modalities={name: modalities[name] for name in hierarchical}
+        or None,
+        transforms=[SamplePreprocessor.from_config(config["preprocessing"])]
+        if config.get("preprocessing")
+        else None,
+    )

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess/common/intrinsics.py

@@ -13,8 +13,9 @@ except ImportError:
 def extract_intrinsics(sample: dict) -> np.ndarray | None:
     """Extract the 3x3 intrinsics matrix from a sample dict.
 
-    Hierarchical modalities are stored as ``sample[name][file_id]``.
-    For intrinsics the convention is ``sample["intrinsics"]["intrinsics"]``.
+    Per-sample intrinsics are stored directly as ``sample["intrinsics"]``.
+    Hierarchical modalities are stored as ``sample[name][file_id]``; for
+    intrinsics the convention is ``sample["intrinsics"]["intrinsics"]``.
     """
     intr_dict = sample.get("intrinsics")
     if intr_dict is None:

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess/common/transform.py

@@ -12,6 +12,15 @@ class Transform(ABC):
     Subclasses declare the modalities they need via class variables and
     implement :meth:`run` to process samples.
 
+    Required modalities:
+        :attr:`REQUIRED_MODALITIES` are per-sample inputs (RGB, depth, masks);
+        they are always joined per sample and anchor the role inference of
+        every other modality.  :attr:`REQUIRED_HIERARCHICAL_MODALITIES` are
+        further required inputs, such as camera intrinsics, that a dataset may
+        store per sample or once per hierarchy level; their role is inferred
+        from the ds-crawler indexes (see
+        :func:`~euler_preprocess.common.dataset.build_dataset`).
+
     Output slots:
         Most transforms produce a single output (the *primary* slot, declared
         via :attr:`OUTPUT_SLOT`).  Transforms that produce additional auxiliary

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess/fog/transform.py

@@ -286,8 +286,9 @@ class FogTransform(Transform):
         "depth":                    np.ndarray (H, W), float in meters
         "semantic_segmentation":    np.ndarray (H, W), boolean sky mask
         "id":         str
-        "intrinsics": dict -- hierarchical modality containing ``"intrinsics"``
-                      key mapping to a (3, 3) camera intrinsics matrix *K*.
+        "intrinsics": (3, 3) camera intrinsics matrix *K* (per-sample
+                      modality), or a dict with an ``"intrinsics"`` key
+                      mapping to *K* (hierarchical modality).
                       When present, planar (z-buffer) depth is converted to
                       radial (Euclidean) depth before fog is applied.
 

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess/radial/transform.py

@@ -15,8 +15,8 @@ from euler_preprocess.common.transform import Transform
 class RadialTransform(Transform):
     """Convert planar (z-buffer) depth to radial (Euclidean) depth.
 
-    Requires camera intrinsics (hierarchical modality ``intrinsics``) to
-    compute the per-pixel ray direction.
+    Requires camera intrinsics (modality ``intrinsics``, stored per sample or
+    per hierarchy level) to compute the per-pixel ray direction.
 
     Config JSON keys:
         ``device`` (str): ``"cpu"`` (default). GPU support reserved for future use.
@@ -61,7 +61,7 @@ class RadialTransform(Transform):
                 if K is None:
                     raise ValueError(
                         f"Sample '{sample.get('id', '?')}' has no intrinsics. "
-                        "RadialTransform requires the 'intrinsics' hierarchical modality."
+                        "RadialTransform requires the 'intrinsics' modality."
                     )
 
                 radial = planar_to_radial_depth(depth, K)

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess/spatial.py

@@ -27,7 +27,15 @@ def export_spatial(config):
     dataset_config = {
         key: value for key, value in config.items() if key != "preprocessing"
     }
-    dataset = build_dataset(dataset_config, set(config.get("modalities", {})))
+    # Bound fields with a spatial layout are per-sample images; the role of the
+    # others (e.g. intrinsics) is inferred from their ds-crawler indexes.
+    per_sample = {
+        name
+        for options in variants.values()
+        for name, binding in options["bindings"]["fields"].items()
+        if "H" in binding["profile"]["layout"]
+    }
+    dataset = build_dataset(dataset_config, per_sample)
     plans = {}
     for variant, options in variants.items():
         bindings = deepcopy(options["bindings"])

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0/euler_preprocess.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 3.15.0
+Version: 3.16.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License-Expression: MIT
@@ -94,9 +94,7 @@ loader.
   "modalities": {
     "rgb": {"path": "/path/to/rgb", "split": "train"},
     "depth": "/path/to/depth",
-    "semantic_segmentation": "/path/to/classSegmentation"
-  },
-  "hierarchical_modalities": {
+    "semantic_segmentation": "/path/to/classSegmentation",
     "intrinsics": {"path": "/path/to/intrinsics"}
   }
 }
@@ -109,17 +107,27 @@ loader.
 | `output_slot` | Optional slot selector when `pipeline.output_targets` has several entries. Defaults to `rgb` for `fog` and `depth` for `sky-depth` / `radial`. |
 | `sample` | Optional 0-based dataset index. Only `dataset[sample]` is transformed — useful for small benchmark slices of large datasets. |
 | `samples` | Optional multi-sample selector: a list of indices (`[0, 10, 20]`) or a slice object (`{"start": 0, "stop": 1000, "step": 2, "count": 100}`). `stop` is exclusive, `count` caps the result. Mutually exclusive with `sample`. |
-| `modalities` | Modalities that participate in sample-ID intersection. Each value is a path string or an object with `path` and optional `split`. |
-| `hierarchical_modalities` | Per-scene data (e.g. intrinsics), same format. Loaded once per scene and cached. |
+| `modalities` | All input modalities. Each value is a path string or an object with `path` and optional `split`. Whether a modality is per-sample or hierarchical is inferred (see below). |
+| `hierarchical_modalities` | Legacy, optional. Same format; entries are pooled with `modalities` and resolved the same way. |
 | `pipeline` | Optional runtime routing block compatible with `euler-inference`. |
 
 **Required modalities per transform:**
 
-| Transform | `modalities` | `hierarchical_modalities` |
+| Transform | Required | Optional |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics`; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
-| `radial` | `depth` | `intrinsics` |
+| `radial` | `depth`, `intrinsics` | — |
+
+**Per-sample vs. hierarchical modalities.** You don't declare a modality's role:
+it is inferred from the ds-crawler indexes. A modality whose qualified IDs
+(hierarchy path plus file ID) match the transform's per-sample inputs is joined per
+sample, e.g. Synscapes' per-image `meta/<id>.json` intrinsics. A modality whose
+files share no sample IDs but sit at or above the samples' hierarchy level is
+matched hierarchically and shared by every sample beneath it, e.g. one intrinsics
+file per scene or camera. A modality that is neither raises an error, and the
+inferred roles are logged. Listing a per-sample modality under
+`hierarchical_modalities` logs a warning and loads it per sample.
 
 When a modality directory contains ds-crawler split files
 (`.ds_crawler/split_<name>.json`), set `split` on that modality to select a subset.
@@ -677,7 +685,7 @@ d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
 ```
 
 The transform config takes no parameters (`{}`); intrinsics are read from the
-`intrinsics` hierarchical modality.
+`intrinsics` modality, stored either per sample or per scene/camera.
 
 CLI runs write a source-backed depth dataset mirroring the input depth modality's layout
 and writer metadata, with `meta.radial_depth` set to `true` in the emitted `index.json`.

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/euler_preprocess.egg-info/SOURCES.txt

@@ -52,6 +52,7 @@ tests/test_fog_aux_outputs.py
 tests/test_fog_aware_auto_exposure.py
 tests/test_fog_cpu_gpu_parity.py
 tests/test_fog_integration.py
+tests/test_modality_roles.py
 tests/test_radial.py
 tests/test_real_drive_sim_scenario_profiles.py
 tests/test_scene_illumination.py

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "euler-preprocess"
-version = "3.15.0"
+version = "3.16.0"
 description = "Physics-based preprocessing (fog, etc.) for RGB+depth datasets"
 readme = "README.md"
 requires-python = ">=3.9"

euler_preprocess-3.16.0/tests/test_modality_roles.py

@@ -0,0 +1,192 @@
+"""Tests for inferring per-sample vs. hierarchical modality roles."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from ds_crawler import DatasetWriter
+from euler_loading import Modality, MultiModalDataset
+from euler_loading.loaders.cpu.generic import write_intrinsics
+from euler_loading.loaders.cpu.generic_dense_depth import write_depth
+
+from euler_preprocess.cli import _run_transform
+from euler_preprocess.common.dataset import build_dataset
+from euler_preprocess.common.intrinsics import extract_intrinsics, planar_to_radial_depth
+from euler_preprocess.common.normalize import normalize_depth
+from euler_preprocess.radial.transform import RadialTransform
+
+FRAMES = ("Scene01/Camera_0/0000", "Scene01/Camera_0/0001", "Scene02/Camera_0/0000")
+
+
+def _K(fx: float) -> np.ndarray:
+    return np.array([[fx, 0.0, 3.0], [0.0, fx, 2.0], [0.0, 0.0, 1.0]], dtype=np.float32)
+
+
+def _write_depth(root: Path, full_ids=FRAMES) -> str:
+    writer = DatasetWriter(
+        root,
+        name="synthetic_depth",
+        type="depth",
+        separator=None,
+        meta={"radial_depth": False, "scale_to_meters": 1.0, "range": [0, 1000]},
+        euler_loading={"loader": "generic_dense_depth", "function": "depth"},
+    )
+    for full_id in full_ids:
+        path = writer.get_path(full_id, full_id.rsplit("/", 1)[-1] + ".npy")
+        write_depth(str(path), np.full((4, 6), 25.0, dtype=np.float32))
+    writer.save_index()
+    return str(root)
+
+
+def _write_intrinsics(root: Path, matrices: dict[str, np.ndarray]) -> str:
+    writer = DatasetWriter(
+        root,
+        name="synthetic_intrinsics",
+        type="intrinsics",
+        separator=None,
+        euler_loading={"loader": "generic", "function": "intrinsics"},
+    )
+    for full_id, K in matrices.items():
+        write_intrinsics(str(writer.get_path(full_id, f"{K[0, 0]:.0f}.npy")), K)
+    writer.save_index()
+    return str(root)
+
+
+def _per_sample_intrinsics(root: Path) -> str:
+    return _write_intrinsics(root, {fid: _K(500.0 + i) for i, fid in enumerate(FRAMES)})
+
+
+def _per_scene_intrinsics(root: Path) -> str:
+    return _write_intrinsics(
+        root, {"Scene01/intrinsics": _K(600.0), "Scene02/intrinsics": _K(700.0)}
+    )
+
+
+def _roles(dataset) -> tuple[set[str], set[str]]:
+    return set(dataset.modality_paths()), set(dataset.hierarchical_modality_paths())
+
+
+def _fx_by_full_id(dataset) -> dict[str, float]:
+    return {
+        dataset[i]["full_id"]: float(extract_intrinsics(dataset[i])[0, 0])
+        for i in range(len(dataset))
+    }
+
+
+@pytest.mark.parametrize("key", ["modalities", "hierarchical_modalities"])
+def test_per_sample_intrinsics_are_joined_per_sample(tmp_path, key):
+    config = {"modalities": {"depth": _write_depth(tmp_path / "depth")}}
+    config.setdefault(key, {})["intrinsics"] = _per_sample_intrinsics(
+        tmp_path / "intrinsics"
+    )
+
+    dataset = build_dataset(
+        config,
+        RadialTransform.REQUIRED_MODALITIES,
+        RadialTransform.REQUIRED_HIERARCHICAL_MODALITIES,
+    )
+
+    assert _roles(dataset) == ({"depth", "intrinsics"}, set())
+    assert _fx_by_full_id(dataset) == {
+        "/Scene01/Camera_0/0000": 500.0,
+        "/Scene01/Camera_0/0001": 501.0,
+        "/Scene02/Camera_0/0000": 502.0,
+    }
+
+
+@pytest.mark.parametrize("key", ["modalities", "hierarchical_modalities"])
+def test_per_scene_intrinsics_are_matched_hierarchically(tmp_path, key):
+    config = {"modalities": {"depth": _write_depth(tmp_path / "depth")}}
+    config.setdefault(key, {})["intrinsics"] = _per_scene_intrinsics(
+        tmp_path / "intrinsics"
+    )
+
+    dataset = build_dataset(
+        config,
+        RadialTransform.REQUIRED_MODALITIES,
+        RadialTransform.REQUIRED_HIERARCHICAL_MODALITIES,
+    )
+
+    assert _roles(dataset) == ({"depth"}, {"intrinsics"})
+    assert _fx_by_full_id(dataset) == {
+        "/Scene01/Camera_0/0000": 600.0,
+        "/Scene01/Camera_0/0001": 600.0,
+        "/Scene02/Camera_0/0000": 700.0,
+    }
+
+
+def test_seed_without_required_modalities_is_the_sample_modality(tmp_path):
+    # The calibration sits at the same level as the frames and is listed first.
+    calibration = _write_intrinsics(
+        tmp_path / "intrinsics", {"Scene01/Camera_0/calibration": _K(800.0)}
+    )
+    depth = _write_depth(tmp_path / "depth", FRAMES[:2])
+
+    dataset = build_dataset(
+        {"modalities": {"intrinsics": calibration, "depth": depth}}, set()
+    )
+
+    assert _roles(dataset) == ({"depth"}, {"intrinsics"})
+    assert len(dataset) == 2
+    assert set(dataset[0]["intrinsics"]) == {"calibration"}
+
+
+def test_modality_neither_per_sample_nor_attached_raises(tmp_path):
+    config = {
+        "modalities": {
+            "depth": _write_depth(tmp_path / "depth"),
+            "intrinsics": _write_intrinsics(
+                tmp_path / "intrinsics", {"Scene99/intrinsics": _K(600.0)}
+            ),
+        },
+    }
+
+    with pytest.raises(ValueError, match="neither joined per sample nor matched"):
+        build_dataset(config, {"depth"})
+
+
+def test_modality_listed_under_both_keys_raises(tmp_path):
+    intrinsics = _per_scene_intrinsics(tmp_path / "intrinsics")
+    config = {
+        "modalities": {"depth": _write_depth(tmp_path / "depth"), "intrinsics": intrinsics},
+        "hierarchical_modalities": {"intrinsics": intrinsics},
+    }
+
+    with pytest.raises(ValueError, match="both 'modalities' and 'hierarchical_modalities'"):
+        build_dataset(config, {"depth"})
+
+
+@pytest.mark.parametrize("layout", [_per_sample_intrinsics, _per_scene_intrinsics])
+def test_radial_cli_infers_intrinsics_role(tmp_path, layout):
+    (tmp_path / "radial.json").write_text("{}", encoding="utf-8")
+    config_path = tmp_path / "dataset.json"
+    config_path.write_text(
+        json.dumps(
+            {
+                "transform_config_path": "radial.json",
+                "output_path": str(tmp_path / "radial_depth"),
+                "modalities": {
+                    "depth": _write_depth(tmp_path / "depth"),
+                    "intrinsics": layout(tmp_path / "intrinsics"),
+                },
+            }
+        ),
+        encoding="utf-8",
+    )
+    source = build_dataset(json.loads(config_path.read_text()), {"depth"})
+
+    assert _run_transform(argparse.Namespace(config=str(config_path)), RadialTransform) == 0
+
+    output = MultiModalDataset({"depth": Modality(str(tmp_path / "radial_depth"))})
+    assert len(output) == len(source) == len(FRAMES)
+    for i in range(len(source)):
+        expected = planar_to_radial_depth(
+            normalize_depth(source[i]["depth"]), extract_intrinsics(source[i])
+        )
+        assert output[i]["full_id"] == source[i]["full_id"]
+        np.testing.assert_allclose(normalize_depth(output[i]["depth"]), expected, rtol=1e-6)

{euler_preprocess-3.15.0 → euler_preprocess-3.16.0}/tests/test_spatial.py

@@ -77,6 +77,17 @@ def test_producer_captures_and_reloads_five_outputs(tmp_path, zip_output):
         )
 
 
+def test_producer_infers_hierarchical_intrinsics_listed_as_modality(tmp_path):
+    config, ds, plan, _data, _original = config_for(tmp_path)
+    config["modalities"].update(config.pop("hierarchical_modalities"))
+    paths = export_spatial(config)
+    ds.enable_transform_capture({"default": plan})
+    expected = ds[1]
+    for field, path in paths.items():
+        sample = MultiModalDataset({field: Modality(path, metadata_scope="saved")})[0]
+        np.testing.assert_array_equal(sample[field], expected[field])
+
+
 def test_common_builder_consumes_preprocessing(tmp_path):
     config, ds, _, _, _ = config_for(tmp_path)
     configured = build_dataset(config, set(config["modalities"]))

euler_preprocess-3.15.0/euler_preprocess/common/dataset.py

@@ -1,73 +0,0 @@
-from __future__ import annotations
-
-
-def _make_modality(entry: str | dict):
-    from euler_loading import Modality
-
-    if isinstance(entry, str):
-        return Modality(entry)
-    return Modality(
-        entry["path"],
-        **{
-            key: entry[key]
-            for key in (
-                "split",
-                "metadata_scope",
-                "collapse_single",
-                "cache",
-                "applies_to",
-                "hierarchy_scope",
-            )
-            if key in entry
-        },
-    )
-
-
-def build_dataset(
-    config: dict,
-    required_modalities: set[str],
-    required_hierarchical: set[str] | None = None,
-):
-    """Build a ``MultiModalDataset`` from a config dict.
-
-    Each modality entry is either a plain path string or a dict with
-    ``path`` and optional ``split``.  Loader resolution (which function to
-    call, which module to use) is handled by euler-loading via the
-    ds-crawler index at each path — point the config at a path whose
-    index declares the function you want (e.g. a ``sky_mask`` index for
-    boolean sky masks vs. a ``class_segmentation`` index for raw class
-    id maps).
-    """
-    from euler_loading import MultiModalDataset, SamplePreprocessor
-
-    raw_modalities = config.get("modalities", {})
-    raw_hierarchical = config.get("hierarchical_modalities", {})
-
-    missing = required_modalities - raw_modalities.keys()
-    if missing:
-        raise ValueError(
-            f"Missing required modalities in config: {', '.join(sorted(missing))}. "
-            f"'modalities' must contain at least: {', '.join(sorted(required_modalities))}"
-        )
-
-    if required_hierarchical:
-        missing_h = required_hierarchical - raw_hierarchical.keys()
-        if missing_h:
-            raise ValueError(
-                f"Missing required hierarchical modalities in config: "
-                f"{', '.join(sorted(missing_h))}. 'hierarchical_modalities' must "
-                f"contain at least: {', '.join(sorted(required_hierarchical))}"
-            )
-
-    modalities = {name: _make_modality(entry) for name, entry in raw_modalities.items()}
-    hierarchical_modalities = {
-        name: _make_modality(entry) for name, entry in raw_hierarchical.items()
-    }
-
-    return MultiModalDataset(
-        modalities=modalities,
-        hierarchical_modalities=hierarchical_modalities or None,
-        transforms=[SamplePreprocessor.from_config(config["preprocessing"])]
-        if config.get("preprocessing")
-        else None,
-    )