euler-loading 2.11.0__tar.gz → 2.12.0__tar.gz

{euler_loading-2.11.0 → euler_loading-2.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-loading
-Version: 2.11.0
+Version: 2.12.0
 Summary: Multi-modal PyTorch dataloader using ds-crawler indices
 Requires-Python: >=3.9
 Requires-Dist: ds-crawler>=2.3.0

{euler_loading-2.11.0 → euler_loading-2.12.0}/README.md

@@ -55,7 +55,7 @@ Frozen dataclass describing one data modality.
 
 | Parameter | Type | Description |
 |-----------|------|-------------|
-| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain ds-crawler metadata, either at `.ds_crawler/` or under the configured `metadata_scope`. |
+| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain ds-crawler metadata, either at `.ds_crawler/` or under the configured `metadata_scope`. Inline selectors are accepted as `/data/ds.zip:train`, `/data/ds.zip#scope=rgb`, or `/data/ds.zip:train#scope=rgb`. |
 | `origin_path` | `str \| None` | Original path before copying/symlinking (e.g. for SLURM staging). Not used by euler-loading itself — useful for experiment logging to retain references to the original dataset location. |
 | `loader` | `Callable[..., Any] \| None` | Receives the file path (or `BinaryIO` buffer for zip-backed modalities) and an optional `meta` dict. Returns loaded data. When `None`, the loader is resolved automatically from the ds-crawler index (see [Automatic loader resolution](#automatic-loader-resolution)). |
 | `writer` | `Callable[..., Any] \| None` | Receives `(path, value, meta)` and writes modality data to disk. When `None`, euler-loading tries to resolve a built-in writer from ds-crawler metadata (`write_<function>` or `write_<suffix>` for `read_<suffix>`). |
@@ -65,7 +65,7 @@ Frozen dataclass describing one data modality.
 | `hierarchy_scope` | `str \| None` | Optional scope label for hierarchical modalities (e.g. `scene_camera`). |
 | `applies_to` | `list[str] \| None` | Optional list of regular modality names a hierarchical modality applies to. |
 | `split` | `str \| None` | Optional inline split name. Loads `.ds_crawler/split_<name>.json` from the modality root (directory or zip) and overlays it on the normal ds-crawler metadata. |
-| `metadata_scope` | `str \| None` | Optional namespace below `.ds_crawler`, e.g. `.ds_crawler/camera_extrinsics/index.json`. Use this when multiple logical modalities share one physical directory or zip. If the scoped artifacts are absent, loading falls back to the legacy root-level `.ds_crawler` layout. |
+| `metadata_scope` | `str \| None` | Optional namespace below `.ds_crawler`, e.g. `.ds_crawler/camera_extrinsics/index.json`. Use this when multiple logical modalities share one physical directory or zip. It can also be supplied inline as `#scope=<metadata_scope>`. If omitted, euler-loading infers it only when the choice is deterministic. If configured artifacts are absent, loading falls back to the legacy root-level `.ds_crawler` layout. |
 | `cache` | `bool \| None` | Opt-in/out for in-memory caching of decoded values (only meaningful for hierarchical modalities; regular modalities are never cached). `True` keeps every distinct loaded file in a process-lifetime dict. `False` re-reads on every access. `None` (default): hierarchical → `True` because small shared calibration files benefit. |
 | `collapse_single` | `bool` | For hierarchical modalities only. When `True`, a sample that matches exactly one hierarchical file returns that loaded value directly instead of `{file_id: value}`. This is useful for shared GT tensors used by multiple augmentations of the same source sample. |
 | `metadata` | `dict[str, Any]` | Optional arbitrary metadata. Keys under `metadata["euler_loading"]` are treated as euler-loading defaults. |
@@ -128,6 +128,12 @@ dataset = MultiModalDataset(
 )
 ```
 
+Path-only callers can use the existing colon form directly:
+
+```python
+Modality("/data/rgb:train")
+```
+
 This works for both directory-backed and zip-backed modalities. The split file only replaces the `dataset` payload; top-level metadata such as dataset type and euler-loading loader hints still come from the canonical ds-crawler index.
 
 ### Scoped ds-crawler Metadata
@@ -165,6 +171,23 @@ dataset = MultiModalDataset(
 )
 ```
 
+Path-only callers can select the same scope inline. The scope selector comes
+after the optional split selector:
+
+```python
+Modality("/data/muses.zip#scope=rgb")
+Modality("/data/muses.zip:train#scope=rgb")
+```
+
+If `metadata_scope` is omitted, euler-loading resolves it automatically only
+when doing so is unambiguous:
+
+1. Explicit `metadata_scope` or `#scope=...` wins.
+2. If root-level `.ds_crawler` metadata exists, the legacy root metadata is used.
+3. If no root metadata exists and exactly one scope exists, that scope is used.
+4. If multiple scopes exist, euler-loading matches the modality dict key, `modality_type`, loader `_modality_meta["type"]`, or scoped `dataset-head.json` `modality.key`.
+5. If the result is still ambiguous, construction raises and lists the available scopes.
+
 This is an additive layout. Existing roots with `.ds_crawler/index.json` continue to load unchanged, and a scoped modality falls back to that legacy location when the scoped artifacts are not present.
 
 ### `MultiModalDataset.modality_paths()`
@@ -368,7 +391,7 @@ Writer resolution uses the same module and function metadata:
 
 ## ds-crawler integration
 
-Each logical modality must have its own ds-crawler index. Usually that means one modality root with its own `.ds_crawler` artifacts. When several logical modalities share one physical directory or zip, put each artifact set under `.ds_crawler/<metadata_scope>/` and set `Modality(..., metadata_scope=...)`.
+Each logical modality must have its own ds-crawler index. Usually that means one modality root with its own `.ds_crawler` artifacts. When several logical modalities share one physical directory or zip, put each artifact set under `.ds_crawler/<metadata_scope>/` and set `Modality(..., metadata_scope=...)` or append `#scope=<metadata_scope>` to the path.
 Files across regular modalities are matched by IDs from those indexes, so **the indexed hierarchy and naming conventions must be consistent** across modalities up to modality-specific parts captured in the config.
 
 Calibration files or other per-scene/per-sequence metadata can be loaded via `hierarchical_modalities`. These files are matched to samples based on their position in the hierarchy — all files at or above a sample's hierarchy level are included and cached for efficiency.

{euler_loading-2.11.0 → euler_loading-2.12.0}/euler_loading/_ds_crawler_utils.py

@@ -3,18 +3,21 @@ from __future__ import annotations
 from collections.abc import Mapping
 from pathlib import Path
 from typing import Any
+from urllib.parse import parse_qsl
 
 from ds_crawler import (
     get_dataset_contract,
+    list_metadata_scopes,
     load_dataset_config,
     load_dataset_split as _load_dataset_split,
     validate_metadata_scope,
 )
-from ds_crawler.zip_utils import read_metadata_json
 from ds_crawler.config import CONFIG_FILENAME as _DS_CRAWLER_CONFIG_FILENAME
 from ds_crawler.zip_utils import (
+    DATASET_HEAD_FILENAME as _DS_CRAWLER_HEAD_FILENAME,
     OUTPUT_FILENAME as _DS_CRAWLER_OUTPUT_FILENAME,
     get_split_filename,
+    read_metadata_json,
     validate_split_name,
 )
 
@@ -63,7 +66,38 @@ def extract_ds_crawler_properties(index_output: Mapping[str, Any]) -> dict[str,
         return {}
 
 
-def parse_path_with_split(path: str) -> tuple[str, str | None]:
+_METADATA_FILENAMES = (
+    _DS_CRAWLER_OUTPUT_FILENAME,
+    _DS_CRAWLER_CONFIG_FILENAME,
+    _DS_CRAWLER_HEAD_FILENAME,
+)
+
+
+def _parse_path_with_scope(path: str) -> tuple[str, str | None]:
+    """Extract an optional ``#scope=<name>`` selector from a path."""
+    candidate_path, sep, fragment = path.rpartition("#")
+    if not sep:
+        return path, None
+
+    params = parse_qsl(fragment, keep_blank_values=True)
+    scope_values = [value for key, value in params if key == "scope"]
+    if not scope_values:
+        return path, None
+
+    unsupported = [key for key, _ in params if key != "scope"]
+    if unsupported:
+        raise ValueError(
+            "Unsupported modality path fragment parameter(s): "
+            f"{', '.join(sorted(set(unsupported)))}. "
+            "Only '#scope=<metadata_scope>' is supported."
+        )
+    if len(scope_values) > 1:
+        raise ValueError("Modality path may contain at most one '#scope=' selector.")
+
+    return candidate_path, validate_metadata_scope(scope_values[0])
+
+
+def _parse_path_with_split(path: str) -> tuple[str, str | None]:
     """Extract an optional inline split suffix from a colon-separated path.
 
     Supports paths like ``/data/ds.zip:train`` where the part after the
@@ -88,16 +122,134 @@ def parse_path_with_split(path: str) -> tuple[str, str | None]:
     return candidate_path, split
 
 
+def parse_modality_path(path: str) -> tuple[str, str | None, str | None]:
+    """Extract inline ``:split`` and ``#scope=...`` selectors from a path."""
+    parsed_path, metadata_scope = _parse_path_with_scope(path)
+    parsed_path, split = _parse_path_with_split(parsed_path)
+    return parsed_path, split, metadata_scope
+
+
+def parse_path_with_split(path: str) -> tuple[str, str | None]:
+    """Extract an optional inline split suffix from a colon-separated path."""
+    parsed_path, split, _ = parse_modality_path(path)
+    return parsed_path, split
+
+
+def _read_metadata_json_safe(
+    dataset_path: Path,
+    filename: str,
+    *,
+    metadata_scope: str | None = None,
+) -> dict[str, Any] | None:
+    try:
+        return read_metadata_json(
+            dataset_path,
+            filename,
+            metadata_scope=metadata_scope,
+        )
+    except FileNotFoundError:
+        return None
+
+
+def _has_root_metadata(dataset_path: Path) -> bool:
+    """Return whether unscoped ds-crawler metadata exists at a dataset root."""
+    return any(
+        _read_metadata_json_safe(dataset_path, filename) is not None
+        for filename in _METADATA_FILENAMES
+    )
+
+
 def _has_scoped_metadata(dataset_path: Path, metadata_scope: str) -> bool:
     """Return whether ds-crawler metadata exists under a configured scope."""
-    for filename in (_DS_CRAWLER_OUTPUT_FILENAME, _DS_CRAWLER_CONFIG_FILENAME):
-        if read_metadata_json(
+    return any(
+        _read_metadata_json_safe(
             dataset_path,
             filename,
             metadata_scope=metadata_scope,
-        ) is not None:
-            return True
-    return False
+        ) is not None
+        for filename in _METADATA_FILENAMES
+    )
+
+
+def _list_metadata_scopes_safe(dataset_path: Path) -> list[str]:
+    try:
+        return list_metadata_scopes(dataset_path)
+    except FileNotFoundError:
+        return []
+
+
+def _read_scope_modality_key(dataset_path: Path, metadata_scope: str) -> str | None:
+    head = _read_metadata_json_safe(
+        dataset_path,
+        _DS_CRAWLER_HEAD_FILENAME,
+        metadata_scope=metadata_scope,
+    )
+    modality = head.get("modality") if isinstance(head, Mapping) else None
+    key = modality.get("key") if isinstance(modality, Mapping) else None
+    return as_non_empty_str(key)
+
+
+def infer_metadata_scope(
+    path: str | Path,
+    *,
+    candidates: list[str | None],
+) -> str | None:
+    """Infer a ds-crawler metadata scope only when the choice is deterministic.
+
+    Legacy root-level metadata always wins for an unscoped modality. If no
+    root metadata exists, a single available scope is selected automatically.
+    With multiple scopes, a scope is selected only when exactly one scope name
+    or scoped ``dataset-head.json`` modality key matches the provided
+    candidates.
+    """
+    dataset_path = Path(path)
+    if _has_root_metadata(dataset_path):
+        return None
+
+    scopes = _list_metadata_scopes_safe(dataset_path)
+    if not scopes:
+        return None
+    if len(scopes) == 1:
+        return scopes[0]
+
+    candidate_set = {
+        candidate
+        for candidate in (as_non_empty_str(value) for value in candidates)
+        if candidate is not None
+    }
+    if not candidate_set:
+        _raise_ambiguous_metadata_scope(path, scopes, candidate_set)
+
+    matches = {
+        scope
+        for scope in scopes
+        if scope in candidate_set
+        or _read_scope_modality_key(dataset_path, scope) in candidate_set
+    }
+    if len(matches) == 1:
+        return next(iter(matches))
+    if len(matches) > 1:
+        _raise_ambiguous_metadata_scope(path, sorted(matches), candidate_set)
+
+    _raise_ambiguous_metadata_scope(path, scopes, candidate_set)
+
+
+def _raise_ambiguous_metadata_scope(
+    path: str | Path,
+    scopes: list[str],
+    candidates: set[str],
+) -> None:
+    candidate_text = (
+        f" Candidates were: {', '.join(sorted(candidates))}."
+        if candidates
+        else ""
+    )
+    raise ValueError(
+        f"Multiple ds-crawler metadata scopes are available for {path!s}: "
+        f"{', '.join(scopes)}.{candidate_text} "
+        "Specify Modality(..., metadata_scope=...) or append "
+        "'#scope=<metadata_scope>' to the path."
+    )
 
 
 def load_dataset_config_for_scope(

{euler_loading-2.11.0 → euler_loading-2.12.0}/euler_loading/dataset.py

@@ -31,8 +31,10 @@ except ImportError:  # pragma: no cover - compatibility with older ds-crawler
     _ds_crawler_get_layout_addon = None
 
 from ._ds_crawler_utils import (
+    as_non_empty_str,
+    infer_metadata_scope,
     load_index_output,
-    parse_path_with_split,
+    parse_modality_path,
     validate_metadata_scope,
 )
 from ._metadata import _build_runlog_entry, _get_ds_crawler_descriptor
@@ -303,6 +305,9 @@ class Modality:
               ``output.json`` from a prior indexing run).  A colon-separated
               split suffix is also accepted (e.g. ``/data/ds.zip:train``);
               the suffix is extracted and used as the ``split`` parameter.
+              A metadata scope may also be selected inline with
+              ``#scope=<metadata_scope>``, including after a split suffix
+              (e.g. ``/data/ds.zip:train#scope=rgb``).
         origin_path: Optional original path string before any copying or symlinking for i.e. slurm staging.  
                 This is not used by euler-loading itself but can be useful for experiment 
                 logging to retain references to the original dataset location.
@@ -337,9 +342,10 @@ class Modality:
                logical modalities to share one physical directory or zip while
                keeping separate ``dataset-head.json``, ``ds-crawler.json``,
                ``index.json``, and split files under
-               ``.ds_crawler/<metadata_scope>/``. When scoped artifacts are
-               absent, loading falls back to the legacy root-level
-               ``.ds_crawler`` layout.
+               ``.ds_crawler/<metadata_scope>/``. If omitted, euler-loading
+               infers a scope only when the choice is deterministic. When
+               configured scoped artifacts are absent, loading falls back to
+               the legacy root-level ``.ds_crawler`` layout.
         cache: Optional opt-in/out for in-memory caching of loaded values.
                Only meaningful for hierarchical modalities; regular
                modalities load once per sample and never cache.
@@ -377,7 +383,7 @@ class Modality:
     metadata: dict[str, Any] = field(default_factory=dict)
 
     def __post_init__(self) -> None:
-        parsed_path, path_split = parse_path_with_split(self.path)
+        parsed_path, path_split, path_metadata_scope = parse_modality_path(self.path)
         if path_split is not None:
             if self.split is not None:
                 raise ValueError(
@@ -385,16 +391,93 @@ class Modality:
                     f"({path_split!r}) but an explicit split={self.split!r} "
                     f"was also provided. Use one or the other, not both."
                 )
+
+        normalized_metadata_scope = (
+            validate_metadata_scope(self.metadata_scope)
+            if self.metadata_scope is not None
+            else None
+        )
+        if path_metadata_scope is not None:
+            if (
+                normalized_metadata_scope is not None
+                and normalized_metadata_scope != path_metadata_scope
+            ):
+                raise ValueError(
+                    f"Modality path {self.path!r} contains an inline "
+                    f"metadata scope ({path_metadata_scope!r}) but an "
+                    f"explicit metadata_scope={self.metadata_scope!r} was "
+                    "also provided. Use one metadata scope selector."
+                )
+            normalized_metadata_scope = path_metadata_scope
+
+        if parsed_path != self.path:
             object.__setattr__(self, "path", parsed_path)
+        if path_split is not None:
             object.__setattr__(self, "split", path_split)
-        if self.metadata_scope is not None:
+        if normalized_metadata_scope is not None:
             object.__setattr__(
                 self,
                 "metadata_scope",
-                validate_metadata_scope(self.metadata_scope),
+                normalized_metadata_scope,
             )
 
 
+def _loader_modality_type(loader: Callable[..., Any] | None) -> str | None:
+    meta = getattr(loader, "_modality_meta", None)
+    if not isinstance(meta, Mapping):
+        return None
+    return as_non_empty_str(meta.get("type"))
+
+
+def _metadata_modality_type(modality: Modality) -> str | None:
+    euler_loading = modality.metadata.get("euler_loading")
+    if not isinstance(euler_loading, Mapping):
+        return None
+    return as_non_empty_str(euler_loading.get("modality_type"))
+
+
+def _metadata_scope_candidates(name: str, modality: Modality) -> list[str | None]:
+    return [
+        name,
+        modality.modality_type,
+        _metadata_modality_type(modality),
+        _loader_modality_type(modality.loader),
+    ]
+
+
+def _resolve_metadata_scope(name: str, modality: Modality) -> Modality:
+    if modality.metadata_scope is not None:
+        return modality
+
+    metadata_scope = infer_metadata_scope(
+        modality.path,
+        candidates=_metadata_scope_candidates(name, modality),
+    )
+    if metadata_scope is None:
+        return modality
+    return replace(modality, metadata_scope=metadata_scope)
+
+
+def _resolve_metadata_scopes(
+    modalities: dict[str, Modality],
+) -> dict[str, Modality]:
+    return {
+        name: _resolve_metadata_scope(name, modality)
+        for name, modality in modalities.items()
+    }
+
+
+def _modality_selector_suffix(modality: Modality) -> str:
+    selectors = []
+    if modality.split is not None:
+        selectors.append(f"split={modality.split}")
+    if modality.metadata_scope is not None:
+        selectors.append(f"metadata_scope={modality.metadata_scope}")
+    if not selectors:
+        return ""
+    return f" ({', '.join(selectors)})"
+
+
 class MultiModalDataset(_BaseDataset):
     """PyTorch Dataset that loads synchronized multi-modal samples.
 
@@ -581,6 +664,7 @@ class MultiModalDataset(_BaseDataset):
         if not modalities:
             raise ValueError("At least one modality must be provided.")
 
+        modalities = _resolve_metadata_scopes(modalities)
         indexed: dict[str, dict[str, Any]] = {}
         layouts: dict[str, dict[str, Any] | None] = {}
         for name, modality in modalities.items():
@@ -663,8 +747,12 @@ class MultiModalDataset(_BaseDataset):
         if not modalities:
             raise ValueError("At least one modality must be provided.")
 
+        modalities = _resolve_metadata_scopes(modalities)
+        hierarchical_modalities = _resolve_metadata_scopes(
+            hierarchical_modalities or {}
+        )
         self._modalities = modalities
-        self._hierarchical_modalities = hierarchical_modalities or {}
+        self._hierarchical_modalities = hierarchical_modalities
         self._transforms = transforms or []
         self._index_outputs: dict[str, dict[str, Any]] = {}
         self._hierarchical_index_outputs: dict[str, dict[str, Any]] = {}
@@ -703,7 +791,7 @@ class MultiModalDataset(_BaseDataset):
                 name,
                 len(lookup),
                 modality.path,
-                f" (split={modality.split})" if modality.split is not None else "",
+                _modality_selector_suffix(modality),
             )
 
         # -- Index hierarchical modalities -----------------------------------
@@ -738,7 +826,7 @@ class MultiModalDataset(_BaseDataset):
                 total_files,
                 len(files_by_level),
                 modality.path,
-                f" (split={modality.split})" if modality.split is not None else "",
+                _modality_selector_suffix(modality),
             )
 
         # -- Compute common IDs (intersection across regular modalities) -----

{euler_loading-2.11.0 → euler_loading-2.12.0}/euler_loading/loaders/cpu/real_drive_sim.py

@@ -12,6 +12,9 @@ Return types
   (single-channel class IDs).
 - **sky_mask** – ``np.ndarray`` of shape ``(H, W)`` bool
   (``True`` where class ID == 29).
+- **calibration** – ``dict[str, dict[str, np.ndarray]]`` keyed by sensor
+  name. Each sensor dict contains ``"K"`` (3x3), ``"T"`` (4x4), and
+  ``"distortion"`` (8,).
 
 Usage::
 
@@ -24,6 +27,7 @@ Usage::
 
 from __future__ import annotations
 
+import json
 from typing import Any, BinaryIO, Union
 
 import numpy as np
@@ -109,6 +113,136 @@ def sky_mask(path: Union[str, BinaryIO], meta: dict[str, Any] | None = None, *,
     return np.array(Image.open(path), dtype=np.uint8)[:, :, 0] == _SKY_CLASS_ID
 
 
+# ---------------------------------------------------------------------------
+# Calibration loaders
+# ---------------------------------------------------------------------------
+
+
+def _load_json(path: Union[str, BinaryIO]) -> Any:
+    """Load JSON from a file path or an in-memory buffer."""
+    if isinstance(path, str):
+        with open(path) as f:
+            return json.load(f)
+    return json.load(path)
+
+
+def _quat_to_rotation_matrix(qw: float, qx: float, qy: float, qz: float) -> np.ndarray:
+    """Convert a quaternion ``(qw, qx, qy, qz)`` to a ``(3, 3)`` rotation matrix."""
+    return np.array(
+        [
+            [1 - 2 * (qy * qy + qz * qz), 2 * (qx * qy - qz * qw), 2 * (qx * qz + qy * qw)],
+            [2 * (qx * qy + qz * qw), 1 - 2 * (qx * qx + qz * qz), 2 * (qy * qz - qx * qw)],
+            [2 * (qx * qz - qy * qw), 2 * (qy * qz + qx * qw), 1 - 2 * (qx * qx + qy * qy)],
+        ],
+        dtype=np.float32,
+    )
+
+
+@modality_meta(
+    modality_type="calibration",
+    dtype="dict",
+    hierarchical=True,
+    shape="dict",
+    file_formats=[".json"],
+    meta={"sensors": ["CS_FRONT", "HDL_32E", "HDL_64E"], "keys": ["K", "T", "distortion"]},
+)
+def calibration(path: Union[str, BinaryIO], meta: dict[str, Any] | None = None, *, attributes: dict[str, Any] | None = None) -> dict[str, dict[str, np.ndarray]]:
+    """Load a Real Drive Sim calibration JSON.
+
+    The file contains parallel arrays ``names``, ``intrinsics``, and
+    ``extrinsics`` - one entry per sensor. Returns a dict keyed by sensor
+    name, where each value contains:
+
+    - ``"K"`` - ``(3, 3)`` float32 camera-intrinsics matrix.
+    - ``"T"`` - ``(4, 4)`` float32 extrinsics matrix (rotation + translation).
+    - ``"distortion"`` - ``(8,)`` float32 distortion coefficients
+      ``[k1, k2, p1, p2, k3, k4, k5, k6]``.
+    """
+    data = _load_json(path)
+
+    result: dict[str, dict[str, np.ndarray]] = {}
+    for name, intr, extr in zip(data["names"], data["intrinsics"], data["extrinsics"]):
+        fx, fy = intr["fx"], intr["fy"]
+        cx, cy = intr["cx"], intr["cy"]
+        s = intr["skew"]
+        K = np.array(
+            [[fx, s, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]],
+            dtype=np.float32,
+        )
+
+        rot = extr["rotation"]
+        R = _quat_to_rotation_matrix(rot["qw"], rot["qx"], rot["qy"], rot["qz"])
+        t = np.array(
+            [extr["translation"]["x"], extr["translation"]["y"], extr["translation"]["z"]],
+            dtype=np.float32,
+        )
+        T = np.eye(4, dtype=np.float32)
+        T[:3, :3] = R
+        T[:3, 3] = t
+
+        distortion = np.array(
+            [
+                intr["k1"],
+                intr["k2"],
+                intr["p1"],
+                intr["p2"],
+                intr["k3"],
+                intr["k4"],
+                intr["k5"],
+                intr["k6"],
+            ],
+            dtype=np.float32,
+        )
+
+        result[name] = {"K": K, "T": T, "distortion": distortion}
+    return result
+
+
+@modality_meta(
+    modality_type="all_intrinsics",
+    dtype="dict",
+    hierarchical=True,
+    shape="dict",
+    file_formats=[".json"],
+)
+def all_intrinsics(path: Union[str, BinaryIO], meta: dict[str, Any] | None = None, *, attributes: dict[str, Any] | None = None) -> dict[str, np.ndarray]:
+    """Load only the intrinsics from a Real Drive Sim calibration JSON."""
+    data = _load_json(path)
+    result: dict[str, np.ndarray] = {}
+
+    for name, intr in zip(data["names"], data["intrinsics"]):
+        fx, fy = intr["fx"], intr["fy"]
+        cx, cy = intr["cx"], intr["cy"]
+        s = intr["skew"]
+        K = np.array(
+            [[fx, s, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]],
+            dtype=np.float32,
+        )
+        result[name] = K
+
+    return result
+
+
+@modality_meta(
+    modality_type="intrinsics",
+    dtype="float32",
+    hierarchical=True,
+    shape="3x3",
+    file_formats=[".json"],
+    meta={"sensor": "CS_FRONT"},
+)
+def read_intrinsics(path: Union[str, BinaryIO], meta: dict[str, Any] | None = None, *, attributes: dict[str, Any] | None = None) -> np.ndarray:
+    """Load the intrinsics for a specific sensor from a Real Drive Sim calibration JSON."""
+    sensor = (attributes or {}).get("sensor") or (meta or {}).get("sensor") or "CS_FRONT"
+    intrinsics = all_intrinsics(path)
+    if sensor not in intrinsics:
+        raise KeyError(
+            f"Real Drive Sim calibration has no intrinsics for sensor {sensor!r}. "
+            f"Available sensors: {sorted(intrinsics)}"
+        )
+    return intrinsics[sensor]
+
+
 # ---------------------------------------------------------------------------
 # Writers
 # ---------------------------------------------------------------------------

{euler_loading-2.11.0 → euler_loading-2.12.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "euler-loading"
-version = "2.11.0"
+version = "2.12.0"
 description = "Multi-modal PyTorch dataloader using ds-crawler indices"
 requires-python = ">=3.9"
 dependencies = [

{euler_loading-2.11.0 → euler_loading-2.12.0}/tests/test_dataset.py

@@ -343,6 +343,7 @@ class TestScopedMetadataLoading:
                 ]
             }
         }
+        _write_metadata(root, "index.json", rgb_index)
         _write_metadata(
             root,
             "index.json",
@@ -412,6 +413,170 @@ class TestScopedMetadataLoading:
             "metadata_scope": "rgb",
         }
 
+    def test_metadata_scope_inferred_from_single_available_scope(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        rgb_index = _flat_index("rgb", ["f001"])
+        _write_metadata(root, "index.json", rgb_index, scope="rgb")
+
+        def mock_index(path, **kwargs):
+            assert kwargs.get("metadata_scope") == "rgb"
+            return rgb_index
+
+        with patch(
+            "euler_loading.dataset.index_dataset_from_path",
+            side_effect=mock_index,
+        ):
+            ds = MultiModalDataset(
+                modalities={
+                    "image": Modality(str(root), loader=dummy_loader),
+                },
+            )
+
+        assert len(ds) == 1
+        assert ds.modality_paths()["image"] == {
+            "path": str(root),
+            "origin_path": None,
+            "metadata_scope": "rgb",
+        }
+
+    def test_metadata_scope_inferred_from_modality_name(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        rgb_index = _flat_index("rgb", ["f001"])
+        depth_index = _flat_index("depth", ["f001"])
+        _write_metadata(root, "index.json", rgb_index, scope="rgb")
+        _write_metadata(root, "index.json", depth_index, scope="depth")
+
+        def mock_index(path, **kwargs):
+            assert kwargs.get("metadata_scope") == "rgb"
+            return rgb_index
+
+        with patch(
+            "euler_loading.dataset.index_dataset_from_path",
+            side_effect=mock_index,
+        ):
+            ds = MultiModalDataset(
+                modalities={
+                    "rgb": Modality(str(root), loader=dummy_loader),
+                },
+            )
+
+        assert ds.modality_paths()["rgb"]["metadata_scope"] == "rgb"
+
+    def test_metadata_scope_inferred_from_loader_modality_type(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        rgb_index = _flat_index("rgb", ["f001"])
+        depth_index = _flat_index("depth", ["f001"])
+        _write_metadata(root, "index.json", rgb_index, scope="rgb")
+        _write_metadata(root, "index.json", depth_index, scope="depth")
+
+        def rgb_loader(path, meta=None):
+            return dummy_loader(path, meta)
+
+        rgb_loader._modality_meta = {"type": "rgb"}  # type: ignore[attr-defined]
+
+        def mock_index(path, **kwargs):
+            assert kwargs.get("metadata_scope") == "rgb"
+            return rgb_index
+
+        with patch(
+            "euler_loading.dataset.index_dataset_from_path",
+            side_effect=mock_index,
+        ):
+            ds = MultiModalDataset(
+                modalities={
+                    "image": Modality(str(root), loader=rgb_loader),
+                },
+            )
+
+        assert ds.modality_paths()["image"]["metadata_scope"] == "rgb"
+
+    def test_metadata_scope_inferred_from_scoped_head_modality_key(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        rgb_index = _flat_index("rgb", ["f001"])
+        depth_index = _flat_index("depth", ["f001"])
+        _write_metadata(root, "index.json", rgb_index, scope="scope-a")
+        _write_metadata(
+            root,
+            "dataset-head.json",
+            {"dataset": {"name": "MUSES"}, "modality": {"key": "rgb"}},
+            scope="scope-a",
+        )
+        _write_metadata(root, "index.json", depth_index, scope="scope-b")
+        _write_metadata(
+            root,
+            "dataset-head.json",
+            {"dataset": {"name": "MUSES"}, "modality": {"key": "depth"}},
+            scope="scope-b",
+        )
+
+        def mock_index(path, **kwargs):
+            assert kwargs.get("metadata_scope") == "scope-a"
+            return rgb_index
+
+        with patch(
+            "euler_loading.dataset.index_dataset_from_path",
+            side_effect=mock_index,
+        ):
+            ds = MultiModalDataset(
+                modalities={
+                    "image": Modality(
+                        str(root),
+                        loader=dummy_loader,
+                        modality_type="rgb",
+                    ),
+                },
+            )
+
+        assert ds.modality_paths()["image"]["metadata_scope"] == "scope-a"
+
+    def test_multiple_metadata_scopes_without_match_raises(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        _write_metadata(root, "index.json", _flat_index("rgb", ["f001"]), scope="rgb")
+        _write_metadata(
+            root,
+            "index.json",
+            _flat_index("depth", ["f001"]),
+            scope="depth",
+        )
+
+        with pytest.raises(ValueError, match="Multiple ds-crawler metadata scopes"):
+            MultiModalDataset(
+                modalities={
+                    "image": Modality(str(root), loader=dummy_loader),
+                },
+            )
+
+    def test_root_metadata_prevents_scope_inference(self, tmp_path):
+        root = tmp_path / "muses"
+        root.mkdir()
+        rgb_index = _flat_index("rgb", ["f001"])
+        _write_metadata(root, "index.json", rgb_index)
+        _write_metadata(root, "index.json", _flat_index("depth", ["f001"]), scope="rgb")
+
+        def mock_index(path, **kwargs):
+            assert "metadata_scope" not in kwargs
+            return rgb_index
+
+        with patch(
+            "euler_loading.dataset.index_dataset_from_path",
+            side_effect=mock_index,
+        ):
+            ds = MultiModalDataset(
+                modalities={
+                    "rgb": Modality(str(root), loader=dummy_loader),
+                },
+            )
+
+        assert ds.modality_paths()["rgb"] == {
+            "path": str(root),
+            "origin_path": None,
+        }
+
 
 class TestPathWithColonSplit:
     """Colon-separated path:split syntax."""
@@ -436,14 +601,39 @@ class TestPathWithColonSplit:
         assert mod.path == "/data/ds"
         assert mod.split == "train"
 
+    def test_scope_extracted_from_path(self):
+        mod = Modality("/data/ds.zip#scope=rgb")
+        assert mod.path == "/data/ds.zip"
+        assert mod.split is None
+        assert mod.metadata_scope == "rgb"
+
+    def test_split_and_scope_extracted_from_path(self):
+        mod = Modality("/data/ds.zip:train#scope=rgb")
+        assert mod.path == "/data/ds.zip"
+        assert mod.split == "train"
+        assert mod.metadata_scope == "rgb"
+
+    def test_inline_scope_can_match_explicit_scope(self):
+        mod = Modality("/data/ds.zip#scope=rgb", metadata_scope=" rgb ")
+        assert mod.path == "/data/ds.zip"
+        assert mod.metadata_scope == "rgb"
+
     def test_colon_and_explicit_split_raises(self):
         with pytest.raises(ValueError, match="inline split"):
             Modality("/data/ds:train", split="val")
 
+    def test_inline_scope_conflict_raises(self):
+        with pytest.raises(ValueError, match="inline metadata scope"):
+            Modality("/data/ds.zip#scope=rgb", metadata_scope="depth")
+
     def test_invalid_metadata_scope_raises(self):
         with pytest.raises(ValueError, match="metadata_scope"):
             Modality("/data/ds", metadata_scope="bad/scope")
 
+    def test_invalid_inline_metadata_scope_raises(self):
+        with pytest.raises(ValueError, match="metadata_scope"):
+            Modality("/data/ds.zip#scope=bad/scope")
+
     def test_windows_drive_letter_not_treated_as_split(self):
         mod = Modality("C:\\data\\ds")
         assert mod.path == "C:\\data\\ds"

{euler_loading-2.11.0 → euler_loading-2.12.0}/tests/test_loaders.py

@@ -813,6 +813,47 @@ class TestRDSCalibration:
         assert self.result["CS_FRONT"]["distortion"].dtype == torch.float32
 
 
+class TestRDSCPUCalibration:
+    """CPU calibration loader parses the Real Drive Sim JSON format."""
+
+    @pytest.fixture(autouse=True)
+    def _load(self):
+        self.result = cpu_rds.calibration(_RDS_CALIB_PATH)
+
+    def test_returns_dict_keyed_by_sensor_name(self):
+        assert set(self.result.keys()) == {"CS_FRONT", "HDL_32E", "HDL_64E"}
+
+    def test_each_sensor_has_expected_keys(self):
+        for sensor in self.result.values():
+            assert set(sensor.keys()) == {"K", "T", "distortion"}
+
+    def test_intrinsics_shape(self):
+        assert self.result["CS_FRONT"]["K"].shape == (3, 3)
+
+    def test_intrinsics_dtype(self):
+        assert self.result["CS_FRONT"]["K"].dtype == np.float32
+
+    def test_intrinsics_values(self):
+        K = self.result["CS_FRONT"]["K"]
+        assert np.isclose(K[0, 0], 2262.52001953125)
+        assert np.isclose(K[1, 1], 2265.3017578125)
+        assert np.isclose(K[0, 2], 1096.97998046875)
+        assert np.isclose(K[1, 2], 513.1370239257812)
+        assert K[2, 2] == 1.0
+
+    def test_read_intrinsics_returns_front_camera_matrix(self):
+        K = cpu_rds.read_intrinsics(_RDS_CALIB_PATH)
+        assert K.shape == (3, 3)
+        assert K.dtype == np.float32
+        assert np.isclose(K[0, 0], 2262.52001953125)
+
+    def test_read_intrinsics_can_select_sensor_from_attributes(self):
+        K = cpu_rds.read_intrinsics(_RDS_CALIB_PATH, attributes={"sensor": "HDL_32E"})
+        assert K.shape == (3, 3)
+        assert K.dtype == np.float32
+        assert K[0, 0] == 0.0
+
+
 # ---------------------------------------------------------------------------
 # Writer round-trip tests
 # ---------------------------------------------------------------------------