euler-loading 2.11.0__tar.gz → 2.14.0__tar.gz

{euler_loading-2.11.0 → euler_loading-2.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-loading
-Version: 2.11.0
+Version: 2.14.0
 Summary: Multi-modal PyTorch dataloader using ds-crawler indices
 Requires-Python: >=3.9
 Requires-Dist: ds-crawler>=2.3.0

{euler_loading-2.11.0 → euler_loading-2.14.0}/README.md

@@ -55,7 +55,7 @@ Frozen dataclass describing one data modality.
 
 | Parameter | Type | Description |
 |-----------|------|-------------|
-| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain ds-crawler metadata, either at `.ds_crawler/` or under the configured `metadata_scope`. |
+| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain ds-crawler metadata, either at `.ds_crawler/` or under the configured `metadata_scope`. Inline selectors are accepted as `/data/ds.zip:train`, `/data/ds.zip#scope=rgb`, or `/data/ds.zip:train#scope=rgb`. |
 | `origin_path` | `str \| None` | Original path before copying/symlinking (e.g. for SLURM staging). Not used by euler-loading itself — useful for experiment logging to retain references to the original dataset location. |
 | `loader` | `Callable[..., Any] \| None` | Receives the file path (or `BinaryIO` buffer for zip-backed modalities) and an optional `meta` dict. Returns loaded data. When `None`, the loader is resolved automatically from the ds-crawler index (see [Automatic loader resolution](#automatic-loader-resolution)). |
 | `writer` | `Callable[..., Any] \| None` | Receives `(path, value, meta)` and writes modality data to disk. When `None`, euler-loading tries to resolve a built-in writer from ds-crawler metadata (`write_<function>` or `write_<suffix>` for `read_<suffix>`). |
@@ -65,7 +65,7 @@ Frozen dataclass describing one data modality.
 | `hierarchy_scope` | `str \| None` | Optional scope label for hierarchical modalities (e.g. `scene_camera`). |
 | `applies_to` | `list[str] \| None` | Optional list of regular modality names a hierarchical modality applies to. |
 | `split` | `str \| None` | Optional inline split name. Loads `.ds_crawler/split_<name>.json` from the modality root (directory or zip) and overlays it on the normal ds-crawler metadata. |
-| `metadata_scope` | `str \| None` | Optional namespace below `.ds_crawler`, e.g. `.ds_crawler/camera_extrinsics/index.json`. Use this when multiple logical modalities share one physical directory or zip. If the scoped artifacts are absent, loading falls back to the legacy root-level `.ds_crawler` layout. |
+| `metadata_scope` | `str \| None` | Optional namespace below `.ds_crawler`, e.g. `.ds_crawler/camera_extrinsics/index.json`. Use this when multiple logical modalities share one physical directory or zip. It can also be supplied inline as `#scope=<metadata_scope>`. If omitted, euler-loading infers it only when the choice is deterministic. If configured artifacts are absent, loading falls back to the legacy root-level `.ds_crawler` layout. |
 | `cache` | `bool \| None` | Opt-in/out for in-memory caching of decoded values (only meaningful for hierarchical modalities; regular modalities are never cached). `True` keeps every distinct loaded file in a process-lifetime dict. `False` re-reads on every access. `None` (default): hierarchical → `True` because small shared calibration files benefit. |
 | `collapse_single` | `bool` | For hierarchical modalities only. When `True`, a sample that matches exactly one hierarchical file returns that loaded value directly instead of `{file_id: value}`. This is useful for shared GT tensors used by multiple augmentations of the same source sample. |
 | `metadata` | `dict[str, Any]` | Optional arbitrary metadata. Keys under `metadata["euler_loading"]` are treated as euler-loading defaults. |
@@ -128,6 +128,12 @@ dataset = MultiModalDataset(
 )
 ```
 
+Path-only callers can use the existing colon form directly:
+
+```python
+Modality("/data/rgb:train")
+```
+
 This works for both directory-backed and zip-backed modalities. The split file only replaces the `dataset` payload; top-level metadata such as dataset type and euler-loading loader hints still come from the canonical ds-crawler index.
 
 ### Scoped ds-crawler Metadata
@@ -165,6 +171,23 @@ dataset = MultiModalDataset(
 )
 ```
 
+Path-only callers can select the same scope inline. The scope selector comes
+after the optional split selector:
+
+```python
+Modality("/data/muses.zip#scope=rgb")
+Modality("/data/muses.zip:train#scope=rgb")
+```
+
+If `metadata_scope` is omitted, euler-loading resolves it automatically only
+when doing so is unambiguous:
+
+1. Explicit `metadata_scope` or `#scope=...` wins.
+2. If root-level `.ds_crawler` metadata exists, the legacy root metadata is used.
+3. If no root metadata exists and exactly one scope exists, that scope is used.
+4. If multiple scopes exist, euler-loading matches the modality dict key, `modality_type`, loader `_modality_meta["type"]`, or scoped `dataset-head.json` `modality.key`.
+5. If the result is still ambiguous, construction raises and lists the available scopes.
+
 This is an additive layout. Existing roots with `.ds_crawler/index.json` continue to load unchanged, and a scoped modality falls back to that legacy location when the scoped artifacts are not present.
 
 ### `MultiModalDataset.modality_paths()`
@@ -358,7 +381,7 @@ When `Modality.loader` is `None`, euler-loading resolves the loader from the ds-
 }
 ```
 
-`loader` is the module name (`vkitti2`, `real_drive_sim`, `muses`, or `generic_dense_depth`) and `function` is the function within that module. The GPU variant is used by default.
+`loader` is the module name (`vkitti2`, `real_drive_sim`, `muses`, `princeton_dense`, or `generic_dense_depth`) and `function` is the function within that module. The GPU variant is used by default.
 
 Writer resolution uses the same module and function metadata:
 
@@ -368,7 +391,7 @@ Writer resolution uses the same module and function metadata:
 
 ## ds-crawler integration
 
-Each logical modality must have its own ds-crawler index. Usually that means one modality root with its own `.ds_crawler` artifacts. When several logical modalities share one physical directory or zip, put each artifact set under `.ds_crawler/<metadata_scope>/` and set `Modality(..., metadata_scope=...)`.
+Each logical modality must have its own ds-crawler index. Usually that means one modality root with its own `.ds_crawler` artifacts. When several logical modalities share one physical directory or zip, put each artifact set under `.ds_crawler/<metadata_scope>/` and set `Modality(..., metadata_scope=...)` or append `#scope=<metadata_scope>` to the path.
 Files across regular modalities are matched by IDs from those indexes, so **the indexed hierarchy and naming conventions must be consistent** across modalities up to modality-specific parts captured in the config.
 
 Calibration files or other per-scene/per-sequence metadata can be loaded via `hierarchical_modalities`. These files are matched to samples based on their position in the hierarchy — all files at or above a sample's hierarchy level are included and cached for efficiency.
@@ -507,6 +530,15 @@ All built-in loaders accept both filesystem paths (`str`) and in-memory buffers
 | `read_intrinsics` | RGB camera intrinsics from MUSES `calib.json` as a `(3, 3)` matrix; defaults to `sensor="rgb"` |
 | `read_extrinsics` | Camera extrinsics from MUSES `calib.json` as a `(4, 4)` matrix; defaults to `transform="lidar2rgb"` for sparse-depth reprojection into RGB pixels |
 
+### Princeton DENSE / SeeingThroughFog (`euler_loading.loaders.princeton_dense`)
+
+| Function | Description |
+|----------|-------------|
+| `rgb` | Left/right stereo 12-bit Bayer TIFF as RGB float32, normalised to [0, 1] |
+| `sparse_depth` | Lidar `.bin` point cloud as `(N, 5)` float32: `x, y, z, intensity, ring` |
+| `read_intrinsics` | Left stereo camera intrinsics `K` from `calib_cam_stereo_left.json` as a `(3, 3)` matrix |
+| `read_extrinsics` | Transform from HDL64 lidar frame `lidar_hdl64_s3_roof` to left camera optical frame `cam_stereo_left_optical` from `calib_tf_tree_full.json` as a `(4, 4)` matrix |
+
 ### Generic Dense Depth (`euler_loading.loaders.gpu.generic_dense_depth`)
 
 A format-agnostic loader that infers the loading strategy from the file extension. Useful for datasets that don't have a dedicated loader module.
@@ -518,7 +550,7 @@ A format-agnostic loader that infers the loading strategy from the file extensio
 | `sky_mask` | Binary mask by comparing pixels against `meta["sky_mask"]` (`[R, G, B]`). Requires `meta` |
 | `read_intrinsics` | Returns `meta["intrinsics"]` as a `(3, 3)` tensor. Ignores path; requires `meta` |
 
-CPU variants of all loaders live under `euler_loading.loaders.cpu.{vkitti2,real_drive_sim,muses,generic_dense_depth}`.
+CPU variants of all loaders live under `euler_loading.loaders.cpu.{vkitti2,real_drive_sim,muses,princeton_dense,generic_dense_depth}`.
 
 ### Flattening hierarchical modalities
 

{euler_loading-2.11.0 → euler_loading-2.14.0}/euler_loading/_ds_crawler_utils.py

@@ -3,18 +3,21 @@ from __future__ import annotations
 from collections.abc import Mapping
 from pathlib import Path
 from typing import Any
+from urllib.parse import parse_qsl
 
 from ds_crawler import (
     get_dataset_contract,
+    list_metadata_scopes,
     load_dataset_config,
     load_dataset_split as _load_dataset_split,
     validate_metadata_scope,
 )
-from ds_crawler.zip_utils import read_metadata_json
 from ds_crawler.config import CONFIG_FILENAME as _DS_CRAWLER_CONFIG_FILENAME
 from ds_crawler.zip_utils import (
+    DATASET_HEAD_FILENAME as _DS_CRAWLER_HEAD_FILENAME,
     OUTPUT_FILENAME as _DS_CRAWLER_OUTPUT_FILENAME,
     get_split_filename,
+    read_metadata_json,
     validate_split_name,
 )
 
@@ -63,7 +66,38 @@ def extract_ds_crawler_properties(index_output: Mapping[str, Any]) -> dict[str,
         return {}
 
 
-def parse_path_with_split(path: str) -> tuple[str, str | None]:
+_METADATA_FILENAMES = (
+    _DS_CRAWLER_OUTPUT_FILENAME,
+    _DS_CRAWLER_CONFIG_FILENAME,
+    _DS_CRAWLER_HEAD_FILENAME,
+)
+
+
+def _parse_path_with_scope(path: str) -> tuple[str, str | None]:
+    """Extract an optional ``#scope=<name>`` selector from a path."""
+    candidate_path, sep, fragment = path.rpartition("#")
+    if not sep:
+        return path, None
+
+    params = parse_qsl(fragment, keep_blank_values=True)
+    scope_values = [value for key, value in params if key == "scope"]
+    if not scope_values:
+        return path, None
+
+    unsupported = [key for key, _ in params if key != "scope"]
+    if unsupported:
+        raise ValueError(
+            "Unsupported modality path fragment parameter(s): "
+            f"{', '.join(sorted(set(unsupported)))}. "
+            "Only '#scope=<metadata_scope>' is supported."
+        )
+    if len(scope_values) > 1:
+        raise ValueError("Modality path may contain at most one '#scope=' selector.")
+
+    return candidate_path, validate_metadata_scope(scope_values[0])
+
+
+def _parse_path_with_split(path: str) -> tuple[str, str | None]:
     """Extract an optional inline split suffix from a colon-separated path.
 
     Supports paths like ``/data/ds.zip:train`` where the part after the
@@ -88,16 +122,134 @@ def parse_path_with_split(path: str) -> tuple[str, str | None]:
     return candidate_path, split
 
 
+def parse_modality_path(path: str) -> tuple[str, str | None, str | None]:
+    """Extract inline ``:split`` and ``#scope=...`` selectors from a path."""
+    parsed_path, metadata_scope = _parse_path_with_scope(path)
+    parsed_path, split = _parse_path_with_split(parsed_path)
+    return parsed_path, split, metadata_scope
+
+
+def parse_path_with_split(path: str) -> tuple[str, str | None]:
+    """Extract an optional inline split suffix from a colon-separated path."""
+    parsed_path, split, _ = parse_modality_path(path)
+    return parsed_path, split
+
+
+def _read_metadata_json_safe(
+    dataset_path: Path,
+    filename: str,
+    *,
+    metadata_scope: str | None = None,
+) -> dict[str, Any] | None:
+    try:
+        return read_metadata_json(
+            dataset_path,
+            filename,
+            metadata_scope=metadata_scope,
+        )
+    except FileNotFoundError:
+        return None
+
+
+def _has_root_metadata(dataset_path: Path) -> bool:
+    """Return whether unscoped ds-crawler metadata exists at a dataset root."""
+    return any(
+        _read_metadata_json_safe(dataset_path, filename) is not None
+        for filename in _METADATA_FILENAMES
+    )
+
+
 def _has_scoped_metadata(dataset_path: Path, metadata_scope: str) -> bool:
     """Return whether ds-crawler metadata exists under a configured scope."""
-    for filename in (_DS_CRAWLER_OUTPUT_FILENAME, _DS_CRAWLER_CONFIG_FILENAME):
-        if read_metadata_json(
+    return any(
+        _read_metadata_json_safe(
             dataset_path,
             filename,
             metadata_scope=metadata_scope,
-        ) is not None:
-            return True
-    return False
+        ) is not None
+        for filename in _METADATA_FILENAMES
+    )
+
+
+def _list_metadata_scopes_safe(dataset_path: Path) -> list[str]:
+    try:
+        return list_metadata_scopes(dataset_path)
+    except FileNotFoundError:
+        return []
+
+
+def _read_scope_modality_key(dataset_path: Path, metadata_scope: str) -> str | None:
+    head = _read_metadata_json_safe(
+        dataset_path,
+        _DS_CRAWLER_HEAD_FILENAME,
+        metadata_scope=metadata_scope,
+    )
+    modality = head.get("modality") if isinstance(head, Mapping) else None
+    key = modality.get("key") if isinstance(modality, Mapping) else None
+    return as_non_empty_str(key)
+
+
+def infer_metadata_scope(
+    path: str | Path,
+    *,
+    candidates: list[str | None],
+) -> str | None:
+    """Infer a ds-crawler metadata scope only when the choice is deterministic.
+
+    Legacy root-level metadata always wins for an unscoped modality. If no
+    root metadata exists, a single available scope is selected automatically.
+    With multiple scopes, a scope is selected only when exactly one scope name
+    or scoped ``dataset-head.json`` modality key matches the provided
+    candidates.
+    """
+    dataset_path = Path(path)
+    if _has_root_metadata(dataset_path):
+        return None
+
+    scopes = _list_metadata_scopes_safe(dataset_path)
+    if not scopes:
+        return None
+    if len(scopes) == 1:
+        return scopes[0]
+
+    candidate_set = {
+        candidate
+        for candidate in (as_non_empty_str(value) for value in candidates)
+        if candidate is not None
+    }
+    if not candidate_set:
+        _raise_ambiguous_metadata_scope(path, scopes, candidate_set)
+
+    matches = {
+        scope
+        for scope in scopes
+        if scope in candidate_set
+        or _read_scope_modality_key(dataset_path, scope) in candidate_set
+    }
+    if len(matches) == 1:
+        return next(iter(matches))
+    if len(matches) > 1:
+        _raise_ambiguous_metadata_scope(path, sorted(matches), candidate_set)
+
+    _raise_ambiguous_metadata_scope(path, scopes, candidate_set)
+
+
+def _raise_ambiguous_metadata_scope(
+    path: str | Path,
+    scopes: list[str],
+    candidates: set[str],
+) -> None:
+    candidate_text = (
+        f" Candidates were: {', '.join(sorted(candidates))}."
+        if candidates
+        else ""
+    )
+    raise ValueError(
+        f"Multiple ds-crawler metadata scopes are available for {path!s}: "
+        f"{', '.join(scopes)}.{candidate_text} "
+        "Specify Modality(..., metadata_scope=...) or append "
+        "'#scope=<metadata_scope>' to the path."
+    )
 
 
 def load_dataset_config_for_scope(

{euler_loading-2.11.0 → euler_loading-2.14.0}/euler_loading/_resolution.py

@@ -24,6 +24,8 @@ _LOADER_MODULES: dict[str, str] = {
     "generic_dense_depth": "euler_loading.loaders.gpu.generic_dense_depth",
     "generic": "euler_loading.loaders.gpu.generic",
     "muses": "euler_loading.loaders.gpu.muses",
+    "princeton_dense": "euler_loading.loaders.gpu.princeton_dense",
+    "seeing_through_fog": "euler_loading.loaders.gpu.princeton_dense",
 }
 
 

{euler_loading-2.11.0 → euler_loading-2.14.0}/euler_loading/dataset.py

@@ -31,8 +31,10 @@ except ImportError:  # pragma: no cover - compatibility with older ds-crawler
     _ds_crawler_get_layout_addon = None
 
 from ._ds_crawler_utils import (
+    as_non_empty_str,
+    infer_metadata_scope,
     load_index_output,
-    parse_path_with_split,
+    parse_modality_path,
     validate_metadata_scope,
 )
 from ._metadata import _build_runlog_entry, _get_ds_crawler_descriptor
@@ -303,6 +305,9 @@ class Modality:
               ``output.json`` from a prior indexing run).  A colon-separated
               split suffix is also accepted (e.g. ``/data/ds.zip:train``);
               the suffix is extracted and used as the ``split`` parameter.
+              A metadata scope may also be selected inline with
+              ``#scope=<metadata_scope>``, including after a split suffix
+              (e.g. ``/data/ds.zip:train#scope=rgb``).
         origin_path: Optional original path string before any copying or symlinking for i.e. slurm staging.  
                 This is not used by euler-loading itself but can be useful for experiment 
                 logging to retain references to the original dataset location.
@@ -337,9 +342,10 @@ class Modality:
                logical modalities to share one physical directory or zip while
                keeping separate ``dataset-head.json``, ``ds-crawler.json``,
                ``index.json``, and split files under
-               ``.ds_crawler/<metadata_scope>/``. When scoped artifacts are
-               absent, loading falls back to the legacy root-level
-               ``.ds_crawler`` layout.
+               ``.ds_crawler/<metadata_scope>/``. If omitted, euler-loading
+               infers a scope only when the choice is deterministic. When
+               configured scoped artifacts are absent, loading falls back to
+               the legacy root-level ``.ds_crawler`` layout.
         cache: Optional opt-in/out for in-memory caching of loaded values.
                Only meaningful for hierarchical modalities; regular
                modalities load once per sample and never cache.
@@ -377,7 +383,7 @@ class Modality:
     metadata: dict[str, Any] = field(default_factory=dict)
 
     def __post_init__(self) -> None:
-        parsed_path, path_split = parse_path_with_split(self.path)
+        parsed_path, path_split, path_metadata_scope = parse_modality_path(self.path)
         if path_split is not None:
             if self.split is not None:
                 raise ValueError(
@@ -385,16 +391,93 @@ class Modality:
                     f"({path_split!r}) but an explicit split={self.split!r} "
                     f"was also provided. Use one or the other, not both."
                 )
+
+        normalized_metadata_scope = (
+            validate_metadata_scope(self.metadata_scope)
+            if self.metadata_scope is not None
+            else None
+        )
+        if path_metadata_scope is not None:
+            if (
+                normalized_metadata_scope is not None
+                and normalized_metadata_scope != path_metadata_scope
+            ):
+                raise ValueError(
+                    f"Modality path {self.path!r} contains an inline "
+                    f"metadata scope ({path_metadata_scope!r}) but an "
+                    f"explicit metadata_scope={self.metadata_scope!r} was "
+                    "also provided. Use one metadata scope selector."
+                )
+            normalized_metadata_scope = path_metadata_scope
+
+        if parsed_path != self.path:
             object.__setattr__(self, "path", parsed_path)
+        if path_split is not None:
             object.__setattr__(self, "split", path_split)
-        if self.metadata_scope is not None:
+        if normalized_metadata_scope is not None:
             object.__setattr__(
                 self,
                 "metadata_scope",
-                validate_metadata_scope(self.metadata_scope),
+                normalized_metadata_scope,
             )
 
 
+def _loader_modality_type(loader: Callable[..., Any] | None) -> str | None:
+    meta = getattr(loader, "_modality_meta", None)
+    if not isinstance(meta, Mapping):
+        return None
+    return as_non_empty_str(meta.get("type"))
+
+
+def _metadata_modality_type(modality: Modality) -> str | None:
+    euler_loading = modality.metadata.get("euler_loading")
+    if not isinstance(euler_loading, Mapping):
+        return None
+    return as_non_empty_str(euler_loading.get("modality_type"))
+
+
+def _metadata_scope_candidates(name: str, modality: Modality) -> list[str | None]:
+    return [
+        name,
+        modality.modality_type,
+        _metadata_modality_type(modality),
+        _loader_modality_type(modality.loader),
+    ]
+
+
+def _resolve_metadata_scope(name: str, modality: Modality) -> Modality:
+    if modality.metadata_scope is not None:
+        return modality
+
+    metadata_scope = infer_metadata_scope(
+        modality.path,
+        candidates=_metadata_scope_candidates(name, modality),
+    )
+    if metadata_scope is None:
+        return modality
+    return replace(modality, metadata_scope=metadata_scope)
+
+
+def _resolve_metadata_scopes(
+    modalities: dict[str, Modality],
+) -> dict[str, Modality]:
+    return {
+        name: _resolve_metadata_scope(name, modality)
+        for name, modality in modalities.items()
+    }
+
+
+def _modality_selector_suffix(modality: Modality) -> str:
+    selectors = []
+    if modality.split is not None:
+        selectors.append(f"split={modality.split}")
+    if modality.metadata_scope is not None:
+        selectors.append(f"metadata_scope={modality.metadata_scope}")
+    if not selectors:
+        return ""
+    return f" ({', '.join(selectors)})"
+
+
 class MultiModalDataset(_BaseDataset):
     """PyTorch Dataset that loads synchronized multi-modal samples.
 
@@ -581,6 +664,7 @@ class MultiModalDataset(_BaseDataset):
         if not modalities:
             raise ValueError("At least one modality must be provided.")
 
+        modalities = _resolve_metadata_scopes(modalities)
         indexed: dict[str, dict[str, Any]] = {}
         layouts: dict[str, dict[str, Any] | None] = {}
         for name, modality in modalities.items():
@@ -663,8 +747,12 @@ class MultiModalDataset(_BaseDataset):
         if not modalities:
             raise ValueError("At least one modality must be provided.")
 
+        modalities = _resolve_metadata_scopes(modalities)
+        hierarchical_modalities = _resolve_metadata_scopes(
+            hierarchical_modalities or {}
+        )
         self._modalities = modalities
-        self._hierarchical_modalities = hierarchical_modalities or {}
+        self._hierarchical_modalities = hierarchical_modalities
         self._transforms = transforms or []
         self._index_outputs: dict[str, dict[str, Any]] = {}
         self._hierarchical_index_outputs: dict[str, dict[str, Any]] = {}
@@ -703,7 +791,7 @@ class MultiModalDataset(_BaseDataset):
                 name,
                 len(lookup),
                 modality.path,
-                f" (split={modality.split})" if modality.split is not None else "",
+                _modality_selector_suffix(modality),
             )
 
         # -- Index hierarchical modalities -----------------------------------
@@ -738,7 +826,7 @@ class MultiModalDataset(_BaseDataset):
                 total_files,
                 len(files_by_level),
                 modality.path,
-                f" (split={modality.split})" if modality.split is not None else "",
+                _modality_selector_suffix(modality),
             )
 
         # -- Compute common IDs (intersection across regular modalities) -----

{euler_loading-2.11.0 → euler_loading-2.14.0}/euler_loading/loaders/__init__.py

@@ -27,4 +27,8 @@ Available submodules:
 - :mod:`euler_loading.loaders.cpu.real_drive_sim` — Real Drive Sim (numpy)
 - :mod:`euler_loading.loaders.gpu.muses` — MUSES (torch)
 - :mod:`euler_loading.loaders.cpu.muses` — MUSES (numpy)
+- :mod:`euler_loading.loaders.gpu.princeton_dense` — Princeton DENSE /
+  SeeingThroughFog (torch)
+- :mod:`euler_loading.loaders.cpu.princeton_dense` — Princeton DENSE /
+  SeeingThroughFog (numpy)
 """