euler-loading 2.9.0__tar.gz → 2.11.0__tar.gz

{euler_loading-2.9.0 → euler_loading-2.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-loading
-Version: 2.9.0
+Version: 2.11.0
 Summary: Multi-modal PyTorch dataloader using ds-crawler indices
 Requires-Python: >=3.9
 Requires-Dist: ds-crawler>=2.3.0

{euler_loading-2.9.0 → euler_loading-2.11.0}/README.md

@@ -55,7 +55,7 @@ Frozen dataclass describing one data modality.
 
 | Parameter | Type | Description |
 |-----------|------|-------------|
-| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain a `ds-crawler.config` or cached `output.json`. |
+| `path` | `str` | Absolute path to the modality root directory or `.zip` archive. Must contain ds-crawler metadata, either at `.ds_crawler/` or under the configured `metadata_scope`. |
 | `origin_path` | `str \| None` | Original path before copying/symlinking (e.g. for SLURM staging). Not used by euler-loading itself — useful for experiment logging to retain references to the original dataset location. |
 | `loader` | `Callable[..., Any] \| None` | Receives the file path (or `BinaryIO` buffer for zip-backed modalities) and an optional `meta` dict. Returns loaded data. When `None`, the loader is resolved automatically from the ds-crawler index (see [Automatic loader resolution](#automatic-loader-resolution)). |
 | `writer` | `Callable[..., Any] \| None` | Receives `(path, value, meta)` and writes modality data to disk. When `None`, euler-loading tries to resolve a built-in writer from ds-crawler metadata (`write_<function>` or `write_<suffix>` for `read_<suffix>`). |
@@ -65,6 +65,7 @@ Frozen dataclass describing one data modality.
 | `hierarchy_scope` | `str \| None` | Optional scope label for hierarchical modalities (e.g. `scene_camera`). |
 | `applies_to` | `list[str] \| None` | Optional list of regular modality names a hierarchical modality applies to. |
 | `split` | `str \| None` | Optional inline split name. Loads `.ds_crawler/split_<name>.json` from the modality root (directory or zip) and overlays it on the normal ds-crawler metadata. |
+| `metadata_scope` | `str \| None` | Optional namespace below `.ds_crawler`, e.g. `.ds_crawler/camera_extrinsics/index.json`. Use this when multiple logical modalities share one physical directory or zip. If the scoped artifacts are absent, loading falls back to the legacy root-level `.ds_crawler` layout. |
 | `cache` | `bool \| None` | Opt-in/out for in-memory caching of decoded values (only meaningful for hierarchical modalities; regular modalities are never cached). `True` keeps every distinct loaded file in a process-lifetime dict. `False` re-reads on every access. `None` (default): hierarchical → `True` because small shared calibration files benefit. |
 | `collapse_single` | `bool` | For hierarchical modalities only. When `True`, a sample that matches exactly one hierarchical file returns that loaded value directly instead of `{file_id: value}`. This is useful for shared GT tensors used by multiple augmentations of the same source sample. |
 | `metadata` | `dict[str, Any]` | Optional arbitrary metadata. Keys under `metadata["euler_loading"]` are treated as euler-loading defaults. |
@@ -129,13 +130,50 @@ dataset = MultiModalDataset(
 
 This works for both directory-backed and zip-backed modalities. The split file only replaces the `dataset` payload; top-level metadata such as dataset type and euler-loading loader hints still come from the canonical ds-crawler index.
 
+### Scoped ds-crawler Metadata
+
+When a single physical root or zip contains files for several logical modalities, keep one ds-crawler artifact set per modality under `.ds_crawler/<metadata_scope>/`:
+
+```
+muses.zip
+  calib.json
+  frames/...
+  lidar/...
+  .ds_crawler/rgb/dataset-head.json
+  .ds_crawler/rgb/ds-crawler.json
+  .ds_crawler/rgb/index.json
+  .ds_crawler/camera_extrinsics/dataset-head.json
+  .ds_crawler/camera_extrinsics/ds-crawler.json
+  .ds_crawler/camera_extrinsics/index.json
+```
+
+Then point multiple modalities at the same `path` and select the appropriate metadata scope:
+
+```python
+dataset = MultiModalDataset(
+    modalities={
+        "rgb": Modality("/data/muses.zip", metadata_scope="rgb"),
+        "sparse_depth": Modality("/data/muses.zip", metadata_scope="sparse_depth"),
+    },
+    hierarchical_modalities={
+        "camera_extrinsics": Modality(
+            "/data/muses.zip",
+            metadata_scope="camera_extrinsics",
+            collapse_single=True,
+        ),
+    },
+)
+```
+
+This is an additive layout. Existing roots with `.ds_crawler/index.json` continue to load unchanged, and a scoped modality falls back to that legacy location when the scoped artifacts are not present.
+
 ### `MultiModalDataset.modality_paths()`
 
-Returns a dict mapping each regular modality name to `{"path": ..., "origin_path": ...}` and includes `split` when configured.
+Returns a dict mapping each regular modality name to `{"path": ..., "origin_path": ...}` and includes `split` and `metadata_scope` when configured.
 
 ### `MultiModalDataset.hierarchical_modality_paths()`
 
-Returns a dict mapping each hierarchical modality name to `{"path": ..., "origin_path": ...}` and includes `split` when configured.
+Returns a dict mapping each hierarchical modality name to `{"path": ..., "origin_path": ...}` and includes `split` and `metadata_scope` when configured.
 
 ### `MultiModalDataset.get_modality_metadata(modality_name)`
 
@@ -145,7 +183,7 @@ Returns the ds-crawler metadata dict for the given modality.
 
 PyTorch `Dataset`. On construction it:
 
-1. Runs `ds_crawler.index_dataset_from_path()` for every modality (regular and hierarchical).
+1. Loads each modality's ds-crawler index from `.ds_crawler/<metadata_scope>/index.json` when `metadata_scope` is configured and present; otherwise uses the legacy root-level ds-crawler lookup.
 2. Computes the **sorted intersection** of file IDs across all regular modalities.
 3. Logs warnings for unmatched files; raises `ValueError` when the intersection is empty.
 
@@ -155,6 +193,17 @@ PyTorch `Dataset`. On construction it:
 | `hierarchical_modalities` | `dict[str, Modality] \| None` | Optional modalities whose files live at intermediate hierarchy levels (e.g. per-scene intrinsics). These do **not** participate in ID intersection. Each sample will contain a dict `{file_id: loaded_result}` with all files at or above the sample's hierarchy level. Results are cached so shared files are parsed only once. |
 | `transforms` | `list[Callable[[dict], dict]] \| None` | Applied in order after loading. Each receives and returns the full sample dict. |
 
+For hierarchical modalities, the ds-crawler file ID is the key in
+`{file_id: loaded_result}`. If files at different ancestor levels use the same
+ID, the deepest matching file wins. This gives calibration files natural
+inheritance semantics: a root-level `intrinsics` file can apply to every sample,
+while a scene- or camera-level `intrinsics` file with the same ID overrides it
+for descendants. Use ds-crawler's `indexing.id.override` when the physical file
+name is incidental, such as `calib.json` or a UUID, and you want a stable
+semantic key like `intrinsics`, `extrinsics`, or `calibration`. Do not use an
+override when multiple distinct files at the same hierarchy level should all be
+returned.
+
 #### Sample dict
 
 `dataset[i]` returns:
@@ -319,11 +368,15 @@ Writer resolution uses the same module and function metadata:
 
 ## ds-crawler integration
 
-Every modality root must be independently indexable by ds-crawler.
-Place a `ds-crawler.config` in the root of each modality directory (or zip archive) — ds-crawler will then parse the directory tree and assign each file an ID derived from its path properties.
-Files across modalities are matched by these IDs, so **the directory structure must be consistent** across modalities (identical hierarchy and naming conventions up to the modality-specific parts captured in the config).
+Each logical modality must have its own ds-crawler index. Usually that means one modality root with its own `.ds_crawler` artifacts. When several logical modalities share one physical directory or zip, put each artifact set under `.ds_crawler/<metadata_scope>/` and set `Modality(..., metadata_scope=...)`.
+Files across regular modalities are matched by IDs from those indexes, so **the indexed hierarchy and naming conventions must be consistent** across modalities up to modality-specific parts captured in the config.
 
 Calibration files or other per-scene/per-sequence metadata can be loaded via `hierarchical_modalities`. These files are matched to samples based on their position in the hierarchy — all files at or above a sample's hierarchy level are included and cached for efficiency.
+Root-level calibration files, for example `calib.json` at the top of a shared
+zip, should be indexed by ds-crawler without a hierarchy block. euler-loading
+treats that root file as an ancestor of every sample. If the modality uses
+`collapse_single=True`, the loaded calibration value is returned directly as
+long as exactly one hierarchical file matches the sample.
 
 ## Layout-aware augmentation loading
 
@@ -447,9 +500,12 @@ All built-in loaders accept both filesystem paths (`str`) and in-memory buffers
 | `reference_rgb` | Clear-weather reference RGB image as float32, normalised to [0, 1] |
 | `semantic_segmentation` | Single-channel Cityscapes `labelIds` or `labelTrainIds` PNG as class IDs |
 | `semantic_segmentation_color` | Cityscapes `labelColor` PNG as RGB uint8 labels |
+| `sky_mask` | Binary mask from Cityscapes labels; uses sky `trainId=10` for `labelTrainIds`, `labelId=23` for `labelIds`, or RGB `meta["sky_class"]` such as `[0, 0, 23]` |
 | `panoptic_segmentation` | COCO-style RGB panoptic PNG decoded to integer segment IDs |
 | `lidar_point_cloud` / `point_cloud` | Lidar `.bin` file as `(N, 6)` float64: `x, y, z, intensity, ring, timestamp` |
 | `sparse_depth` | Alias for MUSES lidar points for sparse-depth style workflows |
+| `read_intrinsics` | RGB camera intrinsics from MUSES `calib.json` as a `(3, 3)` matrix; defaults to `sensor="rgb"` |
+| `read_extrinsics` | Camera extrinsics from MUSES `calib.json` as a `(4, 4)` matrix; defaults to `transform="lidar2rgb"` for sparse-depth reprojection into RGB pixels |
 
 ### Generic Dense Depth (`euler_loading.loaders.gpu.generic_dense_depth`)
 

{euler_loading-2.9.0 → euler_loading-2.11.0}/euler_loading/_ds_crawler_utils.py

@@ -1,17 +1,22 @@
 from __future__ import annotations
 
-import re
 from collections.abc import Mapping
 from pathlib import Path
 from typing import Any
 
-from ds_crawler import get_dataset_contract
+from ds_crawler import (
+    get_dataset_contract,
+    load_dataset_config,
+    load_dataset_split as _load_dataset_split,
+    validate_metadata_scope,
+)
 from ds_crawler.zip_utils import read_metadata_json
-
-try:
-    from ds_crawler import load_dataset_split as _load_dataset_split
-except ImportError:
-    _load_dataset_split = None
+from ds_crawler.config import CONFIG_FILENAME as _DS_CRAWLER_CONFIG_FILENAME
+from ds_crawler.zip_utils import (
+    OUTPUT_FILENAME as _DS_CRAWLER_OUTPUT_FILENAME,
+    get_split_filename,
+    validate_split_name,
+)
 
 
 def as_non_empty_str(value: Any) -> str | None:
@@ -58,9 +63,6 @@ def extract_ds_crawler_properties(index_output: Mapping[str, Any]) -> dict[str,
         return {}
 
 
-_SPLIT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]*$")
-
-
 def parse_path_with_split(path: str) -> tuple[str, str | None]:
     """Extract an optional inline split suffix from a colon-separated path.
 
@@ -78,36 +80,55 @@ def parse_path_with_split(path: str) -> tuple[str, str | None]:
     candidate_split = path[colon_pos + 1:]
     candidate_path = path[:colon_pos]
 
-    if not candidate_split or not _SPLIT_NAME_PATTERN.match(candidate_split):
+    try:
+        split = validate_split_name(candidate_split)
+    except ValueError:
         return path, None
 
-    return candidate_path, candidate_split
+    return candidate_path, split
 
 
-def validate_split_name(split_name: str) -> str:
-    """Validate and normalize an inline split name."""
-    if not isinstance(split_name, str):
-        raise ValueError("split_name must be a string")
-    normalized = split_name.strip()
-    if not normalized:
-        raise ValueError("split_name must be a non-empty string")
-    if not _SPLIT_NAME_PATTERN.match(normalized):
-        raise ValueError(
-            "split_name may only contain letters, digits, '.', '-', or '_'"
-        )
-    return normalized
+def _has_scoped_metadata(dataset_path: Path, metadata_scope: str) -> bool:
+    """Return whether ds-crawler metadata exists under a configured scope."""
+    for filename in (_DS_CRAWLER_OUTPUT_FILENAME, _DS_CRAWLER_CONFIG_FILENAME):
+        if read_metadata_json(
+            dataset_path,
+            filename,
+            metadata_scope=metadata_scope,
+        ) is not None:
+            return True
+    return False
 
 
-def get_split_filename(split_name: str) -> str:
-    """Return the ds-crawler metadata filename for *split_name*."""
-    normalized = validate_split_name(split_name)
-    return f"split_{normalized}.json"
+def load_dataset_config_for_scope(
+    path: str | Path,
+    *,
+    metadata_scope: str | None,
+    load_dataset_config_fn: Any = load_dataset_config,
+    fallback_to_legacy: bool = True,
+) -> Any:
+    """Load a ds-crawler config, delegating scoped layout handling to ds-crawler."""
+    dataset_path = Path(path)
+    if metadata_scope is None:
+        return load_dataset_config_fn({"path": str(path)})
+
+    normalized_scope = validate_metadata_scope(metadata_scope)
+    try:
+        return load_dataset_config_fn(
+            {"path": str(path)},
+            metadata_scope=normalized_scope,
+        )
+    except FileNotFoundError:
+        if fallback_to_legacy and not _has_scoped_metadata(dataset_path, normalized_scope):
+            return load_dataset_config_fn({"path": str(path)})
+        raise
 
 
 def load_index_output(
     path: str | Path,
     *,
     split: str | None,
+    metadata_scope: str | None = None,
     index_dataset_from_path_fn: Any,
     strict: bool = False,
     save_index: bool = False,
@@ -115,12 +136,39 @@ def load_index_output(
 ) -> dict[str, Any]:
     """Load a ds-crawler output, optionally overlaying an inline split.
 
-    When the installed ds-crawler version exposes ``load_dataset_split()``, it
-    is used directly. Older versions are supported by loading the canonical
-    output via ``index_dataset_from_path()`` and replacing ``output["dataset"]``
-    with the contents of ``.ds_crawler/split_<name>.json``.
+    Scoped metadata is loaded through ds-crawler's native ``metadata_scope``
+    support. If a scope is configured but no scoped metadata exists at all,
+    euler-loading still falls back to the legacy root-level lookup for
+    backward compatibility with older datasets.
     """
     dataset_path = Path(path)
+    normalized_scope = (
+        validate_metadata_scope(metadata_scope)
+        if metadata_scope is not None
+        else None
+    )
+
+    if normalized_scope is not None:
+        try:
+            if split is not None:
+                return _load_dataset_split(
+                    path,
+                    validate_split_name(split),
+                    strict=strict,
+                    save_index=save_index,
+                    force_reindex=force_reindex,
+                    metadata_scope=normalized_scope,
+                )
+            return index_dataset_from_path_fn(
+                path,
+                strict=strict,
+                save_index=save_index,
+                force_reindex=force_reindex,
+                metadata_scope=normalized_scope,
+            )
+        except FileNotFoundError:
+            if _has_scoped_metadata(dataset_path, normalized_scope):
+                raise
 
     if split is None:
         return index_dataset_from_path_fn(
@@ -131,19 +179,18 @@ def load_index_output(
         )
 
     normalized_split = validate_split_name(split)
-    if _load_dataset_split is not None:
-        try:
-            return _load_dataset_split(
-                path,
-                normalized_split,
-                strict=strict,
-                save_index=save_index,
-                force_reindex=force_reindex,
-            )
-        except FileNotFoundError:
-            # Fallback for mocked tests or callers that provide synthetic index
-            # loaders without a full ds-crawler config on disk.
-            pass
+    try:
+        return _load_dataset_split(
+            path,
+            normalized_split,
+            strict=strict,
+            save_index=save_index,
+            force_reindex=force_reindex,
+        )
+    except FileNotFoundError:
+        # Fallback for mocked tests or callers that provide synthetic index
+        # loaders without a full ds-crawler config on disk.
+        pass
 
     base_output = index_dataset_from_path_fn(
         path,

{euler_loading-2.9.0 → euler_loading-2.11.0}/euler_loading/_metadata.py

@@ -12,6 +12,7 @@ from ._ds_crawler_utils import (
     extract_ds_crawler_properties,
     first_non_empty,
     first_non_empty_list,
+    load_dataset_config_for_scope,
 )
 
 if TYPE_CHECKING:
@@ -66,6 +67,8 @@ def _build_runlog_entry(
     entry: dict[str, Any] = {"path": modality.path}
     if modality.split is not None:
         entry["split"] = modality.split
+    if modality.metadata_scope is not None:
+        entry["metadata_scope"] = modality.metadata_scope
     if used_as is not None:
         entry["used_as"] = used_as
     if slot is not None:
@@ -125,22 +128,26 @@ def _resolve_euler_loading_property(
 def _get_ds_crawler_descriptor(
     *,
     path: str,
+    metadata_scope: str | None = None,
     index_output: Mapping[str, Any] | None,
     cache: dict[str, dict[str, Any]],
     load_dataset_config_fn: Callable[[dict[str, Any]], Any],
 ) -> dict[str, Any]:
-    if path not in cache:
-        cache[path] = _read_ds_crawler_descriptor(
+    cache_key = f"{path}::{metadata_scope or ''}"
+    if cache_key not in cache:
+        cache[cache_key] = _read_ds_crawler_descriptor(
             path=path,
+            metadata_scope=metadata_scope,
             index_output=index_output,
             load_dataset_config_fn=load_dataset_config_fn,
         )
-    return cache[path]
+    return cache[cache_key]
 
 
 def _read_ds_crawler_descriptor(
     *,
     path: str,
+    metadata_scope: str | None,
     index_output: Mapping[str, Any] | None,
     load_dataset_config_fn: Callable[[dict[str, Any]], Any],
 ) -> dict[str, Any]:
@@ -164,7 +171,11 @@ def _read_ds_crawler_descriptor(
                     descriptor["hierarchy_regex"] = hierarchy_regex
 
     try:
-        cfg = load_dataset_config_fn({"path": path})
+        cfg = load_dataset_config_for_scope(
+            path,
+            metadata_scope=metadata_scope,
+            load_dataset_config_fn=load_dataset_config_fn,
+        )
     except Exception:
         return descriptor
 

{euler_loading-2.9.0 → euler_loading-2.11.0}/euler_loading/dataset.py

@@ -30,7 +30,11 @@ try:
 except ImportError:  # pragma: no cover - compatibility with older ds-crawler
     _ds_crawler_get_layout_addon = None
 
-from ._ds_crawler_utils import load_index_output, parse_path_with_split
+from ._ds_crawler_utils import (
+    load_index_output,
+    parse_path_with_split,
+    validate_metadata_scope,
+)
 from ._metadata import _build_runlog_entry, _get_ds_crawler_descriptor
 from ._resolution import (
     _resolve_loader,
@@ -328,6 +332,14 @@ class Modality:
         split: Optional inline split name. When set, euler-loading reads the
                full ds-crawler metadata from the modality root and overlays the
                dataset payload from ``.ds_crawler/split_<name>.json``.
+        metadata_scope: Optional subdirectory below ``.ds_crawler`` to use for
+               this modality's ds-crawler artifacts. This allows multiple
+               logical modalities to share one physical directory or zip while
+               keeping separate ``dataset-head.json``, ``ds-crawler.json``,
+               ``index.json``, and split files under
+               ``.ds_crawler/<metadata_scope>/``. When scoped artifacts are
+               absent, loading falls back to the legacy root-level
+               ``.ds_crawler`` layout.
         cache: Optional opt-in/out for in-memory caching of loaded values.
                Only meaningful for hierarchical modalities; regular
                modalities load once per sample and never cache.
@@ -359,6 +371,7 @@ class Modality:
     hierarchy_scope: str | None = None
     applies_to: list[str] | None = None
     split: str | None = None
+    metadata_scope: str | None = None
     cache: bool | None = None
     collapse_single: bool = False
     metadata: dict[str, Any] = field(default_factory=dict)
@@ -374,6 +387,12 @@ class Modality:
                 )
             object.__setattr__(self, "path", parsed_path)
             object.__setattr__(self, "split", path_split)
+        if self.metadata_scope is not None:
+            object.__setattr__(
+                self,
+                "metadata_scope",
+                validate_metadata_scope(self.metadata_scope),
+            )
 
 
 class MultiModalDataset(_BaseDataset):
@@ -433,6 +452,8 @@ class MultiModalDataset(_BaseDataset):
             entry: dict[str, str | None] = {"path": path, "origin_path": origin}
             if mod.split is not None:
                 entry["split"] = mod.split
+            if mod.metadata_scope is not None:
+                entry["metadata_scope"] = mod.metadata_scope
             res[name] = entry
         return res
 
@@ -450,6 +471,8 @@ class MultiModalDataset(_BaseDataset):
             entry: dict[str, str | None] = {"path": path, "origin_path": origin}
             if mod.split is not None:
                 entry["split"] = mod.split
+            if mod.metadata_scope is not None:
+                entry["metadata_scope"] = mod.metadata_scope
             res[name] = entry
         return res
 
@@ -496,6 +519,7 @@ class MultiModalDataset(_BaseDataset):
         for name, modality in self._modalities.items():
             descriptor = _get_ds_crawler_descriptor(
                 path=modality.path,
+                metadata_scope=modality.metadata_scope,
                 index_output=self._index_outputs.get(name),
                 cache=descriptor_cache,
                 load_dataset_config_fn=load_dataset_config,
@@ -513,6 +537,7 @@ class MultiModalDataset(_BaseDataset):
         for name, modality in self._hierarchical_modalities.items():
             descriptor = _get_ds_crawler_descriptor(
                 path=modality.path,
+                metadata_scope=modality.metadata_scope,
                 index_output=self._hierarchical_index_outputs.get(name),
                 cache=descriptor_cache,
                 load_dataset_config_fn=load_dataset_config,
@@ -562,6 +587,7 @@ class MultiModalDataset(_BaseDataset):
             index = load_index_output(
                 modality.path,
                 split=modality.split,
+                metadata_scope=modality.metadata_scope,
                 index_dataset_from_path_fn=index_dataset_from_path,
             )
             indexed[name] = index
@@ -653,6 +679,7 @@ class MultiModalDataset(_BaseDataset):
             index = load_index_output(
                 modality.path,
                 split=modality.split,
+                metadata_scope=modality.metadata_scope,
                 index_dataset_from_path_fn=index_dataset_from_path,
             )
             self._index_outputs[name] = index
@@ -688,6 +715,7 @@ class MultiModalDataset(_BaseDataset):
             index = load_index_output(
                 modality.path,
                 split=modality.split,
+                metadata_scope=modality.metadata_scope,
                 index_dataset_from_path_fn=index_dataset_from_path,
             )
             self._hierarchical_index_outputs[name] = index