euler-loading 2.5.0__tar.gz → 2.7.0__tar.gz

{euler_loading-2.5.0 → euler_loading-2.7.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: euler-loading
-Version: 2.5.0
+Version: 2.7.0
 Summary: Multi-modal PyTorch dataloader using ds-crawler indices
 Requires-Python: >=3.9
-Requires-Dist: ds-crawler
+Requires-Dist: ds-crawler>=2.3.0
 Requires-Dist: euler-dataset-contract
 Requires-Dist: numpy
 Requires-Dist: pillow

{euler_loading-2.5.0 → euler_loading-2.7.0}/README.md

@@ -4,7 +4,7 @@ Multi-modal PyTorch `Dataset` that synchronises files across arbitrary dataset m
 
 Each modality points at a directory (or `.zip` archive) that carries its own `ds-crawler.config` (or cached `output.json`).
 ds-crawler indexes the directory tree, discovers files, and exposes hierarchical metadata (path properties, calibration files, …).
-euler-loading then **intersects file IDs** across all modalities so that every sample contains exactly one file per modality. Additional hierarchical data (e.g. per-scene calibration files) can be loaded via `hierarchical_modalities`.
+euler-loading then **intersects file IDs** across all modalities so that every sample contains exactly one file per modality. Additional hierarchical data (e.g. per-scene calibration files) can be loaded via `hierarchical_modalities`. For augmentation-style datasets — many augmented files per ground-truth sample — `keyed_modalities` joins each augmented sample to its single GT file via the value encoded in the augmented sample's deepest hierarchy key.
 How a file is actually **loaded** (image, depth map, point cloud, …) is configurable per modality — either supply a `Callable` or let euler-loading resolve a built-in loader automatically from the ds-crawler config.
 Writer functions can be resolved the same way, so inference outputs can be written back in dataset-native formats.
 
@@ -65,6 +65,8 @@ Frozen dataclass describing one data modality.
 | `hierarchy_scope` | `str \| None` | Optional scope label for hierarchical modalities (e.g. `scene_camera`). |
 | `applies_to` | `list[str] \| None` | Optional list of regular modality names a hierarchical modality applies to. |
 | `split` | `str \| None` | Optional inline split name. Loads `.ds_crawler/split_<name>.json` from the modality root (directory or zip) and overlays it on the normal ds-crawler metadata. |
+| `keyed_by` | `Mapping[str, str] \| None` | Optional join configuration used only when this modality is passed under `MultiModalDataset(keyed_modalities=...)`. Recognised keys: `key_name` (named-group prefix at the regular modality's deepest hierarchy key, e.g. `"file_id"`; auto-detected from the anchor's data when omitted) and `modality` (anchor regular-modality name; auto-inferred when there's exactly one regular). Both keys are optional — the kwarg can be left unset entirely. |
+| `cache` | `bool \| None` | Opt-in/out for in-memory caching of decoded values (only meaningful for hierarchical and keyed modalities; regular modalities are never cached). `True` keeps every distinct loaded file in a process-lifetime dict. `False` re-reads on every access. `None` (default): hierarchical → `True` (small calibration files benefit), keyed → `False` (safe for the augmentation case where the keyed modality holds full per-sample tensors and unbounded caching would OOM). |
 | `metadata` | `dict[str, Any]` | Optional arbitrary metadata. Keys under `metadata["euler_loading"]` are treated as euler-loading defaults. |
 
 The loader is the **only** place where domain-specific I/O happens.
@@ -135,23 +137,30 @@ Returns a dict mapping each regular modality name to `{"path": ..., "origin_path
 
 Returns a dict mapping each hierarchical modality name to `{"path": ..., "origin_path": ...}` and includes `split` when configured.
 
+### `MultiModalDataset.keyed_modality_paths()`
+
+Returns a dict mapping each keyed modality name to `{"path": ..., "origin_path": ...}` plus `keyed_by_modality` (the resolved anchor) and `keyed_by_key_name` (the resolved or auto-detected key name). Includes `split` when configured. Useful for verifying which prefix the auto-detection picked.
+
 ### `MultiModalDataset.get_modality_metadata(modality_name)`
 
 Returns the ds-crawler metadata dict for the given modality.
 
-### `MultiModalDataset(modalities, hierarchical_modalities=None, transforms=None)`
+### `MultiModalDataset(modalities, hierarchical_modalities=None, transforms=None, keyed_modalities=None, strict_keyed=False)`
 
 PyTorch `Dataset`. On construction it:
 
-1. Runs `ds_crawler.index_dataset_from_path()` for every modality (regular and hierarchical).
+1. Runs `ds_crawler.index_dataset_from_path()` for every modality (regular, hierarchical, and keyed).
 2. Computes the **sorted intersection** of file IDs across all regular modalities.
-3. Logs warnings for unmatched files; raises `ValueError` when the intersection is empty.
+3. Validates keyed-modality joins for every common id; drops samples whose join misses (or raises when `strict_keyed=True`).
+4. Logs warnings for unmatched files; raises `ValueError` when the intersection is empty or all samples are dropped by keyed-join validation.
 
 | Parameter | Type | Description |
 |-----------|------|-------------|
 | `modalities` | `dict[str, Modality]` | At least one entry required. Keys become the sample dict keys. These modalities participate in ID intersection. |
 | `hierarchical_modalities` | `dict[str, Modality] \| None` | Optional modalities whose files live at intermediate hierarchy levels (e.g. per-scene intrinsics). These do **not** participate in ID intersection. Each sample will contain a dict `{file_id: loaded_result}` with all files at or above the sample's hierarchy level. Results are cached so shared files are parsed only once. |
+| `keyed_modalities` | `dict[str, Modality] \| None` | Optional modalities joined to a regular sample by the value of the regular sample's deepest hierarchy key. See [Keyed modalities](#keyed-modalities) below. Contributes a single loaded value per sample (not a dict). Caching is **off by default** for keyed modalities (per-sample tensors can be large); set `Modality(..., cache=True)` to opt in when the keyed dataset is small enough to fit in memory. |
 | `transforms` | `list[Callable[[dict], dict]] \| None` | Applied in order after loading. Each receives and returns the full sample dict. |
+| `strict_keyed` | `bool` | When `True`, missing or mis-decoded keyed-modality joins raise immediately at construction instead of warning + dropping the affected samples. Default `False`. |
 
 #### Sample dict
 
@@ -166,16 +175,26 @@ PyTorch `Dataset`. On construction it:
         ...
     },
     ...
+    "<keyed_modality_name>": <loader result>,  # one entry per keyed modality (single value)
+    ...
     "id":          str,                   # file ID (leaf only, shared across modalities)
     "full_id":     str,                   # full hierarchical path including file ID (e.g. "/scene/camera/frame")
-    "meta":        {                      # per-modality ds-crawler file entries (regular modalities only)
-        "<modality_name>": {"id": ..., "path": ..., "path_properties": ..., "basename_properties": ...},
+    "meta":        {                      # per-modality ds-crawler file entries
+        "<modality_name>": {"id": ..., "path": ..., "path_properties": ..., "basename_properties": ..., "attributes": ...},
+        ...
+    },
+    "attributes":  {                      # per-modality top-level surface for file_entry["attributes"]
+        "<modality_name>": {...},                # for regular and keyed modalities (single dict)
+        "<hierarchical_modality_name>": {        # for hierarchical modalities (one dict per matched file)
+            "<file_id>": {...},
+            ...
+        },
         ...
     },
 }
 ```
 
-Hierarchical modality results are cached so shared files are parsed only once.
+Hierarchical modality results are cached by default; keyed modality results are not. Both can be flipped via `Modality(..., cache=True|False)` — see the [`cache` row in the `Modality` table](#modalitypath--loadernone-metadatanone) and [Keyed modalities](#keyed-modalities) for details on when each default is the right choice.
 
 ### `FileRecord`
 
@@ -315,6 +334,98 @@ Files across modalities are matched by these IDs, so **the directory structure m
 
 Calibration files or other per-scene/per-sequence metadata can be loaded via `hierarchical_modalities`. These files are matched to samples based on their position in the hierarchy — all files at or above a sample's hierarchy level are included and cached for efficiency.
 
+## Keyed modalities
+
+A **keyed modality** joins each regular sample to a single file in another dataset by reading the *value* of the regular sample's deepest hierarchy key. This is the right tool when one modality augments samples of another at a different hierarchy depth — typically: many augmented files per ground-truth sample.
+
+### When to use it
+
+Layout where the augmented modality nests files under an extra `file_id:<id>` level, while the GT keeps the file id as the filename stem:
+
+```
+augmented_rgb/
+  scene_000000/CS_FRONT/file_id:000000000000000025/mor_10m.png
+  scene_000000/CS_FRONT/file_id:000000000000000025/mor_20m.png
+  scene_000000/CS_FRONT/file_id:000000000000000026/mor_10m.png
+  ...
+gt_depth/
+  scene_000000/CS_FRONT/000000000000000025.png
+  scene_000000/CS_FRONT/000000000000000026.png
+  ...
+```
+
+ds-crawler indexes both layouts unchanged; the augmented modality's `indexing.hierarchy.separator` must be `":"` (or any single separator) so the deepest key `"file_id:000…025"` decodes into `("file_id", "000…025")`.
+
+### Wiring
+
+```python
+from euler_loading import Modality, MultiModalDataset
+
+dataset = MultiModalDataset(
+    modalities={
+        "rgb_aug": Modality("/data/augmented_rgb", loader=load_rgb),
+    },
+    keyed_modalities={
+        "depth": Modality("/data/gt_depth", loader=load_depth),
+    },
+)
+
+sample = dataset[0]
+# sample["rgb_aug"]   – the per-aug RGB
+# sample["depth"]     – the GT depth for this file_id (single value, shared across augs)
+# sample["full_id"]   – e.g. "/scene_000000/CS_FRONT/file_id:000…025/mor_10m"
+# sample["id"]        – the augmentation's leaf id (e.g. "mor_10m")
+```
+
+`keyed_by` is optional — both `key_name` (the named-group prefix at the regular modality's deepest hierarchy key) and `modality` (the anchor regular modality) are auto-detected when unambiguous. Set them explicitly when:
+
+- there are multiple regular modalities (anchor must be picked: `keyed_by={"modality": "rgb_aug", "key_name": "file_id"}`); or
+- the anchor's deepest hierarchy keys mix multiple prefixes (e.g. some `file_id:…` and some `frame:…`).
+
+### Sample shape
+
+A keyed modality contributes a **single loaded value** per sample, unlike hierarchical modalities which return `{file_id: loader_result}`. The join returns exactly one record by construction (one GT per augmented sample's file id at the parent hierarchy prefix), so a dict would be misleading.
+
+### How the join works
+
+For each common id in the regular modalities, euler-loading:
+
+1. Looks up the anchor record's `hierarchy_path`, e.g. `(scene_000000, CS_FRONT, file_id:000…025)`.
+2. Splits the deepest key on the regular modality's `indexing.hierarchy.separator` → `("file_id", "000…025")`.
+3. Verifies the key name matches the configured (or auto-detected) `key_name`.
+4. Looks up the keyed modality's record at hierarchy `(scene_000000, CS_FRONT)` with `id == "000…025"`.
+
+### Caching
+
+Caching for keyed modalities is **off by default**, which differs from hierarchical modalities (default on). The reason is workload shape: hierarchical files are usually tiny calibration / metadata blobs that you'd happily keep in memory forever, whereas a keyed modality typically holds the same kind of full per-sample tensor as a regular modality. Caching every decoded GT depth tensor in a 500 GB dataset would OOM the process.
+
+Override per modality:
+
+```python
+keyed_modalities={
+    # Small GT (a few hundred MB total): worth caching to avoid re-reads
+    # across augmentations of the same file_id.
+    "depth": Modality("/data/gt_depth", loader=load_depth, cache=True),
+    # Large GT (multi-hundred-GB total): keep it off, re-read each access.
+    "lidar": Modality("/data/gt_lidar", loader=load_lidar),  # cache=False default
+}
+```
+
+When `cache=True`, the GT for `file_id:000…025` is decoded once and reused across every augmentation that points at it. When `cache=False` (the default), every sample access re-reads and re-decodes the file.
+
+### Validation and missing joins
+
+Construction-time validation runs the decoding for every common id. Samples whose decode or lookup fails are dropped from the dataset with a per-modality warning summarising the count. Pass `strict_keyed=True` to raise instead — useful in pipelines where dropped samples should be a hard error. Common error cases (with precise messages):
+
+- key-name mismatch (deepest key starts with a different prefix);
+- regular modality with no `indexing.hierarchy.separator`;
+- ambiguous auto-detection (multiple distinct deepest-key prefixes in the anchor);
+- regular modality with no hierarchy at all.
+
+### Writing keyed-modality outputs
+
+`MultiModalDataset.write_sample(...)` accepts keyed modality names alongside regular ones. The destination path is derived from the keyed record (the GT's relative path), so prediction outputs land in the GT-shape layout — *not* under a synthetic `file_id:` subdirectory.
+
 ## DenseDepthLoader protocol
 
 `euler_loading.DenseDepthLoader` is a `runtime_checkable` Protocol defining the loader contract for dense-depth datasets. A conforming module must expose:

euler_loading-2.7.0/docs/loader-attributes.md

@@ -0,0 +1,183 @@
+# Design note: passing per-file `attributes` to loaders
+
+Status: **implemented**
+
+This is a follow-up to the per-file `attributes` field added in
+ds-crawler (`file_entry["attributes"]`) and surfaced on samples in
+euler-loading (`sample["attributes"][modality]` and
+`sample["meta"][modality]["attributes"]`).
+
+That change persists arbitrary per-file metadata through the index
+schema and exposes it on the loaded sample. Loader callables can now opt
+in to receiving the per-file dict by accepting an `attributes=` keyword.
+Legacy loaders that accept only `(path, meta)` keep receiving the old
+call shape.
+
+## When this matters
+
+Cases where the loader genuinely needs `attributes` rather than the
+consumer reading it post-hoc on the sample:
+
+- Per-file noise level / scale that must be applied during decode
+  (e.g. depth scale-to-meters that varies per scan).
+- Per-file masks / clip ranges for normalization.
+- Per-file lookup keys that select an external resource (e.g. a
+  per-file calibration tag pointing into a calibration database).
+- Anything where the value can't be reconstructed once the file is
+  decoded into a tensor.
+
+If `attributes` is purely descriptive (training weight, source tag,
+log fields), keep it on `sample["attributes"][...]` and don't change
+loader signatures.
+
+## Loader signature
+
+`euler_loading/loaders/contracts.py`:
+
+```python
+def rgb(
+    self,
+    path: str | BinaryIO,
+    meta: dict[str, Any] | None = None,
+    *,
+    attributes: dict[str, Any] | None = None,
+) -> torch.Tensor: ...
+```
+
+Implemented behavior:
+
+- `dataset.py` probes each resolved loader once with `inspect.signature`
+  and caches whether it accepts `attributes` or `**kwargs`.
+- Compatible loaders receive a copied per-file attributes dict (or
+  `None` when the file entry has no attributes).
+- Legacy loaders that do not accept the keyword receive the original
+  `(path, meta)` call.
+- Hierarchical-modality cache keys include the attributes payload for
+  opted-in loaders, so the same path with different attributes can be
+  decoded differently.
+- Built-in CPU/GPU loaders accept `attributes=None`. Most ignore it;
+  `generic_dense_depth.depth` consumes
+  `attributes["scale_to_meters_override"]` when present.
+
+### Rejected alternative: merge into `meta`
+
+Extend `meta` from "modality-level meta" to "merged meta" — keep the
+original keys plus a reserved `meta["__attributes__"]` (or
+`meta["entry"]`) sub-dict. No signature change.
+
+Pros: no signature change, no probe.
+Cons: namespace collision risk; loaders parsing `meta` in surprising
+ways may misinterpret the new key. Hides the source of each field.
+Harder to type. Implicit contract.
+
+### Rejected alternative: pass the full file entry
+
+Replace `meta` with the file entry dict (carrying `attributes`,
+`path_properties`, etc.). Strictly more information, but breaks every
+existing loader.
+
+Pros: maximum flexibility.
+Cons: gratuitous churn, explicit migration of every loader.
+
+## Implementation notes
+
+### 1. Update contracts
+
+```python
+# euler_loading/loaders/contracts.py
+class DenseDepthLoader(Protocol):
+    def rgb(
+        self,
+        path: Union[str, BinaryIO],
+        meta: dict[str, Any] | None = None,
+        *,
+        attributes: dict[str, Any] | None = None,
+    ) -> torch.Tensor: ...
+    # ... and parallel updates to depth / sky_mask / read_intrinsics ...
+```
+
+### 2. Loader feature probe
+
+The utility that identifies whether each loader accepts the
+`attributes` kwarg:
+
+```python
+# euler_loading/_resolution.py
+import inspect
+
+def loader_accepts_attributes(loader: Callable[..., Any]) -> bool:
+    try:
+        sig = inspect.signature(loader)
+    except (TypeError, ValueError):
+        return False
+    params = sig.parameters
+    if "attributes" in params:
+        return True
+    return any(
+        p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()
+    )
+```
+
+Cache the result on the resolved-loaders dict at construction time so
+the probe runs once per modality.
+
+### 3. Update call sites
+
+In `MultiModalDataset.__getitem__`:
+
+```python
+loader = self._resolved_loaders[name]
+file_attrs = _get_file_attributes(record.file_entry)
+if self._loaders_accept_attributes[name]:
+    sample[name] = loader(file_or_path, modality_meta, attributes=file_attrs)
+else:
+    sample[name] = loader(file_or_path, modality_meta)
+```
+
+The hierarchical-modality branch follows the same rule. For opted-in
+loaders, the hierarchical cache key includes a serialized attributes
+fragment so identical paths with different per-file attributes are
+loaded independently.
+
+### 4. Update built-in loaders
+
+For each module under `euler_loading/loaders/{cpu,gpu}/`, every `read_*`
+and modality function accepts `*, attributes: ... = None`. Most ignore
+it. `generic_dense_depth.depth` consumes
+`attributes.get("scale_to_meters_override")`.
+
+### 5. Tests
+
+Mirror the existing `tests/test_writing.py` and
+`tests/test_dataset.py` structure. Minimum coverage:
+
+- A loader that accepts `attributes=` receives the per-file dict.
+- A loader that doesn't is called with the legacy 2-arg form (no
+  `TypeError`).
+- A loader with `**kwargs` is treated as accepting attributes.
+- Hierarchical-modality cache: same hierarchical file with two different
+  attributes loads twice.
+- Round-trip: file entry attributes → loader receives them → if the
+  loader passes them through, they appear on `sample["attributes"]`
+  (already the case post the existing change, but pin it).
+
+## Backwards compatibility
+
+- Stability of the existing 2-arg signature is preserved by feature
+  probing. Loaders that never opt in keep working forever.
+- The `attributes` argument is keyword-only — no position collision.
+- `sample["attributes"]` and `sample["meta"][name]["attributes"]`
+  surfaces are unchanged by this work; they already exist.
+
+## Open questions for the implementer
+
+1. Should the feature probe also be done for *writers*? Today writers
+   take `(path, value, meta)`. If we want symmetry, writers gain
+   `attributes=` too. Same Option 1 treatment.
+2. Should `loader_accepts_attributes` look at the function's
+   `__wrapped__` chain to handle `functools.wraps`-decorated loaders?
+   (Probably yes, but only if a real case appears.)
+3. Worth adding a `FileContext` dataclass — `(meta, attributes,
+   file_entry)` — to consolidate future fields rather than growing kwargs
+   one at a time? Defer this until we have a second per-file thing to
+   add; one new field doesn't justify a wrapper type.

{euler_loading-2.5.0 → euler_loading-2.7.0}/euler_loading/_resolution.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import importlib
+import inspect
 import logging
 from collections.abc import Mapping
 from types import ModuleType
@@ -68,6 +69,32 @@ def resolve_writer_module(name: str) -> ModuleType:
     return resolve_loader_module(name)
 
 
+def loader_accepts_attributes(loader: Callable[..., Any]) -> bool:
+    """Return whether *loader* can accept an ``attributes=`` keyword.
+
+    Loaders historically accepted ``(path, meta=None)``.  Per-file
+    attributes are opt-in so legacy loaders keep receiving the old call
+    shape, while loaders declaring ``attributes`` or ``**kwargs`` receive
+    the new context.
+    """
+    try:
+        signature = inspect.signature(loader)
+    except (TypeError, ValueError):
+        return False
+
+    attributes_param = signature.parameters.get("attributes")
+    if (
+        attributes_param is not None
+        and attributes_param.kind is not inspect.Parameter.POSITIONAL_ONLY
+    ):
+        return True
+
+    return any(
+        param.kind is inspect.Parameter.VAR_KEYWORD
+        for param in signature.parameters.values()
+    )
+
+
 def _resolve_loader(
     *,
     modality_name: str,

{euler_loading-2.5.0 → euler_loading-2.7.0}/euler_loading/_writing.py

@@ -244,15 +244,29 @@ def _write_value_to_destination(
     full_id: str,
     basename: str,
     relative_path: str,
-    source_meta: Mapping[str, Any] | None,
+    source_entry: Mapping[str, Any] | None,
+    entry_attributes: Mapping[str, Any] | None = None,
     create_dirs: bool,
 ) -> str:
+    """Write *value* to the destination and (when applicable) record a
+    ds-crawler file entry.
+
+    ``meta`` is the modality-level metadata passed to the loader/writer
+    callable.  ``entry_attributes`` is the per-file ``attributes`` dict
+    written onto the ds-crawler file entry; it is a no-op for plain
+    filesystem destinations (no index to record into).
+    """
+    entry_attributes_dict = (
+        dict(entry_attributes) if entry_attributes is not None else None
+    )
+
     if isinstance(destination, ZipDatasetWriter):
         if supports_stream_target(writer):
             with destination.open(
                 full_id,
                 basename,
-                source_meta=dict(source_meta or {}),
+                source_entry=dict(source_entry or {}),
+                attributes=entry_attributes_dict,
             ) as stream:
                 _set_stream_name(stream, basename)
                 writer(stream, value, meta)
@@ -264,7 +278,8 @@ def _write_value_to_destination(
                     full_id,
                     basename,
                     temp_path.read_bytes(),
-                    source_meta=dict(source_meta or {}),
+                    source_entry=dict(source_entry or {}),
+                    attributes=entry_attributes_dict,
                 )
         _register_destination_rel_path(destination, relative_path)
         return _destination_location(destination, relative_path)
@@ -273,12 +288,21 @@ def _write_value_to_destination(
         target_path = destination.get_path(
             full_id,
             basename,
-            source_meta=dict(source_meta or {}),
+            source_entry=dict(source_entry or {}),
+            attributes=entry_attributes_dict,
         )
         writer(str(target_path), value, meta)
         _register_destination_rel_path(destination, relative_path)
         return str(target_path)
 
+    if entry_attributes_dict:
+        logger.debug(
+            "Filesystem destination %s has no index — entry attributes "
+            "for %s will not be persisted.",
+            destination,
+            relative_path,
+        )
+
     target_path = Path(destination) / relative_path
     if create_dirs:
         target_path.parent.mkdir(parents=True, exist_ok=True)