euler-loading 2.28.0__tar.gz → 2.29.0__tar.gz

{euler_loading-2.28.0 → euler_loading-2.29.0}/CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## Unreleased
+
+- `Modality.function` picks a callable from the loader module the dataset head
+  names, instead of its `function`; the writer follows it.
+  `Modality(path, function="sky_mask")` reads a `class_segmentation` index as
+  boolean sky masks. It is exclusive with `Modality.loader`.
+- The `sky_mask` loaders and writers of `real_drive_sim`, `vkitti2`,
+  `synscapes`, `muses` and `generic_dense_depth` read the sky definition from
+  the contract field `skyclass` (the sky label in the file's RGB view;
+  `[v, v, v]` for a single-channel label): first from the file's attributes,
+  then from the dataset head's meta. Their old hard-coded values are now
+  defaults, and they log the definition they use once.
+- A `null` `skyclass`, the default from euler-dataset-contract's next release,
+  declares no sky: the loaders fall back to a deprecated key or their default,
+  and `generic` / `generic_dense_depth` raise. `[0, 0, 0]`, the contract's old
+  default, is still taken literally, but a head that declares it logs a warning
+  naming it the old default, in place of the warning about differing from the
+  loader's default. Loaders without a default warn too.
+- Deprecate the per-loader sky keys `sky_class_id`, `sky_color`, `sky_class`
+  and `sky_mask` in favour of `skyclass`. They are still read when no
+  `skyclass` is given and raise a `DeprecationWarning`; when `skyclass` is
+  also given it wins, and the ignored key is logged as a warning.
+- Warn when the resolved sky differs from a loader's built-in default.
+- Reject values a loader cannot match instead of returning empty masks: a
+  non-grey `skyclass` for MUSES label PNGs (as for Synscapes and `generic`),
+  and anything but `[id, 0, 0]` for Real Drive Sim.
+- The dry run decodes segmentation heads with their loader module's
+  `sky_mask` (at least `SKY_SAMPLES` files), reports the sky fraction, and
+  warns when no sampled file has sky or one is more than 90% sky.
+- `real_drive_sim.sky_mask` compares the PNG's RGB value with `skyclass`
+  instead of only the red channel; for its own data, where sky is
+  `(29, 0, 0)`, the masks are unchanged.
+- Add `generic.sky_mask` for `.npy`/`.npz` class-id maps, which requires
+  `skyclass`.
+- The torch `sky_mask` loaders and writers of `real_drive_sim`, `vkitti2`,
+  `muses` and `generic_dense_depth` wrap the NumPy ones instead of
+  duplicating them.
+- `muses.sky_mask`'s declared default meta is `skyclass: [23, 23, 23]`; the
+  old `sky_class: [0, 0, 23]` matched no pixel of a `labelIds` PNG.
+- `MultiModalDataset` logs each modality's effective loader, noting when
+  `Modality.function` overrides the index.
+
 ## 2.27.0 (unreleased)
 
 - Add a dry-run command: `euler-loading <path>` (also `python -m

{euler_loading-2.28.0 → euler_loading-2.29.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: euler-loading
-Version: 2.28.0
+Version: 2.29.0
 Summary: Multi-modal PyTorch dataloader using ds-crawler indices
 Project-URL: Homepage, https://github.com/d-rothen/euler-loading
 Project-URL: Repository, https://github.com/d-rothen/euler-loading
@@ -169,8 +169,11 @@ RGB contract looks like this:
 ```
 
 `loader` selects a built-in loader module and `function` selects the callable
-inside it. Automatic resolution uses the GPU variant; pass a loader explicitly
-when you want the CPU variant or a custom callable. See
+inside it. `Modality(path, function="sky_mask")` picks another callable from
+the same module, e.g. boolean sky masks from a `class_segmentation` index,
+with the sky taken from the head's `skyclass`. Automatic resolution uses the
+GPU variant; pass a loader explicitly when you want the CPU variant or a
+custom callable. See
 [Automatic loader resolution](docs/loaders.md#automatic-loader-resolution) for
 the full contract and writer rules.
 

{euler_loading-2.28.0 → euler_loading-2.29.0}/README.md

@@ -130,8 +130,11 @@ RGB contract looks like this:
 ```
 
 `loader` selects a built-in loader module and `function` selects the callable
-inside it. Automatic resolution uses the GPU variant; pass a loader explicitly
-when you want the CPU variant or a custom callable. See
+inside it. `Modality(path, function="sky_mask")` picks another callable from
+the same module, e.g. boolean sky masks from a `class_segmentation` index,
+with the sky taken from the head's `skyclass`. Automatic resolution uses the
+GPU variant; pass a loader explicitly when you want the CPU variant or a
+custom callable. See
 [Automatic loader resolution](docs/loaders.md#automatic-loader-resolution) for
 the full contract and writer rules.
 

{euler_loading-2.28.0 → euler_loading-2.29.0}/docs/loaders.md

@@ -120,6 +120,14 @@ An explicit `Modality.loader` always takes precedence. Automatic resolution
 uses the GPU variant; import and pass a CPU loader explicitly when NumPy output
 is required.
 
+`Modality.function` keeps the head's `loader` module but picks another callable
+in it, and the writer follows it. `Modality(path, function="sky_mask")` reads a
+`class_segmentation` index as boolean sky masks, taking the sky definition from
+the head's `skyclass` (see [Sky masks](#sky-masks)). It cannot be combined with
+`Modality.loader`. Each modality's effective loader is logged when the dataset
+is built, e.g. `loaded with real_drive_sim.sky_mask (the index declares
+'class_segmentation')`.
+
 For compatibility, euler-loading can still read the legacy top-level
 `euler_loading` mapping from older indexes. New datasets should declare the
 addon in `dataset-head.json` as shown above.
@@ -210,6 +218,8 @@ that addresses it.
 | `decoded 0/2` with per-file `FAILED` | the files are indexed and present, but the declared loader cannot decode them |
 | `(hierarchical)` after the loader | the declared function reads per-scene data; pass that modality as `hierarchical_modalities`, not `modalities` |
 | `error index: ...` | ds-crawler could not produce an index — a missing split, an unreadable archive, or a dataset head the contract rejects |
+| `sky skyclass [29, 0, 0]: sky in 8/8 files, ...` | a segmentation head's files, decoded with its loader module's `sky_mask` (at least `SKY_SAMPLES` = 8 files) |
+| a warning that `sky_mask` finds no sky, or marks a file as more than 90% sky | the head's `skyclass` probably names the wrong label; an `error sky_mask: ...` means the loader rejects it outright |
 | a warning about shared file IDs | the modalities below the path have no file ID in common, so they cannot be intersected into one `MultiModalDataset` |
 
 The same report is available as an API, which is what the command is a thin
@@ -226,6 +236,46 @@ if not report.ok:
 report.to_dict()   # the same structure --json prints
 ```
 
+## Sky masks
+
+The dataset contract requires every `semantic_segmentation` head to declare
+`modality.meta.skyclass`: the sky label as it appears in the segmentation file
+viewed as RGB. That is the sky colour of a colour-coded map, and `[v, v, v]`
+for a single-channel label `v`:
+
+| Dataset | Encoding | `skyclass` |
+|---|---|---|
+| Virtual KITTI 2 | RGB class colours | `[90, 200, 255]` |
+| Real Drive Sim | class ID in the red channel of an RGBA PNG | `[29, 0, 0]` |
+| Synscapes | single-channel Cityscapes label IDs | `[23, 23, 23]` |
+| MUSES `labelIds` | single-channel Cityscapes label IDs | `[23, 23, 23]` |
+| Cityscapes train IDs, e.g. `generic` `.npy` maps | single-channel | `[10, 10, 10]` |
+
+`skyclass: null`, the contract's default, declares no sky. `[0, 0, 0]` is taken
+literally, as label 0 or black; contract releases before `null` used it as the
+default, so heads written then can carry it without anyone having declared it.
+A head that declares `[0, 0, 0]` is therefore logged as a warning.
+
+Every `sky_mask` loader (and the matching `write_sky_mask`) resolves the sky in
+this order, skipping a `null` `skyclass`, and logs the definition it uses once:
+
+1. `skyclass` in the file's ds-crawler attributes,
+2. `skyclass` in the dataset head's meta,
+3. a deprecated per-loader key — `sky_class_id` (Synscapes, Real Drive Sim,
+   MUSES), `sky_color` (VKITTI2), `sky_class` / `sky_mask` (MUSES) or
+   `sky_mask` (generic dense depth) — which raises a `DeprecationWarning`,
+4. the loader's built-in default; `generic.sky_mask` and
+   `generic_dense_depth.sky_mask` have none and require `skyclass`.
+
+A deprecated key that `skyclass` overrides is logged as a warning, saying
+whether it named the same sky. So is a resolved sky that differs from the
+loader's default. Values a loader cannot match raise instead of silently
+finding no sky: a non-grey `skyclass` for single-channel maps (Synscapes,
+MUSES label PNGs, `generic`), and anything but `[id, 0, 0]` for Real Drive Sim,
+which stores class IDs in the red channel. The [dry run](#dry-running-loaders)
+decodes segmentation heads with `sky_mask` and warns about suspicious sky
+fractions.
+
 ## Loader protocols
 
 `DenseDepthLoader` is a `runtime_checkable` Protocol defining the loader
@@ -276,7 +326,7 @@ annotations on the loader functions themselves.
 | `class_segmentation` | CHW / HWC | int64 | RGB-encoded class mask |
 | `instance_segmentation` | CHW / HWC | int64 | RGB-encoded instance mask |
 | `scene_flow` | CHW / HWC | float32 | Optical/scene flow, normalised to `[0, 1]` |
-| `sky_mask` | 1HW / HW | bool | Sky colour `[90, 200, 255]` in the class mask |
+| `sky_mask` | 1HW / HW | bool | Head `skyclass` colour in the class mask, default `[90, 200, 255]` |
 | `read_intrinsics` | 3×3 | float32 | From a `.txt` file · *hierarchical* |
 | `read_extrinsics` | N×1 | float32 | From a `.txt` file · *hierarchical* |
 
@@ -290,14 +340,14 @@ Writers exist for every modality above.
 | `depth` | 1HW / HW | float32 | `img/depth/<id>.exr`, `Z` channel, planar depth in metres |
 | `class_segmentation` | 1HW / HW | int64 | `img/class/<id>.png`, original Cityscapes label IDs |
 | `instance_segmentation` | 1HW / HW | int64 | `img/instance/<id>.png`, decoded as `R + 256 * G + 65536 * B` |
-| `sky_mask` | 1HW / HW | bool | Sky label in `img/class/<id>.png`; ID `23` by default, or `meta['sky_class_id']` |
+| `sky_mask` | 1HW / HW | bool | Sky label in `img/class/<id>.png`: the grey `skyclass`, ID `23` by default |
 | `read_intrinsics` | 3×3 | float32 | `camera.intrinsic` from `meta/<id>.json`: `fx`, `fy`, `u0`, `v0` |
 | `read_extrinsics` | 4×4 | float32 | `camera.extrinsic` from `meta/<id>.json`: `x`, `y`, `z`, `pitch`, `roll`, `yaw` |
 
 Writers exist for every modality above. `write_depth` needs a filesystem path
 because OpenEXR cannot write to a stream; dataset writers hand it a temporary
 file automatically, so zip outputs work unchanged. `write_sky_mask` emits a
-class image labelling sky with `meta['sky_class_id']`, default 23, and
+class image labelling sky with the label `sky_mask` reads, and
 `write_intrinsics` and `write_extrinsics` merge into one `meta/<id>.json`
 rather than overwriting each other or the file's `scene` and `instance`
 blocks.
@@ -384,7 +434,7 @@ same resolution. Loaders preserve the stored resolution.
 | `reference_rgb` | CHW / HWC | float32 | Clear-weather reference frame, `[0, 1]` |
 | `semantic_segmentation` | 1HW / HW | int64 | Cityscapes `labelIds` or `labelTrainIds` |
 | `semantic_segmentation_color` | CHW / HWC | uint8 | Cityscapes `labelColor` PNG |
-| `sky_mask` | 1HW / HW | bool | `trainId=10`, `labelId=23`, or RGB `meta["sky_class"]` |
+| `sky_mask` | 1HW / HW | bool | `skyclass` (grey for label PNGs, a colour for RGB class images); default `trainId=10` / `labelId=23` |
 | `panoptic_segmentation` | 1HW / HW | int64 | COCO-style RGB panoptic PNG → segment IDs |
 | `lidar_point_cloud` | (N, 6) | float64 | `.bin`: `x, y, z, intensity, ring, timestamp` |
 | `sparse_depth` | (N, 6) | float64 | Same points, for sparse-depth workflows |
@@ -400,7 +450,7 @@ same resolution. Loaders preserve the stored resolution.
 | `rgb` | CHW / HWC | float32 | PNG, `[0, 1]` |
 | `depth` | 1HW / HW | float32 | `.npz`, metres |
 | `class_segmentation` | 1HW / HW | int64 | Class IDs from the red channel of an RGBA PNG |
-| `sky_mask` | 1HW / HW | bool | Class ID `29` |
+| `sky_mask` | 1HW / HW | bool | Head `skyclass` colour, default `[29, 0, 0]` (class ID 29) |
 | `calibration` | dict | — | `{sensor: {"K": (3,3), "T": (4,4), "distortion": (8,)}}` · *hierarchical* |
 | `all_intrinsics` | dict | — | `{sensor: (3, 3)}` · *hierarchical* |
 | `read_intrinsics` | 3×3 | float32 | Defaults to `CS_FRONT` · *hierarchical* |
@@ -427,7 +477,7 @@ Useful for datasets without a dedicated module.
 |---|---|---|---|
 | `rgb` | CHW / HWC | float32 | Image (`.png`, `.jpg`, `.bmp`, `.tif`) or NumPy (`.npy`, `.npz`), `[0, 1]` |
 | `depth` | 1HW / HW | float32 | Image or NumPy, returned as-is; honours `attributes["scale_to_meters_override"]` |
-| `sky_mask` | 1HW / HW | bool | Compares pixels against `meta["sky_mask"]` (`[R, G, B]`) — requires `meta` |
+| `sky_mask` | 1HW / HW | bool | Compares pixels against the `skyclass` colour — required |
 | `read_intrinsics` | 3×3 | float32 | Returns `meta["intrinsics"]`, ignores the path — requires `meta` · *hierarchical* |
 
 ### Generic NumPy modalities — `generic`
@@ -444,8 +494,9 @@ assumptions beyond shape and dtype.
 | `atmospheric_light` | CHW / HWC | float32 |
 | `scattering_coefficient` | HW | float32 |
 | `semantic_segmentation` | HW | uint8 |
+| `sky_mask` | HW | bool (head `skyclass` label, required) |
 | `instance_segmentation` | HW | uint16 |
 | `intrinsics` | 3×3 | float32 |
 | `sh_coeffs` | (N, C) | float32 |
 
-Writers exist for every function in this module.
+Writers exist for every function in this module except `sky_mask`.

{euler_loading-2.28.0 → euler_loading-2.29.0}/euler_loading/_resolution.py

@@ -142,8 +142,10 @@ def _resolve_loader(
 ) -> Callable[..., Any]:
     """Return the effective loader for a modality.
 
-    An explicit ``Modality.loader`` always wins; *variant* only selects
-    between the built-in GPU and CPU modules during automatic resolution.
+    An explicit ``Modality.loader`` always wins; ``Modality.function`` replaces
+    the dataset head's function within its loader module. *variant* only
+    selects between the built-in GPU and CPU modules during automatic
+    resolution.
     """
     if modality.loader is not None:
         return modality.loader
@@ -156,7 +158,8 @@ def _resolve_loader(
             f"'addons.euler_loading' entry."
         )
 
-    for key in ("loader", "function"):
+    required = ("loader",) if modality.function is not None else ("loader", "function")
+    for key in required:
         if key not in euler_loading_meta:
             raise ValueError(
                 f"Modality {modality_name!r}: no explicit loader provided and "
@@ -165,7 +168,7 @@ def _resolve_loader(
             )
 
     module_name: str = euler_loading_meta["loader"]
-    func_name: str = euler_loading_meta["function"]
+    func_name: str = modality.function or euler_loading_meta["function"]
 
     module = resolve_loader_module(module_name, variant=variant)
 
@@ -184,6 +187,24 @@ def _resolve_loader(
     return func
 
 
+def describe_loader(
+    loader: Callable[..., Any],
+    *,
+    modality: Modality,
+    index: Mapping[str, Any],
+) -> str:
+    """Name the effective loader for logs, e.g. ``real_drive_sim.sky_mask``."""
+    module = getattr(loader, "__module__", None) or ""
+    name = getattr(loader, "__qualname__", None) or repr(loader)
+    label = f"{module.rsplit('.', 1)[-1]}.{name}" if module else name
+    if modality.loader is not None:
+        return f"{label} (explicit loader)"
+    declared = as_non_empty_str((_get_euler_loading_meta(index) or {}).get("function"))
+    if modality.function is not None and declared not in (None, modality.function):
+        return f"{label} (the index declares {declared!r})"
+    return label
+
+
 def _writer_name_candidates(read_function_name: str, explicit: Any) -> list[str]:
     names: list[str] = []
     explicit_name = as_non_empty_str(explicit)
@@ -220,7 +241,9 @@ def _resolve_writer(
         return None
 
     module_name = as_non_empty_str(euler_loading_meta.get("loader"))
-    read_function_name = as_non_empty_str(euler_loading_meta.get("function"))
+    read_function_name = as_non_empty_str(
+        modality.function or euler_loading_meta.get("function")
+    )
     if module_name is None or read_function_name is None:
         return None
 
@@ -234,9 +257,12 @@ def _resolve_writer(
         )
         return None
 
+    # The head's writer_function pairs with its own function, not an override.
     writer_names = _writer_name_candidates(
         read_function_name,
-        euler_loading_meta.get("writer_function"),
+        None
+        if modality.function is not None
+        else euler_loading_meta.get("writer_function"),
     )
     for writer_name in writer_names:
         writer = getattr(module, writer_name, None)

{euler_loading-2.28.0 → euler_loading-2.29.0}/euler_loading/cli.py

@@ -148,6 +148,9 @@ def _render_modality(check: ModalityCheck) -> list[str]:
                 detail = f"FAILED  {sample.error}"
             lines.append(f"{_INDENT * 2}{name}  ->  {detail}")
 
+    if check.sky:
+        lines.append(_field("sky", check.sky))
+
     for warning in check.warnings:
         lines.append(_field("warning", warning))
     for error in check.errors:

{euler_loading-2.28.0 → euler_loading-2.29.0}/euler_loading/dataset.py

@@ -43,6 +43,7 @@ from ._metadata import _build_runlog_entry, _get_ds_crawler_descriptor
 from ._resolution import (
     _resolve_loader,
     _resolve_writer,
+    describe_loader,
     loader_accepts_attributes,
     resolve_loader_module,
     resolve_writer_module,
@@ -369,6 +370,12 @@ class Modality:
                   ``metadata["euler_loading"]`` are treated as defaults for
                   roles and run logging. They do not select a loader; automatic
                   loader resolution uses the dataset-head addon described above.
+        function: Optional name of the loader function to use from the module
+                  that the dataset head's ``addons.euler_loading.loader``
+                  names, instead of its ``function``. For example
+                  ``"sky_mask"`` reads a ``class_segmentation`` index as
+                  boolean sky masks. The writer follows the same function.
+                  Mutually exclusive with *loader*.
     """
 
     path: str
@@ -385,8 +392,14 @@ class Modality:
     cache: bool | None = None
     collapse_single: bool = False
     metadata: dict[str, Any] = field(default_factory=dict)
+    function: str | None = None
 
     def __post_init__(self) -> None:
+        if self.loader is not None and self.function is not None:
+            raise ValueError(
+                f"Modality {self.path!r} sets both loader and function "
+                f"({self.function!r}); pass one of them."
+            )
         parsed_path, path_split, path_metadata_scope = parse_modality_path(self.path)
         if path_split is not None:
             if self.split is not None:
@@ -791,11 +804,14 @@ class MultiModalDataset(_BaseDataset):
             }
             self._lookups[name] = lookup
             logger.info(
-                "Modality '%s': indexed %d files from %s%s",
+                "Modality '%s': indexed %d files from %s%s, loaded with %s",
                 name,
                 len(lookup),
                 modality.path,
                 _modality_selector_suffix(modality),
+                describe_loader(
+                    self._resolved_loaders[name], modality=modality, index=index
+                ),
             )
 
         # -- Index hierarchical modalities -----------------------------------
@@ -825,12 +841,15 @@ class MultiModalDataset(_BaseDataset):
             total_files = sum(len(f) for f in files_by_level.values())
             logger.info(
                 "Hierarchical modality '%s': indexed %d files across %d "
-                "hierarchy levels from %s%s",
+                "hierarchy levels from %s%s, loaded with %s",
                 name,
                 total_files,
                 len(files_by_level),
                 modality.path,
                 _modality_selector_suffix(modality),
+                describe_loader(
+                    self._resolved_loaders[name], modality=modality, index=index
+                ),
             )
 
         # -- Compute common IDs (intersection across regular modalities) -----