euler-preprocess 3.13.0__tar.gz → 3.16.0__tar.gz

euler_preprocess-3.16.0/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 3.14.0 (unreleased)
+
+Consume shared preprocessing config and add the opt-in spatial producer, explicit variants/mappings and captured source-backed outputs. Preserve ordinary pipeline APIs.

euler_preprocess-3.16.0/MANIFEST.in

@@ -0,0 +1,5 @@
+recursive-exclude tests/fixtures *.zip
+
+include CHANGELOG.md
+recursive-include docs *.md
+recursive-include examples *.json

{euler_preprocess-3.13.0 → euler_preprocess-3.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 3.13.0
+Version: 3.16.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
 License-Expression: MIT
@@ -9,7 +9,9 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: Pillow
-Requires-Dist: euler-loading
+Requires-Dist: euler-loading>=2.24.0
+Requires-Dist: ds-crawler>=2.11.0
+Requires-Dist: euler-dataset-contract>=0.5.0
 Provides-Extra: gpu
 Requires-Dist: torch; extra == "gpu"
 Provides-Extra: progress
@@ -92,9 +94,7 @@ loader.
   "modalities": {
     "rgb": {"path": "/path/to/rgb", "split": "train"},
     "depth": "/path/to/depth",
-    "semantic_segmentation": "/path/to/classSegmentation"
-  },
-  "hierarchical_modalities": {
+    "semantic_segmentation": "/path/to/classSegmentation",
     "intrinsics": {"path": "/path/to/intrinsics"}
   }
 }
@@ -107,17 +107,27 @@ loader.
 | `output_slot` | Optional slot selector when `pipeline.output_targets` has several entries. Defaults to `rgb` for `fog` and `depth` for `sky-depth` / `radial`. |
 | `sample` | Optional 0-based dataset index. Only `dataset[sample]` is transformed — useful for small benchmark slices of large datasets. |
 | `samples` | Optional multi-sample selector: a list of indices (`[0, 10, 20]`) or a slice object (`{"start": 0, "stop": 1000, "step": 2, "count": 100}`). `stop` is exclusive, `count` caps the result. Mutually exclusive with `sample`. |
-| `modalities` | Modalities that participate in sample-ID intersection. Each value is a path string or an object with `path` and optional `split`. |
-| `hierarchical_modalities` | Per-scene data (e.g. intrinsics), same format. Loaded once per scene and cached. |
+| `modalities` | All input modalities. Each value is a path string or an object with `path` and optional `split`. Whether a modality is per-sample or hierarchical is inferred (see below). |
+| `hierarchical_modalities` | Legacy, optional. Same format; entries are pooled with `modalities` and resolved the same way. |
 | `pipeline` | Optional runtime routing block compatible with `euler-inference`. |
 
 **Required modalities per transform:**
 
-| Transform | `modalities` | `hierarchical_modalities` |
+| Transform | Required | Optional |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics`; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
-| `radial` | `depth` | `intrinsics` |
+| `radial` | `depth`, `intrinsics` | — |
+
+**Per-sample vs. hierarchical modalities.** You don't declare a modality's role:
+it is inferred from the ds-crawler indexes. A modality whose qualified IDs
+(hierarchy path plus file ID) match the transform's per-sample inputs is joined per
+sample, e.g. Synscapes' per-image `meta/<id>.json` intrinsics. A modality whose
+files share no sample IDs but sit at or above the samples' hierarchy level is
+matched hierarchically and shared by every sample beneath it, e.g. one intrinsics
+file per scene or camera. A modality that is neither raises an error, and the
+inferred roles are logged. Listing a per-sample modality under
+`hierarchical_modalities` logs a warning and loads it per sample.
 
 When a modality directory contains ds-crawler split files
 (`.ds_crawler/split_<name>.json`), set `split` on that modality to select a subset.
@@ -675,7 +685,7 @@ d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
 ```
 
 The transform config takes no parameters (`{}`); intrinsics are read from the
-`intrinsics` hierarchical modality.
+`intrinsics` modality, stored either per sample or per scene/camera.
 
 CLI runs write a source-backed depth dataset mirroring the input depth modality's layout
 and writer metadata, with `meta.radial_depth` set to `true` in the emitted `index.json`.
@@ -686,3 +696,5 @@ Standalone `RadialTransform(...)` usage keeps the legacy `.npy` output behaviour
 ## License
 
 [MIT](LICENSE) © Daniel Rothenpieler
+
+See [Phase 2 spatial](docs/spatial.md) for the opt-in captured spatial workflow (3.14.0, unreleased).

euler_preprocess-3.13.0/euler_preprocess.egg-info/PKG-INFO → euler_preprocess-3.16.0/README.md

@@ -1,21 +1,3 @@
-Metadata-Version: 2.4
-Name: euler-preprocess
-Version: 3.13.0
-Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
-Author-email: Daniel Rothenpieler <rothenpielerdaniel@gmail.com>
-License-Expression: MIT
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy
-Requires-Dist: Pillow
-Requires-Dist: euler-loading
-Provides-Extra: gpu
-Requires-Dist: torch; extra == "gpu"
-Provides-Extra: progress
-Requires-Dist: tqdm; extra == "progress"
-Dynamic: license-file
-
 <!-- euler header — shared across the euler packages.
      Per package, change only: the <h1>, the tagline, and the badge URLs. -->
 <p align="center">
@@ -92,9 +74,7 @@ loader.
   "modalities": {
     "rgb": {"path": "/path/to/rgb", "split": "train"},
     "depth": "/path/to/depth",
-    "semantic_segmentation": "/path/to/classSegmentation"
-  },
-  "hierarchical_modalities": {
+    "semantic_segmentation": "/path/to/classSegmentation",
     "intrinsics": {"path": "/path/to/intrinsics"}
   }
 }
@@ -107,17 +87,27 @@ loader.
 | `output_slot` | Optional slot selector when `pipeline.output_targets` has several entries. Defaults to `rgb` for `fog` and `depth` for `sky-depth` / `radial`. |
 | `sample` | Optional 0-based dataset index. Only `dataset[sample]` is transformed — useful for small benchmark slices of large datasets. |
 | `samples` | Optional multi-sample selector: a list of indices (`[0, 10, 20]`) or a slice object (`{"start": 0, "stop": 1000, "step": 2, "count": 100}`). `stop` is exclusive, `count` caps the result. Mutually exclusive with `sample`. |
-| `modalities` | Modalities that participate in sample-ID intersection. Each value is a path string or an object with `path` and optional `split`. |
-| `hierarchical_modalities` | Per-scene data (e.g. intrinsics), same format. Loaded once per scene and cached. |
+| `modalities` | All input modalities. Each value is a path string or an object with `path` and optional `split`. Whether a modality is per-sample or hierarchical is inferred (see below). |
+| `hierarchical_modalities` | Legacy, optional. Same format; entries are pooled with `modalities` and resolved the same way. |
 | `pipeline` | Optional runtime routing block compatible with `euler-inference`. |
 
 **Required modalities per transform:**
 
-| Transform | `modalities` | `hierarchical_modalities` |
+| Transform | Required | Optional |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics`; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
-| `radial` | `depth` | `intrinsics` |
+| `radial` | `depth`, `intrinsics` | — |
+
+**Per-sample vs. hierarchical modalities.** You don't declare a modality's role:
+it is inferred from the ds-crawler indexes. A modality whose qualified IDs
+(hierarchy path plus file ID) match the transform's per-sample inputs is joined per
+sample, e.g. Synscapes' per-image `meta/<id>.json` intrinsics. A modality whose
+files share no sample IDs but sit at or above the samples' hierarchy level is
+matched hierarchically and shared by every sample beneath it, e.g. one intrinsics
+file per scene or camera. A modality that is neither raises an error, and the
+inferred roles are logged. Listing a per-sample modality under
+`hierarchical_modalities` logs a warning and loads it per sample.
 
 When a modality directory contains ds-crawler split files
 (`.ds_crawler/split_<name>.json`), set `split` on that modality to select a subset.
@@ -675,7 +665,7 @@ d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
 ```
 
 The transform config takes no parameters (`{}`); intrinsics are read from the
-`intrinsics` hierarchical modality.
+`intrinsics` modality, stored either per sample or per scene/camera.
 
 CLI runs write a source-backed depth dataset mirroring the input depth modality's layout
 and writer metadata, with `meta.radial_depth` set to `true` in the emitted `index.json`.
@@ -686,3 +676,5 @@ Standalone `RadialTransform(...)` usage keeps the legacy `.npy` output behaviour
 ## License
 
 [MIT](LICENSE) © Daniel Rothenpieler
+
+See [Phase 2 spatial](docs/spatial.md) for the opt-in captured spatial workflow (3.14.0, unreleased).

euler_preprocess-3.16.0/docs/spatial.md

@@ -0,0 +1,61 @@
+# Deterministic spatial exports (3.14.0, unreleased)
+
+`euler-preprocess spatial --config examples/spatial-five-field.json` is the
+opt-in crop/resize producer. It uses loading 2.24.0, crawler 2.11.0 and contract
+0.5.0. These source versions must be installed together. No benchmark evaluation
+is enabled. Fog/model/value effects are not labelled replayable spatial exports.
+
+The example config expects five source modality datasets. Change their paths to
+your sources, and supply decoded profiles, image plane, qualified calibration
+selection and source revision assertions that describe those files. The producer
+hashes the actual source content itself; it does not accept supplied hashes as
+independently checked. Paths in CLI configs are resolved against the config file.
+
+```python
+from euler_preprocess.spatial import export_spatial
+paths = export_spatial(config)
+```
+
+`preprocessing` uses the shared `SamplePreprocessor` config. `bindings` has
+`fields`, `image_planes`, `revisions` (source alias to revision digest) and an
+optional explicit CPU `backend` (`torch_cpu` by default). `outputs` selects
+fields; each needs `path`, a distinct derived `dataset_id`, and `revision`.
+Optional output options are `metadata_scope`, `resume` (directories only), and
+`encoding: {"id":"npy"}` or a supported PNG codec, e.g.
+`{"id":"png_depth16","scale":0.001}`. NPY preserves K/rays exactly. PNG RGB
+expects normalized float32 values, masks require bool, and depth requires meters.
+Clipping is rejected. Codec parameters, profiles and real dimensions are stored
+alongside the bytes; source metadata cannot override them.
+
+`spatial_variants` replaces top-level preprocessing/bindings with explicit named
+`{preprocessing, bindings}` entries. Every variant is frozen in the output plan
+before writing. Variants can select different qualified calibrations or concrete
+geometries. A `samples` list selects source indices. Optional
+`mapping: {variant: {source_full_id: output_full_id}}` supplies exact output IDs.
+Otherwise a single variant preserves source IDs; multiple variants append the
+explicit variant ID as a hierarchy component. Conflicting mappings are rejected.
+The producer computes the complete expected output set before starting writes.
+Use explicit mappings/subsets, rather than inferring relationships from filenames.
+
+Each sample carries the actual producer receipt into `SourceBackedOutputBackend`.
+`prepare_output_backend` and `prepare_output_backends` also accept explicitly
+configured `materialization_outputs` for Python integrations; each output option
+includes a frozen `derivation` and optional `expected_full_ids`. The dataset must
+already have source-backed transform capture enabled. An arbitrary transform's
+changed values will fail the receipt checks. Ordinary attributes are preserved,
+while the new receipt and output encoding metadata are authoritative.
+
+`receipt_sidecars: true` moves receipts to relative content-addressed records.
+Failed encoding cannot register a receipt. Finalization verifies source-backed
+replay, output bytes, expected IDs and references before publishing each logical
+output dataset. There is no all-or-nothing transaction across several output
+roots: each head is independently complete for its one selected modality.
+Directory resume is idempotent with the same source/content/recipe/revision;
+ZIP supports one owner/one finalization. Interrupted outputs do not advertise
+complete metadata.
+
+The common dataset builder now consumes `preprocessing` for ordinary pipelines
+as well. Existing callable pipelines retain their APIs. Strict receipts are
+created only by the explicit spatial/materialization path. Reload through
+`MultiModalDataset` loads stored bytes once; explicit loading replay and typed
+`CalibrationView` resolve the original geometry without evaluator heuristics.

{euler_preprocess-3.13.0 → euler_preprocess-3.16.0}/euler_preprocess/cli.py

@@ -244,6 +244,23 @@ def _cmd_radial(args: argparse.Namespace) -> int:
     return _run_transform(args, RadialTransform)
 
 
+def _cmd_spatial(args: argparse.Namespace) -> int:
+    from euler_preprocess.spatial import export_spatial
+
+    config_path = Path(args.config).resolve()
+    config = json.loads(config_path.read_text(encoding="utf-8"))
+    for group in ("modalities", "hierarchical_modalities"):
+        for key, entry in config.get(group, {}).items():
+            if isinstance(entry, str):
+                config[group][key] = str(_resolve(entry, config_path.parent))
+            else:
+                entry["path"] = str(_resolve(entry["path"], config_path.parent))
+    for entry in config["outputs"].values():
+        entry["path"] = str(_resolve(entry["path"], config_path.parent))
+    export_spatial(config)
+    return 0
+
+
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(
         description="Preprocessing transforms for RGB+depth datasets.",
@@ -281,6 +298,11 @@ def parse_args() -> argparse.Namespace:
     )
     radial_parser.set_defaults(func=_cmd_radial)
 
+    spatial_parser = subparsers.add_parser(
+        "spatial", parents=[common], help="Export deterministic crop/resize with verified execution receipts.",
+    )
+    spatial_parser.set_defaults(func=_cmd_spatial)
+
     args = parser.parse_args()
     if not hasattr(args, "func"):
         parser.print_help()

euler_preprocess-3.16.0/euler_preprocess/common/dataset.py

@@ -0,0 +1,212 @@
+from __future__ import annotations
+
+from collections import Counter
+
+from euler_preprocess.common.logging import get_logger
+
+
+def _make_modality(entry: str | dict):
+    from euler_loading import Modality
+
+    if isinstance(entry, str):
+        return Modality(entry)
+    return Modality(
+        entry["path"],
+        **{
+            key: entry[key]
+            for key in (
+                "split",
+                "metadata_scope",
+                "collapse_single",
+                "cache",
+                "applies_to",
+                "hierarchy_scope",
+            )
+            if key in entry
+        },
+    )
+
+
+def _load_index_tree(name: str, modality) -> dict:
+    """Load a modality's ds-crawler tree the same way ``MultiModalDataset`` does."""
+    from ds_crawler import index_dataset_from_path
+    from euler_loading._ds_crawler_utils import load_index_output
+    from euler_loading.dataset import _get_index_tree, _resolve_metadata_scope
+
+    modality = _resolve_metadata_scope(name, modality)
+    index = load_index_output(
+        modality.path,
+        split=modality.split,
+        metadata_scope=modality.metadata_scope,
+        index_dataset_from_path_fn=index_dataset_from_path,
+    )
+    return _get_index_tree(index)
+
+
+def _check_attachment(
+    name: str, seed: str, tree: dict, samples_per_level: Counter
+) -> None:
+    """Check that hierarchical modality *name* applies to the samples by tree position."""
+    from euler_loading.indexing import collect_hierarchical_files, match_hierarchical_files
+
+    files_by_level = collect_hierarchical_files(tree)
+    attached = {
+        level: len(match_hierarchical_files(level, files_by_level))
+        for level in samples_per_level
+    }
+    covered = sum(count for level, count in samples_per_level.items() if attached[level])
+    if not covered:
+        raise ValueError(
+            f"Modality '{name}' shares no sample IDs with '{seed}' and none of its "
+            "files sit at or above the hierarchy level of any sample, so it can be "
+            "neither joined per sample nor matched hierarchically. Check that its "
+            f"ds-crawler IDs and hierarchy follow the same conventions as '{seed}'."
+        )
+    logger = get_logger()
+    total = sum(samples_per_level.values())
+    log = logger.info if covered == total else logger.warning
+    log(
+        "Modality '%s' is hierarchical (no sample IDs shared with '%s'); it applies "
+        "to %d/%d samples by hierarchy position.",
+        name,
+        seed,
+        covered,
+        total,
+    )
+    most = max(attached.values())
+    if most > 1:
+        logger.warning(
+            "Hierarchical modality '%s' attaches up to %d files to one sample. If it "
+            "holds per-sample data, its ds-crawler IDs do not match '%s'.",
+            name,
+            most,
+            seed,
+        )
+
+
+def _resolve_roles(
+    modalities: dict, per_sample: set[str]
+) -> tuple[list[str], list[str]]:
+    """Split modality names into regular and hierarchical ones.
+
+    The role follows from the ds-crawler indexes.  A modality is regular
+    (joined per sample) when its qualified file IDs -- hierarchy path plus
+    file ID -- overlap those of the seed modality.  Otherwise it is
+    hierarchical (matched by tree position, like per-scene intrinsics) and
+    must sit at or above the hierarchy level of at least one sample.
+
+    *per_sample* modalities are regular by contract and the first one is the
+    seed.  Without one, the seed is the modality aligned with the most others,
+    then the one with the most files, then the first listed.
+    """
+    from euler_loading.indexing import collect_files
+
+    names = list(modalities)
+    if len(names) < 2 or per_sample.issuperset(names):
+        return names, []
+
+    trees = {name: _load_index_tree(name, modalities[name]) for name in names}
+    # Qualified sample ID -> hierarchy path, keyed like euler-loading's lookups.
+    sample_ids = {}
+    for name, tree in trees.items():
+        sample_ids[name] = {
+            "/".join(r.hierarchy_path + (r.file_entry["id"],)): r.hierarchy_path
+            for r in collect_files(tree)
+        }
+
+    def aligned(name: str, other: str) -> bool:
+        return not sample_ids[name].keys().isdisjoint(sample_ids[other].keys())
+
+    anchors = [name for name in names if name in per_sample]
+    seed = anchors[0] if anchors else max(
+        names,
+        key=lambda name: (
+            sum(aligned(other, name) for other in names),
+            len(sample_ids[name]),
+        ),
+    )
+    regular = [
+        name
+        for name in names
+        if name in per_sample or name == seed or aligned(name, seed)
+    ]
+    hierarchical = [name for name in names if name not in regular]
+    get_logger().info(
+        "Modality roles inferred from ds-crawler indexes: per-sample %s; "
+        "hierarchical %s",
+        ", ".join(regular),
+        ", ".join(hierarchical) or "none",
+    )
+
+    # Without common samples MultiModalDataset raises its own, clearer error.
+    common = set(sample_ids[seed]).intersection(*(sample_ids[name] for name in regular))
+    if common:
+        samples_per_level = Counter(sample_ids[seed][sample_id] for sample_id in common)
+        for name in hierarchical:
+            _check_attachment(name, seed, trees[name], samples_per_level)
+    return regular, hierarchical
+
+
+def build_dataset(
+    config: dict,
+    required_modalities: set[str],
+    required_hierarchical: set[str] | None = None,
+):
+    """Build a ``MultiModalDataset`` from a config dict.
+
+    Each modality entry is either a plain path string or a dict with
+    ``path`` and optional ``split``.  Loader resolution (which function to
+    call, which module to use) is handled by euler-loading via the
+    ds-crawler index at each path — point the config at a path whose
+    index declares the function you want (e.g. a ``sky_mask`` index for
+    boolean sky masks vs. a ``class_segmentation`` index for raw class
+    id maps).
+
+    Every modality can be listed under ``modalities``: whether it is joined
+    per sample or matched hierarchically (e.g. per-scene intrinsics) is
+    inferred from the ds-crawler indexes.  Entries under the legacy
+    ``hierarchical_modalities`` key are pooled with them and resolved the
+    same way.  *required_modalities* are per-sample inputs and anchor that
+    inference; *required_hierarchical* names further required modalities,
+    whose role is inferred like any other.
+    """
+    from euler_loading import MultiModalDataset, SamplePreprocessor
+
+    raw_modalities = config.get("modalities", {})
+    raw_hierarchical = config.get("hierarchical_modalities", {})
+
+    duplicated = raw_modalities.keys() & raw_hierarchical.keys()
+    if duplicated:
+        raise ValueError(
+            "Modalities listed under both 'modalities' and "
+            f"'hierarchical_modalities': {', '.join(sorted(duplicated))}"
+        )
+    configured = {**raw_modalities, **raw_hierarchical}
+
+    required = set(required_modalities) | set(required_hierarchical or ())
+    missing = required - configured.keys()
+    if missing:
+        raise ValueError(
+            f"Missing required modalities in config: {', '.join(sorted(missing))}. "
+            f"'modalities' must contain at least: {', '.join(sorted(required))}"
+        )
+
+    modalities = {name: _make_modality(entry) for name, entry in configured.items()}
+    regular, hierarchical = _resolve_roles(modalities, set(required_modalities))
+    logger = get_logger()
+    for name in regular:
+        if name in raw_hierarchical:
+            logger.warning(
+                "Modality '%s' is listed under 'hierarchical_modalities' but holds "
+                "per-sample data; loading it per sample.",
+                name,
+            )
+
+    return MultiModalDataset(
+        modalities={name: modalities[name] for name in regular},
+        hierarchical_modalities={name: modalities[name] for name in hierarchical}
+        or None,
+        transforms=[SamplePreprocessor.from_config(config["preprocessing"])]
+        if config.get("preprocessing")
+        else None,
+    )

{euler_preprocess-3.13.0 → euler_preprocess-3.16.0}/euler_preprocess/common/intrinsics.py

@@ -13,8 +13,9 @@ except ImportError:
 def extract_intrinsics(sample: dict) -> np.ndarray | None:
     """Extract the 3x3 intrinsics matrix from a sample dict.
 
-    Hierarchical modalities are stored as ``sample[name][file_id]``.
-    For intrinsics the convention is ``sample["intrinsics"]["intrinsics"]``.
+    Per-sample intrinsics are stored directly as ``sample["intrinsics"]``.
+    Hierarchical modalities are stored as ``sample[name][file_id]``; for
+    intrinsics the convention is ``sample["intrinsics"]["intrinsics"]``.
     """
     intr_dict = sample.get("intrinsics")
     if intr_dict is None:

{euler_preprocess-3.13.0 → euler_preprocess-3.16.0}/euler_preprocess/common/output.py

@@ -8,19 +8,20 @@ from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any
 
-from ds_crawler import DatasetWriter, ZipDatasetWriter
+from ds_crawler import DatasetWriter, ValidatedDatasetWriter, ZipDatasetWriter
 from ds_crawler.zip_utils import (
     METADATA_DIR,
-    OUTPUT_FILENAME as DS_CRAWLER_INDEX_FILENAME,
     get_metadata_entry_name,
 )
+from ds_crawler.zip_utils import (
+    OUTPUT_FILENAME as DS_CRAWLER_INDEX_FILENAME,
+)
 from euler_loading import MultiModalDataset
 from euler_loading.dataset import create_dataset_writer_from_index
 from euler_loading.loaders._writer_utils import supports_stream_target
 
 from euler_preprocess.common.io import OutputWriter
 
-
 PIPELINE_OUTPUTS_MANIFEST_VERSION = 1
 _PIPELINE_OUTPUT_STORAGE_KINDS = {"directory", "zip", "file"}
 _IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".bmp", ".tif", ".tiff"}
@@ -420,9 +421,32 @@ class SourceBackedOutputBackend:
         output_full_id: str | None = None,
         output_basename: str | None = None,
         attributes: dict[str, Any] | None = None,
+        variant_id: str | None = None,
+        sidecar: bool = False,
     ) -> Path:
         del default_path
 
+        if isinstance(self.dataset_writer, ValidatedDatasetWriter):
+            from euler_loading import write_materialized
+
+            provenance = sample.get("provenance")
+            if variant_id is not None and (
+                not provenance or provenance["variant_id"] != variant_id
+            ):
+                raise ValueError("output variant disagrees with captured execution")
+            source_attrs = dict(
+                sample.get("attributes", {}).get(self.source_modality, {})
+            )
+            source_attrs.pop("euler_transforms", None)
+            source_attrs.pop("output_encoding", None)
+            source_attrs.update(attributes or {})
+            return Path(write_materialized(
+                self.dataset_writer, value, provenance,
+                output_full_id=output_full_id or sample["full_id"],
+                output_basename=output_basename,
+                attributes=source_attrs, sidecar=sidecar,
+            ))
+
         sample_id = sample.get("id", "?")
         full_id = str(output_full_id or sample.get("full_id") or f"/{sample_id}")
         meta = sample.get("meta")
@@ -505,15 +529,11 @@ class SourceBackedOutputBackend:
 
     def set_hierarchy_separator(self, separator: str) -> None:
         """Set the writer hierarchy separator used for future entries."""
-        setattr(self.dataset_writer, "_separator", separator)
+        self.dataset_writer.set_hierarchy_separator(separator)
 
     def add_head_addon(self, name: str, payload: dict[str, Any]) -> None:
         """Add a dataset-head addon before the writer saves its artifacts."""
-        head = getattr(self.dataset_writer, "_dataset_head", None)
-        addons = getattr(head, "addons", None)
-        if not isinstance(addons, dict):
-            raise RuntimeError("Unsupported dataset writer head object")
-        addons[name] = dict(payload)
+        self.dataset_writer.set_head_addon(name, payload)
         self.index_overrides[name] = dict(payload)
 
     def write_json(self, path: Path, data: dict[str, Any]) -> None:
@@ -522,17 +542,13 @@ class SourceBackedOutputBackend:
         )
 
     def finalize(self) -> None:
-        self.dataset_writer.save_index(filename=DS_CRAWLER_INDEX_FILENAME)
-        _patch_output_index(
-            self.dataset_writer,
-            self.index_overrides,
-            filename=DS_CRAWLER_INDEX_FILENAME,
-        )
+        if isinstance(self.dataset_writer, ValidatedDatasetWriter):
+            self.dataset_writer.finalize()
+        else:
+            self.dataset_writer.save_index(filename=DS_CRAWLER_INDEX_FILENAME)
+            _patch_output_index(self.dataset_writer, self.index_overrides, filename=DS_CRAWLER_INDEX_FILENAME)
         if self.pipeline_manifest_path and self.pipeline_manifest_targets:
-            _write_pipeline_outputs_manifest(
-                self.pipeline_manifest_path,
-                self.pipeline_manifest_targets,
-            )
+            _write_pipeline_outputs_manifest(self.pipeline_manifest_path, self.pipeline_manifest_targets)
 
 
 def _resolve_output_slot(
@@ -644,6 +660,10 @@ def prepare_output_backend(
             f"{transform_class.__name__} does not declare SOURCE_MODALITY"
         )
 
+    materialized = config.get("materialization_outputs", {}).get(source_modality)
+    if materialized is not None:
+        return prepare_materialized_backend(dataset, source_modality, materialized)
+
     pipeline = parse_pipeline_config(config)
     pipeline_target = _select_pipeline_target(config, pipeline, transform_class)
     root = _resolve_output_root(config, pipeline, pipeline_target)
@@ -834,6 +854,12 @@ def prepare_output_backends(
     ``OUTPUT_SLOTS`` order.
     """
 
+    if config.get("materialization_outputs"):
+        return {
+            field: prepare_materialized_backend(dataset, field, options)
+            for field, options in config["materialization_outputs"].items()
+        }
+
     primary_slot = _resolve_primary_slot(transform_class)
     primary_backend = prepare_output_backend(config, dataset, transform_class)
     backends: dict[str, SourceBackedOutputBackend] = {primary_slot: primary_backend}
@@ -859,11 +885,7 @@ def prepare_output_backends(
 
     # Aggregate every slot we actually wrote into the manifest the primary
     # backend will emit, so a single manifest documents the full set.
-    if (
-        pipeline is not None
-        and pipeline.outputs_manifest_path
-        and len(backends) > 1
-    ):
+    if pipeline is not None and pipeline.outputs_manifest_path and len(backends) > 1:
         manifest_targets: list[PipelineOutputTargetConfig] = list(
             primary_backend.pipeline_manifest_targets
         )
@@ -876,3 +898,29 @@ def prepare_output_backends(
         primary_backend.pipeline_manifest_targets = manifest_targets
 
     return backends
+
+
+def prepare_materialized_backend(dataset, field, options):
+    """Explicit bound-output slot. Arbitrary producer changes fail receipt checks."""
+    from euler_loading import output_encoding
+
+    codec_config = options.get("encoding", {"id": "npy"})
+    codec = output_encoding(codec_config["id"], scale=codec_config.get("scale"))
+    writer = dataset.create_output_writer(
+        field,
+        options["path"],
+        derivation=options["derivation"],
+        dataset_id=options["dataset_id"],
+        revision=options["revision"],
+        encoding=codec,
+        expected_full_ids=options.get("expected_full_ids"),
+        metadata_scope=options.get("metadata_scope"),
+        resume=options.get("resume", False),
+    )
+    return SourceBackedOutputBackend(
+        source_modality=field,
+        root=Path(options["path"]),
+        dataset_writer=writer,
+        modality_writer=None,
+        modality_meta=None,
+    )