euler-preprocess 2.3.0__tar.gz → 3.0.0__tar.gz

{euler_preprocess-2.3.0 → euler_preprocess-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 2.3.0
+Version: 3.0.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
@@ -98,7 +98,7 @@ starting at index 10 with stride 5.
 
 | Transform | `modalities` | `hierarchical_modalities` |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | — (intrinsics optional) |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
 | `radial` | `depth` | `intrinsics` |
 
@@ -150,6 +150,8 @@ Controls the fog simulation.
   "contrast_threshold": 0.05,
   "device": "cpu",
   "gpu_batch_size": 4,
+  "capture": { "preset": "camera" },
+  "camera_profile": "dashcam",
   "augmentations": { ... },
   "selection": { ... },
   "models": { ... }
@@ -165,8 +167,130 @@ Controls the fog simulation.
 | `contrast_threshold` | Threshold *C_t* used in the visibility-to-attenuation conversion (default `0.05`). |
 | `device` | `"cpu"`, `"cuda"`, `"mps"`, or `"gpu"` (alias for cuda). |
 | `gpu_batch_size` | Batch size when running on GPU. Uniform-model samples are batched; heterogeneous samples are processed individually. |
+| `capture` / `capture_artifacts` | Optional post-fog camera artifact pipeline. Omit it or set `{"stages": []}` for the legacy no-op path. Set `true`, `{"preset": "camera"}`, or a custom `stages` list to enable optics, raw sensor, ISP, and compression artifacts. |
+| `camera_profile` | Optional named or inline camera profile whose stage defaults are merged into the capture stack before per-stage overrides. Built-ins are `"default"`, `"generic"`, `"dashcam"`, and `"low_light_fog"`. |
+| `camera_profiles` | Optional map of project-specific named profiles. Use this for calibrated lens, sensor, ISP, and transport settings. |
 | `augmentations` | Optional stepped augmentation set. When present, every input sample produces every configured augmentation and uses the file-id hierarchy output layout described below. |
 
+### Processing Pipeline
+
+Fog generation is split into two phases:
+
+1. Ideal scene rendering: physics-based fog and auxiliary `scattering_coefficient`
+   / `atmospheric_light` maps are computed.
+2. Capture artifacts: camera-specific effects are applied to the rendered RGB
+   only. Physical fog maps stay stable while the RGB output can receive
+   exposure shifts, lens blur, vignetting, raw sensor noise, Bayer/demosaicing
+   artifacts, ISP tone/gamma/sharpening, and JPEG/resize/quantization effects.
+
+This keeps physical fog maps stable while making the RGB output extensible for
+real-camera simulation.
+
+### Capture Artifact Stack
+
+Enable the recommended camera stack with:
+
+```json
+"capture": { "preset": "camera" }
+```
+
+or equivalently:
+
+```json
+"capture": true
+```
+
+For tighter control, provide explicit stages in camera order:
+
+```json
+"camera_profiles": {
+  "real_drive_front": {
+    "optics": {
+      "lens_distortion": -0.012,
+      "vignetting_strength": 0.18,
+      "windshield_haze": {"enabled": true, "probability": 0.55}
+    },
+    "sensor": {
+      "bayer_pattern": "RGGB",
+      "iso": {"dist": "choice", "values": [200, 400, 800]},
+      "base_iso": 100,
+      "full_well_electrons": [14000, 12000, 13000],
+      "read_noise_electrons": {"dist": "uniform", "min": 2.0, "max": 6.0},
+      "black_level": [0.003, 0.0035, 0.003],
+      "white_level": [1.0, 0.995, 1.0],
+      "adc_bit_depth": 12,
+      "post_demosaic_bit_depth": 12
+    },
+    "isp": {
+      "tone_map": "reinhard",
+      "gamma": "srgb",
+      "denoise_sigma": 0.2,
+      "sharpen_amount": 0.2,
+      "saturation": 0.9
+    },
+    "transport": {
+      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
+      "bit_depth": 8
+    }
+  }
+},
+"camera_profile": "real_drive_front",
+"capture": {
+  "stages": [
+    {
+      "type": "optics",
+      "blur_sigma": {"dist": "uniform", "min": 0.2, "max": 0.8},
+      "vignetting_strength": 0.15,
+      "windshield_haze": {"enabled": true, "probability": 0.4},
+      "droplets": {"enabled": false}
+    },
+    {
+      "type": "sensor",
+      "input_space": "srgb",
+      "exposure_gain": {"dist": "uniform", "min": 0.85, "max": 1.2},
+      "row_noise_sigma": 0.003
+    },
+    {
+      "type": "isp",
+      "tone_map": "reinhard",
+      "gamma": "srgb",
+      "denoise_sigma": 0.2,
+      "sharpen_amount": 0.2,
+      "saturation": 0.9
+    },
+    {
+      "type": "transport",
+      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
+      "bit_depth": 8
+    }
+  ]
+}
+```
+
+Supported stage types:
+
+| Stage | Main effects |
+|---|---|
+| `optics` | Defocus/MTF blur, motion blur, bloom, veiling glare, vignetting, chromatic aberration, lens distortion, windshield haze, optional droplets. |
+| `sensor` | Exposure, white balance, camera matrix, Bayer mosaic, shot/read noise, fixed-pattern noise, row/column banding, hot/dead pixels, bilinear demosaic. |
+| `isp` | Denoising, color correction, tone mapping, sRGB/gamma, local contrast, sharpening halos, saturation shifts. |
+| `transport` | Crop/resize, bit-depth quantization, JPEG round-trip compression. |
+| `exposure` | Lightweight standalone exposure and white-balance stage for simple custom chains. |
+
+Any stage can define `condition_profiles` to sample coherent per-image settings
+before the stage runs. This is useful for exposure states where ISO, exposure
+gain, read noise, banding, and dark/fog noise modulation should move together:
+
+```json
+{
+  "type": "sensor",
+  "condition_profiles": [
+    {"name": "clean_daylight", "weight": 0.25, "exposure_gain": 1.0, "iso": 100},
+    {"name": "underexposed_noisy", "weight": 0.25, "exposure_gain": 0.65, "iso": 1600}
+  ]
+}
+```
+
 ### Fog Model
 
 The core equation is the **Koschmieder model** (atmospheric scattering):
@@ -224,6 +348,39 @@ When `airlight` is `"dcp_heuristic"`, you can optionally add:
 - `white_bias + cool_bias` must be `<= 1`.
 - The tint bias preserves the estimated airlight luminance, so it shifts colour without silently changing fog density.
 
+### Airlight Intensity Dampening
+
+Estimated airlight is dampened by default as fog density increases. This keeps
+strong fog closer to the low, grey lighting seen in real in-car fog footage
+instead of letting DCP-style estimates wash dense fog toward white.
+
+Each fog model can override the dampening curve:
+
+```json
+"airlight_dampening": {
+  "enabled": true,
+  "apply_to": "estimated",
+  "reference_visibility_m": 80.0,
+  "min_factor": 0.45,
+  "max_factor": 1.0,
+  "strength": 1.0
+}
+```
+
+The factor is:
+`min_factor + (max_factor - min_factor) / (1 + strength * beta / reference_beta)`.
+`reference_beta` is either `reference_scattering_coefficient` /
+`reference_beta`, or it is derived from `reference_visibility_m` using the
+model's contrast threshold. The default applies only when `atmospheric_light`
+uses an estimated airlight method (`"from_sky"`, `"dcp"`, or
+`"dcp_heuristic"`); literal RGB `atmospheric_light` values stay exact unless
+`apply_to` is set to `"all"`. Set `"enabled": false` or `apply_to: "none"` to
+preserve the previous undampened behavior.
+
+For `heterogeneous_ls` and `heterogeneous_k_ls`, the Perlin atmospheric-light
+field is sampled around the dampened base airlight, so the spatial variation
+does not reintroduce the old over-illuminated look.
+
 ### Model Selection
 
 Each image is assigned a fog model via the `selection` block:
@@ -264,7 +421,10 @@ Each model specifies a `visibility_m` distribution from which a visibility dista
 | `lognormal` | `mean`, `sigma`, optional `min`/`max` | Log-normal. |
 | `choice` | `values`, optional `weights` | Discrete weighted choice. |
 
-The sampled visibility *V* is converted to the attenuation coefficient: **k = -ln(C_t) / V**.
+The sampled visibility *V* is converted to the attenuation coefficient:
+**k = -ln(C_t) / V**. This happens once per output image. For
+`heterogeneous_k` and `heterogeneous_k_ls`, that sampled value is the base
+coefficient that is then spatially modulated by the noise field.
 
 ### Stepped Augmentations
 
@@ -355,7 +515,31 @@ The noise field (values in [0, 1]) is mapped to a factor field:
 `contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
 extreme local fog density. When `normalize_to_mean` is `true`, the factor field
 is rescaled so its spatial mean equals 1.0, preserving the overall fog density
-while introducing spatial variation.
+while introducing spatial variation. In other words, with heterogeneous `k`:
+`k(x) = k_sampled * factor(x)`. If `visibility_m` / MOR was sampled from a
+distribution, `k_sampled` is the coefficient derived from that one sampled MOR.
+With `normalize_to_mean: true`, the arithmetic mean of the per-pixel `k` map
+equals `k_sampled`; the median is not forced to match. With
+`normalize_to_mean: false`, the map mean shifts by the mean of the factor field.
+
+For `heterogeneous_ls` / `heterogeneous_k_ls`, `ls_hetero` can also include a
+weak view-direction illumination prior. This modulates the atmospheric-light
+field itself, so the rendered effect is still gated by fog transmittance:
+
+```json
+"ls_hetero": {
+  "ls_gradient": {
+    "enabled": true,
+    "probability": 0.65,
+    "axis": "vertical",
+    "top_factor": {"dist": "uniform", "min": 1.03, "max": 1.14},
+    "bottom_factor": {"dist": "uniform", "min": 0.88, "max": 0.99},
+    "gamma": {"dist": "uniform", "min": 0.85, "max": 1.6},
+    "normalize_to_mean": true,
+    "fog_opacity_weight": 0.65
+  }
+}
+```
 
 | Parameter | Effect |
 |---|---|
@@ -366,16 +550,19 @@ while introducing spatial variation.
 | `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
 | `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
 | `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
+| `ls_gradient` | Optional `L_s` top-to-bottom or left-to-right factor field. Keep it weak and probabilistic to avoid a deterministic image-position shortcut. |
 
 ### Fog Output
 
 CLI runs write a source-backed RGB dataset. The output keeps the source RGB
-dataset's relative paths, basenames, extensions, and `output.json` metadata so
-the result stays loadable by `euler-loading`:
+dataset's relative paths, basenames, extensions, and ds-crawler metadata so the
+result stays loadable by `euler-loading`:
 
 ```
 <output_path>/
-  .ds_crawler/output.json
+  .ds_crawler/dataset-head.json
+  .ds_crawler/ds-crawler.json
+  .ds_crawler/index.json
   Scene01/
     Camera_0/
       00000.png
@@ -390,7 +577,9 @@ the source file id instead:
 
 ```
 <output_path>/
-  .ds_crawler/output.json
+  .ds_crawler/dataset-head.json
+  .ds_crawler/ds-crawler.json
+  .ds_crawler/index.json
   Scene01/
     Camera_0/
       00000/
@@ -446,6 +635,6 @@ No special parameters are required. The transform reads intrinsics from the `int
 ### Radial Output
 
 CLI runs write a source-backed depth dataset mirroring the input depth
-modality's layout and writer metadata. The emitted `output.json` also flips
+modality's layout and writer metadata. The emitted `index.json` also flips
 `meta.radial_depth` to `true`. Standalone/direct `RadialTransform(...)` usage
 keeps the legacy `.npy` output behavior.

{euler_preprocess-2.3.0 → euler_preprocess-3.0.0}/README.md

@@ -84,7 +84,7 @@ starting at index 10 with stride 5.
 
 | Transform | `modalities` | `hierarchical_modalities` |
 |---|---|---|
-| `fog` | `rgb`, `depth`, `semantic_segmentation` | — (intrinsics optional) |
+| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
 | `sky-depth` | `depth`, `semantic_segmentation` | — |
 | `radial` | `depth` | `intrinsics` |
 
@@ -136,6 +136,8 @@ Controls the fog simulation.
   "contrast_threshold": 0.05,
   "device": "cpu",
   "gpu_batch_size": 4,
+  "capture": { "preset": "camera" },
+  "camera_profile": "dashcam",
   "augmentations": { ... },
   "selection": { ... },
   "models": { ... }
@@ -151,8 +153,130 @@ Controls the fog simulation.
 | `contrast_threshold` | Threshold *C_t* used in the visibility-to-attenuation conversion (default `0.05`). |
 | `device` | `"cpu"`, `"cuda"`, `"mps"`, or `"gpu"` (alias for cuda). |
 | `gpu_batch_size` | Batch size when running on GPU. Uniform-model samples are batched; heterogeneous samples are processed individually. |
+| `capture` / `capture_artifacts` | Optional post-fog camera artifact pipeline. Omit it or set `{"stages": []}` for the legacy no-op path. Set `true`, `{"preset": "camera"}`, or a custom `stages` list to enable optics, raw sensor, ISP, and compression artifacts. |
+| `camera_profile` | Optional named or inline camera profile whose stage defaults are merged into the capture stack before per-stage overrides. Built-ins are `"default"`, `"generic"`, `"dashcam"`, and `"low_light_fog"`. |
+| `camera_profiles` | Optional map of project-specific named profiles. Use this for calibrated lens, sensor, ISP, and transport settings. |
 | `augmentations` | Optional stepped augmentation set. When present, every input sample produces every configured augmentation and uses the file-id hierarchy output layout described below. |
 
+### Processing Pipeline
+
+Fog generation is split into two phases:
+
+1. Ideal scene rendering: physics-based fog and auxiliary `scattering_coefficient`
+   / `atmospheric_light` maps are computed.
+2. Capture artifacts: camera-specific effects are applied to the rendered RGB
+   only. Physical fog maps stay stable while the RGB output can receive
+   exposure shifts, lens blur, vignetting, raw sensor noise, Bayer/demosaicing
+   artifacts, ISP tone/gamma/sharpening, and JPEG/resize/quantization effects.
+
+This keeps physical fog maps stable while making the RGB output extensible for
+real-camera simulation.
+
+### Capture Artifact Stack
+
+Enable the recommended camera stack with:
+
+```json
+"capture": { "preset": "camera" }
+```
+
+or equivalently:
+
+```json
+"capture": true
+```
+
+For tighter control, provide explicit stages in camera order:
+
+```json
+"camera_profiles": {
+  "real_drive_front": {
+    "optics": {
+      "lens_distortion": -0.012,
+      "vignetting_strength": 0.18,
+      "windshield_haze": {"enabled": true, "probability": 0.55}
+    },
+    "sensor": {
+      "bayer_pattern": "RGGB",
+      "iso": {"dist": "choice", "values": [200, 400, 800]},
+      "base_iso": 100,
+      "full_well_electrons": [14000, 12000, 13000],
+      "read_noise_electrons": {"dist": "uniform", "min": 2.0, "max": 6.0},
+      "black_level": [0.003, 0.0035, 0.003],
+      "white_level": [1.0, 0.995, 1.0],
+      "adc_bit_depth": 12,
+      "post_demosaic_bit_depth": 12
+    },
+    "isp": {
+      "tone_map": "reinhard",
+      "gamma": "srgb",
+      "denoise_sigma": 0.2,
+      "sharpen_amount": 0.2,
+      "saturation": 0.9
+    },
+    "transport": {
+      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
+      "bit_depth": 8
+    }
+  }
+},
+"camera_profile": "real_drive_front",
+"capture": {
+  "stages": [
+    {
+      "type": "optics",
+      "blur_sigma": {"dist": "uniform", "min": 0.2, "max": 0.8},
+      "vignetting_strength": 0.15,
+      "windshield_haze": {"enabled": true, "probability": 0.4},
+      "droplets": {"enabled": false}
+    },
+    {
+      "type": "sensor",
+      "input_space": "srgb",
+      "exposure_gain": {"dist": "uniform", "min": 0.85, "max": 1.2},
+      "row_noise_sigma": 0.003
+    },
+    {
+      "type": "isp",
+      "tone_map": "reinhard",
+      "gamma": "srgb",
+      "denoise_sigma": 0.2,
+      "sharpen_amount": 0.2,
+      "saturation": 0.9
+    },
+    {
+      "type": "transport",
+      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
+      "bit_depth": 8
+    }
+  ]
+}
+```
+
+Supported stage types:
+
+| Stage | Main effects |
+|---|---|
+| `optics` | Defocus/MTF blur, motion blur, bloom, veiling glare, vignetting, chromatic aberration, lens distortion, windshield haze, optional droplets. |
+| `sensor` | Exposure, white balance, camera matrix, Bayer mosaic, shot/read noise, fixed-pattern noise, row/column banding, hot/dead pixels, bilinear demosaic. |
+| `isp` | Denoising, color correction, tone mapping, sRGB/gamma, local contrast, sharpening halos, saturation shifts. |
+| `transport` | Crop/resize, bit-depth quantization, JPEG round-trip compression. |
+| `exposure` | Lightweight standalone exposure and white-balance stage for simple custom chains. |
+
+Any stage can define `condition_profiles` to sample coherent per-image settings
+before the stage runs. This is useful for exposure states where ISO, exposure
+gain, read noise, banding, and dark/fog noise modulation should move together:
+
+```json
+{
+  "type": "sensor",
+  "condition_profiles": [
+    {"name": "clean_daylight", "weight": 0.25, "exposure_gain": 1.0, "iso": 100},
+    {"name": "underexposed_noisy", "weight": 0.25, "exposure_gain": 0.65, "iso": 1600}
+  ]
+}
+```
+
 ### Fog Model
 
 The core equation is the **Koschmieder model** (atmospheric scattering):
@@ -210,6 +334,39 @@ When `airlight` is `"dcp_heuristic"`, you can optionally add:
 - `white_bias + cool_bias` must be `<= 1`.
 - The tint bias preserves the estimated airlight luminance, so it shifts colour without silently changing fog density.
 
+### Airlight Intensity Dampening
+
+Estimated airlight is dampened by default as fog density increases. This keeps
+strong fog closer to the low, grey lighting seen in real in-car fog footage
+instead of letting DCP-style estimates wash dense fog toward white.
+
+Each fog model can override the dampening curve:
+
+```json
+"airlight_dampening": {
+  "enabled": true,
+  "apply_to": "estimated",
+  "reference_visibility_m": 80.0,
+  "min_factor": 0.45,
+  "max_factor": 1.0,
+  "strength": 1.0
+}
+```
+
+The factor is:
+`min_factor + (max_factor - min_factor) / (1 + strength * beta / reference_beta)`.
+`reference_beta` is either `reference_scattering_coefficient` /
+`reference_beta`, or it is derived from `reference_visibility_m` using the
+model's contrast threshold. The default applies only when `atmospheric_light`
+uses an estimated airlight method (`"from_sky"`, `"dcp"`, or
+`"dcp_heuristic"`); literal RGB `atmospheric_light` values stay exact unless
+`apply_to` is set to `"all"`. Set `"enabled": false` or `apply_to: "none"` to
+preserve the previous undampened behavior.
+
+For `heterogeneous_ls` and `heterogeneous_k_ls`, the Perlin atmospheric-light
+field is sampled around the dampened base airlight, so the spatial variation
+does not reintroduce the old over-illuminated look.
+
 ### Model Selection
 
 Each image is assigned a fog model via the `selection` block:
@@ -250,7 +407,10 @@ Each model specifies a `visibility_m` distribution from which a visibility dista
 | `lognormal` | `mean`, `sigma`, optional `min`/`max` | Log-normal. |
 | `choice` | `values`, optional `weights` | Discrete weighted choice. |
 
-The sampled visibility *V* is converted to the attenuation coefficient: **k = -ln(C_t) / V**.
+The sampled visibility *V* is converted to the attenuation coefficient:
+**k = -ln(C_t) / V**. This happens once per output image. For
+`heterogeneous_k` and `heterogeneous_k_ls`, that sampled value is the base
+coefficient that is then spatially modulated by the noise field.
 
 ### Stepped Augmentations
 
@@ -341,7 +501,31 @@ The noise field (values in [0, 1]) is mapped to a factor field:
 `contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
 extreme local fog density. When `normalize_to_mean` is `true`, the factor field
 is rescaled so its spatial mean equals 1.0, preserving the overall fog density
-while introducing spatial variation.
+while introducing spatial variation. In other words, with heterogeneous `k`:
+`k(x) = k_sampled * factor(x)`. If `visibility_m` / MOR was sampled from a
+distribution, `k_sampled` is the coefficient derived from that one sampled MOR.
+With `normalize_to_mean: true`, the arithmetic mean of the per-pixel `k` map
+equals `k_sampled`; the median is not forced to match. With
+`normalize_to_mean: false`, the map mean shifts by the mean of the factor field.
+
+For `heterogeneous_ls` / `heterogeneous_k_ls`, `ls_hetero` can also include a
+weak view-direction illumination prior. This modulates the atmospheric-light
+field itself, so the rendered effect is still gated by fog transmittance:
+
+```json
+"ls_hetero": {
+  "ls_gradient": {
+    "enabled": true,
+    "probability": 0.65,
+    "axis": "vertical",
+    "top_factor": {"dist": "uniform", "min": 1.03, "max": 1.14},
+    "bottom_factor": {"dist": "uniform", "min": 0.88, "max": 0.99},
+    "gamma": {"dist": "uniform", "min": 0.85, "max": 1.6},
+    "normalize_to_mean": true,
+    "fog_opacity_weight": 0.65
+  }
+}
+```
 
 | Parameter | Effect |
 |---|---|
@@ -352,16 +536,19 @@ while introducing spatial variation.
 | `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
 | `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
 | `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
+| `ls_gradient` | Optional `L_s` top-to-bottom or left-to-right factor field. Keep it weak and probabilistic to avoid a deterministic image-position shortcut. |
 
 ### Fog Output
 
 CLI runs write a source-backed RGB dataset. The output keeps the source RGB
-dataset's relative paths, basenames, extensions, and `output.json` metadata so
-the result stays loadable by `euler-loading`:
+dataset's relative paths, basenames, extensions, and ds-crawler metadata so the
+result stays loadable by `euler-loading`:
 
 ```
 <output_path>/
-  .ds_crawler/output.json
+  .ds_crawler/dataset-head.json
+  .ds_crawler/ds-crawler.json
+  .ds_crawler/index.json
   Scene01/
     Camera_0/
       00000.png
@@ -376,7 +563,9 @@ the source file id instead:
 
 ```
 <output_path>/
-  .ds_crawler/output.json
+  .ds_crawler/dataset-head.json
+  .ds_crawler/ds-crawler.json
+  .ds_crawler/index.json
   Scene01/
     Camera_0/
       00000/
@@ -432,6 +621,6 @@ No special parameters are required. The transform reads intrinsics from the `int
 ### Radial Output
 
 CLI runs write a source-backed depth dataset mirroring the input depth
-modality's layout and writer metadata. The emitted `output.json` also flips
+modality's layout and writer metadata. The emitted `index.json` also flips
 `meta.radial_depth` to `true`. Standalone/direct `RadialTransform(...)` usage
 keeps the legacy `.npy` output behavior.

{euler_preprocess-2.3.0 → euler_preprocess-3.0.0}/euler_preprocess/common/output.py

@@ -9,6 +9,11 @@ from pathlib import Path
 from typing import Any
 
 from ds_crawler import DatasetWriter, ZipDatasetWriter
+from ds_crawler.zip_utils import (
+    METADATA_DIR,
+    OUTPUT_FILENAME as DS_CRAWLER_INDEX_FILENAME,
+    get_metadata_entry_name,
+)
 from euler_loading import MultiModalDataset
 from euler_loading.dataset import create_dataset_writer_from_index
 from euler_loading.loaders._writer_utils import supports_stream_target
@@ -216,12 +221,18 @@ def _merge_top_level_mapping(target: dict[str, Any], overlay: dict[str, Any]) ->
 def _patch_output_index(
     dataset_writer: DatasetWriter | ZipDatasetWriter,
     overrides: dict[str, Any],
+    *,
+    filename: str = DS_CRAWLER_INDEX_FILENAME,
 ) -> None:
     if not overrides:
         return
 
+    metadata_scope = getattr(dataset_writer, "_metadata_scope", None)
     if isinstance(dataset_writer, ZipDatasetWriter):
-        entry_name = ".ds_crawler/output.json"
+        entry_name = get_metadata_entry_name(
+            filename,
+            metadata_scope=metadata_scope,
+        )
         archive_path = Path(dataset_writer.root)
         replacement_name = archive_path.with_suffix(archive_path.suffix + ".tmp")
         with zipfile.ZipFile(archive_path, "r") as source_zip:
@@ -238,7 +249,10 @@ def _patch_output_index(
         replacement_name.replace(archive_path)
         return
 
-    output_path = Path(dataset_writer.root) / ".ds_crawler" / "output.json"
+    output_path = Path(dataset_writer.root) / METADATA_DIR
+    if metadata_scope is not None:
+        output_path = output_path / metadata_scope
+    output_path = output_path / filename
     payload = json.loads(output_path.read_text(encoding="utf-8"))
     _merge_top_level_mapping(payload, overrides)
     output_path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
@@ -508,8 +522,12 @@ class SourceBackedOutputBackend:
         )
 
     def finalize(self) -> None:
-        self.dataset_writer.save_index()
-        _patch_output_index(self.dataset_writer, self.index_overrides)
+        self.dataset_writer.save_index(filename=DS_CRAWLER_INDEX_FILENAME)
+        _patch_output_index(
+            self.dataset_writer,
+            self.index_overrides,
+            filename=DS_CRAWLER_INDEX_FILENAME,
+        )
         if self.pipeline_manifest_path and self.pipeline_manifest_targets:
             _write_pipeline_outputs_manifest(
                 self.pipeline_manifest_path,