euler-preprocess 2.2.0__tar.gz → 2.3.0__tar.gz

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 2.2.0
+Version: 2.3.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
@@ -232,10 +232,10 @@ Each image is assigned a fog model via the `selection` block:
 "selection": {
   "mode": "weighted",
   "weights": {
-    "uniform": 1.0,
-    "heterogeneous_k": 0.0,
-    "heterogeneous_ls": 0.0,
-    "heterogeneous_k_ls": 0.0
+    "uniform": 0.25,
+    "heterogeneous_k": 0.35,
+    "heterogeneous_ls": 0.25,
+    "heterogeneous_k_ls": 0.15
   }
 }
 ```
@@ -309,9 +309,12 @@ variants:
       "scattering_coefficient": 0.15,
       "atmospheric_light": [1.0, 1.0, 1.0],
       "k_hetero": {
-        "scales": "auto",
-        "min_factor": 0.5,
-        "max_factor": 1.5,
+        "scales": "smooth_auto",
+        "correlation_length_fraction": 0.25,
+        "octaves": 3,
+        "min_factor": 0.65,
+        "max_factor": 1.45,
+        "contrast": 0.65,
         "normalize_to_mean": true
       }
     }
@@ -327,26 +330,42 @@ MOR/beta descriptors when available. euler-loading exposes these as
 
 ### Heterogeneous Noise Fields
 
-Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian motion) to generate spatially-varying factor fields:
+Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian
+motion) to generate spatially-varying factor fields. For realistic fog,
+prefer the smooth mode: it keeps Perlin wavelengths tied to the image size,
+then optionally reduces noise contrast and applies a final blur before mapping
+the noise to physical factors.
 
 ```json
 "k_hetero": {
-  "scales": "auto",
-  "min_scale": 2,
+  "scales": "smooth_auto",
+  "correlation_length_fraction": 0.25,
+  "octaves": 3,
   "max_scale": null,
-  "min_factor": 0.0,
-  "max_factor": 1.0,
+  "min_factor": 0.65,
+  "max_factor": 1.45,
+  "contrast": 0.65,
+  "smooth_sigma_fraction": 0.0,
   "normalize_to_mean": true
 }
 ```
 
-The noise field (values in [0, 1]) is mapped to a factor field: `factor(x) = min_factor + (max_factor - min_factor) * noise(x)`. When `normalize_to_mean` is `true`, the factor field is rescaled so its spatial mean equals 1.0, preserving the overall fog density while introducing spatial variation.
+The noise field (values in [0, 1]) is mapped to a factor field:
+`factor(x) = min_factor + (max_factor - min_factor) * noise(x)`.
+`contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
+extreme local fog density. When `normalize_to_mean` is `true`, the factor field
+is rescaled so its spatial mean equals 1.0, preserving the overall fog density
+while introducing spatial variation.
 
 | Parameter | Effect |
 |---|---|
 | `min_factor` / `max_factor` | Range of the multiplicative factor. |
 | `normalize_to_mean` | Rescale factors so the image-wide mean equals the base value. Recommended for `k_hetero`. |
-| `scales` / `min_scale` / `max_scale` | Control spatial frequency content. |
+| `scales: "smooth_auto"` | Build low-frequency Perlin scales from the image size. |
+| `correlation_length_fraction` | Approximate smallest fog feature size as a fraction of the shorter image side. Larger values create smoother gradients. |
+| `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
+| `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
+| `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
 
 ### Fog Output
 

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/README.md

@@ -218,10 +218,10 @@ Each image is assigned a fog model via the `selection` block:
 "selection": {
   "mode": "weighted",
   "weights": {
-    "uniform": 1.0,
-    "heterogeneous_k": 0.0,
-    "heterogeneous_ls": 0.0,
-    "heterogeneous_k_ls": 0.0
+    "uniform": 0.25,
+    "heterogeneous_k": 0.35,
+    "heterogeneous_ls": 0.25,
+    "heterogeneous_k_ls": 0.15
   }
 }
 ```
@@ -295,9 +295,12 @@ variants:
       "scattering_coefficient": 0.15,
       "atmospheric_light": [1.0, 1.0, 1.0],
       "k_hetero": {
-        "scales": "auto",
-        "min_factor": 0.5,
-        "max_factor": 1.5,
+        "scales": "smooth_auto",
+        "correlation_length_fraction": 0.25,
+        "octaves": 3,
+        "min_factor": 0.65,
+        "max_factor": 1.45,
+        "contrast": 0.65,
         "normalize_to_mean": true
       }
     }
@@ -313,26 +316,42 @@ MOR/beta descriptors when available. euler-loading exposes these as
 
 ### Heterogeneous Noise Fields
 
-Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian motion) to generate spatially-varying factor fields:
+Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian
+motion) to generate spatially-varying factor fields. For realistic fog,
+prefer the smooth mode: it keeps Perlin wavelengths tied to the image size,
+then optionally reduces noise contrast and applies a final blur before mapping
+the noise to physical factors.
 
 ```json
 "k_hetero": {
-  "scales": "auto",
-  "min_scale": 2,
+  "scales": "smooth_auto",
+  "correlation_length_fraction": 0.25,
+  "octaves": 3,
   "max_scale": null,
-  "min_factor": 0.0,
-  "max_factor": 1.0,
+  "min_factor": 0.65,
+  "max_factor": 1.45,
+  "contrast": 0.65,
+  "smooth_sigma_fraction": 0.0,
   "normalize_to_mean": true
 }
 ```
 
-The noise field (values in [0, 1]) is mapped to a factor field: `factor(x) = min_factor + (max_factor - min_factor) * noise(x)`. When `normalize_to_mean` is `true`, the factor field is rescaled so its spatial mean equals 1.0, preserving the overall fog density while introducing spatial variation.
+The noise field (values in [0, 1]) is mapped to a factor field:
+`factor(x) = min_factor + (max_factor - min_factor) * noise(x)`.
+`contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
+extreme local fog density. When `normalize_to_mean` is `true`, the factor field
+is rescaled so its spatial mean equals 1.0, preserving the overall fog density
+while introducing spatial variation.
 
 | Parameter | Effect |
 |---|---|
 | `min_factor` / `max_factor` | Range of the multiplicative factor. |
 | `normalize_to_mean` | Rescale factors so the image-wide mean equals the base value. Recommended for `k_hetero`. |
-| `scales` / `min_scale` / `max_scale` | Control spatial frequency content. |
+| `scales: "smooth_auto"` | Build low-frequency Perlin scales from the image size. |
+| `correlation_length_fraction` | Approximate smallest fog feature size as a fraction of the shorter image side. Larger values create smoother gradients. |
+| `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
+| `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
+| `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
 
 ### Fog Output
 

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/euler_preprocess/common/output.py

@@ -447,13 +447,22 @@ class SourceBackedOutputBackend:
             entry_attributes = None
         if attributes:
             entry_attributes = {**(entry_attributes or {}), **attributes}
+        source_entry_for_writer = dict(source_meta_copy)
+        if output_full_id is not None or output_basename is not None:
+            # The caller is intentionally writing a new logical layout
+            # (e.g. source sample -> augmentation).  Let DatasetWriter derive
+            # properties from the new full_id/basename instead of copying the
+            # source file's old path and basename captures.
+            source_entry_for_writer.pop("path_properties", None)
+            source_entry_for_writer.pop("basename_properties", None)
+            source_entry_for_writer.pop("attributes", None)
 
         if isinstance(self.dataset_writer, ZipDatasetWriter):
             if supports_stream_target(self.modality_writer):
                 with self.dataset_writer.open(
                     full_id,
                     basename,
-                    source_entry=source_meta_copy,
+                    source_entry=source_entry_for_writer,
                     attributes=entry_attributes,
                 ) as stream:
                     _set_stream_name(stream, basename)
@@ -466,7 +475,7 @@ class SourceBackedOutputBackend:
                         full_id,
                         basename,
                         temp_path.read_bytes(),
-                        source_entry=source_meta_copy,
+                        source_entry=source_entry_for_writer,
                         attributes=entry_attributes,
                     )
             return Path(f"{self.dataset_writer.root}::{relative_path}")
@@ -474,12 +483,25 @@ class SourceBackedOutputBackend:
         target_path = self.dataset_writer.get_path(
             full_id,
             basename,
-            source_entry=source_meta_copy,
+            source_entry=source_entry_for_writer,
             attributes=entry_attributes,
         )
         self.modality_writer(str(target_path), value, self.modality_meta)
         return target_path
 
+    def set_hierarchy_separator(self, separator: str) -> None:
+        """Set the writer hierarchy separator used for future entries."""
+        setattr(self.dataset_writer, "_separator", separator)
+
+    def add_head_addon(self, name: str, payload: dict[str, Any]) -> None:
+        """Add a dataset-head addon before the writer saves its artifacts."""
+        head = getattr(self.dataset_writer, "_dataset_head", None)
+        addons = getattr(head, "addons", None)
+        if not isinstance(addons, dict):
+            raise RuntimeError("Unsupported dataset writer head object")
+        addons[name] = dict(payload)
+        self.index_overrides[name] = dict(payload)
+
     def write_json(self, path: Path, data: dict[str, Any]) -> None:
         raise RuntimeError(
             "Source-backed outputs do not support auxiliary JSON sidecars."

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/euler_preprocess/fog/models.py

@@ -28,11 +28,13 @@ DEFAULT_MODEL_CONFIGS = {
         "visibility_m": {"dist": "constant", "value": 80.0},
         "atmospheric_light": "from_sky",
         "k_hetero": {
-            "scales": "auto",
-            "min_scale": 2,
+            "scales": "smooth_auto",
+            "correlation_length_fraction": 0.25,
+            "octaves": 3,
             "max_scale": None,
-            "min_factor": 0.0,
-            "max_factor": 1.0,
+            "min_factor": 0.65,
+            "max_factor": 1.45,
+            "contrast": 0.65,
             "normalize_to_mean": True,
         },
     },
@@ -40,11 +42,13 @@ DEFAULT_MODEL_CONFIGS = {
         "visibility_m": {"dist": "constant", "value": 80.0},
         "atmospheric_light": "from_sky",
         "ls_hetero": {
-            "scales": "auto",
-            "min_scale": 2,
+            "scales": "smooth_auto",
+            "correlation_length_fraction": 0.35,
+            "octaves": 3,
             "max_scale": None,
-            "min_factor": 0.0,
-            "max_factor": 1.0,
+            "min_factor": 0.85,
+            "max_factor": 1.08,
+            "contrast": 0.55,
             "normalize_to_mean": False,
         },
     },
@@ -52,19 +56,23 @@ DEFAULT_MODEL_CONFIGS = {
         "visibility_m": {"dist": "constant", "value": 80.0},
         "atmospheric_light": "from_sky",
         "k_hetero": {
-            "scales": "auto",
-            "min_scale": 2,
+            "scales": "smooth_auto",
+            "correlation_length_fraction": 0.25,
+            "octaves": 3,
             "max_scale": None,
-            "min_factor": 0.0,
-            "max_factor": 1.0,
+            "min_factor": 0.65,
+            "max_factor": 1.45,
+            "contrast": 0.65,
             "normalize_to_mean": True,
         },
         "ls_hetero": {
-            "scales": "auto",
-            "min_scale": 2,
+            "scales": "smooth_auto",
+            "correlation_length_fraction": 0.35,
+            "octaves": 3,
             "max_scale": None,
-            "min_factor": 0.0,
-            "max_factor": 1.0,
+            "min_factor": 0.85,
+            "max_factor": 1.08,
+            "contrast": 0.55,
             "normalize_to_mean": False,
         },
     },
@@ -166,6 +174,8 @@ def resolve_scales(
     scales_spec = hetero_cfg.get("scales", "auto")
     scales_spec = sample_value(scales_spec, rng)
     if isinstance(scales_spec, str):
+        if scales_spec == "smooth_auto":
+            return _resolve_smooth_auto_scales(hetero_cfg, height, width, rng)
         if scales_spec != "auto":
             raise ValueError(f"Unsupported scales value: {scales_spec}")
         min_scale = int(sample_value(hetero_cfg.get("min_scale", 2), rng))
@@ -186,6 +196,255 @@ def resolve_scales(
     raise ValueError(f"Unsupported scales spec: {scales_spec}")
 
 
+def _resolve_smooth_auto_scales(
+    hetero_cfg: dict,
+    height: int,
+    width: int,
+    rng: np.random.Generator,
+) -> list[int]:
+    """Resolve low-frequency Perlin scales for realistic fog gradients."""
+    min_dimension = max(1, min(height, width))
+    max_dimension = max(1, max(height, width))
+
+    base_scale = _resolve_scale_alias(
+        hetero_cfg,
+        rng,
+        absolute_keys=("correlation_length", "base_scale", "min_scale"),
+        fraction_keys=(
+            "correlation_length_fraction",
+            "base_scale_fraction",
+            "min_scale_fraction",
+        ),
+        fraction_basis=min_dimension,
+        default=max(4, int(round(min_dimension * 0.25))),
+    )
+    max_scale = _resolve_scale_alias(
+        hetero_cfg,
+        rng,
+        absolute_keys=("max_scale",),
+        fraction_keys=("max_scale_fraction",),
+        fraction_basis=max_dimension,
+        default=max_dimension,
+        allow_none=True,
+    )
+    max_scale = max(base_scale, max_scale)
+
+    octaves = max(
+        1,
+        int(round(_sample_float(hetero_cfg.get("octaves", 3), rng, "octaves"))),
+    )
+    lacunarity = _sample_float(hetero_cfg.get("lacunarity", 2.0), rng, "lacunarity")
+    if lacunarity <= 1.0:
+        raise ValueError(f"lacunarity must be > 1.0, got {lacunarity}")
+
+    scales: list[int] = []
+    scale = float(base_scale)
+    for _ in range(octaves):
+        scales.append(max(1, int(round(scale))))
+        if scale >= max_scale:
+            break
+        scale = min(scale * lacunarity, float(max_scale))
+    return _unique_positive_scales(scales)
+
+
+def _resolve_scale_alias(
+    hetero_cfg: dict,
+    rng: np.random.Generator,
+    *,
+    absolute_keys: tuple[str, ...],
+    fraction_keys: tuple[str, ...],
+    fraction_basis: int,
+    default: int,
+    allow_none: bool = False,
+) -> int:
+    for key in absolute_keys:
+        if key not in hetero_cfg:
+            continue
+        raw_value = hetero_cfg[key]
+        if raw_value is None and allow_none:
+            break
+        return _scale_pixels(raw_value, rng, key)
+    for key in fraction_keys:
+        if key not in hetero_cfg:
+            continue
+        fraction = _sample_float(hetero_cfg[key], rng, key)
+        if fraction <= 0:
+            raise ValueError(f"{key} must be > 0, got {fraction}")
+        return max(1, int(round(float(fraction_basis) * fraction)))
+    return max(1, int(default))
+
+
+def _scale_pixels(value, rng: np.random.Generator, name: str) -> int:
+    scale = _sample_float(value, rng, name)
+    if scale <= 0:
+        raise ValueError(f"{name} must be > 0, got {scale}")
+    return max(1, int(round(scale)))
+
+
+def _sample_float(value, rng: np.random.Generator, name: str) -> float:
+    sampled = sample_value(value, rng)
+    try:
+        return float(sampled)
+    except (TypeError, ValueError) as exc:
+        raise ValueError(f"{name} must resolve to a number, got {sampled!r}") from exc
+
+
+def _unique_positive_scales(scales: list[int]) -> list[int]:
+    unique: list[int] = []
+    seen: set[int] = set()
+    for scale in scales:
+        scale = int(scale)
+        if scale <= 0 or scale in seen:
+            continue
+        seen.add(scale)
+        unique.append(scale)
+    return unique or [1]
+
+
+def prepare_noise_field(
+    noise: np.ndarray,
+    hetero_cfg: dict,
+    rng: np.random.Generator,
+) -> np.ndarray:
+    """Apply optional smoothing and contrast control to a Perlin noise field."""
+    noise = np.asarray(noise, dtype=np.float32)
+    sigma = resolve_smoothing_sigma(hetero_cfg, noise.shape[0], noise.shape[1], rng)
+    if sigma > 0.0:
+        noise = _gaussian_blur_np(noise, sigma)
+    noise = _normalize_noise_np(noise)
+    contrast = resolve_noise_contrast(hetero_cfg, rng)
+    if contrast != 1.0:
+        noise = 0.5 + (noise - 0.5) * contrast
+    return np.clip(noise, 0.0, 1.0).astype(np.float32, copy=False)
+
+
+def prepare_noise_field_torch(
+    noise: "torch.Tensor",
+    hetero_cfg: dict,
+    rng: np.random.Generator,
+) -> "torch.Tensor":
+    """Torch equivalent of :func:`prepare_noise_field`."""
+    height = int(noise.shape[-2])
+    width = int(noise.shape[-1])
+    sigma = resolve_smoothing_sigma(hetero_cfg, height, width, rng)
+    if sigma > 0.0:
+        noise = _gaussian_blur_torch(noise, sigma)
+    noise = _normalize_noise_torch(noise)
+    contrast = resolve_noise_contrast(hetero_cfg, rng)
+    if contrast != 1.0:
+        noise = 0.5 + (noise - 0.5) * contrast
+    return torch.clamp(noise, 0.0, 1.0)
+
+
+def resolve_smoothing_sigma(
+    hetero_cfg: dict,
+    height: int,
+    width: int,
+    rng: np.random.Generator,
+) -> float:
+    for key in ("smooth_sigma", "smoothing_sigma", "blur_sigma"):
+        if key in hetero_cfg:
+            sigma = _sample_float(hetero_cfg[key], rng, key)
+            if sigma < 0:
+                raise ValueError(f"{key} must be >= 0, got {sigma}")
+            return sigma
+    for key in (
+        "smooth_sigma_fraction",
+        "smoothing_sigma_fraction",
+        "blur_sigma_fraction",
+    ):
+        if key in hetero_cfg:
+            fraction = _sample_float(hetero_cfg[key], rng, key)
+            if fraction < 0:
+                raise ValueError(f"{key} must be >= 0, got {fraction}")
+            return fraction * float(max(1, min(height, width)))
+    return 0.0
+
+
+def resolve_noise_contrast(hetero_cfg: dict, rng: np.random.Generator) -> float:
+    raw = hetero_cfg.get("contrast", hetero_cfg.get("noise_contrast", 1.0))
+    contrast = _sample_float(raw, rng, "contrast")
+    if contrast < 0:
+        raise ValueError(f"contrast must be >= 0, got {contrast}")
+    return contrast
+
+
+def _normalize_noise_np(noise: np.ndarray) -> np.ndarray:
+    min_val = float(np.min(noise))
+    max_val = float(np.max(noise))
+    denom = max_val - min_val
+    if denom <= 1e-8:
+        return np.full_like(noise, 0.5, dtype=np.float32)
+    return ((noise - min_val) / denom).astype(np.float32, copy=False)
+
+
+def _normalize_noise_torch(noise: "torch.Tensor") -> "torch.Tensor":
+    min_val = noise.amin()
+    max_val = noise.amax()
+    denom = max_val - min_val
+    if float(denom.item()) <= 1e-8:
+        return torch.full_like(noise, 0.5)
+    return (noise - min_val) / denom
+
+
+def _gaussian_kernel_np(sigma: float) -> np.ndarray:
+    radius = max(1, int(math.ceil(3.0 * sigma)))
+    offsets = np.arange(-radius, radius + 1, dtype=np.float32)
+    kernel = np.exp(-0.5 * (offsets / float(sigma)) ** 2)
+    kernel /= float(kernel.sum())
+    return kernel.astype(np.float32)
+
+
+def _convolve_axis_np(
+    values: np.ndarray,
+    kernel: np.ndarray,
+    axis: int,
+) -> np.ndarray:
+    radius = kernel.shape[0] // 2
+    padding = [(0, 0)] * values.ndim
+    padding[axis] = (radius, radius)
+    padded = np.pad(values, padding, mode="edge")
+    result = np.zeros_like(values, dtype=np.float32)
+    for offset, weight in enumerate(kernel):
+        slices = [slice(None)] * values.ndim
+        slices[axis] = slice(offset, offset + values.shape[axis])
+        result += float(weight) * padded[tuple(slices)]
+    return result
+
+
+def _gaussian_blur_np(noise: np.ndarray, sigma: float) -> np.ndarray:
+    if sigma <= 0.0:
+        return noise
+    kernel = _gaussian_kernel_np(sigma)
+    blurred = _convolve_axis_np(noise, kernel, axis=1)
+    return _convolve_axis_np(blurred, kernel, axis=0)
+
+
+def _gaussian_blur_torch(noise: "torch.Tensor", sigma: float) -> "torch.Tensor":
+    if sigma <= 0.0:
+        return noise
+    radius = max(1, int(math.ceil(3.0 * sigma)))
+    offsets = torch.arange(
+        -radius,
+        radius + 1,
+        device=noise.device,
+        dtype=torch.float32,
+    )
+    kernel = torch.exp(-0.5 * (offsets / float(sigma)) ** 2)
+    kernel = kernel / kernel.sum()
+    x = noise.to(dtype=torch.float32).view(
+        1,
+        1,
+        int(noise.shape[-2]),
+        int(noise.shape[-1]),
+    )
+    x = torch.nn.functional.pad(x, (radius, radius, 0, 0), mode="replicate")
+    x = torch.nn.functional.conv2d(x, kernel.view(1, 1, 1, -1))
+    x = torch.nn.functional.pad(x, (0, 0, radius, radius), mode="replicate")
+    x = torch.nn.functional.conv2d(x, kernel.view(1, 1, -1, 1))
+    return x.view(noise.shape)
+
+
 def modulate_with_noise(
     mean_value: np.ndarray,
     noise: np.ndarray,
@@ -349,6 +608,7 @@ def apply_model(
         k_cfg = model_cfg.get("k_hetero", {})
         k_scales = resolve_scales(k_cfg, height, width, rng)
         k_noise = perlin_fbm(height, width, k_scales, rng)
+        k_noise = prepare_noise_field(k_noise, k_cfg, rng)
         min_factor = float(sample_value(k_cfg.get("min_factor", 1.0), rng))
         max_factor = float(sample_value(k_cfg.get("max_factor", 1.0), rng))
         k_field = modulate_with_noise(
@@ -365,6 +625,7 @@ def apply_model(
         ls_cfg = model_cfg.get("ls_hetero", {})
         ls_scales = resolve_scales(ls_cfg, height, width, rng)
         ls_noise = perlin_fbm(height, width, ls_scales, rng)
+        ls_noise = prepare_noise_field(ls_noise, ls_cfg, rng)
         min_factor = float(sample_value(ls_cfg.get("min_factor", 1.0), rng))
         max_factor = float(sample_value(ls_cfg.get("max_factor", 1.0), rng))
         ls_field = modulate_with_noise(

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/euler_preprocess/fog/transform.py

@@ -40,6 +40,7 @@ from euler_preprocess.fog.models import (
     estimate_airlight_torch,
     modulate_with_noise_torch,
     normalize_atmospheric_light_torch,
+    prepare_noise_field_torch,
     resolve_model_config,
     resolve_scattering_coefficient,
     resolve_scales,
@@ -51,6 +52,33 @@ from euler_loading.loaders.cpu.generic import (
     write_map_3d as _write_map_3d,
 )
 
+try:
+    from ds_crawler import EULER_LAYOUT_ADDON, build_layout_addon
+except ImportError:  # pragma: no cover - compatibility with older ds-crawler
+    EULER_LAYOUT_ADDON = "euler_layout"
+
+    def build_layout_addon(**kwargs):
+        payload: dict[str, Any] = {
+            "version": kwargs.get("version", "1.0"),
+            "sample_axis": {
+                "name": kwargs["sample_axis_name"],
+                "location": kwargs["sample_axis_location"],
+            },
+        }
+        family = kwargs.get("family")
+        if family is not None:
+            payload["family"] = family
+        variant_axis_name = kwargs.get("variant_axis_name")
+        if variant_axis_name is not None:
+            payload["variant_axis"] = {
+                "name": variant_axis_name,
+                "location": kwargs.get("variant_axis_location", "file_id"),
+            }
+        derived_from = kwargs.get("derived_from")
+        if derived_from is not None:
+            payload["derived_from"] = dict(derived_from)
+        return payload
+
 
 SCATTERING_COEFFICIENT_SLOT = "scattering_coefficient"
 ATMOSPHERIC_LIGHT_SLOT = "atmospheric_light"
@@ -174,6 +202,7 @@ class FogTransform(Transform):
             self.config
         )
         self.augmentation_specs = list(self.augmentation_config.specs)
+        self._configure_output_layout_metadata()
         self._written_configs: set[str] = set()
         self.torch_device = None
         self.use_gpu = False
@@ -374,9 +403,57 @@ class FogTransform(Transform):
                     return suffix
         return ".png"
 
+    def _layout_family(self) -> str | None:
+        raw = self.config.get("dataset_family")
+        return raw if isinstance(raw, str) and raw else None
+
+    def _augmentation_hierarchy_separator(self, backend: Any) -> str:
+        separator = getattr(getattr(backend, "dataset_writer", None), "_separator", None)
+        if isinstance(separator, str) and separator and separator != "+":
+            return separator
+        return ":"
+
+    def _configure_output_layout_metadata(self) -> None:
+        """Declare fog outputs as variants grouped by source sample id."""
+        if not self.augmentation_specs:
+            return
+
+        sample_axis_name = self.augmentation_config.file_id_hierarchy_name
+        if not sample_axis_name:
+            return
+
+        for backend in self.output_backends.values():
+            if not getattr(backend, "is_source_backed", False):
+                continue
+
+            separator = self._augmentation_hierarchy_separator(backend)
+            set_separator = getattr(backend, "set_hierarchy_separator", None)
+            if callable(set_separator):
+                set_separator(separator)
+
+            layout = build_layout_addon(
+                family=self._layout_family(),
+                sample_axis_name=sample_axis_name,
+                sample_axis_location="hierarchy",
+                variant_axis_name=self.augmentation_config.attribute_key,
+                variant_axis_location="file_id",
+                derived_from={
+                    "source_modality": getattr(backend, "source_modality", "rgb"),
+                    "source_id_attribute": (
+                        f"{self.augmentation_config.attribute_key}.source_id"
+                    ),
+                    "source_full_id_attribute": (
+                        f"{self.augmentation_config.attribute_key}.source_full_id"
+                    ),
+                },
+            )
+            add_head_addon = getattr(backend, "add_head_addon", None)
+            if callable(add_head_addon):
+                add_head_addon(EULER_LAYOUT_ADDON, layout)
+
     def _file_id_hierarchy_key(self, sample_id: str, backend: Any) -> str:
         name = self.augmentation_config.file_id_hierarchy_name
-        separator = getattr(getattr(backend, "dataset_writer", None), "_separator", None)
+        separator = self._augmentation_hierarchy_separator(backend)
         if name and separator:
             return f"{name}{separator}{sample_id}"
         return sample_id
@@ -635,6 +712,7 @@ class FogTransform(Transform):
                 torch_gen,
                 self.torch_device,
             )
+            k_noise = prepare_noise_field_torch(k_noise, k_cfg, rng)
             min_factor = float(sample_value(k_cfg.get("min_factor", 1.0), rng))
             max_factor = float(sample_value(k_cfg.get("max_factor", 1.0), rng))
             k_field = modulate_with_noise_torch(
@@ -657,6 +735,7 @@ class FogTransform(Transform):
                 torch_gen,
                 self.torch_device,
             )
+            ls_noise = prepare_noise_field_torch(ls_noise, ls_cfg, rng)
             min_factor = float(sample_value(ls_cfg.get("min_factor", 1.0), rng))
             max_factor = float(sample_value(ls_cfg.get("max_factor", 1.0), rng))
             ls_field = modulate_with_noise_torch(

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/euler_preprocess.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-preprocess
-Version: 2.2.0
+Version: 2.3.0
 Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
@@ -232,10 +232,10 @@ Each image is assigned a fog model via the `selection` block:
 "selection": {
   "mode": "weighted",
   "weights": {
-    "uniform": 1.0,
-    "heterogeneous_k": 0.0,
-    "heterogeneous_ls": 0.0,
-    "heterogeneous_k_ls": 0.0
+    "uniform": 0.25,
+    "heterogeneous_k": 0.35,
+    "heterogeneous_ls": 0.25,
+    "heterogeneous_k_ls": 0.15
   }
 }
 ```
@@ -309,9 +309,12 @@ variants:
       "scattering_coefficient": 0.15,
       "atmospheric_light": [1.0, 1.0, 1.0],
       "k_hetero": {
-        "scales": "auto",
-        "min_factor": 0.5,
-        "max_factor": 1.5,
+        "scales": "smooth_auto",
+        "correlation_length_fraction": 0.25,
+        "octaves": 3,
+        "min_factor": 0.65,
+        "max_factor": 1.45,
+        "contrast": 0.65,
         "normalize_to_mean": true
       }
     }
@@ -327,26 +330,42 @@ MOR/beta descriptors when available. euler-loading exposes these as
 
 ### Heterogeneous Noise Fields
 
-Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian motion) to generate spatially-varying factor fields:
+Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian
+motion) to generate spatially-varying factor fields. For realistic fog,
+prefer the smooth mode: it keeps Perlin wavelengths tied to the image size,
+then optionally reduces noise contrast and applies a final blur before mapping
+the noise to physical factors.
 
 ```json
 "k_hetero": {
-  "scales": "auto",
-  "min_scale": 2,
+  "scales": "smooth_auto",
+  "correlation_length_fraction": 0.25,
+  "octaves": 3,
   "max_scale": null,
-  "min_factor": 0.0,
-  "max_factor": 1.0,
+  "min_factor": 0.65,
+  "max_factor": 1.45,
+  "contrast": 0.65,
+  "smooth_sigma_fraction": 0.0,
   "normalize_to_mean": true
 }
 ```
 
-The noise field (values in [0, 1]) is mapped to a factor field: `factor(x) = min_factor + (max_factor - min_factor) * noise(x)`. When `normalize_to_mean` is `true`, the factor field is rescaled so its spatial mean equals 1.0, preserving the overall fog density while introducing spatial variation.
+The noise field (values in [0, 1]) is mapped to a factor field:
+`factor(x) = min_factor + (max_factor - min_factor) * noise(x)`.
+`contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
+extreme local fog density. When `normalize_to_mean` is `true`, the factor field
+is rescaled so its spatial mean equals 1.0, preserving the overall fog density
+while introducing spatial variation.
 
 | Parameter | Effect |
 |---|---|
 | `min_factor` / `max_factor` | Range of the multiplicative factor. |
 | `normalize_to_mean` | Rescale factors so the image-wide mean equals the base value. Recommended for `k_hetero`. |
-| `scales` / `min_scale` / `max_scale` | Control spatial frequency content. |
+| `scales: "smooth_auto"` | Build low-frequency Perlin scales from the image size. |
+| `correlation_length_fraction` | Approximate smallest fog feature size as a fraction of the shorter image side. Larger values create smoother gradients. |
+| `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
+| `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
+| `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
 
 ### Fog Output
 

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "euler-preprocess"
-version = "2.2.0"
+version = "2.3.0"
 description = "Physics-based preprocessing (fog, etc.) for RGB+depth datasets"
 readme = "README.md"
 requires-python = ">=3.9"

{euler_preprocess-2.2.0 → euler_preprocess-2.3.0}/tests/test_fog_aux_outputs.py

@@ -314,15 +314,29 @@ def test_stepped_augmentations_write_file_id_layout_and_attributes(
         (pipeline_root / "foggy_rgb" / ".ds_crawler" / "output.json").read_text()
     )
     node = output_index["dataset"]["children"]["Scene01"]["children"]["Camera_0"]
-    file_id_node = node["children"]["00001"]
+    file_id_node = node["children"]["file_id:00001"]
     entries = {entry["id"]: entry for entry in file_id_node["files"]}
     assert set(entries) == {"mor_10m", "mor_20m"}
+    assert entries["mor_10m"]["path_properties"]["file_id"] == "00001"
+    assert entries["mor_10m"]["basename_properties"]["ext"] == "png"
     attrs = entries["mor_10m"]["attributes"]["fog_augmentation"]
     assert attrs["id"] == "mor_10m"
     assert attrs["source_id"] == "00001"
     assert attrs["meteorological_visibility_m"] == 10.0
     assert attrs["model"] == "uniform"
     np.testing.assert_allclose(attrs["atmospheric_light"], [0.4, 0.5, 0.6])
+    assert output_index["euler_layout"]["sample_axis"] == {
+        "name": "file_id",
+        "location": "hierarchy",
+    }
+    assert output_index["euler_layout"]["variant_axis"] == {
+        "name": "fog_augmentation",
+        "location": "file_id",
+    }
+    output_head = json.loads(
+        (pipeline_root / "foggy_rgb" / ".ds_crawler" / "dataset-head.json").read_text()
+    )
+    assert output_head["addons"]["euler_layout"] == output_index["euler_layout"]
 
 
 def test_only_scattering_target_writes_only_scattering(tmp_path: Path) -> None:
@@ -547,6 +561,60 @@ def test_apply_model_returns_spatial_fields_for_heterogeneous() -> None:
     assert float(k_map.std()) > 0.0
 
 
+def test_smooth_auto_scales_are_image_relative_low_frequency() -> None:
+    """smooth_auto should avoid the pixel-scale octaves that make fog speckly."""
+    from euler_preprocess.fog.models import resolve_scales
+
+    rng = np.random.default_rng(0)
+    cfg = {
+        "scales": "smooth_auto",
+        "correlation_length_fraction": 0.25,
+        "octaves": 4,
+        "max_scale_fraction": 1.0,
+    }
+
+    assert resolve_scales(cfg, height=100, width=200, rng=rng) == [25, 50, 100, 200]
+
+
+def test_smooth_noise_contrast_keeps_heterogeneous_beta_near_mean() -> None:
+    """Low noise contrast keeps spatial fog gradients subtle around the base beta."""
+    from euler_preprocess.fog.models import apply_model
+
+    rng = np.random.default_rng(123)
+    rgb = np.full((80, 120, 3), 0.5, dtype=np.float32)
+    depth = np.full((80, 120), 50.0, dtype=np.float32)
+    estimated = np.array([0.8, 0.8, 0.9], dtype=np.float32)
+    cfg = {
+        "visibility_m": {"dist": "constant", "value": 80.0},
+        "atmospheric_light": "from_sky",
+        "k_hetero": {
+            "scales": "smooth_auto",
+            "correlation_length_fraction": 0.25,
+            "octaves": 3,
+            "min_factor": 0.5,
+            "max_factor": 1.5,
+            "contrast": 0.2,
+            "normalize_to_mean": True,
+        },
+    }
+
+    _, k_mean, _, k_map, _ = apply_model(
+        rgb,
+        depth,
+        "heterogeneous_k",
+        cfg,
+        rng,
+        contrast_threshold_default=0.05,
+        estimated_airlight=estimated,
+    )
+
+    factors = k_map / k_mean
+    assert float(factors.std()) > 0.0
+    assert float(factors.min()) >= 0.75
+    assert float(factors.max()) <= 1.25
+    np.testing.assert_allclose(float(factors.mean()), 1.0, rtol=1e-6)
+
+
 def test_apply_model_accepts_direct_scattering_coefficient() -> None:
     """Stepped configs may specify beta directly instead of MOR/visibility."""
     from euler_preprocess.fog.models import apply_model