justdata 1.0.0__tar.gz → 1.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (137) hide show
  1. justdata-1.2.0/PKG-INFO +1311 -0
  2. justdata-1.2.0/README.md +1268 -0
  3. {justdata-1.0.0 → justdata-1.2.0}/pyproject.toml +9 -2
  4. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/__init__.py +1 -1
  5. justdata-1.2.0/src/justdata/acoustic/_dcase.py +23 -0
  6. justdata-1.2.0/src/justdata/acoustic/_manifests.py +31 -0
  7. justdata-1.2.0/src/justdata/acoustic/_random.py +54 -0
  8. justdata-1.2.0/src/justdata/acoustic/_signal.py +79 -0
  9. justdata-1.2.0/src/justdata/acoustic/augment/_common.py +70 -0
  10. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/batch.py +11 -84
  11. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/patchout.py +26 -7
  12. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/spectrogram.py +13 -93
  13. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/waveform.py +14 -139
  14. justdata-1.2.0/src/justdata/acoustic/batching.py +11 -0
  15. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ast.py +1 -12
  16. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ced.py +2 -9
  17. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/efficientat.py +2 -22
  18. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/passt.py +2 -9
  19. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/noise.py +2 -32
  20. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/registry.py +6 -44
  21. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/dcase2025.py +9 -45
  22. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/ast.py +1 -5
  23. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/log.py +3 -4
  24. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mel.py +1 -14
  25. justdata-1.2.0/src/justdata/acoustic/pipelines.py +572 -0
  26. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/presets.py +10 -2
  27. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/segment.py +1 -12
  28. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/sources.py +20 -63
  29. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/tasks.py +52 -24
  30. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/__init__.py +41 -0
  31. justdata-1.2.0/src/justdata/core/_records.py +66 -0
  32. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/adapters.py +4 -1
  33. justdata-1.2.0/src/justdata/core/cache.py +270 -0
  34. justdata-1.2.0/src/justdata/core/config_resolution.py +58 -0
  35. justdata-1.2.0/src/justdata/core/executed_config.py +77 -0
  36. justdata-1.2.0/src/justdata/core/execution.py +45 -0
  37. justdata-1.2.0/src/justdata/core/finalization.py +350 -0
  38. justdata-1.2.0/src/justdata/core/inventory.py +745 -0
  39. justdata-1.2.0/src/justdata/core/loader.py +1022 -0
  40. justdata-1.2.0/src/justdata/core/metadata.py +727 -0
  41. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/presets.py +33 -2
  42. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/registry.py +136 -6
  43. justdata-1.2.0/src/justdata/core/replay.py +387 -0
  44. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/sources.py +37 -0
  45. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/stats.py +2 -1
  46. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/auto.py +10 -39
  47. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/color.py +117 -2
  48. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/geometric.py +83 -3
  49. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/mixing.py +38 -4
  50. justdata-1.2.0/src/justdata/vision/corruptions/__init__.py +27 -0
  51. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/corruptions/blur.py +38 -1
  52. justdata-1.2.0/src/justdata/vision/corruptions/digital.py +52 -0
  53. justdata-1.2.0/src/justdata/vision/corruptions/noise.py +44 -0
  54. justdata-1.2.0/src/justdata/vision/corruptions/registry.py +471 -0
  55. justdata-1.2.0/src/justdata/vision/corruptions/standard.py +331 -0
  56. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/corruptions/weather.py +47 -1
  57. justdata-1.2.0/src/justdata/vision/encodings/__init__.py +21 -0
  58. justdata-1.2.0/src/justdata/vision/encodings/panoptic_targets.py +320 -0
  59. justdata-1.2.0/src/justdata/vision/encodings/semantic_targets.py +139 -0
  60. justdata-1.2.0/src/justdata/vision/geometry.py +579 -0
  61. justdata-1.2.0/src/justdata/vision/minic.py +137 -0
  62. justdata-1.2.0/src/justdata/vision/pipelines.py +905 -0
  63. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/presets.py +88 -66
  64. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/sources.py +233 -3
  65. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/stages.py +68 -1
  66. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/tasks/classification.py +44 -15
  67. justdata-1.2.0/src/justdata/vision/tasks/panoptic.py +106 -0
  68. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/tasks/segmentation.py +83 -1
  69. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/utils.py +24 -151
  70. justdata-1.0.0/PKG-INFO +0 -860
  71. justdata-1.0.0/README.md +0 -817
  72. justdata-1.0.0/src/justdata/acoustic/batching.py +0 -27
  73. justdata-1.0.0/src/justdata/acoustic/pipelines.py +0 -112
  74. justdata-1.0.0/src/justdata/core/finalization.py +0 -173
  75. justdata-1.0.0/src/justdata/core/loader.py +0 -384
  76. justdata-1.0.0/src/justdata/core/metadata.py +0 -250
  77. justdata-1.0.0/src/justdata/vision/corruptions/__init__.py +0 -16
  78. justdata-1.0.0/src/justdata/vision/corruptions/digital.py +0 -26
  79. justdata-1.0.0/src/justdata/vision/corruptions/noise.py +0 -20
  80. justdata-1.0.0/src/justdata/vision/corruptions/registry.py +0 -113
  81. justdata-1.0.0/src/justdata/vision/minic.py +0 -69
  82. justdata-1.0.0/src/justdata/vision/pipelines.py +0 -62
  83. justdata-1.0.0/src/justdata/vision/tasks/__init__.py +0 -1
  84. {justdata-1.0.0 → justdata-1.2.0}/LICENSE +0 -0
  85. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/__init__.py +0 -0
  86. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/adapters.py +0 -0
  87. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/__init__.py +0 -0
  88. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/augment/configs.py +0 -0
  89. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/channel.py +0 -0
  90. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/__init__.py +0 -0
  91. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/compat/panns.py +0 -0
  92. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/configs.py +0 -0
  93. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/__init__.py +0 -0
  94. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/codec.py +0 -0
  95. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/datasets.py +0 -0
  96. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/device.py +0 -0
  97. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/dynamics.py +0 -0
  98. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/filters.py +0 -0
  99. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/reverb.py +0 -0
  100. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/timefreq.py +0 -0
  101. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/datasets.py +0 -0
  102. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/decoding.py +0 -0
  103. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/eval_views.py +0 -0
  104. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/evals.py +0 -0
  105. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/filters.py +0 -0
  106. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/__init__.py +0 -0
  107. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/kaldi.py +0 -0
  108. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mfcc.py +0 -0
  109. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/pcen.py +0 -0
  110. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/stft.py +0 -0
  111. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/jax.py +0 -0
  112. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/labels.py +0 -0
  113. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/layouts.py +0 -0
  114. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/metadata.py +0 -0
  115. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/normalization.py +0 -0
  116. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/postprocessing.py +0 -0
  117. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/preprocessing.py +0 -0
  118. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/registry.py +0 -0
  119. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/resampling.py +0 -0
  120. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/schema.py +0 -0
  121. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/stages.py +0 -0
  122. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/acoustic/stats.py +0 -0
  123. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/audio/__init__.py +0 -0
  124. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/augmentations.py +0 -0
  125. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/filters.py +0 -0
  126. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/core/label_mixing.py +0 -0
  127. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/corruptions.py +0 -0
  128. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/py.typed +0 -0
  129. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/__init__.py +0 -0
  130. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/__init__.py +0 -0
  131. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/composed.py +0 -0
  132. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/augmentations/registry.py +0 -0
  133. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/datasets.py +0 -0
  134. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/encodings/heatmaps.py +0 -0
  135. {justdata-1.0.0/src/justdata/vision/encodings → justdata-1.2.0/src/justdata/vision/tasks}/__init__.py +0 -0
  136. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/tasks/depth.py +0 -0
  137. {justdata-1.0.0 → justdata-1.2.0}/src/justdata/vision/transforms.py +0 -0
@@ -0,0 +1,1311 @@
1
+ Metadata-Version: 2.4
2
+ Name: justdata
3
+ Version: 1.2.0
4
+ Summary: A TensorFlow-native data pipeline library with a modality-neutral core and first-class computer vision and acoustic recipes
5
+ Author: clementpoiret
6
+ Author-email: clementpoiret <clement@linux.com>
7
+ License-Expression: MIT
8
+ License-File: LICENSE
9
+ Requires-Dist: loguru>=0.7.3
10
+ Requires-Dist: tensorflow>=2.18.1
11
+ Requires-Dist: tensorflow-datasets>=4.9.9
12
+ Requires-Dist: datasets[audio]>=4.5.0 ; extra == 'acoustic'
13
+ Requires-Dist: soundfile>=0.12 ; extra == 'acoustic'
14
+ Requires-Dist: soxr>=0.5 ; extra == 'acoustic'
15
+ Requires-Dist: librosa>=0.11 ; extra == 'acoustic'
16
+ Requires-Dist: torch==2.10.0 ; extra == 'acoustic'
17
+ Requires-Dist: torchcodec==0.10.0 ; extra == 'acoustic'
18
+ Requires-Dist: datasets[audio]>=4.5.0 ; extra == 'dev'
19
+ Requires-Dist: datasets[vision]>=4.5.0 ; extra == 'dev'
20
+ Requires-Dist: filelock>=3.20 ; extra == 'dev'
21
+ Requires-Dist: librosa>=0.11 ; extra == 'dev'
22
+ Requires-Dist: matplotlib>=3.10.8 ; extra == 'dev'
23
+ Requires-Dist: pillow>=12.2.0 ; extra == 'dev'
24
+ Requires-Dist: pytest>=9.0.2 ; extra == 'dev'
25
+ Requires-Dist: ruff==0.15.20 ; extra == 'dev'
26
+ Requires-Dist: soundfile>=0.12 ; extra == 'dev'
27
+ Requires-Dist: soxr>=0.5 ; extra == 'dev'
28
+ Requires-Dist: torch==2.10.0 ; extra == 'dev'
29
+ Requires-Dist: torchcodec==0.10.0 ; extra == 'dev'
30
+ Requires-Dist: torchaudio==2.10.0 ; extra == 'golden'
31
+ Requires-Dist: torchlibrosa==0.1.0 ; extra == 'golden'
32
+ Requires-Dist: datasets[vision]>=4.5.0 ; extra == 'vision'
33
+ Requires-Dist: filelock>=3.20 ; extra == 'vision'
34
+ Requires-Dist: pillow>=12.2.0 ; extra == 'vision'
35
+ Requires-Dist: wilds>=2.0.0 ; extra == 'wilds'
36
+ Requires-Python: >=3.11, <3.14
37
+ Provides-Extra: acoustic
38
+ Provides-Extra: dev
39
+ Provides-Extra: golden
40
+ Provides-Extra: vision
41
+ Provides-Extra: wilds
42
+ Description-Content-Type: text/markdown
43
+
44
+ # justdata
45
+
46
+ A TensorFlow-native data pipeline library with a modality-neutral core and first-class computer vision and acoustic
47
+ recipes. `justdata.core` owns loading, adapter, preset, metadata, padding, and seeded execution machinery;
48
+ `justdata.vision` owns image schemas, transforms, augmentations, corruptions, tasks, and vision presets.
49
+ `justdata.acoustic` owns audio schemas, decoding, resampling, segmentation, frontends, augmentations, corruptions, DCASE
50
+ helpers, stats, metadata/JAX helpers, and acoustic presets.
51
+
52
+ ______________________________________________________________________
53
+
54
+ ## Table of Contents
55
+
56
+ 1. [Installation](#installation)
57
+ 1. [Architecture](#architecture)
58
+ 1. [Data Pipeline Stages](#data-pipeline-stages)
59
+ 1. [Supervised Learning — Training Pipelines](#supervised-learning--training-pipelines)
60
+ 1. [Automatic Augmentation Policies: RandAugment, TrivialAugment, and TrivialAugmentWide](#automatic-augmentation-policies-randaugment-trivialaugment-and-trivialaugmentwide)
61
+ 1. [Supervised Learning — Validation Pipelines](#supervised-learning--validation-pipelines)
62
+ 1. [The `timm` ImageNet Recipes (ResNet Strikes Back: A1/A2/A3)](#the-timm-imagenet-recipes-resnet-strikes-back-a1a2a3)
63
+ 1. [Self-Supervised Learning Pipeline (DINOv2)](#self-supervised-learning-pipeline-dinov2)
64
+ 1. [Registry System](#registry-system)
65
+ 1. [Presets and Smart Merging](#presets-and-smart-merging)
66
+ 1. [Acoustic Pipelines](#acoustic-pipelines)
67
+ 1. [Dataset Adapters](#dataset-adapters)
68
+ 1. [Mini-C Corruption Benchmark](#mini-c-corruption-benchmark)
69
+ 1. [Audio Corruption Benchmark](#audio-corruption-benchmark)
70
+ 1. [Usage](#usage)
71
+ 1. [Development](#development)
72
+
73
+ ______________________________________________________________________
74
+
75
+ ## Installation
76
+
77
+ ```bash
78
+ pip install justdata
79
+ ```
80
+
81
+ **Requirements:** Python 3.11–3.13, TensorFlow ≥ 2.18.1, TensorFlow Datasets ≥ 4.9.9. Install optional modality extras
82
+ with `justdata[vision]` for Hugging Face vision datasets, `justdata[wilds]` for WILDS image classification datasets, or
83
+ `justdata[acoustic]` for audio dataset/source dependencies.
84
+
85
+ ______________________________________________________________________
86
+
87
+ ## Architecture
88
+
89
+ `justdata` is structured around a fixed, four-stage pipeline abstraction. The currently supported vision tasks,
90
+ classification, semantic segmentation, and panoptic segmentation, each expose exactly four composable functions. Object
91
+ detection and depth estimation are future scope and are not registered capabilities.
92
+
93
+ ```
94
+ (preprocess_fn, augment_fn, late_augment_fn, postprocess_fn)
95
+ ```
96
+
97
+ These are assembled by `justdata.core.load_ds` into the following execution graph:
98
+
99
+ ```
100
+ fetch_ds -> adapter -> preprocess -> cache -> augment -> shuffle -> postprocess -> batch -> late_augment -> pad -> prefetch
101
+ ```
102
+
103
+ The strict ordering reflects the execution domain requirements articulated throughout this document: format
104
+ normalization occurs before the optional preprocessing cache; spatial and photometric distortions precede tensor
105
+ conversion and normalization; and batch-level operations (Mixup, CutMix, random erasing) occur after batching on the
106
+ GPU. An optional model-input cache can be inserted after deterministic postprocessing and before batching; for training
107
+ it must be explicitly opted in and is placed before shuffle so epoch order is not frozen.
108
+
109
+ ### TensorFlow execution controls
110
+
111
+ `load_ds` uses `tf.data.AUTOTUNE` for loader-owned maps by default. Advanced callers may set `map_parallel_calls`,
112
+ `private_threadpool_size`, and `max_intra_op_parallelism` to bound TensorFlow input-pipeline concurrency. Omitting these
113
+ arguments preserves the default TensorFlow behavior. The map limit also applies to metadata sidecar writing when
114
+ enabled.
115
+
116
+ ### Core Public API
117
+
118
+ | Function | Description |
119
+ | :-------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |
120
+ | `justdata.core.fetch_ds(dataset_names, splits_info, data_dir)` | Raw dataset loading through registered source loaders. Returns a `tf.data.Dataset` in canonical schema. |
121
+ | `justdata.core.load_ds(...)` | Full pipeline for training or evaluation. Handles caching, augmentation, shuffling, batching, and prefetching. |
122
+ | `justdata.core.admit_inventory(...)` | Strict offline validation of a resolved inventory into a disk snapshot with structured identity and failure reports. |
123
+ | `justdata.core.open_inventory(snapshot_dir)` | Verify and reopen a completed inventory snapshot without original sources. |
124
+ | `justdata.core.load_inventory(admitted, ...)` | Apply the shared pipeline to an admitted inventory, including raw epoch finalization. |
125
+ | `justdata.core.ExecutedConfig` | Immutable, canonical snapshot returned by loaders when `return_config=True`. |
126
+ | `justdata.vision.minic.create_minic_datasets(...)` | Constructs Mini-C corruption benchmark datasets from a shared preprocessed base dataset. |
127
+ | `justdata.acoustic.corruptions.create_audio_corruption_datasets(...)` | Constructs acoustic corruption benchmark datasets from a shared preprocessed base dataset. |
128
+ | `justdata.acoustic.dcase2025.make_source_dataset(...)` | Builds DCASE Task 1 source-domain datasets with split-safety checks. |
129
+
130
+ Import `justdata.vision` before resolving built-in vision datasets or pipelines. Hugging Face vision datasets are
131
+ referenced with the `hf:` prefix (e.g., `hf:cifar10`), and WILDS image classification datasets are referenced with the
132
+ `wilds:` prefix (e.g., `wilds:camelyon17`).
133
+
134
+ Use the [strict local inventory API](docs/inventory.md) when every requested source and record must be accounted for. It
135
+ verifies expected content digests, preserves record IDs and order, and reports declared filtering. Existing `fetch_ds`
136
+ and `load_ds` retain their legacy source-skipping behavior and do not provide the strict admission guarantee.
137
+
138
+ ______________________________________________________________________
139
+
140
+ ## Data Pipeline Stages
141
+
142
+ ### Stage 1: Preprocessing (before cache)
143
+
144
+ Format normalization is performed before the optional decoded-dataset cache. Operations include rank fixing (2D -> 3D),
145
+ CHW -> HWC transposition, grayscale -> RGB expansion, and RGBA -> RGB projection. The output shape is forced to
146
+ `[None, None, 3]`.
147
+
148
+ ### Stage 2: Augmentation (per-sample, training only)
149
+
150
+ Per-sample spatial and photometric augmentations are applied after the optional cache stage, ensuring that each training
151
+ epoch receives independently sampled augmentations. All augmentation functions use `tf.random.split` for stateless
152
+ randomization, enabling full reproducibility.
153
+
154
+ ### Stage 3: Postprocessing (every sample)
155
+
156
+ Resize, channel-wise normalization to zero mean and unit standard deviation, and optional HWC -> CHW transposition. This
157
+ stage runs unconditionally for both training and evaluation.
158
+
159
+ Set `cache_model_inputs=True` with `model_input_cache_path` to cache these resized, normalized tensors on local SSD. For
160
+ `dataset_type="train"`, also set `allow_train_model_input_cache=True` and use it only when the training view is
161
+ deterministic; stochastic crops or photometric augmentation will be materialized into the cache on first fill.
162
+
163
+ ### Stage 4: Late Augmentation (per-batch, training only)
164
+
165
+ Batch-level operations—Mixup, CutMix, and random erasing—are applied after batching. These operations are strictly
166
+ training-only and require a formed batch to operate across the sample dimension.
167
+
168
+ ______________________________________________________________________
169
+
170
+ ## Supervised Learning — Training Pipelines
171
+
172
+ To ensure mathematical correctness, the training pipeline is strictly segregated by execution domain. Spatial and
173
+ photometric distortions are applied to `[0, 255]` image tensors (PIL-equivalent domain) before zero-mean tensor
174
+ normalization.
175
+
176
+ ### 1. Image Domain Augmentations
177
+
178
+ #### Geometric Augmentations
179
+
180
+ - **CIFAR-10/100:** `RandomCrop(32, padding=4, padding_mode='zeros')` -> `RandomHorizontalFlip(p=0.5)`.
181
+ - **ImageNet-1K:** `RandomResizedCrop(size=224)` (or **256**). Scale: **(0.08, 1.0)**, Aspect ratio: **(0.75, 1.33)**,
182
+ Interpolation: Bicubic. -> `RandomHorizontalFlip(p=0.5)`.
183
+
184
+ #### Photometric Augmentations (Mutually Exclusive Branches)
185
+
186
+ The photometric augmentation strategy is architecture-dependent and the two branches are mutually exclusive.
187
+
188
+ - **Modern Branch (ViT / ConvNeXt):** Apply `RandAugment(num_ops=2, magnitude=9)`, `TrivialAugment`, or
189
+ `TrivialAugmentWide` (see the
190
+ [Automatic Augmentation Policies](#automatic-augmentation-policies-randaugment-trivialaugment-and-trivialaugmentwide)
191
+ section for full specifications). Color Jitter is explicitly disabled to prevent redundant and destructive color space
192
+ distortion.
193
+ - **Legacy Branch (ResNet):** Apply `ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1)`. RandAugment is
194
+ disabled.
195
+
196
+ ### 2. Tensor Domain
197
+
198
+ - **Conversion & Scaling:** `ToImage()` -> `ToDtype(float32, scale=True)`. Maps **[0, 255]** -> **[0.0, 1.0]**.
199
+ - **Normalization:**
200
+ - **CIFAR-10:** `mean=(0.4914, 0.4822, 0.4465)`, `std=(0.2023, 0.1994, 0.2010)`
201
+ - **CIFAR-100:** `mean=(0.5071, 0.4867, 0.4408)`, `std=(0.2675, 0.2565, 0.2761)`
202
+ - **ImageNet-1K:** `mean=(0.485, 0.456, 0.406)`, `std=(0.229, 0.224, 0.225)`
203
+ - **Random Erasing (Cutout):** Applied to the normalized tensor. `p=0.25`, `scale=(0.02, 0.33)`, `ratio=(0.3, 3.3)`.
204
+
205
+ ### 3. Mini-Batch Domain
206
+
207
+ Applied across the batch dimension during training.
208
+
209
+ - **Repeated Augmentation (RA):** Enabled for ViTs. Typically **3** repetitions per sample per mini-batch.
210
+ - **Mixup & CutMix:** Controlled by `mixup_prob=1.0` (probability of batch mixing) and `switch_prob=0.5` (probability of
211
+ selecting CutMix over Mixup).
212
+ - **Mixup:** $\tilde{x} = \lambda x_i + (1 - \lambda) x_j$, where $\lambda \sim \mathrm{Beta}(0.8,, 0.8)$.
213
+ - **CutMix:** Replaces a rectangular bounding box region; $\lambda \sim \mathrm{Beta}(1.0,, 1.0)$.
214
+ - *Note: The $\alpha$ values (0.8 and 1.0) reflect the DeiT baseline. Mixup/CutMix parameters are strictly
215
+ recipe-dependent; see the A1/A2/A3 section for ResNet-specific variations.*
216
+ - **Label Smoothing:** Cross-entropy loss modification with $\varepsilon = 0.1$.
217
+
218
+ ______________________________________________________________________
219
+
220
+ ## Automatic Augmentation Policies: RandAugment, TrivialAugment, and TrivialAugmentWide
221
+
222
+ `justdata` provides native TensorFlow implementations of three closely related automatic augmentation policies:
223
+ **RandAugment** (Cubuk et al., 2020), **TrivialAugment** (Müller & Hutter, 2021), and **TrivialAugmentWide** (Müller &
224
+ Hutter, 2021). All three are registered under the `augment_strategy` registry and share a common operation pool and
225
+ magnitude discretization framework.
226
+
227
+ ### 1. Pipeline Position and Domain Constraints
228
+
229
+ To guarantee deterministic reproduction of published results, all three policies are applied exclusively in the **image
230
+ domain** (integer `uint8` tensors, pixel values $\in [0, 255]$), before tensor conversion to floating-point and
231
+ channel-wise normalization. The canonical execution order within the per-sample augmentation stage is:
232
+
233
+ 1. Random Resized Crop (or Random Pad Crop for CIFAR)
234
+ 1. **RandAugment / TrivialAugment / TrivialAugmentWide**
235
+ 1. Random Horizontal Flip *(integrated into the crop strategy)*
236
+ 1. `ToTensor` — scales to float $[0.0, 1.0]$ *(postprocessing stage)*
237
+ 1. `Normalize` — subtracts dataset mean, divides by standard deviation *(postprocessing stage)*
238
+
239
+ Applying photometric and geometric distortions prior to floating-point conversion ensures that operations such as
240
+ `Posterize` and `Solarize`, which are defined on integer pixel arithmetic, remain numerically well-founded, and that
241
+ fill values for geometric operations are expressed in the same integer domain as the source image.
242
+
243
+ ### 2. The RA Operation Space ($K = 14$)
244
+
245
+ All three algorithms draw from the **RA augmentation space**, a fixed pool of $K = 14$ operations. The pool is
246
+ partitioned into two subsets based on magnitude dependency.
247
+
248
+ #### Magnitude-Independent Operations (3 ops)
249
+
250
+ These operations ignore the sampled magnitude $m$ entirely.
251
+
252
+ | Operation | Description |
253
+ | :------------- | :---------------------------------------------------------------------------------------------------------- |
254
+ | `Identity` | Returns the image unmodified. |
255
+ | `AutoContrast` | Linearly scales the pixel intensity histogram so that the darkest pixel maps to 0 and the brightest to 255. |
256
+ | `Equalize` | Equalizes the image histogram per channel using a cumulative distribution function. |
257
+
258
+ #### Magnitude-Dependent Operations (11 ops)
259
+
260
+ These operations scale their physical intensity as a function of the sampled magnitude index $m$.
261
+
262
+ **Sign randomization.** For all geometric operations (`Rotate`, `ShearX`, `ShearY`, `TranslateX`, `TranslateY`) and all
263
+ color enhancement operations (`Brightness`, `Color`, `Contrast`, `Sharpness`), the direction of the applied
264
+ transformation is randomized bidirectionally. Given a raw physical magnitude $v$ computed from $m$, the applied value
265
+ is:
266
+
267
+ $$v' = v \cdot s, \quad s \sim \mathcal{U}\{-1, +1\}$$
268
+
269
+ This sign randomization is sampled independently per operation per sample, using a stateless seed derived from the
270
+ layer's random state.
271
+
272
+ **Geometric fill.** When pixels are shifted outside the image boundary by `Rotate`, `ShearX`, `ShearY`, `TranslateX`, or
273
+ `TranslateY`, the vacated regions are filled with a constant value (default: 128). Geometric operations use the
274
+ implementation's fixed nearest-neighbor interpolation for images.
275
+
276
+ For segmentation pipelines, these geometric operations reuse the same sampled parameters for the image and segmentation
277
+ map. The image uses bilinear interpolation, while the map uses nearest-neighbor interpolation and preserves its integer
278
+ dtype. Vacated map pixels use the configurable `mask_fill_value` (default `255`), which must be treated as an ignore
279
+ label by the training loss. Photometric operations continue to affect only the image.
280
+
281
+ **Solarize semantics.** `Solarize` inverts all pixel values that are greater than or equal to the computed threshold
282
+ $\tau$. Formally, for each pixel $p$:
283
+
284
+ $$p' = \begin{cases} 255 - p & \text{if } p \geq \tau \\ p & \text{otherwise} \end{cases}$$
285
+
286
+ where $\tau = 255 \cdot (1 - m / B)$. At $m = 0$, the threshold is 255 (no pixels inverted); at $m = B$, the threshold
287
+ is 0 (all pixels inverted).
288
+
289
+ ### 3. Magnitude Discretization and Physical Mappings
290
+
291
+ The magnitude scale is a **31-bin discrete framework** with bin indices $m \in \{0, 1, \ldots, 30\}$ and $B = 30$ (the
292
+ maximum bin index). This parameterization is the native scale of the `torchvision` (v0.13+) implementation of both
293
+ RandAugment and TrivialAugmentWide.
294
+
295
+ The table below specifies the physical mapping formula and the operation-specific bound for each of the two spaces. $W$
296
+ and $H$ denote the input image width and height at the moment the augmentation is applied.
297
+
298
+ | Operation | Sign Rand. | Physical Mapping | Standard Bound (RA / TA) | Wide Bound (TA-Wide) |
299
+ | :--------------- | :--------: | :-------------------------------------------------------------- | :-------------------------------- | :-------------------------------- |
300
+ | **Rotate** | Yes | $(m / B) \cdot \text{MaxDeg} \cdot s$ | 30.0° | 135.0° |
301
+ | **TranslateX** | Yes | $(m / B) \cdot \text{MaxPx}_X \cdot s$ | $(150 / 331) \times W$ | 32.0 px |
302
+ | **TranslateY** | Yes | $(m / B) \cdot \text{MaxPx}_Y \cdot s$ | $(150 / 331) \times H$ | 32.0 px |
303
+ | **ShearX** | Yes | $(m / B) \cdot \text{MaxShear} \cdot s$ | 0.3 | 0.99 |
304
+ | **ShearY** | Yes | $(m / B) \cdot \text{MaxShear} \cdot s$ | 0.3 | 0.99 |
305
+ | **Brightness** | Yes | $1.0 + (m / B) \cdot \text{MaxDelta} \cdot s$ | 0.9 | 0.99 |
306
+ | **Color** | Yes | $1.0 + (m / B) \cdot \text{MaxDelta} \cdot s$ | 0.9 | 0.99 |
307
+ | **Contrast** | Yes | $1.0 + (m / B) \cdot \text{MaxDelta} \cdot s$ | 0.9 | 0.99 |
308
+ | **Sharpness** | Yes | $1.0 + (m / B) \cdot \text{MaxDelta} \cdot s$ | 0.9 | 0.99 |
309
+ | **Posterize** | No | $8 - \mathrm{round}\!\left((m / B) \cdot \text{MaxBits}\right)$ | MaxBits = 4 (min 4 bits retained) | MaxBits = 6 (min 2 bits retained) |
310
+ | **Solarize** | No | $255.0 \cdot (1 - m / B)$ | threshold down to 0 | threshold down to 0 |
311
+ | **Identity** | — | — | — | — |
312
+ | **AutoContrast** | — | — | — | — |
313
+ | **Equalize** | — | — | — | — |
314
+
315
+ > **Note on TranslateX / TranslateY:** Translation is a deliberate exception to the "wider bounds" pattern. The TA-Wide
316
+ > space specifies a fixed ceiling of 32 px for both axes. For any image dimension exceeding approximately 71 px, the
317
+ > Standard RA space—whose ceiling scales proportionally with the image dimension—admits a **larger** maximum translation
318
+ > than the Wide space.
319
+
320
+ ### 4. Algorithm Specifications
321
+
322
+ #### 4.1 RandAugment (RA)
323
+
324
+ RandAugment (Cubuk et al., 2020) reduces the search space of AutoAugment from $\mathcal{O}(K^N)$ policies to two scalar
325
+ hyperparameters: the number of sequential operations $N$ and the global magnitude $M$.
326
+
327
+ **Procedure.** Given a training image $x$:
328
+
329
+ 1. Independently sample $N$ operation indices $k_1, \ldots, k_N$ uniformly at random **with replacement** from the pool
330
+ of $K$ operations.
331
+ 1. For each selected operation $T_{k_i}$, map the fixed global magnitude $M$ to a physical parameter via the
332
+ operation-specific formula in the table above, applying sign randomization where applicable.
333
+ 1. Apply the operations sequentially: $x \leftarrow T_{k_N}(\cdots T_{k_1}(x))$.
334
+
335
+ The magnitude $M$ is expressed on the **31-bin scale** ($M \in \{0, \ldots, 30\}$) used throughout `justdata`. An
336
+ optional magnitude perturbation $\sigma > 0$ samples the effective level per layer from $\mathcal{N}(M, \sigma^2)$,
337
+ clipped to $[0, B]$.
338
+
339
+ **API parameters** (`augment_strategy = "rand_augment"` or direct call to `rand_augment`):
340
+
341
+ | Parameter | Type | Default | Description |
342
+ | :------------------- | :------------------ | :------ | :---------------------------------------------------------------------------------------------------- |
343
+ | `num_layers` | `int` | 2 | Number of operations $N$ applied per image. |
344
+ | `magnitude` | `float` | 9.0 | Global magnitude $M$ on the 31-bin scale. |
345
+ | `magnitude_std` | `float` | 0.0 | Per-layer Gaussian magnitude noise $\sigma$. Set to 0.5 to replicate `timm` stochastic magnitude. |
346
+ | `prob_to_apply` | `float \| None` | `None` | If set, each layer is skipped with probability $1 - p$. |
347
+ | `rotate_max` | `float` | 30.0 | Maximum rotation angle in degrees. |
348
+ | `shear_max` | `float` | 0.3 | Maximum shear coefficient (Standard RA space). |
349
+ | `enhance_max` | `float` | 0.9 | Maximum delta for brightness, color, contrast, sharpness. |
350
+ | `posterize_max_bits` | `int` | 4 | Bits removed at maximum magnitude; min retained = $8 - \text{MaxBits}$. |
351
+ | `translate_const` | `float` | 100.0 | Absolute translate ceiling in pixels. For Standard RA, set to $(150/331) \times \text{image\_width}$. |
352
+ | `exclude_ops` | `list[str] \| None` | `None` | Operations to exclude from the sampling pool. |
353
+
354
+ **Published optimal hyperparameters and 31-bin scale conversion.** The original paper (Cubuk et al., 2020) reports
355
+ hyperparameters on an **11-level scale** ($M_{\text{paper}} \in \{0, \ldots, 10\}$). To replicate those results using
356
+ the 31-bin scale employed by `justdata`, apply the conversion $m = M_{\text{paper}} \times 3$.
357
+
358
+ | Dataset | Architecture | $N$ | $M_{\text{paper}}$ | $m$ (31-bin) |
359
+ | :---------- | :--------------------- | :-: | :----------------: | :----------: |
360
+ | CIFAR-10 | WideResNet-28-2 | 3 | 4 | 12 |
361
+ | CIFAR-10 | WideResNet-28-10 | 3 | 5 | 15 |
362
+ | CIFAR-10 | PyramidNet + ShakeDrop | 3 | 7 | 21 |
363
+ | CIFAR-10 | Shake-Shake | 3 | 9 | 27 |
364
+ | ImageNet-1K | ResNet-50 | 2 | 9 | 27 |
365
+ | ImageNet-1K | EfficientNet-B7 | 2 | — | 28† |
366
+
367
+ > †The EfficientNet-B7 value was reported on an extended scale beyond 10; cross-verify against the target codebase
368
+ > before use. The `torchvision` default of `magnitude=9` on the 31-bin scale corresponds to moderate augmentation and is
369
+ > **not** equivalent to the paper's $M_{\text{paper}} = 9$.
370
+
371
+ #### 4.2 TrivialAugment (TA)
372
+
373
+ TrivialAugment (Müller & Hutter, 2021) eliminates hyperparameter search entirely by selecting a single operation and
374
+ sampling its magnitude uniformly at random from the full discrete range on each forward pass.
375
+
376
+ **Procedure.** Given a training image $x$:
377
+
378
+ 1. Sample one operation index $k \sim \mathcal{U}\{1, \ldots, K\}$ uniformly from the 14-op RA pool.
379
+ 1. Sample a magnitude $m \sim \mathcal{U}\{0, 1, \ldots, 30\}$ uniformly from the full 31-bin range.
380
+ 1. Map $m$ to a physical parameter via the **Standard** bounds in the table above, applying sign randomization where
381
+ applicable.
382
+ 1. Apply the single operation: $x \leftarrow T_k(x)$.
383
+
384
+ The proportional translate ceiling is computed dynamically as $(150 / 331) \times W$ (image width), matching the
385
+ Standard RA space definition.
386
+
387
+ **API:** `augment_strategy = "trivial_augment"`. Accepts `translate_const` (float, optional; computed from image width
388
+ if not supplied) and `exclude_ops` (list of strings, optional).
389
+
390
+ #### 4.3 TrivialAugmentWide (TA-Wide)
391
+
392
+ TrivialAugmentWide (Müller & Hutter, 2021) is the native `torchvision` variant of TrivialAugment. It uses the same
393
+ zero-hyperparameter, single-operation protocol as baseline TA, but operates over the **Wide** magnitude bounds,
394
+ substantially expanding the geometric and photometric search range.
395
+
396
+ **Procedure.** Identical to TrivialAugment, with two differences:
397
+
398
+ 1. The **Wide** physical bounds from the table above are applied in place of the Standard bounds.
399
+ 1. The translate ceiling is a fixed **32 px** (not image-proportional).
400
+
401
+ Wide bounds summary:
402
+
403
+ | Transformation | Standard RA / TA | TA-Wide |
404
+ | :--------------------------------------------------------- | :---------------------------- | :-------- |
405
+ | Rotation range | ±30° | **±135°** |
406
+ | Shear range (X and Y) | ±0.3 | **±0.99** |
407
+ | Enhancement delta (Brightness, Color, Contrast, Sharpness) | ±0.9 | **±0.99** |
408
+ | Posterize (minimum bits retained) | 4 | **2** |
409
+ | Translation (fixed ceiling) | $(150/331) \times \text{dim}$ | **32 px** |
410
+
411
+ **API:** `augment_strategy = "trivial_augment_wide"`. Accepts `exclude_ops` (list of strings, optional). The wide bounds
412
+ are fixed by design and cannot be overridden via kwargs; to use custom bounds, call `rand_augment` directly with
413
+ `num_layers=1` and the desired parameters.
414
+
415
+ ### 5. Algorithm Comparison
416
+
417
+ | Property | RandAugment | TrivialAugment | TrivialAugmentWide |
418
+ | :--------------------------------- | :------------------------------------ | :--------------------------------------------- | :--------------------------------------------- |
419
+ | **Operations applied per image** | $N$ (tunable, with replacement) | 1 (fixed) | 1 (fixed) |
420
+ | **Magnitude $m$** | Fixed global $M$, constant across ops | Sampled: $m \sim \mathcal{U}\{0, \ldots, 30\}$ | Sampled: $m \sim \mathcal{U}\{0, \ldots, 30\}$ |
421
+ | **Hyperparameter search required** | Grid search over $(N, M)$ | None | None |
422
+ | **Transformation bounds** | Standard RA space | Standard RA space | Wide space |
423
+ | **Translate ceiling** | Configurable (default 100 px) | $(150/331) \times W$ (image-proportional) | 32 px (fixed) |
424
+ | **Reference implementation** | `torchvision` | `automl/trivialaugment` | `torchvision` |
425
+ | **`justdata` registry key** | `rand_augment` | `trivial_augment` | `trivial_augment_wide` |
426
+
427
+ ### 6. Non-RA Operations
428
+
429
+ `Invert`, `Cutout`, `SolarizeAdd`, and `Grayscale` are outside the fixed 14-op pool and cannot be selected through
430
+ RandAugment, TrivialAugment, or TrivialAugmentWide. The public `exclude_ops` parameter only removes operations from the
431
+ pool; listing these names has no effect. The `cutout_const` argument remains accepted for compatibility and has no
432
+ effect. Random erasing is available through the `random_erasing` late-augmentation stage.
433
+
434
+ ______________________________________________________________________
435
+
436
+ ## Supervised Learning — Validation Pipelines
437
+
438
+ Validation pipelines are strictly deterministic. The objective shifts from regularization to feature preservation and
439
+ scale alignment.
440
+
441
+ ### CIFAR-10 / CIFAR-100 Standard Validation
442
+
443
+ Because CIFAR images are inherently **32×32** and contain minimal background, spatial cropping destroys the primary
444
+ subject.
445
+
446
+ **Pipeline:** `ToImage()` -> `ToDtype(float32, scale=True)` -> `Normalize` (training-set statistics as above).
447
+
448
+ ### ImageNet-1K Standard Baseline (The 0.875 Rule)
449
+
450
+ For standard supervised models and baseline linear probing, the canonical **0.875** crop ratio discards peripheral
451
+ background.
452
+
453
+ **Pipeline:** `Resize(256, interpolation=Bicubic)` -> `CenterCrop(224)` -> `ToImage()` -> `ToDtype(float32, scale=True)`
454
+ -> `Normalize`.
455
+
456
+ ### ImageNet-1K Modern Recipes (FixRes Strategy)
457
+
458
+ Modern recipes correct train-test resolution discrepancies by manipulating the validation crop percentage (`crop_pct`).
459
+
460
+ - **A3 (Light) Validation:** Train at **160×160**, validate at **224×224**. Resize shorter edge to **≈236**, then
461
+ `CenterCrop(224)`.
462
+ - **A1 / A2 (Heavy/Moderate) Validation:** Default test at **224×224** (`crop_pct=1.0`, so `Resize(224)` ->
463
+ `CenterCrop(224)`). Accuracy improves further via FixRes evaluation at **288×288** (`Resize(288)` ->
464
+ `CenterCrop(288)`) as demonstrated in the RSB paper.
465
+
466
+ ______________________________________________________________________
467
+
468
+ ## The `timm` ImageNet Recipes (ResNet Strikes Back: A1/A2/A3)
469
+
470
+ The "ResNet Strikes Back" (RSB) recipes dynamically scale augmentation intensity and training schedules to match model
471
+ capacity.
472
+
473
+ > **Crucial context:** These recipes were designed specifically for **ResNet-family architectures**. ViTs typically use
474
+ > distinct recipes (e.g., DeiT, BEiT) with different Mixup $\alpha$, optimizers (AdamW), and loss functions. While the
475
+ > principle of scaling augmentation with model capacity generalizes, the specific hyperparameters below do not transfer
476
+ > directly to ViTs.
477
+
478
+ | Parameter | A1 (Heavy) | A2 (Moderate) | A3 (Light) |
479
+ | :---------------------- | :--------------------------------------------------- | :--------------------- | :-------------------------------------------- |
480
+ | **Target Architecture** | Large ResNets (e.g., ResNet-152/200) or high compute | ResNet-50 (standard) | ResNet-50 (fast) or smaller (e.g., ResNet-18) |
481
+ | **Training Resolution** | 224 | 224 | **160** (FixRes strategy) |
482
+ | **Test Resolution** | 224 (scales to 288 via FixRes) | 224 | **224** |
483
+ | **Epochs** | **600** | **300** | **100** |
484
+ | **Optimizer** | LAMB | LAMB | LAMB |
485
+ | **LR Schedule** | Cosine with warmup | Cosine with warmup | Cosine with warmup |
486
+ | **Loss Function** | **BCE** (per-class binary) | **BCE** | **CE** (standard) |
487
+ | **RandAugment** | $m \approx 7$, $n = 2$ | $m \approx 6$, $n = 2$ | $m \approx 6$, $n = 2$ |
488
+ | **Random Erasing** | `p=0.35` | `p=0.25` | **Disabled** (`p=0.0`) |
489
+ | **Repeated Aug (RA)** | Enabled (3×) | Enabled (3×) | **Disabled** |
490
+ | **Mixup** $\alpha$ | 0.2 | 0.2 | 0.1 |
491
+ | **CutMix** $\alpha$ | 1.0 | 1.0 | 1.0 |
492
+ | **Stochastic Depth** | Capacity-dependent (e.g., 0.05+) | 0.0 | 0.0 |
493
+ | **EMA** | Yes | Yes | No (or lighter) |
494
+ | **Weight Decay** | 0.02 | 0.02 | 0.02 |
495
+
496
+ *Note: RandAugment parameter $n$ defaults to 2 per the original specification, though exact magnitude strings fluctuate
497
+ across `timm` versions.*
498
+
499
+ ______________________________________________________________________
500
+
501
+ ## Self-Supervised Learning Pipeline (DINOv2)
502
+
503
+ DINOv2 employs a Teacher-Student knowledge distillation framework operating on multi-crop asymmetry to force the
504
+ learning of semantic invariance over low-level frequency matching.
505
+
506
+ ### Multi-Crop Geometric Strategy
507
+
508
+ - **Global Crops (Context):** 2 crops at **224×224**. Scale: **(0.32, 1.0)**. Passed to both Teacher and Student.
509
+ - **Local Crops (Detail):** 8 crops at **96×96** (default). *Use **98×98** ($14 \times 7$) to avoid positional embedding
510
+ interpolation when using ViT-14 backbones.* Scale: **(0.05, 0.32)**. Passed to Student only.
511
+
512
+ ### Asymmetric Pipeline Implementation
513
+
514
+ The per-crop asymmetry is structurally enforced by three distinct `Compose` pipelines rather than conditional branching
515
+ within a single block. Each source image passes through all three pipelines to produce the full 10-crop suite.
516
+
517
+ - **`global_transform_1`**: Global Crop 1. Enforces strict blurring (`p=1.0`), disables solarization (`p=0.0`).
518
+ - **`global_transform_2`**: Global Crop 2. Minimizes blurring (`p=0.1`), enables solarization (`p=0.2`).
519
+ - **`local_transform`**: The 8 Local Crops. Moderate blurring (`p=0.5`), disables solarization (`p=0.0`).
520
+
521
+ ### Execution Order and Domain Separation (Per Crop)
522
+
523
+ | Step | Operation | Parameters / Per-Crop Asymmetry | Domain |
524
+ | :--- | :--------------------- | :--------------------------------------------------------------------------------------------------------------- | :----------------- |
525
+ | 1 | `RandomResizedCrop` | Aspect ratio **(0.75, 1.33)**, Bicubic interpolation. | Image |
526
+ | 2 | `RandomHorizontalFlip` | `p=0.5` (all crops) | Image |
527
+ | 3 | `ColorJitter` | `p=0.8` (all crops). `b=0.4, c=0.4, s=0.2, h=0.1` | Image |
528
+ | 4 | `RandomGrayscale` | `p=0.2` (all crops) | Image |
529
+ | 5 | `GaussianBlur` | $\sigma \sim \mathrm{Uniform}(0.1, 2.0)$, dynamic kernel. Global 1: `p=1.0` · Global 2: `p=0.1` · Local: `p=0.5` | Image |
530
+ | 6 | `Solarization` | Invert pixels $> 128$. Global 1: `p=0.0` · Global 2: `p=0.2` · Local: `p=0.0` | Image |
531
+ | 7 | Convert & Scale | `ToImage()` -> `ToDtype(float32, scale=True)` | Image -> Tensor |
532
+ | 8 | `Normalize` | ImageNet `mean` and `std`. | Tensor (zero-mean) |
533
+
534
+ ### SSL Downstream Evaluation
535
+
536
+ **Image-Level Classification (Linear Probing / $k$-NN):** Standard supervised inference transforms:
537
+ `Resize(256, Bicubic)` -> `CenterCrop(224)` -> `ToImage()` -> `ToDtype(float32)` -> `Normalize`. Features are extracted
538
+ from the `[CLS]` token or a concatenation of `[CLS]` and average-pooled patch tokens.
539
+
540
+ **Dense Tasks (Segmentation / Depth Validation — Patch Alignment):** To avoid dropping boundary pixels or forcing
541
+ complex interpolation during dense evaluation, images are resized to the target scale and padded (reflection or zero) on
542
+ the bottom and right edges such that both height and width are exact multiples of the ViT patch size (e.g., 14). This is
543
+ implemented in `justdata.vision.transforms.pad_to_patch_multiple`.
544
+
545
+ For semantic segmentation with replayable geometry, select `vision/segmentation` with `apply_presets=False` and explicit
546
+ `geometry_kwargs`. It supports paired aspect-preserving training resize and crops, longer-side evaluation caps,
547
+ independent RGB/ignore padding, per-pixel validity, and original-coordinate score restoration. See
548
+ [the dense geometry contract](docs/vision.md#12-replayable-dense-segmentation) for configuration, record fields, replay,
549
+ and batching requirements.
550
+
551
+ For instance-aware labels, select `vision/panoptic_segmentation` with a segment-ID map and category table. It preserves
552
+ thing instances, merges stuff by category, and excludes crowd pixels from panoptic supervision. The
553
+ [panoptic contract](docs/vision.md#13-panoptic-segmentation) and
554
+ [LaRS example](docs/vision.md#14-loading-local-lars-archives) show how to load either semantic or panoptic labels from
555
+ local LaRS v1.0.0 archives and plot five images under the A1/A2/A3 presets with both label views.
556
+
557
+ ______________________________________________________________________
558
+
559
+ ## Registry System
560
+
561
+ All extensible components in `justdata` use a decorator-based registry pattern with thread-safe lookups.
562
+
563
+ | Registry | Decorator | Lookup |
564
+ | :----------------- | :------------------------------------------------ | :------------------------------------------------------------- |
565
+ | Crop strategies | `@register_crop_strategy(name)` | `justdata.vision.augmentations.get_crop_strategy(name)` |
566
+ | Augment strategies | `@register_augment_strategy(name)` | `justdata.vision.augmentations.get_augment_strategy(name)` |
567
+ | Corruptions | `@register_corruption(name, descriptor=...)` | `justdata.vision.corruptions.apply_corruption(...)` |
568
+ | Dataset adapters | `@register_adapter(dataset_name)` | `justdata.core.get_adapter(dataset_name)` |
569
+ | Source loaders | `@register_source_loader(prefix)` | `justdata.core.get_source_loader(dataset_name)` |
570
+ | Pipelines | `@register_pipeline("modality/task")` | `justdata.core.get_pipeline(...)` |
571
+ | Dataset metadata | `register_dataset(name, task_type, modality=...)` | `justdata.core.get_dataset_info(name)` |
572
+ | Audio frontends | `@register_audio_frontend(name)` | `justdata.acoustic.get_audio_frontend(name)` |
573
+ | Audio components | `@register_audio_*` decorators | `justdata.acoustic.get_audio_*`, `list_audio_*`, `has_audio_*` |
574
+
575
+ `get_pipeline` is the high-level resolver: it infers the task type from the dataset name, merges preset defaults with
576
+ user-supplied kwargs (via smart merge; see below), and invokes the appropriate pipeline factory.
577
+
578
+ Custom pipelines may register `@register_pipeline(name, config_resolver=resolver)` when they support strict overrides
579
+ and executed-configuration export. The resolver receives a defensive copy of the merged configuration plus `is_training`
580
+ and returns the resolved four-stage configuration and model-input contract. Legacy factories without a resolver remain
581
+ buildable, but strict resolution and `return_config=True` reject them because their behavior cannot be described
582
+ completely.
583
+
584
+ **Built-in crop strategies:** `random_resized`, `random_resized_hvflip`, `random_pad`, `random_hflip`,
585
+ `resize_random_hflip`, `random_rot90_hflip`.
586
+
587
+ **Built-in augment strategies:** `rand_augment`, `trivial_augment`, `trivial_augment_wide`, `color_jitter`, `none`.
588
+
589
+ Crop-specific parameters are passed through `aug_kwargs["crop_kwargs"]`. `size`, `seed`, `interpolation`, `padding`, and
590
+ `pad_mode` remain controlled by the existing top-level augmentation fields and are rejected inside `crop_kwargs`,
591
+ avoiding ambiguous precedence. For example, the FMoW M1 crop is fully declarative:
592
+
593
+ ```python
594
+ pipeline = get_pipeline(
595
+ dataset="wilds:fmow",
596
+ aug_kwargs={
597
+ "enable": True,
598
+ "image_size": 224,
599
+ "crop_type": "random_resized_hvflip",
600
+ "interpolation": "bicubic",
601
+ "crop_kwargs": {
602
+ "scale": (0.85, 1.0),
603
+ "ratio": (0.90, 1.10),
604
+ "horizontal_flip_probability": 0.5,
605
+ "vertical_flip_probability": 0.5,
606
+ },
607
+ },
608
+ )
609
+ ```
610
+
611
+ The acoustic package registers audio decoders, resamplers, channel strategies, segment strategies, frontends,
612
+ augmentations, normalizations, eval views, postprocessors, and corruptions. Use `list_audio_*` helpers to inspect
613
+ registered acoustic components.
614
+
615
+ ______________________________________________________________________
616
+
617
+ ## Presets and Smart Merging
618
+
619
+ `justdata.vision.presets` stores dataset-specific default kwargs for all four vision pipeline stages. The vision
620
+ `_default` preset (ImageNet statistics, 224px, RandAugment) serves as the fallback for vision only.
621
+
622
+ **Available presets:**
623
+
624
+ | Preset | Description |
625
+ | :---------------- | :-------------------------------------------------------------- |
626
+ | `_default` | ImageNet-1K statistics, 224px, RandAugment, modern branch |
627
+ | `cifar` | 32px, TrivialAugmentWide, CIFAR-10 normalization, no resizing |
628
+ | `cifar100` | 32px, TrivialAugmentWide, CIFAR-100 normalization |
629
+ | `imagenet_resnet` | ImageNet statistics, legacy branch with ColorJitter |
630
+ | `imagenet_a1` | RSB A1: heavy augmentation, BCE loss, 600 epochs |
631
+ | `imagenet_a2` | RSB A2: moderate augmentation, BCE loss, 300 epochs |
632
+ | `imagenet_a3` | RSB A3: light augmentation, CE loss, 160px training |
633
+ | `dinov2` | Asymmetric multi-crop SSL pipeline |
634
+ | `wilds:*` | WILDS benchmark defaults; matching `_strong` presets are opt-in |
635
+
636
+ `merge_with_presets(dataset, user_kwargs)` implements a **smart merge**: user-supplied kwargs that are identical to the
637
+ `_default` preset values are treated as "not explicitly overridden," allowing dataset-specific preset values to take
638
+ precedence. Only kwargs that genuinely differ from the defaults are considered intentional user overrides.
639
+
640
+ For an authoritative recipe, pass `overrides=` to `get_pipeline`. Explicit values win even when they are `False`, zero,
641
+ `None` (where supported), or equal to the modality default. Nested mappings preserve siblings from the selected preset:
642
+
643
+ ```python
644
+ pipeline = get_pipeline(
645
+ dataset="cifar10",
646
+ overrides={
647
+ "aug_kwargs": {"enable": False, "image_size": 224},
648
+ "laug_kwargs": {"enable": False},
649
+ "postproc_kwargs": {
650
+ "image_size": 224,
651
+ "val_resize_size": None,
652
+ "normalization_params": (
653
+ (0.485, 0.456, 0.406),
654
+ (0.229, 0.224, 0.225),
655
+ ),
656
+ },
657
+ },
658
+ )
659
+ ```
660
+
661
+ Supplying `overrides={}` selects strict resolution without changing values. Built-in resolvers reject unknown nested
662
+ keys, missing required settings, contradictory configurations, and attempts to set runtime-owned fields before a loader
663
+ opens its source. Existing keyword arguments retain smart-merge compatibility semantics.
664
+
665
+ Resolved presets are serializable and hashable across modalities. Use `get_resolved_preset(name)` from
666
+ `justdata.vision.presets`, `justdata.acoustic.presets`, or `justdata.core.presets` to obtain an object with `.to_json()`
667
+ and `.hash()`. Hashes use canonical JSON with sorted keys and a 16-character SHA-256 prefix.
668
+
669
+ Acoustic presets are typed with `AudioPreset` and nested frozen config dataclasses for preprocessing, segmentation,
670
+ frontends, labels, normalization, train augment settings, eval views, layout, and metadata policy. The `justdata.audio`
671
+ namespace is a compatibility alias for `justdata.acoustic`.
672
+
673
+ See [docs/presets.md](docs/presets.md) for acoustic preset contracts, including EfficientAT/DyMN, PaSST, and CED.
674
+
675
+ ### Executed configuration snapshots
676
+
677
+ Pass `return_config=True` to `load_ds` or `load_inventory` to receive `(dataset, n_batches, config)`. The immutable
678
+ `ExecutedConfig` records the selected pipeline and preset, resolved four-stage configuration, derived model-input
679
+ contract, train/evaluation mode, randomness, shuffle, batching, metadata, caches, execution limits, prefetching, and
680
+ NumPy conversion. Its canonical versioned encoding is available as `config.to_bytes()`:
681
+
682
+ ```python
683
+ import hashlib
684
+ from pathlib import Path
685
+
686
+ dataset, n_batches, config = load_ds(
687
+ dataset_names_arg="cifar10",
688
+ splits_arg="test",
689
+ dataset_type="validation",
690
+ batch_size=128,
691
+ seed=42,
692
+ pipeline=pipeline,
693
+ num_classes=10,
694
+ return_config=True,
695
+ )
696
+ encoded = config.to_bytes()
697
+ Path("executed-justdata-config.json").write_bytes(encoded)
698
+ full_digest = hashlib.sha256(encoded).hexdigest()
699
+ ```
700
+
701
+ For a JD-01 offline inventory, use the same return option after verified admission:
702
+
703
+ ```python
704
+ import hashlib
705
+ from pathlib import Path
706
+
707
+ from justdata.core import load_inventory, open_inventory
708
+
709
+ admitted = open_inventory("/data/lars-admitted")
710
+ dataset, n_batches, config = load_inventory(
711
+ admitted,
712
+ dataset_type="validation",
713
+ batch_size=8,
714
+ seed=42,
715
+ pipeline=pipeline,
716
+ return_config=True,
717
+ )
718
+ encoded = config.to_bytes()
719
+ Path("lars-justdata-execution.json").write_bytes(encoded)
720
+ full_digest = hashlib.sha256(encoded).hexdigest()
721
+ ```
722
+
723
+ With `return_raw_ds=True`, the loader returns `(prepared, tools, config)` and marks the remaining stages as pending.
724
+ Both `tools["finalize_fn"]` and `tools["finalize_epoch"]` accept `return_config=True` and append a snapshot of their
725
+ actual finalization settings. Snapshot export rejects opaque callback replacements because their behavior cannot be
726
+ represented by configuration data.
727
+
728
+ The snapshot describes JustData's configured execution. A consuming product must separately bind admitted source
729
+ inventory identities, external transformations, exact library revisions, model/checkpoint identities, and other run
730
+ artifacts.
731
+
732
+ ### Numeric identity and metadata sidecars
733
+
734
+ With `metadata_mode="numeric_only"`, pass `metadata_sidecar=` for a complete source mapping prepared before iteration.
735
+ For an admitted inventory, use `MetadataSidecar.from_inventory(admitted)`; for another finite source, use
736
+ `MetadataSidecar.from_metadata(source_records)`. Each real batch row then has `metadata.row_id`, an `int64` join key,
737
+ and `metadata.row_fingerprint`, a numeric check against stale cache entries. Resolve keys through `sidecar.records` only
738
+ where `padding_mask` is true. The sidecar preserves original string IDs and static source metadata while numeric
739
+ geometry remains with each view.
740
+
741
+ For a view with additional string metadata, use the numeric `metadata.view_id` with `row_id` to read
742
+ `sidecar.view_records[(row_id, view_id)]`.
743
+
744
+ `sidecar.write_jsonl(path, policy="create")` saves a complete mapping and `MetadataSidecar.read_jsonl(path)` reopens it.
745
+ The streaming `sidecar_metadata_path=` route resumes and validates an existing file by default;
746
+ `sidecar_metadata_policy="create"` rejects reuse and `"overwrite"` explicitly replaces it. Streaming artifacts can be
747
+ partial after interrupted iteration. Executed configuration records the sidecar policy or full mapping digest.
748
+
749
+ ### Automatic preset resolution
750
+
751
+ When a vision dataset has a registered preset (e.g., `cifar10` -> `cifar`, `cifar100` -> `cifar100`), `get_pipeline`
752
+ automatically applies it. Vision datasets without a dedicated preset fall back to the vision `_default`.
753
+
754
+ ```python
755
+ import justdata.vision
756
+ from justdata.core.registry import get_pipeline
757
+
758
+ # cifar10 automatically gets the 'cifar' preset (32px, TrivialAugmentWide, CIFAR-10 stats)
759
+ pipeline = get_pipeline(dataset="cifar10")
760
+ ```
761
+
762
+ ### Applying a named preset to any dataset
763
+
764
+ The `dataset` argument in `get_pipeline` drives preset lookup, not just task inference. To apply a specific named preset
765
+ to a dataset that does not have its own preset (e.g., using `imagenet_a3` for `imagenette`), pass it as `preset`:
766
+
767
+ ```python
768
+ import justdata.vision
769
+ from justdata.core.loader import load_ds
770
+ from justdata.core.registry import get_pipeline
771
+
772
+ # Resolve the A3 (RSB light) pipeline for imagenette
773
+ pipeline = get_pipeline(
774
+ dataset="imagenette",
775
+ preset="imagenet_a3", # drives preset lookup: 160px train, 224px val, RandAugment m=6
776
+ )
777
+
778
+ train_ds, N = load_ds(
779
+ dataset_names_arg=["imagenette"], # actual dataset to load
780
+ splits_arg={"imagenette": ["train"]},
781
+ dataset_type="train",
782
+ batch_size=128,
783
+ seed=42,
784
+ pipeline=pipeline,
785
+ num_classes=10,
786
+ cache_dataset=False,
787
+ )
788
+ ```
789
+
790
+ The validation pipeline uses the same preset name; the A3 preset's FixRes strategy (`train_image_size=160`,
791
+ `val_resize_size=236`) is applied automatically:
792
+
793
+ ```python
794
+ # The same pipeline can be used for validation; `load_ds` automatically sets `is_training=False` based on `dataset_type`
795
+ # and applies the deterministic center-crop resize (FixRes 236 → CenterCrop 224)
796
+ val_ds, N = load_ds(
797
+ dataset_names_arg=["imagenette"],
798
+ splits_arg={"imagenette": ["validation"]},
799
+ dataset_type="validation",
800
+ batch_size=256,
801
+ seed=0,
802
+ pipeline=pipeline,
803
+ num_classes=10,
804
+ cache_dataset=False,
805
+ )
806
+ ```
807
+
808
+ ______________________________________________________________________
809
+
810
+ ## Acoustic Pipelines
811
+
812
+ Import `justdata.acoustic` before resolving built-in acoustic datasets or pipelines. Acoustic samples use the canonical
813
+ keys `waveform`, `sample_rate`, optional `label`, `features`, `duration`, and `metadata`.
814
+
815
+ ```python
816
+ import justdata.acoustic
817
+ from justdata.core.loader import load_ds
818
+ from justdata.core.registry import get_pipeline
819
+
820
+ pipeline = get_pipeline(
821
+ dataset="dcase2025_task1",
822
+ preset="dcase2025_task1_efficientat_32k_1s",
823
+ )
824
+
825
+ ds, n = load_ds(
826
+ dataset_names_arg=["dcase2025:task1"],
827
+ splits_arg={"dcase2025:task1": ["dev_train_25"]},
828
+ dataset_type="train",
829
+ batch_size=64,
830
+ seed=0,
831
+ pipeline=pipeline,
832
+ num_classes=10,
833
+ cache_dataset=False,
834
+ data_dir="/path/to/dcase",
835
+ metadata_mode="numeric_only",
836
+ as_numpy=True,
837
+ )
838
+ ```
839
+
840
+ Modality documentation:
841
+
842
+ - [docs/vision.md](docs/vision.md): vision schema, four stages, preset contracts, metadata modes, Mini-C, and parity
843
+ matrix.
844
+ - [docs/acoustic.md](docs/acoustic.md): canonical schema, four stages, metadata modes, golden tests, corruption
845
+ benchmark, and parity matrix.
846
+ - [docs/dcase2025.md](docs/dcase2025.md): DCASE Task 1 source/target helpers and split safety.
847
+ - [docs/presets.md](docs/presets.md): vision and acoustic preset selection and hashable contracts.
848
+ - [docs/golden_tests.md](docs/golden_tests.md): optional golden compatibility test workflow.
849
+
850
+ Examples are split by modality under `examples/vision/` and `examples/acoustic/`. Each directory includes loading,
851
+ Hugging Face source, statistics, and corruption benchmark examples.
852
+
853
+ ______________________________________________________________________
854
+
855
+ ## Dataset Adapters
856
+
857
+ Core adapters normalize raw records into the modality-specific schema expected by a selected pipeline. `justdata.core`
858
+ itself is schema-neutral, and the identity adapter is applied when no dataset-specific adapter is registered.
859
+
860
+ ### Vision adapter
861
+
862
+ Vision pipelines expect `image`, optional `label`, and task-specific fields such as `mask`.
863
+
864
+ ```python
865
+ from justdata.core.adapters import register_adapter
866
+
867
+ @register_adapter("my_dataset")
868
+ def my_adapter(sample):
869
+ return {"image": sample["img"], "label": sample["class_id"]}
870
+ ```
871
+
872
+ ### Acoustic adapter
873
+
874
+ Acoustic pipelines expect `waveform`, `sample_rate`, and an optional `label`.
875
+
876
+ ```python
877
+ from justdata.core.adapters import register_adapter
878
+
879
+ @register_adapter("my_audio_dataset")
880
+ def my_audio_adapter(sample):
881
+ result = {
882
+ "waveform": sample["audio"],
883
+ "sample_rate": sample["sampling_rate"],
884
+ }
885
+ if "target" in sample:
886
+ result["label"] = sample["target"]
887
+ return result
888
+ ```
889
+
890
+ ______________________________________________________________________
891
+
892
+ ## Mini-C Corruption Benchmark
893
+
894
+ `create_minic_datasets` constructs finalized corruption evaluation datasets by forking from a preprocessed RGB dataset.
895
+ Corruption runs after decoding and rank/channel normalization, but before resize, float conversion, normalization,
896
+ batching, and padding. Pass `cache_dataset=True` with an explicit disk path when avoiding source re-reads across
897
+ corruptions is worth the storage cost; otherwise the base remains streaming.
898
+
899
+ The original Mini-C family names remain pinned to their `1.0.0` behavior:
900
+
901
+ | Name | Version 1 implementation |
902
+ | :-------- | :-------------------------------------------------------------------- |
903
+ | `blur` | Defocus blur with a disk kernel and alias Gaussian filter |
904
+ | `noise` | Additive Gaussian noise using the legacy absolute standard deviations |
905
+ | `digital` | Nearest-neighbor pixelation |
906
+ | `weather` | Synthetic snow |
907
+
908
+ Five exact-name TensorFlow-native operators are also available at version `1.0.0`: `gaussian_blur`, `gaussian_noise`,
909
+ `jpeg_compression`, `contrast_reduction`, and `brightness_reduction`. Their immutable descriptors contain the severity
910
+ table, domains, dtype/shape contract, seed policy, clipping and rounding policy, backend identity, and codec/filter
911
+ settings. Use `get_corruption_descriptor(name, version)` or `list_corruption_descriptors()` to serialize and fingerprint
912
+ these semantics.
913
+
914
+ For a caller-owned scientific seed, use exact version dispatch directly:
915
+
916
+ ```python
917
+ import tensorflow as tf
918
+
919
+ from justdata.vision.corruptions import (
920
+ apply_corruption,
921
+ get_corruption_descriptor,
922
+ )
923
+
924
+ descriptor = get_corruption_descriptor("gaussian_noise", "1.0.0")
925
+ corrupted = apply_corruption(
926
+ decoded_rgb_uint8,
927
+ name=descriptor.name,
928
+ version=descriptor.version,
929
+ severity=3,
930
+ seed=tf.constant([123, 456], dtype=tf.int32),
931
+ )
932
+ ```
933
+
934
+ The complete seed is supplied by the caller; no global RNG is read. `gaussian_noise@1.0.0` uses TensorFlow Philox. The
935
+ JPEG descriptor fixes encode and decode options and is repeatable for a fixed TensorFlow/codec build, but does not claim
936
+ byte identity across unrelated TensorFlow or codec versions.
937
+
938
+ ```python
939
+ import justdata.vision
940
+ from justdata.core.registry import get_pipeline
941
+ from justdata.vision.minic import create_minic_datasets
942
+
943
+ pipeline = get_pipeline(dataset="imagenet")
944
+
945
+ datasets, n = create_minic_datasets(
946
+ corruption_types=["gaussian_noise", "jpeg_compression", "blur"],
947
+ severity=3,
948
+ dataset_names_arg=["imagenet2012"],
949
+ splits_arg={"imagenet2012": ["validation"]},
950
+ dataset_type="validation",
951
+ batch_size=256,
952
+ seed=0,
953
+ pipeline=pipeline,
954
+ num_classes=1000,
955
+ )
956
+ ```
957
+
958
+ With `metadata_mode="full"`, finalized samples include `corruption`, `corruption_version`, `corruption_identity`,
959
+ `corruption_identity_hash`, `severity`, and `corruption_domain`. `numeric_only` retains the numeric identity hash and
960
+ severity; `none` removes metadata according to the shared core contract. The dataset helper preserves legacy
961
+ enumeration-position seed salting. Consumers with their own sample-level PRNG lineage should call `apply_corruption`
962
+ directly.
963
+
964
+ ______________________________________________________________________
965
+
966
+ ## Audio Corruption Benchmark
967
+
968
+ `create_audio_corruption_datasets` mirrors Mini-C for acoustic evaluation. It applies deterministic severity 1-5
969
+ corruptions in the waveform or spectrogram domain, attaches corruption metadata, runs the normal postprocessing stage,
970
+ and preserves `padding_mask`.
971
+
972
+ ```python
973
+ import justdata.acoustic
974
+ from justdata.acoustic.corruptions.datasets import create_audio_corruption_datasets
975
+ from justdata.core.registry import get_pipeline
976
+
977
+ pipeline = get_pipeline(
978
+ dataset="dcase2025_task1",
979
+ preset="dcase2025_task1_efficientat_32k_1s",
980
+ )
981
+
982
+ datasets, n = create_audio_corruption_datasets(
983
+ corruption_types=["additive_white_noise", "clipping"],
984
+ severity=3,
985
+ base_dataset="dcase2025:task1",
986
+ preset="dcase2025_task1_efficientat_32k_1s",
987
+ split="dev_test",
988
+ pipeline=pipeline,
989
+ batch_size=64,
990
+ seed=0,
991
+ num_classes=10,
992
+ metadata_mode="numeric_only",
993
+ )
994
+ ```
995
+
996
+ ______________________________________________________________________
997
+
998
+ ## Usage
999
+
1000
+ ### Basic Classification Pipeline
1001
+
1002
+ ```python
1003
+ import justdata.vision
1004
+ from justdata.core.loader import load_ds
1005
+ from justdata.core.registry import get_pipeline
1006
+
1007
+ pipeline = get_pipeline(dataset="cifar10")
1008
+
1009
+ train_ds, N = load_ds(
1010
+ dataset_names_arg=["cifar10"],
1011
+ splits_arg={"cifar10": ["train"]},
1012
+ dataset_type="train",
1013
+ batch_size=128,
1014
+ seed=42,
1015
+ pipeline=pipeline,
1016
+ num_classes=10,
1017
+ # CIFAR-10 is small enough for an intentional memory cache.
1018
+ cache_dataset=True,
1019
+ )
1020
+ ```
1021
+
1022
+ ### Reusable, Explicitly Seeded Epochs
1023
+
1024
+ Use `return_raw_ds=True` to prepare the source, pipeline, preprocessing, cache, and filters once, then materialize the
1025
+ complete remaining training pipeline for each epoch:
1026
+
1027
+ ```python
1028
+ prepared_train, tools = load_ds(
1029
+ dataset_names_arg=["cifar10"],
1030
+ splits_arg={"cifar10": ["train"]},
1031
+ dataset_type="train",
1032
+ batch_size=128,
1033
+ seed=0,
1034
+ pipeline=pipeline,
1035
+ num_classes=10,
1036
+ cache_dataset=False,
1037
+ deterministic=True,
1038
+ return_raw_ds=True,
1039
+ )
1040
+
1041
+ for epoch in range(num_epochs):
1042
+ train_iterator, n_batches = tools["finalize_epoch"](
1043
+ prepared_train,
1044
+ seed=derive_epoch_seed(epoch),
1045
+ as_numpy=True,
1046
+ )
1047
+ train_one_epoch(train_iterator)
1048
+ ```
1049
+
1050
+ `finalize_epoch` applies standard augmentation, shuffle, postprocessing, batching, late augmentation, padding, and
1051
+ prefetching. The epoch seed is split into standard- and late-augmentation seeds, with source and batch indices folded
1052
+ in. Shuffle uses the epoch seed with reshuffling disabled on the materialized dataset. With `deterministic=True`,
1053
+ repeating a seed therefore reproduces the epoch independently of previous iterator creation or consumption, including
1054
+ when the returned TensorFlow dataset is iterated more than once.
1055
+
1056
+ For an augmentation callback that depends only on its input sample and supplied seed, pass `augment_is_stateless=True`
1057
+ to `finalize_epoch` to run standard augmentation concurrently using `map_parallel_calls` (or `AUTOTUNE`). With
1058
+ `deterministic=True`, sample order and indexed RNG seeds remain unchanged:
1059
+
1060
+ ```python
1061
+ train_iterator, n_batches = tools["finalize_epoch"](
1062
+ prepared_train,
1063
+ seed=derive_epoch_seed(epoch),
1064
+ augment_is_stateless=True,
1065
+ as_numpy=True,
1066
+ )
1067
+ ```
1068
+
1069
+ This flag is the caller's assertion that the augmentation callback uses no stateful RNG, mutable state, or side effects.
1070
+ It defaults to `False`, preserving serial standard augmentation for deterministic epochs. It does not change the
1071
+ one-shot `load_ds` or `finalize_fn` seed scheduling.
1072
+
1073
+ With `cache_dataset=False`, source samples, preprocessing, and filters still run on every iteration; only loader and
1074
+ graph preparation are reused. A fresh NumPy iterator is created by each `finalize_epoch(..., as_numpy=True)` call.
1075
+
1076
+ `finalize_epoch` rejects `cache_model_inputs=True` when training or evaluation augmentation is enabled because that
1077
+ downstream cache intentionally freezes the first sampled augmented views. Existing one-shot `load_ds` and `finalize_fn`
1078
+ behavior is unchanged.
1079
+
1080
+ For a verified local inventory, `load_replay_epoch` binds an immutable snapshot and executed configuration to an
1081
+ explicit committed-batch cursor. It rebuilds the full epoch before skipping committed batches. See
1082
+ [`docs/inventory.md`](docs/inventory.md#deterministic-epoch-replay) for checkpoint and resume usage. Count real rows
1083
+ with `count_real_examples(batch)` or the returned epoch's `remaining_examples`; padded rows have a false `padding_mask`.
1084
+ The epoch path also accepts `prefetch=False` when finalization must precede a caller-controlled prefetch stage.
1085
+
1086
+ ### Bounded input execution and protected caches
1087
+
1088
+ Pass `prefetch=1` (or another positive batch count) to `load_ds`, `load_inventory`, or `finalize_fn` to bound final
1089
+ prefetch. `finalize_epoch` inherits the configured value unless overridden. `False` disables prefetch; `True` retains
1090
+ automatic tuning. A bounded input profile also sets positive integer `map_parallel_calls`, `private_threadpool_size`,
1091
+ and `max_intra_op_parallelism`, plus a finite shuffle buffer. The executed configuration records the effective values.
1092
+ Replay prefetches after skipping committed batches and defaults to one batch.
1093
+
1094
+ When another framework uses the accelerator, configure TensorFlow before creating tensors or datasets:
1095
+
1096
+ ```python
1097
+ from justdata.core import configure_tensorflow_cpu
1098
+
1099
+ device_report = configure_tensorflow_cpu(intra_op_threads=1, inter_op_threads=1)
1100
+ assert device_report["logical_gpus"] == []
1101
+ ```
1102
+
1103
+ The report includes physical and logical device names and process thread settings. The helper raises if the runtime has
1104
+ initialized too early for the requested change. Core, vision, acoustic, and audio imports permit this setup.
1105
+
1106
+ `CachePolicy` enables identity-checked file caches on the existing preprocessing and model-input cache stages. The cache
1107
+ path is a directory whose parent exists. The input identity must cover ordered content. `load_inventory` binds its
1108
+ verified manifest automatically; other sources require a content-derived `input_identity`. Callers must declare
1109
+ deterministic callbacks because arbitrary callbacks cannot be inspected for stateful random operations.
1110
+
1111
+ ```python
1112
+ from justdata.core import CachePolicy, inspect_cache, load_inventory
1113
+
1114
+ policy = CachePolicy(
1115
+ max_bytes=1_073_741_824,
1116
+ max_examples=256,
1117
+ materialization="lazy", # choose "eager" to finish before returning
1118
+ callbacks_are_deterministic=True,
1119
+ )
1120
+ batches, count, config = load_inventory(
1121
+ admitted, "validation", 2, 17,
1122
+ pipeline=pipeline,
1123
+ cache_dataset=True,
1124
+ cache_path="/existing/cache-parent/prepared",
1125
+ cache_policy=policy,
1126
+ return_config=True,
1127
+ map_parallel_calls=2,
1128
+ private_threadpool_size=2,
1129
+ max_intra_op_parallelism=1,
1130
+ prefetch=1,
1131
+ )
1132
+ for batch in batches:
1133
+ consume(batch)
1134
+ assert policy.status("preprocess")["state"] == "complete"
1135
+ assert inspect_cache("/existing/cache-parent/prepared")["state"] == "complete"
1136
+ ```
1137
+
1138
+ Lazy caches publish completion only after the source is fully exhausted. Eager caches finish before loading returns.
1139
+ Both modes record count, byte size, checksum, and a versioned stage/configuration identity. Changed input,
1140
+ normalization, geometry, or augmentation configuration rejects reuse at the same path. An incomplete or incompatible
1141
+ cache raises `CacheError`; choose a new path after inspecting or clearing the caller-owned artifact. Cache errors expose
1142
+ `code` and `path`. During lazy iteration TensorFlow may wrap the exception; `policy.status(stage)` retains its
1143
+ structured failure code. Quotas bound record count and serialized record bytes. Protected model-input caches require
1144
+ evaluation without active augmentation; preprocessing caches remain before per-epoch augmentation. Existing cache flags
1145
+ without a policy retain their behavior.
1146
+
1147
+ ### Loading a Hugging Face Dataset
1148
+
1149
+ ```python
1150
+ train_ds, n_batches = load_ds(
1151
+ dataset_names_arg=["hf:cifar10"],
1152
+ splits_arg={"hf:cifar10": ["train"]},
1153
+ cache_dataset=False,
1154
+ ...
1155
+ )
1156
+ ```
1157
+
1158
+ `load_ds` returns the dataset and its batch cardinality. The cardinality is a Python integer when TensorFlow can
1159
+ determine it and `None` for unknown or infinite pipelines.
1160
+
1161
+ For remote/downloaded sources, `data_dir` is a cache root. justdata namespaces source-owned caches under it: `tfds/`,
1162
+ `hf/vision/`, `hf/acoustic/`, `wilds/`, and `zenodo/`. If omitted, the root is `~/.cache/justdata`.
1163
+
1164
+ Loader caches are separate from source-owned caches. `cache_dataset/cache_path` caches decoded, preprocessed samples
1165
+ before augmentation. Loader caching is off by default, so datasets remain streaming when `cache_dataset` is omitted. Set
1166
+ `cache_dataset=True` with an empty `cache_path` to opt into memory caching, or provide a nonempty path to cache on the
1167
+ filesystem. Large datasets should use an explicit disk path or remain uncached. `cache_model_inputs` and
1168
+ `model_input_cache_path` cache resized, normalized model inputs before batching. Use different paths for the two cache
1169
+ stages. This default changes performance, not output values.
1170
+
1171
+ ### Loading a WILDS Dataset
1172
+
1173
+ WILDS image classification datasets use the `wilds:` source prefix. Source options such as FMoW temporal split schemes
1174
+ live in the dataset string; when using a `splits_arg` dictionary, reuse that exact string as the key. Downloads are
1175
+ disabled by default (`download=false`). To download through WILDS under `data_dir/wilds`, add `download=true` to the
1176
+ dataset string. Use `&` between query options, not a second `?`. The automatic presets are benchmark-faithful:
1177
+ Camelyon17 96 px, FMoW 224 px, iWildCam 448 px, and RxRx1 256 px with per-image channel standardization. Use
1178
+ `preset="wilds:fmow_strong"` or another `_strong` name to opt into stronger training augmentation.
1179
+
1180
+ ```python
1181
+ local_ssd = "/local_ssd/justdata"
1182
+ wilds_ds = "wilds:fmow?split_scheme=time_after_2016&download=true"
1183
+ pipeline = get_pipeline(dataset="wilds:fmow")
1184
+
1185
+ train_ds, N = load_ds(
1186
+ dataset_names_arg=[wilds_ds],
1187
+ splits_arg={wilds_ds: ["train"]},
1188
+ dataset_type="train",
1189
+ batch_size=32,
1190
+ seed=42,
1191
+ pipeline=pipeline,
1192
+ num_classes=62,
1193
+ data_dir=f"{local_ssd}/sources",
1194
+ cache_dataset=True,
1195
+ cache_path=f"{local_ssd}/decoded/fmow-train",
1196
+ cache_model_inputs=True,
1197
+ model_input_cache_path=f"{local_ssd}/model-inputs/fmow-train-224",
1198
+ allow_train_model_input_cache=True,
1199
+ metadata_mode="numeric_only",
1200
+ )
1201
+ ```
1202
+
1203
+ FMoW source inventories can opt into authoritative sequence identity and acquisition time without changing the default
1204
+ WILDS signature:
1205
+
1206
+ ```python
1207
+ from justdata.core import fetch_ds
1208
+
1209
+ fmow = (
1210
+ "wilds:fmow?split_scheme=official&version=1.1&download=false"
1211
+ "&source_metadata=location_id,timestamp"
1212
+ )
1213
+ inventory = fetch_ds([fmow], {fmow: ["train", "id_val", "id_test", "val"]})
1214
+
1215
+ for sample in inventory.as_numpy_iterator():
1216
+ source = sample["metadata"]["wilds_source"]
1217
+ location_id = source["location_id"] # UTF-8 bytes
1218
+ timestamp = source["timestamp"] # UTF-8 bytes
1219
+ ```
1220
+
1221
+ The option accepts only `location_id` and `timestamp`. JustData maps every emitted `wilds_index` through WILDS'
1222
+ authoritative `full_idxs` mapping: `location_id` is the exact original FMoW sequence-directory basename, and `timestamp`
1223
+ preserves the source timezone-aware ISO-8601 text. Invalid fields, target-equivalent fields, missing source columns, or
1224
+ unreliable mappings fail closed. The `wilds_source` mapping is absent without the opt-in; it is retained by
1225
+ `metadata_mode="full"` and removed by `numeric_only`.
1226
+
1227
+ The base `wilds:fmow` preset has no stochastic per-sample training augmentation, so train model-input caching is safe
1228
+ when explicitly opted in. Do not use train model-input caching with `wilds:fmow_strong` unless freezing the first pass
1229
+ of stochastic augmentation is intentional.
1230
+
1231
+ ### Applying a Named Preset
1232
+
1233
+ To use a specific preset for any dataset, pass the preset name as `preset`:
1234
+
1235
+ ```python
1236
+ # Load imagenette with the RSB A3 (light) recipe: 160px training, RandAugment m=6, Mixup α=0.1
1237
+ pipeline = get_pipeline(
1238
+ dataset="imagenette",
1239
+ preset="imagenet_a3", # apply the A3 preset to the imagenette task
1240
+ )
1241
+
1242
+ train_ds, N = load_ds(
1243
+ dataset_names_arg=["imagenette"],
1244
+ splits_arg={"imagenette": ["train"]},
1245
+ dataset_type="train",
1246
+ batch_size=128,
1247
+ seed=42,
1248
+ pipeline=pipeline,
1249
+ num_classes=10,
1250
+ cache_dataset=False,
1251
+ )
1252
+ ```
1253
+
1254
+ ### Overriding Preset Parameters
1255
+
1256
+ ```python
1257
+ pipeline = get_pipeline(
1258
+ dataset="imagenet",
1259
+ aug_kwargs={"augment_type": "trivial_augment"}, # overrides preset default
1260
+ )
1261
+ ```
1262
+
1263
+ ### DINOv2 SSL Pipeline
1264
+
1265
+ ```python
1266
+ pipeline = get_pipeline(
1267
+ preset="dinov2",
1268
+ pipeline_name="vision/classification",
1269
+ )
1270
+ ```
1271
+
1272
+ ______________________________________________________________________
1273
+
1274
+ ## Development
1275
+
1276
+ This project uses [devenv](https://devenv.sh/) (Nix-based) with `uv` for Python dependency management.
1277
+
1278
+ ```bash
1279
+ devenv shell # enter the development environment
1280
+ direnv allow # approve automatic devenv activation once
1281
+ uv sync # install/update dependencies
1282
+ uv run pytest # run all tests
1283
+ uv run ruff check . # run lint checks
1284
+ uv run ruff format --check . # check formatting
1285
+ uv run mdformat --check --wrap 120 ./*.md benchmarks docs examples src tests # check Markdown formatting
1286
+ uv run pytest tests/path/to/test_file.py::test_name # run a single test
1287
+ ```
1288
+
1289
+ The project targets Python 3.12 (see `.python-version`). `LD_LIBRARY_PATH` is configured by devenv for native libraries.
1290
+
1291
+ See [Performance checks](benchmarks/README.md) for reproducible CPU benchmarks and the performance audit results.
1292
+
1293
+ GitLab runs `sh tests/doctor/test_lint.sh` in a pinned Debian-based uv image. This installs only the `lint` dependency
1294
+ group and checks Python with Ruff and Markdown with mdformat using the GFM, footnote, and MyST plugins. The image
1295
+ provides the standard Linux dynamic loader required by Ruff's PyPI executable, which the Nix image lacks.
1296
+
1297
+ ### Releases
1298
+
1299
+ GitLab builds a wheel and source distribution after lint, the default test suite, and all Python 3.11–3.13
1300
+ package-support jobs pass. Build artifacts are retained for one week. Tag pipelines publish those artifacts to
1301
+ [PyPI](https://pypi.org/project/justdata/) after the existing manual `golden` job also passes; start that job in GitLab
1302
+ to complete a release.
1303
+
1304
+ Before tagging, update `pyproject.toml`, `src/justdata/__init__.py`, and `CHANGELOG.md`, then run `uv lock`. Push a tag
1305
+ matching the package version, with an optional `v` prefix (for example, `v1.2.0` or `1.2.0`). Publishing rejects tags
1306
+ that disagree with either version declaration. Branch pipelines build artifacts without publishing.
1307
+
1308
+ Configure a [PyPI GitLab trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) for the
1309
+ `justdata` project with namespace `rhizome-labs/public`, project `justdata`, pipeline path `.gitlab-ci.yml`, and
1310
+ environment `release`. The `publish` job requests a GitLab ID token with audience `pypi` and requires trusted
1311
+ publishing; no stored PyPI API token is needed.