justdata 1.1.0__tar.gz → 1.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (128) hide show
  1. {justdata-1.1.0 → justdata-1.2.0}/PKG-INFO +245 -64
  2. {justdata-1.1.0 → justdata-1.2.0}/README.md +244 -63
  3. {justdata-1.1.0 → justdata-1.2.0}/pyproject.toml +5 -1
  4. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/__init__.py +1 -1
  5. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_signal.py +1 -1
  6. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/batch.py +1 -1
  7. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/spectrogram.py +1 -1
  8. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/waveform.py +1 -1
  9. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/registry.py +2 -2
  10. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/ast.py +1 -1
  11. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/log.py +3 -4
  12. justdata-1.2.0/src/justdata/acoustic/pipelines.py +572 -0
  13. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/presets.py +10 -2
  14. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/__init__.py +41 -0
  15. justdata-1.2.0/src/justdata/core/_records.py +66 -0
  16. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/adapters.py +4 -1
  17. justdata-1.2.0/src/justdata/core/cache.py +270 -0
  18. justdata-1.2.0/src/justdata/core/config_resolution.py +58 -0
  19. justdata-1.2.0/src/justdata/core/executed_config.py +77 -0
  20. justdata-1.2.0/src/justdata/core/execution.py +45 -0
  21. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/finalization.py +100 -15
  22. justdata-1.2.0/src/justdata/core/inventory.py +745 -0
  23. justdata-1.2.0/src/justdata/core/loader.py +1022 -0
  24. justdata-1.2.0/src/justdata/core/metadata.py +727 -0
  25. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/presets.py +33 -2
  26. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/registry.py +136 -6
  27. justdata-1.2.0/src/justdata/core/replay.py +387 -0
  28. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/color.py +110 -0
  29. justdata-1.2.0/src/justdata/vision/encodings/__init__.py +21 -0
  30. justdata-1.2.0/src/justdata/vision/encodings/panoptic_targets.py +320 -0
  31. justdata-1.2.0/src/justdata/vision/encodings/semantic_targets.py +139 -0
  32. justdata-1.2.0/src/justdata/vision/geometry.py +579 -0
  33. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/minic.py +1 -1
  34. justdata-1.2.0/src/justdata/vision/pipelines.py +905 -0
  35. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/presets.py +52 -2
  36. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/stages.py +64 -0
  37. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/classification.py +1 -1
  38. justdata-1.2.0/src/justdata/vision/tasks/panoptic.py +106 -0
  39. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/segmentation.py +83 -1
  40. justdata-1.1.0/src/justdata/acoustic/pipelines.py +0 -112
  41. justdata-1.1.0/src/justdata/core/loader.py +0 -482
  42. justdata-1.1.0/src/justdata/core/metadata.py +0 -255
  43. justdata-1.1.0/src/justdata/vision/pipelines.py +0 -62
  44. justdata-1.1.0/src/justdata/vision/tasks/__init__.py +0 -1
  45. {justdata-1.1.0 → justdata-1.2.0}/LICENSE +0 -0
  46. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/__init__.py +0 -0
  47. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_dcase.py +0 -0
  48. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_manifests.py +0 -0
  49. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_random.py +0 -0
  50. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/adapters.py +0 -0
  51. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/__init__.py +0 -0
  52. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/_common.py +0 -0
  53. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/configs.py +0 -0
  54. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/patchout.py +0 -0
  55. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/batching.py +0 -0
  56. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/channel.py +0 -0
  57. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/__init__.py +0 -0
  58. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ast.py +0 -0
  59. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ced.py +0 -0
  60. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/efficientat.py +0 -0
  61. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/panns.py +0 -0
  62. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/passt.py +0 -0
  63. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/configs.py +0 -0
  64. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/__init__.py +0 -0
  65. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/codec.py +0 -0
  66. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/datasets.py +0 -0
  67. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/device.py +0 -0
  68. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/dynamics.py +0 -0
  69. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/filters.py +0 -0
  70. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/noise.py +0 -0
  71. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/reverb.py +0 -0
  72. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/timefreq.py +0 -0
  73. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/datasets.py +0 -0
  74. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/dcase2025.py +0 -0
  75. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/decoding.py +0 -0
  76. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/eval_views.py +0 -0
  77. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/evals.py +0 -0
  78. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/filters.py +0 -0
  79. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/__init__.py +0 -0
  80. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/kaldi.py +0 -0
  81. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mel.py +0 -0
  82. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mfcc.py +0 -0
  83. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/pcen.py +0 -0
  84. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/stft.py +0 -0
  85. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/jax.py +0 -0
  86. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/labels.py +0 -0
  87. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/layouts.py +0 -0
  88. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/metadata.py +0 -0
  89. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/normalization.py +0 -0
  90. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/postprocessing.py +0 -0
  91. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/preprocessing.py +0 -0
  92. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/registry.py +0 -0
  93. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/resampling.py +0 -0
  94. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/schema.py +0 -0
  95. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/segment.py +0 -0
  96. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/sources.py +0 -0
  97. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/stages.py +0 -0
  98. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/stats.py +0 -0
  99. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/tasks.py +0 -0
  100. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/audio/__init__.py +0 -0
  101. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/augmentations.py +0 -0
  102. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/filters.py +0 -0
  103. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/label_mixing.py +0 -0
  104. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/sources.py +0 -0
  105. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/stats.py +0 -0
  106. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/corruptions.py +0 -0
  107. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/py.typed +0 -0
  108. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/__init__.py +0 -0
  109. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/__init__.py +0 -0
  110. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/auto.py +0 -0
  111. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/composed.py +0 -0
  112. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/geometric.py +0 -0
  113. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/mixing.py +0 -0
  114. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/registry.py +0 -0
  115. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/__init__.py +0 -0
  116. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/blur.py +0 -0
  117. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/digital.py +0 -0
  118. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/noise.py +0 -0
  119. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/registry.py +0 -0
  120. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/standard.py +0 -0
  121. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/weather.py +0 -0
  122. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/datasets.py +0 -0
  123. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/encodings/heatmaps.py +0 -0
  124. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/sources.py +0 -0
  125. {justdata-1.1.0/src/justdata/vision/encodings → justdata-1.2.0/src/justdata/vision/tasks}/__init__.py +0 -0
  126. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/depth.py +0 -0
  127. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/transforms.py +0 -0
  128. {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/utils.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: justdata
3
- Version: 1.1.0
3
+ Version: 1.2.0
4
4
  Summary: A TensorFlow-native data pipeline library with a modality-neutral core and first-class computer vision and acoustic recipes
5
5
  Author: clementpoiret
6
6
  Author-email: clementpoiret <clement@linux.com>
@@ -87,8 +87,8 @@ ______________________________________________________________________
87
87
  ## Architecture
88
88
 
89
89
  `justdata` is structured around a fixed, four-stage pipeline abstraction. The currently supported vision tasks,
90
- classification and segmentation, each expose exactly four composable functions. Object detection and depth estimation
91
- are future scope and are not registered capabilities.
90
+ classification, semantic segmentation, and panoptic segmentation, each expose exactly four composable functions. Object
91
+ detection and depth estimation are future scope and are not registered capabilities.
92
92
 
93
93
  ```
94
94
  (preprocess_fn, augment_fn, late_augment_fn, postprocess_fn)
@@ -108,25 +108,33 @@ it must be explicitly opted in and is placed before shuffle so epoch order is no
108
108
 
109
109
  ### TensorFlow execution controls
110
110
 
111
- `load_ds` uses `tf.data.AUTOTUNE` for loader-owned maps by default. Advanced callers may set
112
- `map_parallel_calls`, `private_threadpool_size`, and `max_intra_op_parallelism` to bound TensorFlow input-pipeline
113
- concurrency. Omitting these arguments preserves the default TensorFlow behavior.
114
- The map limit also applies to metadata sidecar writing when enabled.
111
+ `load_ds` uses `tf.data.AUTOTUNE` for loader-owned maps by default. Advanced callers may set `map_parallel_calls`,
112
+ `private_threadpool_size`, and `max_intra_op_parallelism` to bound TensorFlow input-pipeline concurrency. Omitting these
113
+ arguments preserves the default TensorFlow behavior. The map limit also applies to metadata sidecar writing when
114
+ enabled.
115
115
 
116
116
  ### Core Public API
117
117
 
118
- | Function | Description |
119
- | :-------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------- |
120
- | `justdata.core.fetch_ds(dataset_names, splits_info, data_dir)` | Raw dataset loading through registered source loaders. Returns a `tf.data.Dataset` in canonical schema. |
121
- | `justdata.core.load_ds(...)` | Full pipeline for training or evaluation. Handles caching, augmentation, shuffling, batching, and prefetching. |
122
- | `justdata.vision.minic.create_minic_datasets(...)` | Constructs Mini-C corruption benchmark datasets from a shared preprocessed base dataset. |
123
- | `justdata.acoustic.corruptions.create_audio_corruption_datasets(...)` | Constructs acoustic corruption benchmark datasets from a shared preprocessed base dataset. |
124
- | `justdata.acoustic.dcase2025.make_source_dataset(...)` | Builds DCASE Task 1 source-domain datasets with split-safety checks. |
118
+ | Function | Description |
119
+ | :-------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |
120
+ | `justdata.core.fetch_ds(dataset_names, splits_info, data_dir)` | Raw dataset loading through registered source loaders. Returns a `tf.data.Dataset` in canonical schema. |
121
+ | `justdata.core.load_ds(...)` | Full pipeline for training or evaluation. Handles caching, augmentation, shuffling, batching, and prefetching. |
122
+ | `justdata.core.admit_inventory(...)` | Strict offline validation of a resolved inventory into a disk snapshot with structured identity and failure reports. |
123
+ | `justdata.core.open_inventory(snapshot_dir)` | Verify and reopen a completed inventory snapshot without original sources. |
124
+ | `justdata.core.load_inventory(admitted, ...)` | Apply the shared pipeline to an admitted inventory, including raw epoch finalization. |
125
+ | `justdata.core.ExecutedConfig` | Immutable, canonical snapshot returned by loaders when `return_config=True`. |
126
+ | `justdata.vision.minic.create_minic_datasets(...)` | Constructs Mini-C corruption benchmark datasets from a shared preprocessed base dataset. |
127
+ | `justdata.acoustic.corruptions.create_audio_corruption_datasets(...)` | Constructs acoustic corruption benchmark datasets from a shared preprocessed base dataset. |
128
+ | `justdata.acoustic.dcase2025.make_source_dataset(...)` | Builds DCASE Task 1 source-domain datasets with split-safety checks. |
125
129
 
126
130
  Import `justdata.vision` before resolving built-in vision datasets or pipelines. Hugging Face vision datasets are
127
131
  referenced with the `hf:` prefix (e.g., `hf:cifar10`), and WILDS image classification datasets are referenced with the
128
132
  `wilds:` prefix (e.g., `wilds:camelyon17`).
129
133
 
134
+ Use the [strict local inventory API](docs/inventory.md) when every requested source and record must be accounted for. It
135
+ verifies expected content digests, preserves record IDs and order, and reports declared filtering. Existing `fetch_ds`
136
+ and `load_ds` retain their legacy source-skipping behavior and do not provide the strict admission guarantee.
137
+
130
138
  ______________________________________________________________________
131
139
 
132
140
  ## Data Pipeline Stages
@@ -419,9 +427,9 @@ are fixed by design and cannot be overridden via kwargs; to use custom bounds, c
419
427
  ### 6. Non-RA Operations
420
428
 
421
429
  `Invert`, `Cutout`, `SolarizeAdd`, and `Grayscale` are outside the fixed 14-op pool and cannot be selected through
422
- RandAugment, TrivialAugment, or TrivialAugmentWide. The public `exclude_ops` parameter only removes operations from
423
- the pool; listing these names has no effect. The `cutout_const` argument remains accepted for compatibility and
424
- has no effect. Random erasing is available through the `random_erasing` late-augmentation stage.
430
+ RandAugment, TrivialAugment, or TrivialAugmentWide. The public `exclude_ops` parameter only removes operations from the
431
+ pool; listing these names has no effect. The `cutout_const` argument remains accepted for compatibility and has no
432
+ effect. Random erasing is available through the `random_erasing` late-augmentation stage.
425
433
 
426
434
  ______________________________________________________________________
427
435
 
@@ -534,6 +542,18 @@ complex interpolation during dense evaluation, images are resized to the target
534
542
  the bottom and right edges such that both height and width are exact multiples of the ViT patch size (e.g., 14). This is
535
543
  implemented in `justdata.vision.transforms.pad_to_patch_multiple`.
536
544
 
545
+ For semantic segmentation with replayable geometry, select `vision/segmentation` with `apply_presets=False` and explicit
546
+ `geometry_kwargs`. It supports paired aspect-preserving training resize and crops, longer-side evaluation caps,
547
+ independent RGB/ignore padding, per-pixel validity, and original-coordinate score restoration. See
548
+ [the dense geometry contract](docs/vision.md#12-replayable-dense-segmentation) for configuration, record fields, replay,
549
+ and batching requirements.
550
+
551
+ For instance-aware labels, select `vision/panoptic_segmentation` with a segment-ID map and category table. It preserves
552
+ thing instances, merges stuff by category, and excludes crowd pixels from panoptic supervision. The
553
+ [panoptic contract](docs/vision.md#13-panoptic-segmentation) and
554
+ [LaRS example](docs/vision.md#14-loading-local-lars-archives) show how to load either semantic or panoptic labels from
555
+ local LaRS v1.0.0 archives and plot five images under the A1/A2/A3 presets with both label views.
556
+
537
557
  ______________________________________________________________________
538
558
 
539
559
  ## Registry System
@@ -555,6 +575,12 @@ All extensible components in `justdata` use a decorator-based registry pattern w
555
575
  `get_pipeline` is the high-level resolver: it infers the task type from the dataset name, merges preset defaults with
556
576
  user-supplied kwargs (via smart merge; see below), and invokes the appropriate pipeline factory.
557
577
 
578
+ Custom pipelines may register `@register_pipeline(name, config_resolver=resolver)` when they support strict overrides
579
+ and executed-configuration export. The resolver receives a defensive copy of the merged configuration plus `is_training`
580
+ and returns the resolved four-stage configuration and model-input contract. Legacy factories without a resolver remain
581
+ buildable, but strict resolution and `return_config=True` reject them because their behavior cannot be described
582
+ completely.
583
+
558
584
  **Built-in crop strategies:** `random_resized`, `random_resized_hvflip`, `random_pad`, `random_hflip`,
559
585
  `resize_random_hflip`, `random_rot90_hflip`.
560
586
 
@@ -611,6 +637,31 @@ ______________________________________________________________________
611
637
  `_default` preset values are treated as "not explicitly overridden," allowing dataset-specific preset values to take
612
638
  precedence. Only kwargs that genuinely differ from the defaults are considered intentional user overrides.
613
639
 
640
+ For an authoritative recipe, pass `overrides=` to `get_pipeline`. Explicit values win even when they are `False`, zero,
641
+ `None` (where supported), or equal to the modality default. Nested mappings preserve siblings from the selected preset:
642
+
643
+ ```python
644
+ pipeline = get_pipeline(
645
+ dataset="cifar10",
646
+ overrides={
647
+ "aug_kwargs": {"enable": False, "image_size": 224},
648
+ "laug_kwargs": {"enable": False},
649
+ "postproc_kwargs": {
650
+ "image_size": 224,
651
+ "val_resize_size": None,
652
+ "normalization_params": (
653
+ (0.485, 0.456, 0.406),
654
+ (0.229, 0.224, 0.225),
655
+ ),
656
+ },
657
+ },
658
+ )
659
+ ```
660
+
661
+ Supplying `overrides={}` selects strict resolution without changing values. Built-in resolvers reject unknown nested
662
+ keys, missing required settings, contradictory configurations, and attempts to set runtime-owned fields before a loader
663
+ opens its source. Existing keyword arguments retain smart-merge compatibility semantics.
664
+
614
665
  Resolved presets are serializable and hashable across modalities. Use `get_resolved_preset(name)` from
615
666
  `justdata.vision.presets`, `justdata.acoustic.presets`, or `justdata.core.presets` to obtain an object with `.to_json()`
616
667
  and `.hash()`. Hashes use canonical JSON with sorted keys and a 16-character SHA-256 prefix.
@@ -621,6 +672,80 @@ namespace is a compatibility alias for `justdata.acoustic`.
621
672
 
622
673
  See [docs/presets.md](docs/presets.md) for acoustic preset contracts, including EfficientAT/DyMN, PaSST, and CED.
623
674
 
675
+ ### Executed configuration snapshots
676
+
677
+ Pass `return_config=True` to `load_ds` or `load_inventory` to receive `(dataset, n_batches, config)`. The immutable
678
+ `ExecutedConfig` records the selected pipeline and preset, resolved four-stage configuration, derived model-input
679
+ contract, train/evaluation mode, randomness, shuffle, batching, metadata, caches, execution limits, prefetching, and
680
+ NumPy conversion. Its canonical versioned encoding is available as `config.to_bytes()`:
681
+
682
+ ```python
683
+ import hashlib
684
+ from pathlib import Path
685
+
686
+ dataset, n_batches, config = load_ds(
687
+ dataset_names_arg="cifar10",
688
+ splits_arg="test",
689
+ dataset_type="validation",
690
+ batch_size=128,
691
+ seed=42,
692
+ pipeline=pipeline,
693
+ num_classes=10,
694
+ return_config=True,
695
+ )
696
+ encoded = config.to_bytes()
697
+ Path("executed-justdata-config.json").write_bytes(encoded)
698
+ full_digest = hashlib.sha256(encoded).hexdigest()
699
+ ```
700
+
701
+ For a JD-01 offline inventory, use the same return option after verified admission:
702
+
703
+ ```python
704
+ import hashlib
705
+ from pathlib import Path
706
+
707
+ from justdata.core import load_inventory, open_inventory
708
+
709
+ admitted = open_inventory("/data/lars-admitted")
710
+ dataset, n_batches, config = load_inventory(
711
+ admitted,
712
+ dataset_type="validation",
713
+ batch_size=8,
714
+ seed=42,
715
+ pipeline=pipeline,
716
+ return_config=True,
717
+ )
718
+ encoded = config.to_bytes()
719
+ Path("lars-justdata-execution.json").write_bytes(encoded)
720
+ full_digest = hashlib.sha256(encoded).hexdigest()
721
+ ```
722
+
723
+ With `return_raw_ds=True`, the loader returns `(prepared, tools, config)` and marks the remaining stages as pending.
724
+ Both `tools["finalize_fn"]` and `tools["finalize_epoch"]` accept `return_config=True` and append a snapshot of their
725
+ actual finalization settings. Snapshot export rejects opaque callback replacements because their behavior cannot be
726
+ represented by configuration data.
727
+
728
+ The snapshot describes JustData's configured execution. A consuming product must separately bind admitted source
729
+ inventory identities, external transformations, exact library revisions, model/checkpoint identities, and other run
730
+ artifacts.
731
+
732
+ ### Numeric identity and metadata sidecars
733
+
734
+ With `metadata_mode="numeric_only"`, pass `metadata_sidecar=` for a complete source mapping prepared before iteration.
735
+ For an admitted inventory, use `MetadataSidecar.from_inventory(admitted)`; for another finite source, use
736
+ `MetadataSidecar.from_metadata(source_records)`. Each real batch row then has `metadata.row_id`, an `int64` join key,
737
+ and `metadata.row_fingerprint`, a numeric check against stale cache entries. Resolve keys through `sidecar.records` only
738
+ where `padding_mask` is true. The sidecar preserves original string IDs and static source metadata while numeric
739
+ geometry remains with each view.
740
+
741
+ For a view with additional string metadata, use the numeric `metadata.view_id` with `row_id` to read
742
+ `sidecar.view_records[(row_id, view_id)]`.
743
+
744
+ `sidecar.write_jsonl(path, policy="create")` saves a complete mapping and `MetadataSidecar.read_jsonl(path)` reopens it.
745
+ The streaming `sidecar_metadata_path=` route resumes and validates an existing file by default;
746
+ `sidecar_metadata_policy="create"` rejects reuse and `"overwrite"` explicitly replaces it. Streaming artifacts can be
747
+ partial after interrupted iteration. Executed configuration records the sidecar policy or full mapping digest.
748
+
624
749
  ### Automatic preset resolution
625
750
 
626
751
  When a vision dataset has a registered preset (e.g., `cifar10` -> `cifar`, `cifar100` -> `cifar100`), `get_pipeline`
@@ -896,9 +1021,8 @@ train_ds, N = load_ds(
896
1021
 
897
1022
  ### Reusable, Explicitly Seeded Epochs
898
1023
 
899
- Use `return_raw_ds=True` to prepare the source, pipeline, preprocessing, cache,
900
- and filters once, then materialize the complete remaining training pipeline for
901
- each epoch:
1024
+ Use `return_raw_ds=True` to prepare the source, pipeline, preprocessing, cache, and filters once, then materialize the
1025
+ complete remaining training pipeline for each epoch:
902
1026
 
903
1027
  ```python
904
1028
  prepared_train, tools = load_ds(
@@ -923,17 +1047,14 @@ for epoch in range(num_epochs):
923
1047
  train_one_epoch(train_iterator)
924
1048
  ```
925
1049
 
926
- `finalize_epoch` applies standard augmentation, shuffle, postprocessing,
927
- batching, late augmentation, padding, and prefetching. The epoch seed is split
928
- into standard- and late-augmentation seeds, with source and batch indices folded
929
- in. Shuffle uses the epoch seed with reshuffling disabled on the materialized
930
- dataset. With `deterministic=True`, repeating a seed therefore reproduces the
931
- epoch independently of previous iterator creation or consumption, including
1050
+ `finalize_epoch` applies standard augmentation, shuffle, postprocessing, batching, late augmentation, padding, and
1051
+ prefetching. The epoch seed is split into standard- and late-augmentation seeds, with source and batch indices folded
1052
+ in. Shuffle uses the epoch seed with reshuffling disabled on the materialized dataset. With `deterministic=True`,
1053
+ repeating a seed therefore reproduces the epoch independently of previous iterator creation or consumption, including
932
1054
  when the returned TensorFlow dataset is iterated more than once.
933
1055
 
934
- For an augmentation callback that depends only on its input sample and supplied
935
- seed, pass `augment_is_stateless=True` to `finalize_epoch` to run standard
936
- augmentation concurrently using `map_parallel_calls` (or `AUTOTUNE`). With
1056
+ For an augmentation callback that depends only on its input sample and supplied seed, pass `augment_is_stateless=True`
1057
+ to `finalize_epoch` to run standard augmentation concurrently using `map_parallel_calls` (or `AUTOTUNE`). With
937
1058
  `deterministic=True`, sample order and indexed RNG seeds remain unchanged:
938
1059
 
939
1060
  ```python
@@ -945,20 +1066,84 @@ train_iterator, n_batches = tools["finalize_epoch"](
945
1066
  )
946
1067
  ```
947
1068
 
948
- This flag is the caller's assertion that the augmentation callback uses no
949
- stateful RNG, mutable state, or side effects. It defaults to `False`, preserving
950
- serial standard augmentation for deterministic epochs. It does not change the
1069
+ This flag is the caller's assertion that the augmentation callback uses no stateful RNG, mutable state, or side effects.
1070
+ It defaults to `False`, preserving serial standard augmentation for deterministic epochs. It does not change the
951
1071
  one-shot `load_ds` or `finalize_fn` seed scheduling.
952
1072
 
953
- With `cache_dataset=False`, source samples, preprocessing, and filters still run
954
- on every iteration; only loader and graph preparation are reused. A fresh NumPy
955
- iterator is created by each `finalize_epoch(..., as_numpy=True)` call.
1073
+ With `cache_dataset=False`, source samples, preprocessing, and filters still run on every iteration; only loader and
1074
+ graph preparation are reused. A fresh NumPy iterator is created by each `finalize_epoch(..., as_numpy=True)` call.
956
1075
 
957
- `finalize_epoch` rejects `cache_model_inputs=True` when training or evaluation
958
- augmentation is enabled because that downstream cache intentionally freezes the
959
- first sampled augmented views. Existing one-shot `load_ds` and `finalize_fn`
1076
+ `finalize_epoch` rejects `cache_model_inputs=True` when training or evaluation augmentation is enabled because that
1077
+ downstream cache intentionally freezes the first sampled augmented views. Existing one-shot `load_ds` and `finalize_fn`
960
1078
  behavior is unchanged.
961
1079
 
1080
+ For a verified local inventory, `load_replay_epoch` binds an immutable snapshot and executed configuration to an
1081
+ explicit committed-batch cursor. It rebuilds the full epoch before skipping committed batches. See
1082
+ [`docs/inventory.md`](docs/inventory.md#deterministic-epoch-replay) for checkpoint and resume usage. Count real rows
1083
+ with `count_real_examples(batch)` or the returned epoch's `remaining_examples`; padded rows have a false `padding_mask`.
1084
+ The epoch path also accepts `prefetch=False` when finalization must precede a caller-controlled prefetch stage.
1085
+
1086
+ ### Bounded input execution and protected caches
1087
+
1088
+ Pass `prefetch=1` (or another positive batch count) to `load_ds`, `load_inventory`, or `finalize_fn` to bound final
1089
+ prefetch. `finalize_epoch` inherits the configured value unless overridden. `False` disables prefetch; `True` retains
1090
+ automatic tuning. A bounded input profile also sets positive integer `map_parallel_calls`, `private_threadpool_size`,
1091
+ and `max_intra_op_parallelism`, plus a finite shuffle buffer. The executed configuration records the effective values.
1092
+ Replay prefetches after skipping committed batches and defaults to one batch.
1093
+
1094
+ When another framework uses the accelerator, configure TensorFlow before creating tensors or datasets:
1095
+
1096
+ ```python
1097
+ from justdata.core import configure_tensorflow_cpu
1098
+
1099
+ device_report = configure_tensorflow_cpu(intra_op_threads=1, inter_op_threads=1)
1100
+ assert device_report["logical_gpus"] == []
1101
+ ```
1102
+
1103
+ The report includes physical and logical device names and process thread settings. The helper raises if the runtime has
1104
+ initialized too early for the requested change. Core, vision, acoustic, and audio imports permit this setup.
1105
+
1106
+ `CachePolicy` enables identity-checked file caches on the existing preprocessing and model-input cache stages. The cache
1107
+ path is a directory whose parent exists. The input identity must cover ordered content. `load_inventory` binds its
1108
+ verified manifest automatically; other sources require a content-derived `input_identity`. Callers must declare
1109
+ deterministic callbacks because arbitrary callbacks cannot be inspected for stateful random operations.
1110
+
1111
+ ```python
1112
+ from justdata.core import CachePolicy, inspect_cache, load_inventory
1113
+
1114
+ policy = CachePolicy(
1115
+ max_bytes=1_073_741_824,
1116
+ max_examples=256,
1117
+ materialization="lazy", # choose "eager" to finish before returning
1118
+ callbacks_are_deterministic=True,
1119
+ )
1120
+ batches, count, config = load_inventory(
1121
+ admitted, "validation", 2, 17,
1122
+ pipeline=pipeline,
1123
+ cache_dataset=True,
1124
+ cache_path="/existing/cache-parent/prepared",
1125
+ cache_policy=policy,
1126
+ return_config=True,
1127
+ map_parallel_calls=2,
1128
+ private_threadpool_size=2,
1129
+ max_intra_op_parallelism=1,
1130
+ prefetch=1,
1131
+ )
1132
+ for batch in batches:
1133
+ consume(batch)
1134
+ assert policy.status("preprocess")["state"] == "complete"
1135
+ assert inspect_cache("/existing/cache-parent/prepared")["state"] == "complete"
1136
+ ```
1137
+
1138
+ Lazy caches publish completion only after the source is fully exhausted. Eager caches finish before loading returns.
1139
+ Both modes record count, byte size, checksum, and a versioned stage/configuration identity. Changed input,
1140
+ normalization, geometry, or augmentation configuration rejects reuse at the same path. An incomplete or incompatible
1141
+ cache raises `CacheError`; choose a new path after inspecting or clearing the caller-owned artifact. Cache errors expose
1142
+ `code` and `path`. During lazy iteration TensorFlow may wrap the exception; `policy.status(stage)` retains its
1143
+ structured failure code. Quotas bound record count and serialized record bytes. Protected model-input caches require
1144
+ evaluation without active augmentation; preprocessing caches remain before per-epoch augmentation. Existing cache flags
1145
+ without a policy retain their behavior.
1146
+
962
1147
  ### Loading a Hugging Face Dataset
963
1148
 
964
1149
  ```python
@@ -1015,8 +1200,8 @@ train_ds, N = load_ds(
1015
1200
  )
1016
1201
  ```
1017
1202
 
1018
- FMoW source inventories can opt into authoritative sequence identity and
1019
- acquisition time without changing the default WILDS signature:
1203
+ FMoW source inventories can opt into authoritative sequence identity and acquisition time without changing the default
1204
+ WILDS signature:
1020
1205
 
1021
1206
  ```python
1022
1207
  from justdata.core import fetch_ds
@@ -1033,13 +1218,11 @@ for sample in inventory.as_numpy_iterator():
1033
1218
  timestamp = source["timestamp"] # UTF-8 bytes
1034
1219
  ```
1035
1220
 
1036
- The option accepts only `location_id` and `timestamp`. JustData maps every
1037
- emitted `wilds_index` through WILDS' authoritative `full_idxs` mapping:
1038
- `location_id` is the exact original FMoW sequence-directory basename, and
1039
- `timestamp` preserves the source timezone-aware ISO-8601 text. Invalid fields,
1040
- target-equivalent fields, missing source columns, or unreliable mappings fail
1041
- closed. The `wilds_source` mapping is absent without the opt-in; it is retained
1042
- by `metadata_mode="full"` and removed by `numeric_only`.
1221
+ The option accepts only `location_id` and `timestamp`. JustData maps every emitted `wilds_index` through WILDS'
1222
+ authoritative `full_idxs` mapping: `location_id` is the exact original FMoW sequence-directory basename, and `timestamp`
1223
+ preserves the source timezone-aware ISO-8601 text. Invalid fields, target-equivalent fields, missing source columns, or
1224
+ unreliable mappings fail closed. The `wilds_source` mapping is absent without the opt-in; it is retained by
1225
+ `metadata_mode="full"` and removed by `numeric_only`.
1043
1226
 
1044
1227
  The base `wilds:fmow` preset has no stochastic per-sample training augmentation, so train model-input caching is safe
1045
1228
  when explicitly opted in. Do not use train model-input caching with `wilds:fmow_strong` unless freezing the first pass
@@ -1099,6 +1282,7 @@ uv sync # install/update dependencies
1099
1282
  uv run pytest # run all tests
1100
1283
  uv run ruff check . # run lint checks
1101
1284
  uv run ruff format --check . # check formatting
1285
+ uv run mdformat --check --wrap 120 ./*.md benchmarks docs examples src tests # check Markdown formatting
1102
1286
  uv run pytest tests/path/to/test_file.py::test_name # run a single test
1103
1287
  ```
1104
1288
 
@@ -1107,24 +1291,21 @@ The project targets Python 3.12 (see `.python-version`). `LD_LIBRARY_PATH` is co
1107
1291
  See [Performance checks](benchmarks/README.md) for reproducible CPU benchmarks and the performance audit results.
1108
1292
 
1109
1293
  GitLab runs `sh tests/doctor/test_lint.sh` in a pinned Debian-based uv image. This installs only the `lint` dependency
1110
- group and provides the standard Linux dynamic loader required by Ruff's PyPI executable, which the Nix image lacks.
1294
+ group and checks Python with Ruff and Markdown with mdformat using the GFM, footnote, and MyST plugins. The image
1295
+ provides the standard Linux dynamic loader required by Ruff's PyPI executable, which the Nix image lacks.
1111
1296
 
1112
1297
  ### Releases
1113
1298
 
1114
- GitLab builds a wheel and source distribution after lint, the default test suite,
1115
- and all Python 3.11–3.13 package-support jobs pass. Build artifacts are retained
1116
- for one week. Tag pipelines publish those artifacts to
1117
- [PyPI](https://pypi.org/project/justdata/) after the existing manual `golden` job
1118
- also passes; start that job in GitLab to complete a release.
1119
-
1120
- Before tagging, update `pyproject.toml`, `src/justdata/__init__.py`, and
1121
- `CHANGELOG.md`, then run `uv lock`. Push a tag matching the package version, with
1122
- an optional `v` prefix (for example, `v1.1.0` or `1.1.0`). Publishing rejects tags
1123
- that disagree with either version declaration. Branch pipelines build artifacts
1124
- without publishing.
1125
-
1126
- Configure a [PyPI GitLab trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
1127
- for the `justdata` project with namespace `rhizome-labs/public`, project
1128
- `justdata`, pipeline path `.gitlab-ci.yml`, and environment `release`. The
1129
- `publish` job requests a GitLab ID token with audience `pypi` and requires trusted
1299
+ GitLab builds a wheel and source distribution after lint, the default test suite, and all Python 3.11–3.13
1300
+ package-support jobs pass. Build artifacts are retained for one week. Tag pipelines publish those artifacts to
1301
+ [PyPI](https://pypi.org/project/justdata/) after the existing manual `golden` job also passes; start that job in GitLab
1302
+ to complete a release.
1303
+
1304
+ Before tagging, update `pyproject.toml`, `src/justdata/__init__.py`, and `CHANGELOG.md`, then run `uv lock`. Push a tag
1305
+ matching the package version, with an optional `v` prefix (for example, `v1.2.0` or `1.2.0`). Publishing rejects tags
1306
+ that disagree with either version declaration. Branch pipelines build artifacts without publishing.
1307
+
1308
+ Configure a [PyPI GitLab trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) for the
1309
+ `justdata` project with namespace `rhizome-labs/public`, project `justdata`, pipeline path `.gitlab-ci.yml`, and
1310
+ environment `release`. The `publish` job requests a GitLab ID token with audience `pypi` and requires trusted
1130
1311
  publishing; no stored PyPI API token is needed.