justdata 1.1.0__tar.gz → 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {justdata-1.1.0 → justdata-1.2.0}/PKG-INFO +245 -64
- {justdata-1.1.0 → justdata-1.2.0}/README.md +244 -63
- {justdata-1.1.0 → justdata-1.2.0}/pyproject.toml +5 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/__init__.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_signal.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/batch.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/spectrogram.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/waveform.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/registry.py +2 -2
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/ast.py +1 -1
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/log.py +3 -4
- justdata-1.2.0/src/justdata/acoustic/pipelines.py +572 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/presets.py +10 -2
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/__init__.py +41 -0
- justdata-1.2.0/src/justdata/core/_records.py +66 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/adapters.py +4 -1
- justdata-1.2.0/src/justdata/core/cache.py +270 -0
- justdata-1.2.0/src/justdata/core/config_resolution.py +58 -0
- justdata-1.2.0/src/justdata/core/executed_config.py +77 -0
- justdata-1.2.0/src/justdata/core/execution.py +45 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/finalization.py +100 -15
- justdata-1.2.0/src/justdata/core/inventory.py +745 -0
- justdata-1.2.0/src/justdata/core/loader.py +1022 -0
- justdata-1.2.0/src/justdata/core/metadata.py +727 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/presets.py +33 -2
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/registry.py +136 -6
- justdata-1.2.0/src/justdata/core/replay.py +387 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/color.py +110 -0
- justdata-1.2.0/src/justdata/vision/encodings/__init__.py +21 -0
- justdata-1.2.0/src/justdata/vision/encodings/panoptic_targets.py +320 -0
- justdata-1.2.0/src/justdata/vision/encodings/semantic_targets.py +139 -0
- justdata-1.2.0/src/justdata/vision/geometry.py +579 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/minic.py +1 -1
- justdata-1.2.0/src/justdata/vision/pipelines.py +905 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/presets.py +52 -2
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/stages.py +64 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/classification.py +1 -1
- justdata-1.2.0/src/justdata/vision/tasks/panoptic.py +106 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/segmentation.py +83 -1
- justdata-1.1.0/src/justdata/acoustic/pipelines.py +0 -112
- justdata-1.1.0/src/justdata/core/loader.py +0 -482
- justdata-1.1.0/src/justdata/core/metadata.py +0 -255
- justdata-1.1.0/src/justdata/vision/pipelines.py +0 -62
- justdata-1.1.0/src/justdata/vision/tasks/__init__.py +0 -1
- {justdata-1.1.0 → justdata-1.2.0}/LICENSE +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_dcase.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_manifests.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/_random.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/adapters.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/_common.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/configs.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/augment/patchout.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/batching.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/channel.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ast.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/ced.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/efficientat.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/panns.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/compat/passt.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/configs.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/codec.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/datasets.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/device.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/dynamics.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/filters.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/noise.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/reverb.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/corruptions/timefreq.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/datasets.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/dcase2025.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/decoding.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/eval_views.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/evals.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/filters.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/kaldi.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mel.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/mfcc.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/pcen.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/frontends/stft.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/jax.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/labels.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/layouts.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/metadata.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/normalization.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/postprocessing.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/preprocessing.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/registry.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/resampling.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/schema.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/segment.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/sources.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/stages.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/stats.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/acoustic/tasks.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/audio/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/augmentations.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/filters.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/label_mixing.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/sources.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/core/stats.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/corruptions.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/py.typed +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/auto.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/composed.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/geometric.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/mixing.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/augmentations/registry.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/blur.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/digital.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/noise.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/registry.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/standard.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/corruptions/weather.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/datasets.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/encodings/heatmaps.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/sources.py +0 -0
- {justdata-1.1.0/src/justdata/vision/encodings → justdata-1.2.0/src/justdata/vision/tasks}/__init__.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/tasks/depth.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/transforms.py +0 -0
- {justdata-1.1.0 → justdata-1.2.0}/src/justdata/vision/utils.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: justdata
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.2.0
|
|
4
4
|
Summary: A TensorFlow-native data pipeline library with a modality-neutral core and first-class computer vision and acoustic recipes
|
|
5
5
|
Author: clementpoiret
|
|
6
6
|
Author-email: clementpoiret <clement@linux.com>
|
|
@@ -87,8 +87,8 @@ ______________________________________________________________________
|
|
|
87
87
|
## Architecture
|
|
88
88
|
|
|
89
89
|
`justdata` is structured around a fixed, four-stage pipeline abstraction. The currently supported vision tasks,
|
|
90
|
-
classification and segmentation, each expose exactly four composable functions. Object
|
|
91
|
-
are future scope and are not registered capabilities.
|
|
90
|
+
classification, semantic segmentation, and panoptic segmentation, each expose exactly four composable functions. Object
|
|
91
|
+
detection and depth estimation are future scope and are not registered capabilities.
|
|
92
92
|
|
|
93
93
|
```
|
|
94
94
|
(preprocess_fn, augment_fn, late_augment_fn, postprocess_fn)
|
|
@@ -108,25 +108,33 @@ it must be explicitly opted in and is placed before shuffle so epoch order is no
|
|
|
108
108
|
|
|
109
109
|
### TensorFlow execution controls
|
|
110
110
|
|
|
111
|
-
`load_ds` uses `tf.data.AUTOTUNE` for loader-owned maps by default. Advanced callers may set
|
|
112
|
-
`
|
|
113
|
-
|
|
114
|
-
|
|
111
|
+
`load_ds` uses `tf.data.AUTOTUNE` for loader-owned maps by default. Advanced callers may set `map_parallel_calls`,
|
|
112
|
+
`private_threadpool_size`, and `max_intra_op_parallelism` to bound TensorFlow input-pipeline concurrency. Omitting these
|
|
113
|
+
arguments preserves the default TensorFlow behavior. The map limit also applies to metadata sidecar writing when
|
|
114
|
+
enabled.
|
|
115
115
|
|
|
116
116
|
### Core Public API
|
|
117
117
|
|
|
118
|
-
| Function | Description
|
|
119
|
-
| :-------------------------------------------------------------------- |
|
|
120
|
-
| `justdata.core.fetch_ds(dataset_names, splits_info, data_dir)` | Raw dataset loading through registered source loaders. Returns a `tf.data.Dataset` in canonical schema.
|
|
121
|
-
| `justdata.core.load_ds(...)` | Full pipeline for training or evaluation. Handles caching, augmentation, shuffling, batching, and prefetching.
|
|
122
|
-
| `justdata.
|
|
123
|
-
| `justdata.
|
|
124
|
-
| `justdata.
|
|
118
|
+
| Function | Description |
|
|
119
|
+
| :-------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |
|
|
120
|
+
| `justdata.core.fetch_ds(dataset_names, splits_info, data_dir)` | Raw dataset loading through registered source loaders. Returns a `tf.data.Dataset` in canonical schema. |
|
|
121
|
+
| `justdata.core.load_ds(...)` | Full pipeline for training or evaluation. Handles caching, augmentation, shuffling, batching, and prefetching. |
|
|
122
|
+
| `justdata.core.admit_inventory(...)` | Strict offline validation of a resolved inventory into a disk snapshot with structured identity and failure reports. |
|
|
123
|
+
| `justdata.core.open_inventory(snapshot_dir)` | Verify and reopen a completed inventory snapshot without original sources. |
|
|
124
|
+
| `justdata.core.load_inventory(admitted, ...)` | Apply the shared pipeline to an admitted inventory, including raw epoch finalization. |
|
|
125
|
+
| `justdata.core.ExecutedConfig` | Immutable, canonical snapshot returned by loaders when `return_config=True`. |
|
|
126
|
+
| `justdata.vision.minic.create_minic_datasets(...)` | Constructs Mini-C corruption benchmark datasets from a shared preprocessed base dataset. |
|
|
127
|
+
| `justdata.acoustic.corruptions.create_audio_corruption_datasets(...)` | Constructs acoustic corruption benchmark datasets from a shared preprocessed base dataset. |
|
|
128
|
+
| `justdata.acoustic.dcase2025.make_source_dataset(...)` | Builds DCASE Task 1 source-domain datasets with split-safety checks. |
|
|
125
129
|
|
|
126
130
|
Import `justdata.vision` before resolving built-in vision datasets or pipelines. Hugging Face vision datasets are
|
|
127
131
|
referenced with the `hf:` prefix (e.g., `hf:cifar10`), and WILDS image classification datasets are referenced with the
|
|
128
132
|
`wilds:` prefix (e.g., `wilds:camelyon17`).
|
|
129
133
|
|
|
134
|
+
Use the [strict local inventory API](docs/inventory.md) when every requested source and record must be accounted for. It
|
|
135
|
+
verifies expected content digests, preserves record IDs and order, and reports declared filtering. Existing `fetch_ds`
|
|
136
|
+
and `load_ds` retain their legacy source-skipping behavior and do not provide the strict admission guarantee.
|
|
137
|
+
|
|
130
138
|
______________________________________________________________________
|
|
131
139
|
|
|
132
140
|
## Data Pipeline Stages
|
|
@@ -419,9 +427,9 @@ are fixed by design and cannot be overridden via kwargs; to use custom bounds, c
|
|
|
419
427
|
### 6. Non-RA Operations
|
|
420
428
|
|
|
421
429
|
`Invert`, `Cutout`, `SolarizeAdd`, and `Grayscale` are outside the fixed 14-op pool and cannot be selected through
|
|
422
|
-
RandAugment, TrivialAugment, or TrivialAugmentWide. The public `exclude_ops` parameter only removes operations from
|
|
423
|
-
|
|
424
|
-
|
|
430
|
+
RandAugment, TrivialAugment, or TrivialAugmentWide. The public `exclude_ops` parameter only removes operations from the
|
|
431
|
+
pool; listing these names has no effect. The `cutout_const` argument remains accepted for compatibility and has no
|
|
432
|
+
effect. Random erasing is available through the `random_erasing` late-augmentation stage.
|
|
425
433
|
|
|
426
434
|
______________________________________________________________________
|
|
427
435
|
|
|
@@ -534,6 +542,18 @@ complex interpolation during dense evaluation, images are resized to the target
|
|
|
534
542
|
the bottom and right edges such that both height and width are exact multiples of the ViT patch size (e.g., 14). This is
|
|
535
543
|
implemented in `justdata.vision.transforms.pad_to_patch_multiple`.
|
|
536
544
|
|
|
545
|
+
For semantic segmentation with replayable geometry, select `vision/segmentation` with `apply_presets=False` and explicit
|
|
546
|
+
`geometry_kwargs`. It supports paired aspect-preserving training resize and crops, longer-side evaluation caps,
|
|
547
|
+
independent RGB/ignore padding, per-pixel validity, and original-coordinate score restoration. See
|
|
548
|
+
[the dense geometry contract](docs/vision.md#12-replayable-dense-segmentation) for configuration, record fields, replay,
|
|
549
|
+
and batching requirements.
|
|
550
|
+
|
|
551
|
+
For instance-aware labels, select `vision/panoptic_segmentation` with a segment-ID map and category table. It preserves
|
|
552
|
+
thing instances, merges stuff by category, and excludes crowd pixels from panoptic supervision. The
|
|
553
|
+
[panoptic contract](docs/vision.md#13-panoptic-segmentation) and
|
|
554
|
+
[LaRS example](docs/vision.md#14-loading-local-lars-archives) show how to load either semantic or panoptic labels from
|
|
555
|
+
local LaRS v1.0.0 archives and plot five images under the A1/A2/A3 presets with both label views.
|
|
556
|
+
|
|
537
557
|
______________________________________________________________________
|
|
538
558
|
|
|
539
559
|
## Registry System
|
|
@@ -555,6 +575,12 @@ All extensible components in `justdata` use a decorator-based registry pattern w
|
|
|
555
575
|
`get_pipeline` is the high-level resolver: it infers the task type from the dataset name, merges preset defaults with
|
|
556
576
|
user-supplied kwargs (via smart merge; see below), and invokes the appropriate pipeline factory.
|
|
557
577
|
|
|
578
|
+
Custom pipelines may register `@register_pipeline(name, config_resolver=resolver)` when they support strict overrides
|
|
579
|
+
and executed-configuration export. The resolver receives a defensive copy of the merged configuration plus `is_training`
|
|
580
|
+
and returns the resolved four-stage configuration and model-input contract. Legacy factories without a resolver remain
|
|
581
|
+
buildable, but strict resolution and `return_config=True` reject them because their behavior cannot be described
|
|
582
|
+
completely.
|
|
583
|
+
|
|
558
584
|
**Built-in crop strategies:** `random_resized`, `random_resized_hvflip`, `random_pad`, `random_hflip`,
|
|
559
585
|
`resize_random_hflip`, `random_rot90_hflip`.
|
|
560
586
|
|
|
@@ -611,6 +637,31 @@ ______________________________________________________________________
|
|
|
611
637
|
`_default` preset values are treated as "not explicitly overridden," allowing dataset-specific preset values to take
|
|
612
638
|
precedence. Only kwargs that genuinely differ from the defaults are considered intentional user overrides.
|
|
613
639
|
|
|
640
|
+
For an authoritative recipe, pass `overrides=` to `get_pipeline`. Explicit values win even when they are `False`, zero,
|
|
641
|
+
`None` (where supported), or equal to the modality default. Nested mappings preserve siblings from the selected preset:
|
|
642
|
+
|
|
643
|
+
```python
|
|
644
|
+
pipeline = get_pipeline(
|
|
645
|
+
dataset="cifar10",
|
|
646
|
+
overrides={
|
|
647
|
+
"aug_kwargs": {"enable": False, "image_size": 224},
|
|
648
|
+
"laug_kwargs": {"enable": False},
|
|
649
|
+
"postproc_kwargs": {
|
|
650
|
+
"image_size": 224,
|
|
651
|
+
"val_resize_size": None,
|
|
652
|
+
"normalization_params": (
|
|
653
|
+
(0.485, 0.456, 0.406),
|
|
654
|
+
(0.229, 0.224, 0.225),
|
|
655
|
+
),
|
|
656
|
+
},
|
|
657
|
+
},
|
|
658
|
+
)
|
|
659
|
+
```
|
|
660
|
+
|
|
661
|
+
Supplying `overrides={}` selects strict resolution without changing values. Built-in resolvers reject unknown nested
|
|
662
|
+
keys, missing required settings, contradictory configurations, and attempts to set runtime-owned fields before a loader
|
|
663
|
+
opens its source. Existing keyword arguments retain smart-merge compatibility semantics.
|
|
664
|
+
|
|
614
665
|
Resolved presets are serializable and hashable across modalities. Use `get_resolved_preset(name)` from
|
|
615
666
|
`justdata.vision.presets`, `justdata.acoustic.presets`, or `justdata.core.presets` to obtain an object with `.to_json()`
|
|
616
667
|
and `.hash()`. Hashes use canonical JSON with sorted keys and a 16-character SHA-256 prefix.
|
|
@@ -621,6 +672,80 @@ namespace is a compatibility alias for `justdata.acoustic`.
|
|
|
621
672
|
|
|
622
673
|
See [docs/presets.md](docs/presets.md) for acoustic preset contracts, including EfficientAT/DyMN, PaSST, and CED.
|
|
623
674
|
|
|
675
|
+
### Executed configuration snapshots
|
|
676
|
+
|
|
677
|
+
Pass `return_config=True` to `load_ds` or `load_inventory` to receive `(dataset, n_batches, config)`. The immutable
|
|
678
|
+
`ExecutedConfig` records the selected pipeline and preset, resolved four-stage configuration, derived model-input
|
|
679
|
+
contract, train/evaluation mode, randomness, shuffle, batching, metadata, caches, execution limits, prefetching, and
|
|
680
|
+
NumPy conversion. Its canonical versioned encoding is available as `config.to_bytes()`:
|
|
681
|
+
|
|
682
|
+
```python
|
|
683
|
+
import hashlib
|
|
684
|
+
from pathlib import Path
|
|
685
|
+
|
|
686
|
+
dataset, n_batches, config = load_ds(
|
|
687
|
+
dataset_names_arg="cifar10",
|
|
688
|
+
splits_arg="test",
|
|
689
|
+
dataset_type="validation",
|
|
690
|
+
batch_size=128,
|
|
691
|
+
seed=42,
|
|
692
|
+
pipeline=pipeline,
|
|
693
|
+
num_classes=10,
|
|
694
|
+
return_config=True,
|
|
695
|
+
)
|
|
696
|
+
encoded = config.to_bytes()
|
|
697
|
+
Path("executed-justdata-config.json").write_bytes(encoded)
|
|
698
|
+
full_digest = hashlib.sha256(encoded).hexdigest()
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
For a JD-01 offline inventory, use the same return option after verified admission:
|
|
702
|
+
|
|
703
|
+
```python
|
|
704
|
+
import hashlib
|
|
705
|
+
from pathlib import Path
|
|
706
|
+
|
|
707
|
+
from justdata.core import load_inventory, open_inventory
|
|
708
|
+
|
|
709
|
+
admitted = open_inventory("/data/lars-admitted")
|
|
710
|
+
dataset, n_batches, config = load_inventory(
|
|
711
|
+
admitted,
|
|
712
|
+
dataset_type="validation",
|
|
713
|
+
batch_size=8,
|
|
714
|
+
seed=42,
|
|
715
|
+
pipeline=pipeline,
|
|
716
|
+
return_config=True,
|
|
717
|
+
)
|
|
718
|
+
encoded = config.to_bytes()
|
|
719
|
+
Path("lars-justdata-execution.json").write_bytes(encoded)
|
|
720
|
+
full_digest = hashlib.sha256(encoded).hexdigest()
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
With `return_raw_ds=True`, the loader returns `(prepared, tools, config)` and marks the remaining stages as pending.
|
|
724
|
+
Both `tools["finalize_fn"]` and `tools["finalize_epoch"]` accept `return_config=True` and append a snapshot of their
|
|
725
|
+
actual finalization settings. Snapshot export rejects opaque callback replacements because their behavior cannot be
|
|
726
|
+
represented by configuration data.
|
|
727
|
+
|
|
728
|
+
The snapshot describes JustData's configured execution. A consuming product must separately bind admitted source
|
|
729
|
+
inventory identities, external transformations, exact library revisions, model/checkpoint identities, and other run
|
|
730
|
+
artifacts.
|
|
731
|
+
|
|
732
|
+
### Numeric identity and metadata sidecars
|
|
733
|
+
|
|
734
|
+
With `metadata_mode="numeric_only"`, pass `metadata_sidecar=` for a complete source mapping prepared before iteration.
|
|
735
|
+
For an admitted inventory, use `MetadataSidecar.from_inventory(admitted)`; for another finite source, use
|
|
736
|
+
`MetadataSidecar.from_metadata(source_records)`. Each real batch row then has `metadata.row_id`, an `int64` join key,
|
|
737
|
+
and `metadata.row_fingerprint`, a numeric check against stale cache entries. Resolve keys through `sidecar.records` only
|
|
738
|
+
where `padding_mask` is true. The sidecar preserves original string IDs and static source metadata while numeric
|
|
739
|
+
geometry remains with each view.
|
|
740
|
+
|
|
741
|
+
For a view with additional string metadata, use the numeric `metadata.view_id` with `row_id` to read
|
|
742
|
+
`sidecar.view_records[(row_id, view_id)]`.
|
|
743
|
+
|
|
744
|
+
`sidecar.write_jsonl(path, policy="create")` saves a complete mapping and `MetadataSidecar.read_jsonl(path)` reopens it.
|
|
745
|
+
The streaming `sidecar_metadata_path=` route resumes and validates an existing file by default;
|
|
746
|
+
`sidecar_metadata_policy="create"` rejects reuse and `"overwrite"` explicitly replaces it. Streaming artifacts can be
|
|
747
|
+
partial after interrupted iteration. Executed configuration records the sidecar policy or full mapping digest.
|
|
748
|
+
|
|
624
749
|
### Automatic preset resolution
|
|
625
750
|
|
|
626
751
|
When a vision dataset has a registered preset (e.g., `cifar10` -> `cifar`, `cifar100` -> `cifar100`), `get_pipeline`
|
|
@@ -896,9 +1021,8 @@ train_ds, N = load_ds(
|
|
|
896
1021
|
|
|
897
1022
|
### Reusable, Explicitly Seeded Epochs
|
|
898
1023
|
|
|
899
|
-
Use `return_raw_ds=True` to prepare the source, pipeline, preprocessing, cache,
|
|
900
|
-
|
|
901
|
-
each epoch:
|
|
1024
|
+
Use `return_raw_ds=True` to prepare the source, pipeline, preprocessing, cache, and filters once, then materialize the
|
|
1025
|
+
complete remaining training pipeline for each epoch:
|
|
902
1026
|
|
|
903
1027
|
```python
|
|
904
1028
|
prepared_train, tools = load_ds(
|
|
@@ -923,17 +1047,14 @@ for epoch in range(num_epochs):
|
|
|
923
1047
|
train_one_epoch(train_iterator)
|
|
924
1048
|
```
|
|
925
1049
|
|
|
926
|
-
`finalize_epoch` applies standard augmentation, shuffle, postprocessing,
|
|
927
|
-
|
|
928
|
-
|
|
929
|
-
|
|
930
|
-
dataset. With `deterministic=True`, repeating a seed therefore reproduces the
|
|
931
|
-
epoch independently of previous iterator creation or consumption, including
|
|
1050
|
+
`finalize_epoch` applies standard augmentation, shuffle, postprocessing, batching, late augmentation, padding, and
|
|
1051
|
+
prefetching. The epoch seed is split into standard- and late-augmentation seeds, with source and batch indices folded
|
|
1052
|
+
in. Shuffle uses the epoch seed with reshuffling disabled on the materialized dataset. With `deterministic=True`,
|
|
1053
|
+
repeating a seed therefore reproduces the epoch independently of previous iterator creation or consumption, including
|
|
932
1054
|
when the returned TensorFlow dataset is iterated more than once.
|
|
933
1055
|
|
|
934
|
-
For an augmentation callback that depends only on its input sample and supplied
|
|
935
|
-
|
|
936
|
-
augmentation concurrently using `map_parallel_calls` (or `AUTOTUNE`). With
|
|
1056
|
+
For an augmentation callback that depends only on its input sample and supplied seed, pass `augment_is_stateless=True`
|
|
1057
|
+
to `finalize_epoch` to run standard augmentation concurrently using `map_parallel_calls` (or `AUTOTUNE`). With
|
|
937
1058
|
`deterministic=True`, sample order and indexed RNG seeds remain unchanged:
|
|
938
1059
|
|
|
939
1060
|
```python
|
|
@@ -945,20 +1066,84 @@ train_iterator, n_batches = tools["finalize_epoch"](
|
|
|
945
1066
|
)
|
|
946
1067
|
```
|
|
947
1068
|
|
|
948
|
-
This flag is the caller's assertion that the augmentation callback uses no
|
|
949
|
-
|
|
950
|
-
serial standard augmentation for deterministic epochs. It does not change the
|
|
1069
|
+
This flag is the caller's assertion that the augmentation callback uses no stateful RNG, mutable state, or side effects.
|
|
1070
|
+
It defaults to `False`, preserving serial standard augmentation for deterministic epochs. It does not change the
|
|
951
1071
|
one-shot `load_ds` or `finalize_fn` seed scheduling.
|
|
952
1072
|
|
|
953
|
-
With `cache_dataset=False`, source samples, preprocessing, and filters still run
|
|
954
|
-
|
|
955
|
-
iterator is created by each `finalize_epoch(..., as_numpy=True)` call.
|
|
1073
|
+
With `cache_dataset=False`, source samples, preprocessing, and filters still run on every iteration; only loader and
|
|
1074
|
+
graph preparation are reused. A fresh NumPy iterator is created by each `finalize_epoch(..., as_numpy=True)` call.
|
|
956
1075
|
|
|
957
|
-
`finalize_epoch` rejects `cache_model_inputs=True` when training or evaluation
|
|
958
|
-
|
|
959
|
-
first sampled augmented views. Existing one-shot `load_ds` and `finalize_fn`
|
|
1076
|
+
`finalize_epoch` rejects `cache_model_inputs=True` when training or evaluation augmentation is enabled because that
|
|
1077
|
+
downstream cache intentionally freezes the first sampled augmented views. Existing one-shot `load_ds` and `finalize_fn`
|
|
960
1078
|
behavior is unchanged.
|
|
961
1079
|
|
|
1080
|
+
For a verified local inventory, `load_replay_epoch` binds an immutable snapshot and executed configuration to an
|
|
1081
|
+
explicit committed-batch cursor. It rebuilds the full epoch before skipping committed batches. See
|
|
1082
|
+
[`docs/inventory.md`](docs/inventory.md#deterministic-epoch-replay) for checkpoint and resume usage. Count real rows
|
|
1083
|
+
with `count_real_examples(batch)` or the returned epoch's `remaining_examples`; padded rows have a false `padding_mask`.
|
|
1084
|
+
The epoch path also accepts `prefetch=False` when finalization must precede a caller-controlled prefetch stage.
|
|
1085
|
+
|
|
1086
|
+
### Bounded input execution and protected caches
|
|
1087
|
+
|
|
1088
|
+
Pass `prefetch=1` (or another positive batch count) to `load_ds`, `load_inventory`, or `finalize_fn` to bound final
|
|
1089
|
+
prefetch. `finalize_epoch` inherits the configured value unless overridden. `False` disables prefetch; `True` retains
|
|
1090
|
+
automatic tuning. A bounded input profile also sets positive integer `map_parallel_calls`, `private_threadpool_size`,
|
|
1091
|
+
and `max_intra_op_parallelism`, plus a finite shuffle buffer. The executed configuration records the effective values.
|
|
1092
|
+
Replay prefetches after skipping committed batches and defaults to one batch.
|
|
1093
|
+
|
|
1094
|
+
When another framework uses the accelerator, configure TensorFlow before creating tensors or datasets:
|
|
1095
|
+
|
|
1096
|
+
```python
|
|
1097
|
+
from justdata.core import configure_tensorflow_cpu
|
|
1098
|
+
|
|
1099
|
+
device_report = configure_tensorflow_cpu(intra_op_threads=1, inter_op_threads=1)
|
|
1100
|
+
assert device_report["logical_gpus"] == []
|
|
1101
|
+
```
|
|
1102
|
+
|
|
1103
|
+
The report includes physical and logical device names and process thread settings. The helper raises if the runtime has
|
|
1104
|
+
initialized too early for the requested change. Core, vision, acoustic, and audio imports permit this setup.
|
|
1105
|
+
|
|
1106
|
+
`CachePolicy` enables identity-checked file caches on the existing preprocessing and model-input cache stages. The cache
|
|
1107
|
+
path is a directory whose parent exists. The input identity must cover ordered content. `load_inventory` binds its
|
|
1108
|
+
verified manifest automatically; other sources require a content-derived `input_identity`. Callers must declare
|
|
1109
|
+
deterministic callbacks because arbitrary callbacks cannot be inspected for stateful random operations.
|
|
1110
|
+
|
|
1111
|
+
```python
|
|
1112
|
+
from justdata.core import CachePolicy, inspect_cache, load_inventory
|
|
1113
|
+
|
|
1114
|
+
policy = CachePolicy(
|
|
1115
|
+
max_bytes=1_073_741_824,
|
|
1116
|
+
max_examples=256,
|
|
1117
|
+
materialization="lazy", # choose "eager" to finish before returning
|
|
1118
|
+
callbacks_are_deterministic=True,
|
|
1119
|
+
)
|
|
1120
|
+
batches, count, config = load_inventory(
|
|
1121
|
+
admitted, "validation", 2, 17,
|
|
1122
|
+
pipeline=pipeline,
|
|
1123
|
+
cache_dataset=True,
|
|
1124
|
+
cache_path="/existing/cache-parent/prepared",
|
|
1125
|
+
cache_policy=policy,
|
|
1126
|
+
return_config=True,
|
|
1127
|
+
map_parallel_calls=2,
|
|
1128
|
+
private_threadpool_size=2,
|
|
1129
|
+
max_intra_op_parallelism=1,
|
|
1130
|
+
prefetch=1,
|
|
1131
|
+
)
|
|
1132
|
+
for batch in batches:
|
|
1133
|
+
consume(batch)
|
|
1134
|
+
assert policy.status("preprocess")["state"] == "complete"
|
|
1135
|
+
assert inspect_cache("/existing/cache-parent/prepared")["state"] == "complete"
|
|
1136
|
+
```
|
|
1137
|
+
|
|
1138
|
+
Lazy caches publish completion only after the source is fully exhausted. Eager caches finish before loading returns.
|
|
1139
|
+
Both modes record count, byte size, checksum, and a versioned stage/configuration identity. Changed input,
|
|
1140
|
+
normalization, geometry, or augmentation configuration rejects reuse at the same path. An incomplete or incompatible
|
|
1141
|
+
cache raises `CacheError`; choose a new path after inspecting or clearing the caller-owned artifact. Cache errors expose
|
|
1142
|
+
`code` and `path`. During lazy iteration TensorFlow may wrap the exception; `policy.status(stage)` retains its
|
|
1143
|
+
structured failure code. Quotas bound record count and serialized record bytes. Protected model-input caches require
|
|
1144
|
+
evaluation without active augmentation; preprocessing caches remain before per-epoch augmentation. Existing cache flags
|
|
1145
|
+
without a policy retain their behavior.
|
|
1146
|
+
|
|
962
1147
|
### Loading a Hugging Face Dataset
|
|
963
1148
|
|
|
964
1149
|
```python
|
|
@@ -1015,8 +1200,8 @@ train_ds, N = load_ds(
|
|
|
1015
1200
|
)
|
|
1016
1201
|
```
|
|
1017
1202
|
|
|
1018
|
-
FMoW source inventories can opt into authoritative sequence identity and
|
|
1019
|
-
|
|
1203
|
+
FMoW source inventories can opt into authoritative sequence identity and acquisition time without changing the default
|
|
1204
|
+
WILDS signature:
|
|
1020
1205
|
|
|
1021
1206
|
```python
|
|
1022
1207
|
from justdata.core import fetch_ds
|
|
@@ -1033,13 +1218,11 @@ for sample in inventory.as_numpy_iterator():
|
|
|
1033
1218
|
timestamp = source["timestamp"] # UTF-8 bytes
|
|
1034
1219
|
```
|
|
1035
1220
|
|
|
1036
|
-
The option accepts only `location_id` and `timestamp`. JustData maps every
|
|
1037
|
-
|
|
1038
|
-
|
|
1039
|
-
`
|
|
1040
|
-
|
|
1041
|
-
closed. The `wilds_source` mapping is absent without the opt-in; it is retained
|
|
1042
|
-
by `metadata_mode="full"` and removed by `numeric_only`.
|
|
1221
|
+
The option accepts only `location_id` and `timestamp`. JustData maps every emitted `wilds_index` through WILDS'
|
|
1222
|
+
authoritative `full_idxs` mapping: `location_id` is the exact original FMoW sequence-directory basename, and `timestamp`
|
|
1223
|
+
preserves the source timezone-aware ISO-8601 text. Invalid fields, target-equivalent fields, missing source columns, or
|
|
1224
|
+
unreliable mappings fail closed. The `wilds_source` mapping is absent without the opt-in; it is retained by
|
|
1225
|
+
`metadata_mode="full"` and removed by `numeric_only`.
|
|
1043
1226
|
|
|
1044
1227
|
The base `wilds:fmow` preset has no stochastic per-sample training augmentation, so train model-input caching is safe
|
|
1045
1228
|
when explicitly opted in. Do not use train model-input caching with `wilds:fmow_strong` unless freezing the first pass
|
|
@@ -1099,6 +1282,7 @@ uv sync # install/update dependencies
|
|
|
1099
1282
|
uv run pytest # run all tests
|
|
1100
1283
|
uv run ruff check . # run lint checks
|
|
1101
1284
|
uv run ruff format --check . # check formatting
|
|
1285
|
+
uv run mdformat --check --wrap 120 ./*.md benchmarks docs examples src tests # check Markdown formatting
|
|
1102
1286
|
uv run pytest tests/path/to/test_file.py::test_name # run a single test
|
|
1103
1287
|
```
|
|
1104
1288
|
|
|
@@ -1107,24 +1291,21 @@ The project targets Python 3.12 (see `.python-version`). `LD_LIBRARY_PATH` is co
|
|
|
1107
1291
|
See [Performance checks](benchmarks/README.md) for reproducible CPU benchmarks and the performance audit results.
|
|
1108
1292
|
|
|
1109
1293
|
GitLab runs `sh tests/doctor/test_lint.sh` in a pinned Debian-based uv image. This installs only the `lint` dependency
|
|
1110
|
-
group and
|
|
1294
|
+
group and checks Python with Ruff and Markdown with mdformat using the GFM, footnote, and MyST plugins. The image
|
|
1295
|
+
provides the standard Linux dynamic loader required by Ruff's PyPI executable, which the Nix image lacks.
|
|
1111
1296
|
|
|
1112
1297
|
### Releases
|
|
1113
1298
|
|
|
1114
|
-
GitLab builds a wheel and source distribution after lint, the default test suite,
|
|
1115
|
-
|
|
1116
|
-
|
|
1117
|
-
|
|
1118
|
-
|
|
1119
|
-
|
|
1120
|
-
|
|
1121
|
-
|
|
1122
|
-
|
|
1123
|
-
|
|
1124
|
-
|
|
1125
|
-
|
|
1126
|
-
Configure a [PyPI GitLab trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
|
|
1127
|
-
for the `justdata` project with namespace `rhizome-labs/public`, project
|
|
1128
|
-
`justdata`, pipeline path `.gitlab-ci.yml`, and environment `release`. The
|
|
1129
|
-
`publish` job requests a GitLab ID token with audience `pypi` and requires trusted
|
|
1299
|
+
GitLab builds a wheel and source distribution after lint, the default test suite, and all Python 3.11–3.13
|
|
1300
|
+
package-support jobs pass. Build artifacts are retained for one week. Tag pipelines publish those artifacts to
|
|
1301
|
+
[PyPI](https://pypi.org/project/justdata/) after the existing manual `golden` job also passes; start that job in GitLab
|
|
1302
|
+
to complete a release.
|
|
1303
|
+
|
|
1304
|
+
Before tagging, update `pyproject.toml`, `src/justdata/__init__.py`, and `CHANGELOG.md`, then run `uv lock`. Push a tag
|
|
1305
|
+
matching the package version, with an optional `v` prefix (for example, `v1.2.0` or `1.2.0`). Publishing rejects tags
|
|
1306
|
+
that disagree with either version declaration. Branch pipelines build artifacts without publishing.
|
|
1307
|
+
|
|
1308
|
+
Configure a [PyPI GitLab trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) for the
|
|
1309
|
+
`justdata` project with namespace `rhizome-labs/public`, project `justdata`, pipeline path `.gitlab-ci.yml`, and
|
|
1310
|
+
environment `release`. The `publish` job requests a GitLab ID token with audience `pypi` and requires trusted
|
|
1130
1311
|
publishing; no stored PyPI API token is needed.
|