loaderx 2.2.1__tar.gz → 2.3.5__tar.gz

{loaderx-2.2.1/loaderx.egg-info → loaderx-2.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.2.1
+Version: 2.3.5
 Summary: Rebuildable high-performance ordered record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
@@ -21,6 +21,8 @@ License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: cffi
 Requires-Dist: msgpack
+Provides-Extra: converter
+Requires-Dist: datasets[vision]>=2.19; extra == "converter"
 Dynamic: license-file
 
 # Loaderx
@@ -73,9 +75,9 @@ from loaderx.dataloader import DataLoader
 
 data = np.load('data.npy', mmap_mode='r')
 label = np.load('label.npy', mmap_mode='r')
-with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
+with Dense.create('train_data', data.dtype, data.shape[1:], codec='zstd') as ds:
     ds.append(data)
-with Dense.create('train_label', label.dtype, label.shape[1:]) as ds:
+with Dense.create('train_label', label.dtype, label.shape[1:], codec='zstd') as ds:
     ds.append(label)
 
 data_store = Dense.open('train_data')
@@ -100,13 +102,70 @@ at the same indices, so record ``i`` lines up across them.  The ``transform``
 callback is the collate step — reshape, cast, stack — where ``values`` is the
 plain dense batch ready for the model.
 
+### Offline Hugging Face conversion
+
+The optional converter uses Hugging Face Datasets for remote discovery,
+download, caching, revision handling and source-format decoding. It then writes
+explicit typed Zrecord streams, so training needs neither `datasets` nor Arrow:
+
+```bash
+pip install 'loaderx[converter]'
+```
+
+```python
+from loaderx.converter import convert, huggingface
+
+dataset = huggingface("ylecun/mnist", revision="main", token=None)
+convert(
+    dataset,
+    "mnist",
+    codec="zstd",
+    batch_size=256,
+)
+```
+
+`huggingface` resolves the requested branch, tag or commit to an immutable
+snapshot, then downloads and returns the repository's DatasetDict. Repositories
+without one default config require `config=...`. `convert` is the independent
+persistence stage: it accepts that DatasetDict or one already normalized
+Dataset. `batch_size` bounds records per Python batch; actual bytes also depend
+on record sizes and Ragged packing.
+For private or gated repositories, pass `token="hf_..."`; `None` uses the
+standard HF environment, local login, or anonymous access.
+Static or regularly batched numeric arrays become Dense stores, while dynamic
+arrays become Ragged stores; source dtype and record shape are preserved.
+
+Zrecord persists numeric tensors rather than Python objects. A standard HF
+feature that cannot be represented losslessly fails with its column name;
+conversion does not filter, pad, tokenize, reshape, or run a transform stage.
+Arrow dtypes, including float64 and uint64, are preserved. Only the current
+batch is formatted; `with_format` never materializes the complete dataset.
+
+The result groups aligned streams under one published root:
+
+```text
+mnist/
+  train/
+    image/
+    label/
+  test/
+    image/
+    label/
+```
+
+Conversion happens under a temporary sibling directory. The root is renamed
+into place only after all stores close and their record counts agree; a failed
+build that raises a Python exception is discarded. A process kill may leave the
+hidden temporary sibling for manual cleanup. No collection manifest is added;
+the output is only split directories containing ordinary column Zrecord stores.
+
 ### Creating a dense store
 ```python
 import numpy as np
 from loaderx.zrecord import Dense
 
 data = np.load('data.npy', mmap_mode='r')
-with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
+with Dense.create('train_data', data.dtype, data.shape[1:], codec='zstd') as ds:
     ds.append(data)
 ```
 One record per slice along axis 0; a 1-D array (the usual shape of a label set)
@@ -132,7 +191,7 @@ engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
 strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
 iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
 ``np.frombuffer(raw, dtype=np.uint8)`` and stored in a
-``Ragged.create(path, dtype=np.uint8, ndim=1)`` rather than creating a second
+``Ragged.create(path, dtype=np.uint8, ndim=1, codec="zstd")`` rather than creating a second
 public storage API.
 
 ### Records
@@ -147,7 +206,7 @@ import numpy as np
 from loaderx.zrecord import Dense
 
 data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
-with Dense.create('data', data.dtype, data.shape[1:]) as ds:
+with Dense.create('data', data.dtype, data.shape[1:], codec='zstd') as ds:
     ds.append(data)
 ds = Dense.open('data')
 ds[0, 5, 2]                          # (3, 2, 4) — shape from the persisted schema
@@ -173,7 +232,7 @@ batch is the model's call — a plain numpy loop, wherever you need it:
 from loaderx.zrecord import Ragged
 
 seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
-with Ragged.create('tokens', np.int32, ndim=1) as rs:
+with Ragged.create('tokens', np.int32, ndim=1, codec='zstd') as rs:
     rs.append(seqs)                       # dtype/rank fixed; lengths remain per-record
 rs = Ragged.open('tokens')
 
@@ -260,8 +319,10 @@ warmup belongs outside build or loader benchmark timing.
 ### Creating containers
 
 ``Dense.create`` and ``Ragged.create`` return append-only ordered-sequence
-builders. ``append`` is explicit — one input batch extends the logical sequence
-without exposing native physical completion order.
+builders. The ``codec`` keyword is required: callers must explicitly select
+``"raw"``, ``"zstd"``, or ``"zstd_dict"``. ``append`` is explicit — one input
+batch extends the logical sequence without exposing native physical completion
+order.
 Dense append is synchronous and borrows an already-contiguous ndarray without a
 snapshot copy. Ragged append consumes its iterable once into one owned packed
 buffer, then completes the native append before returning. Nothing is inferred.
@@ -269,12 +330,13 @@ buffer, then completes the native append before returning. Nothing is inferred.
 ```python
 from loaderx.zrecord import Dense, Ragged
 
-ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))
+ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28),
+                  codec='zstd', data_shards=4)
 ds.append(images[i:i + 1024])        # synchronous native batch; returns None
 ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
 ds.close()                            # publish before opening
 
-tok = Ragged.create('tokens', dtype=np.int32, ndim=1)
+tok = Ragged.create('tokens', dtype=np.int32, ndim=1, codec='zstd')
 tok.append([seq_a, seq_b, seq_c])
 tok.close()
 
@@ -282,6 +344,11 @@ with Dense.open('mnist/x') as ds:
     first_four = ds[:4]               # opened containers are read-only
 ```
 
+`data_shards` is the keyword-only write-parallelism setting. It is persisted in
+the native Header, must be in `1..255`, and defaults to four. Values near the
+writer lane count spread payload writes across independent files; opening a
+completed container discovers the value automatically.
+
 Dense and Ragged append both report native errors in the current call and return
 only after accepting the batch. A writer cannot be read, and a reader cannot be
 appended to. ``close()`` on a writer publishes the container Header;
@@ -295,7 +362,7 @@ schemas contain ``dtype`` and ``ndim``; Dense additionally contains
 appended record to have that rank, while every dimension length may vary.
 Structured, subarray, object, metadata-bearing, and zero-itemsize dtypes are not
 supported: their semantics do not round-trip through one canonical NumPy dtype
-string. The encoded schema has 4064 bytes available in the fixed 4096-byte
+string. The encoded schema has 2040 bytes available in the fixed 4096-byte
 metadata page. Ragged schema size is fixed; Dense schema size grows
 only with the integer ``item_shape``, so the physical limit is far above any
 practical NumPy array rank.
@@ -310,7 +377,7 @@ published without one.
 ### Codec notes
 
 `"zstd"` compresses each record independently with plain zstd (level 3). Use it
-for general-purpose compression — it is fast and the default.
+for general-purpose compression; it is fast and must be selected explicitly.
 
 `"zstd_dict"` trains a shared dictionary on a sample of the data before writing
 any record, then compresses every record against it at level 15. The dictionary
@@ -404,11 +471,11 @@ from loaderx.zrecord import Dense, Ragged
 from loaderx.dataloader import DataLoader
 
 root = "xsub/train"
-with Dense.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
+with Dense.create(root + "/joint", joint.dtype, joint.shape[1:], codec="zstd") as s:
     s.append(joint)
-with Dense.create(root + "/label", label.dtype, label.shape[1:]) as s:
+with Dense.create(root + "/label", label.dtype, label.shape[1:], codec="zstd") as s:
     s.append(label)
-with Ragged.create(root + "/token", np.int32, ndim=1) as s:
+with Ragged.create(root + "/token", np.int32, ndim=1, codec="zstd") as s:
     s.append(seqs)
 
 streams = {
@@ -502,8 +569,8 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 ### Methodology
 
 The comparison matrix began as one complete run on a warm page cache. All
-Zrecord Store rows and the two loaderx identity rows were refreshed on 2.2.0
-after the libzstd 1.5.7 and runtime-dispatched Huffman kernel update. Unchanged
+Zrecord Store rows and the two loaderx identity rows were refreshed on 2.3.0
+with the default `data_shards=4`. Unchanged
 optional Store backends, sampler rows and external loader rows remain from the
 immediately preceding 2.0.9 run. These are not three-run medians: an unexpected result is traced
 separately instead of being hidden by repeated aggregation. Within each store workload every
@@ -583,9 +650,9 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 6796 MiB/s | 10816 MiB/s | 75.3 | 4.09 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 102 MiB/s | 11844 MiB/s | 82.5 | 3.61 ms | 16.2 MiB | 22.10x |
-| zrecord-raw | 2801 MiB/s | 14655 MiB/s | 102.1 | 2.92 ms | 358.9 MiB | 1.00x |
+| zrecord-zstd | 7172 MiB/s | 11081 MiB/s | 77.2 | 4.38 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 104 MiB/s | 12042 MiB/s | 83.9 | 4.09 ms | 16.2 MiB | 22.10x |
+| zrecord-raw | 5721 MiB/s | 14457 MiB/s | 100.7 | 3.16 ms | 358.9 MiB | 1.00x |
 | npy-mmap-raw | 2004 MiB/s | 4749 MiB/s | 33.1 | 11.08 ms | 358.9 MiB | 1.00x |
 | hdf5-raw | 2350 MiB/s | 1836 MiB/s | 12.8 | 29.68 ms | 359.0 MiB | 1.00x |
 | hdf5-gzip | 277 MiB/s | 657 MiB/s | 4.6 | 62.53 ms | 26.1 MiB | 13.73x |
@@ -599,8 +666,8 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 | tiledb-raw | 743 MiB/s | 630 MiB/s | 4.4 | 66.01 ms | 359.0 MiB | 1.00x |
 | tiledb-zstd | 1498 MiB/s | 1503 MiB/s | 10.5 | 27.82 ms | 26.9 MiB | 13.36x |
 
-At 147 KiB per record, Zrecord-raw reaches 14.4 GiB/s and is 3.1x npy-mmap-raw;
-plain zstd gathers at 10.6 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+At 147 KiB per record, Zrecord-raw reaches 14.1 GiB/s and is 3.0x npy-mmap-raw;
+plain zstd gathers at 10.8 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -618,9 +685,9 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2339 MiB/s | 9507 MiB/s | 57.1 | 5.16 ms | 26.9 MiB | 15.49x |
-| zrecord-zstdict | 102 MiB/s | 10336 MiB/s | 62.1 | 4.85 ms | 18.2 MiB | 22.98x |
-| zrecord-raw | 1420 MiB/s | 12929 MiB/s | 77.6 | 3.86 ms | 417.2 MiB | 1.00x |
+| zrecord-zstd | 2220 MiB/s | 9444 MiB/s | 56.7 | 5.67 ms | 26.9 MiB | 15.48x |
+| zrecord-zstdict | 102 MiB/s | 10345 MiB/s | 62.1 | 5.15 ms | 18.2 MiB | 22.97x |
+| zrecord-raw | 2171 MiB/s | 12356 MiB/s | 74.2 | 4.22 ms | 417.2 MiB | 1.00x |
 | hdf5-raw | 1451 MiB/s | 1059 MiB/s | 6.4 | 44.91 ms | 418.0 MiB | 1.00x |
 | hdf5-gzip | 250 MiB/s | 126 MiB/s | 0.7 | 359.23 ms | 29.9 MiB | 13.94x |
 | lmdb-raw | 1717 MiB/s | 7269 MiB/s | 43.6 | 7.93 ms | 422.2 MiB | 0.99x |
@@ -633,16 +700,16 @@ list or a one-dimensional variable-length abstraction is not enough.
 | tiledb-raw | 392 MiB/s | 48 MiB/s | 0.3 | 941.09 ms | 417.2 MiB | 1.00x |
 | tiledb-zstd | 675 MiB/s | 138 MiB/s | 0.8 | 328.05 ms | 27.0 MiB | 15.46x |
 
-Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
-delivers 9.3 GiB/s of
+Zrecord-raw is 1.7x LMDB and 3.1x Arrow IPC in logical gather. Zrecord-zstd
+delivers 9.2 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
 and row-group formats pay read amplification on random records.
 
 The shared generator makes compression ratios directly comparable across
-contracts: Zrecord zstd is 14.69x Dense versus 15.49x Ragged, and zstdict is
-22.10x versus 22.98x. The remaining difference comes from the H/W distribution
+contracts: Zrecord zstd is 14.69x Dense versus 15.48x Ragged, and zstdict is
+22.10x versus 22.97x. The remaining difference comes from the H/W distribution
 and Ragged shape metadata, not a different image entropy model.
 
 ### Small Token Records
@@ -662,9 +729,9 @@ accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 1103 MiB/s | 2083 MiB/s | 1066.3 | 0.33 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 70 MiB/s | 2160 MiB/s | 1105.7 | 0.32 ms | 154.4 MiB | 2.53x |
-| zrecord-raw | 2178 MiB/s | 6862 MiB/s | 3513.2 | 0.10 ms | 393.7 MiB | 0.99x |
+| zrecord-zstd | 989 MiB/s | 2012 MiB/s | 1030.3 | 0.34 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 62 MiB/s | 2079 MiB/s | 1064.6 | 0.32 ms | 154.4 MiB | 2.53x |
+| zrecord-raw | 4542 MiB/s | 6391 MiB/s | 3272.3 | 0.11 ms | 393.7 MiB | 0.99x |
 | npy-mmap-raw | 1587 MiB/s | 9070 MiB/s | 4644.0 | 0.08 ms | 390.6 MiB | 1.00x |
 | lmdb-raw | 639 MiB/s | 1104 MiB/s | 565.0 | 0.71 ms | 786.3 MiB | 0.50x |
 | arrow-ipc-raw | 2068 MiB/s | 225 MiB/s | 115.0 | 2.81 ms | 390.8 MiB | 1.00x |
@@ -672,9 +739,9 @@ accumulate at least two seconds.
 | arrayrecord-zstd | 127 MiB/s | 135 MiB/s | 69.4 | 4.45 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.51 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.27 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-1.07–1.11 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.03–1.06 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
@@ -686,16 +753,16 @@ with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 271 MiB/s | 336 MiB/s | 634.2 | 0.54 ms | 67.4 MiB | 1.57x |
-| zrecord-zstdict | 58 MiB/s | 364 MiB/s | 688.2 | 0.50 ms | 49.1 MiB | 2.16x |
-| zrecord-raw | 546 MiB/s | 420 MiB/s | 794.2 | 0.44 ms | 110.5 MiB | 0.96x |
+| zrecord-zstd | 287 MiB/s | 315 MiB/s | 594.6 | 0.60 ms | 67.4 MiB | 1.57x |
+| zrecord-zstdict | 54 MiB/s | 332 MiB/s | 627.6 | 0.58 ms | 49.1 MiB | 2.16x |
+| zrecord-raw | 536 MiB/s | 389 MiB/s | 734.9 | 0.50 ms | 110.5 MiB | 0.96x |
 | lmdb-raw | 318 MiB/s | 113 MiB/s | 214.3 | 1.48 ms | 153.0 MiB | 0.69x |
 | arrow-ipc-raw | 562 MiB/s | 44 MiB/s | 83.1 | 4.00 ms | 109.9 MiB | 0.96x |
 | arrayrecord-raw | 288 MiB/s | 26 MiB/s | 49.3 | 7.39 ms | 118.8 MiB | 0.89x |
 | arrayrecord-zstd | 62 MiB/s | 33 MiB/s | 61.5 | 5.58 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 634–794 krecords/s with 0.44–0.54 ms p95;
+Zrecord's three codecs return 595–735 krecords/s with 0.50–0.60 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -742,8 +809,8 @@ and Grain reads ArrayRecord.
 
 | loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
 |---|---|---|---:|---:|---:|---:|---:|
-| **loaderx** | threads | zrecord-zstd | 166.3 | 15.95 ms | 985 MiB | 986 MiB | 988 MiB |
-| loaderx-raw | threads | zrecord-raw | 208.4 | 13.56 ms | 987 MiB | 987 MiB | 989 MiB |
+| **loaderx** | threads | zrecord-zstd | 171.5 | 12.69 ms | 984 MiB | 985 MiB | 988 MiB |
+| loaderx-raw | threads | zrecord-raw | 200.0 | 12.50 ms | 993 MiB | 993 MiB | 996 MiB |
 | torch | fork | npy-mmap-raw | 109.4 | 32.70 ms | 1783 MiB | 1889 MiB | 6396 MiB |
 | torch-spawn | spawn | npy-mmap-raw | 111.6 | 30.69 ms | 2835 MiB | 2913 MiB | 4880 MiB |
 | grain | processes | arrayrecord-zstd | 46.6 | 93.48 ms | 1847 MiB | 1946 MiB | 2065 MiB |
@@ -753,8 +820,8 @@ the transform threads overlap Python-side collation with the next gather. The
 memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
 threads inside one process, so workers share one interpreter, one NumPy runtime
 and one set of gather buffers. With source geometry and entropy held constant,
-raw is 1.25x compressed loaderx; compressed loaderx is 1.52x Torch fork, 1.49x
-Torch spawn and 3.57x Grain, while raw is 1.90x, 1.87x and 4.47x faster.
+raw is 1.17x compressed loaderx; compressed loaderx is 1.57x Torch fork, 1.54x
+Torch spawn and 3.68x Grain, while raw is 1.83x, 1.79x and 4.29x faster.
 Torch's aggregate RSS is high because
 Linux fork mappings are counted repeatedly; it is not a total-memory ratio
 against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
@@ -828,8 +895,8 @@ format: in the current Dense structured-vision workload, plain zstd reaches
 **Loader results combine architecture and storage.** loaderx uses threads and
 never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
 torch uses finite shuffled epochs, worker processes and shared-memory handoff.
-Here compressed loaderx is 1.52x Torch fork, 1.49x Torch spawn and 3.57x Grain;
-raw loaderx is 1.90x, 1.87x and 4.47x faster, respectively.
+Here compressed loaderx is 1.57x Torch fork, 1.54x Torch spawn and 3.68x Grain;
+raw loaderx is 1.83x, 1.79x and 4.29x faster, respectively.
 Storage also differs per loader — each reads from what it was
 built for — so the loader table is a different comparison from either store
 table, not a rerun.
@@ -890,57 +957,13 @@ The loader rows used Numba 0.67.0, Torch 2.13.0 and Grain 0.2.18; all benchmark
 are pinned in `scripts/requirements-bench.txt`. Temporary stores used the ordinary
 disk-backed `/tmp` filesystem, not `/dev/shm`.
 
-## Real-data verification: NTU RGB-D skeletons
-
-The vision tables above are synthetic; the token tables use real WikiText-103.
-As a separate historical ground-truth check,
-loaderx was run end to end on **NTU RGB-D skeleton data** — 114,480 raw `.skeleton` files,
-120 action classes, 25 joints — processed into the ST-GCN `N C T V M` layout
-(per-sample `(3, 300, 25, 2)` float32) for the `xsub`/`xview` protocols. The
-`.npy` outputs of the standard preprocessing pipeline were treated as ground
-truth. This verification was not rerun with the synthetic benchmarks above;
-its throughput is retained as a separate historical 12-core result.
-
-**Correctness — the read path is bit-exact against the ground truth:**
-
-* Full scan of all **228,356 records** (joint float32 + label int64, all four
-  splits) through `Dense`: byte-for-byte identical to the reference npy.
-* A `DataLoader` over joint + label + an index stream, run under all three
-  sampler modes (`sequential`, `iid`, `cyclic`): every received batch is
-  bit-exact to the ground truth at its own declared indices, and the streams
-  stay index-aligned.
-* Sampler semantics hold on real index spaces: `sequential` walks in order,
-  `cyclic` draws a full cycle without replacement, `iid` is deterministic per
-  seed.
-
-**Storage — zstd on this data:**
-
-| store                 | on disk | ratio |
-|-----------------------|---------|-------|
-| npy (raw float32)     | 6.4 GB  | 1.00x |
-| zrecord raw           | 6.86 GB | 1.00x |
-| zrecord zstd          | 0.79 GB | 8.66x |
-| zrecord zstd_dict     | 0.75 GB | 9.13x |
-
-Sizes above are for one split (`xview/val`, 38,132 records); across all four
-splits the zstd `joint` stores total 4.97 GB against 41 GB of raw npy (~8x).
-
-**Throughput** (180 KB per record, warm page cache, 12 physical cores):
-
-| path                                | throughput |
-|-------------------------------------|-----------|
-| random-batch gather, zstd store     | 4.1–4.5 GiB/s |
-| same, npy-mmap fancy indexing       | 0.6–1.3 GiB/s |
-| DataLoader, 4 prefetch threads      | 5.7–6.6 GiB/s (123–144 batches/s) |
-
-zstd decompression reads ~8x fewer bytes than raw storage, so the compressed
-store gathers *faster* than the raw one (zstd 4587 MiB/s vs raw 1792 MiB/s on
-the same split).
-
-**The npy intermediate is optional.** Parse the skeleton files in parallel and
-feed each fixed-shape ndarray produced by the parser directly to
-`Dense.append`. This writes the store in one pass with no npy staging or
-second read; zrecord bounds its compression working memory independently.
+## Real-data verification
+
+Dataset-specific preprocessing and verification remain in the
+**[DataPipe](https://codeberg.org/eoelab/DataPipe)** repository. The built-in
+converter covers standardized Hugging Face datasets; DataPipe handles sources
+without a common remote protocol and implements derived modalities as loader
+transforms without NumPy dump intermediates.
 
 ## Current Limitations
 * Single-host only; multi-host training is not supported.
@@ -1055,9 +1078,9 @@ license to trust storage or the operating system: native code still validates
     normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only
 defend against bypassing the public Python API do not belong in zrecord.
 
-1. `RecordEngine` stores N logically ordered records. Parallel chunks may finish
-   and occupy `data.zr` in a different physical order, but logical ID is the
-   stable append position and `RecordLoc[ID]` preserves it. Index and slice
+1. `RecordEngine` stores N logically ordered records. Payload record `i` belongs
+   to `data_{i % data_shards}.zr`; logical ID remains the stable append position
+   and `RecordLoc[ID]` preserves its shard-local offset. Index and slice
    operations are implemented as ordered gathers over those positions.
 2. It hands the container layer a dense sequence space: records are exactly
    `0..N-1`. Named streams are composed dynamically by a plain Python dict;
@@ -1120,24 +1143,33 @@ the authority for irreplaceable data. Keep authoritative source data and reprodu
 build scripts; after an interrupted build, storage failure, incompatible implementation
 change, or content change, rebuild a complete container at a new path.
 
-Native storage uses a fixed file set:
+Native storage uses one metadata file and a create-time-fixed payload file set:
 ```
 store/
-  ├── meta.zr      4096-byte static Header/schema page + RecordLoc table
-  ├── data.zr      payload stream
+  ├── meta.zr      4096-byte Header/schema/tails page + RecordLoc table
+  ├── data_0.zr    payload records where ID % data_shards == 0
+  ├── ...
+  ├── data_{N-1}.zr final static payload shard
   └── dict.zr      zstd dictionary (only in dict stores)
 ```
 
+`data_shards` is a write-performance parameter in `1..255`, fixed by `create`
+and recovered automatically by `open`. The default is four; practical values are
+usually 2, 4, 8 or 16, near the writer lane count. More files spread positional
+writes across payload inodes but consume one descriptor each. This physical
+striping does not change record IDs, order, codec, or read results.
+
 ### Metadata (meta.zr)
 
 Files are read and written **positionally** — pread/pwrite at computed offsets,
 no mmap. `meta.zr` starts with one fixed 4096-byte static page: a naturally
-aligned 32-byte Header, then the opaque MsgPack schema and unused zero padding.
+aligned 16-byte Header, 255 shard-local u64 tails at bytes 16..2055, then up to
+2040 bytes of opaque MsgPack schema at bytes 2056..4095.
 An array of 16-byte `RecordLoc`s starts at offset 4096. Record `i` is one
 pread/pwrite at `4096 + i * 16`; there is no variable table base, segment
 mapping, or rollover fd table.
 
-**1. Python schema** — bytes `32..32+schema_length` are exactly one immutable
+**1. Python schema** — bytes `2056..2056+schema_length` are exactly one immutable
 MsgPack object. Both stores contain ``dtype`` and ``ndim``; Dense additionally
 contains ``item_shape``. Native create persists these bytes together with
 the physical container but does not decode them. Open acquires the native lifetime
@@ -1147,30 +1179,36 @@ dtype/item_shape and passed to the native handle as runtime geometry; it is not
 independently persisted as a second authority. There is no format version or
 legacy kind dispatch.
 
-**2. Physical header** — the first 32 bytes of `meta.zr`. The format
+**2. Physical header** — the first 16 bytes of `meta.zr`. The format
 deliberately carries no payload or metadata checksum.
 
 * `codec` is the store's one compression method, stamped at creation and
   immutable — there is no per-record tag anywhere.
 * `length` (u64) is the physical record count; it equals logical length for both
   dense stores and inline ragged stores.
-* `tail_offset` (u64) is the absolute committed frontier in `data.zr`.
-* `schema_length` (u32) is the occupied prefix of the static schema area and
-  must be in `1..4064`.
+* `schema_length` (u16) is the occupied prefix of the static schema area and
+  must be in `1..2040`.
+* `data_shards` (u8) is the static payload file count and must be in `1..255`.
 
 ```zig
 const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
 const Header = extern struct {
     length: u64,
-    tail_offset: u64,
-    schema_length: u32,
-    reserved: [11]u8,
+    schema_length: u16,
+    data_shards: u8,
     codec: u8,
+    reserved: [4]u8,
 };
 ```
 
-**3. Record table** — contiguous 16-byte entries start at offset 4096 in
-`meta.zr` and grow as location runs are written. `offset` is an absolute byte offset in `data.zr`;
+**3. Shard frontiers** — tail slot `s` at `16 + s * 8` is the committed
+end of `data_s.zr`. Unused slots among the 255 fixed u64 entries are zero.
+Open requires every data file to be at least its persisted tail; locations may
+not cross that shard-local frontier.
+
+**4. Record table** — contiguous 16-byte entries start at offset 4096 in
+`meta.zr` and grow as location windows are written. `offset` is local to
+`data_{ID % data_shards}.zr`;
 `phys_length`/`logic_length` are the stored and original sizes. The
 codec is not here: it is the header's, so a record is stored exactly the way the
 store is declared.
@@ -1185,9 +1223,9 @@ const RecordLoc = extern struct {
 
 There is no liveness flag. Every entry below `length` is a record.
 
-**4. No fixed record-count cap.** The table and payload stream grow naturally in
-their fixed files. The practical bounds are the u64 count, supported positional
-file offsets, 2 GiB per record, and disk.
+**5. No fixed record-count cap.** The table and payload streams grow naturally.
+The practical bounds are the u64 count, supported positional file offsets,
+2 GiB per record, descriptor budget, and disk.
 
 ## Executor
 
@@ -1202,34 +1240,36 @@ while `Ragged` supplies offsets for its shape-prefixed records.
 The private ABI turns those inputs into compile-time record sources and
 destinations; the engine has one append and one gather operation. Its shared
 opaque handle remains private and carries no typed-store geometry.
-* Compressed append: workers claim record chunks and lease process-bounded
-  ``ExecutionSlot`` scratch, reserve physical offsets in completion order through a short frontier
-  lock, and issue positional payload writes in parallel. Logical IDs remain in
-  the loc table, so physical completion order does not change random gather.
-  The caller waits for every payload before publishing the new locations and
-  in-process length. Append does not overwrite the on-disk Header and is
-  intentionally lazy. Writer ``close()`` writes the current Header and closes
-  the files, making the page-cache state available for read-only open.
-  Every compressed frame owns one absolute range in `data.zr`; positional writes extend the file to the
-  current frontier. Python budgets the process-wide executor at three quarters
+* Append validates the complete call before physical I/O, then processes bounded
+  logical windows. Within a window, record positions are divided by
+  `ID % data_shards` and planned as bounded shard-local chunks. Fixed Executor
+  lanes dynamically claim those chunks, encode or pack them, briefly lock only
+  the selected shard's tail reservation, then issue positional payload writes
+  directly. Multiple lanes may write non-overlapping ranges of one shard; the
+  static file set spreads that pressure across inodes without limiting codec
+  concurrency to the shard count. Workers fill disjoint entries in one
+  contiguous location buffer. A payload barrier precedes one contiguous
+  `meta.zr` location write. All windows must succeed before the in-process
+  length and per-shard tails advance. Writer `close()` truncates each data file
+  to its committed tail and publishes the 4096-byte static page.
+* Compressed shard tasks lease process-bounded `ExecutionSlot` scratch and write
+  independent frames in bounded subchunks. Python budgets the process-wide executor at three quarters
   of the logical CPUs available to the process, leaving headroom for packing,
   transforms, and the caller without encoding a platform-specific thread count.
   Each producer configures its CCtx or shared immutable CDict once, then starts
   every independent record frame with `ZSTD_compress2`.
-* Raw append preserves the stronger invariant already supplied by Python: every
-  record in one Dense or packed Ragged batch is a boundary inside one contiguous
-  source buffer. After locating the records, the engine consumes that buffer in
-  plan order and passes each contiguous planned range directly to
-  `writePositionalAll` on `data.zr`. There is no per-record iovec construction,
-  byte-budget flush, payload copy, or platform-specific syscall path. Zig's
-  `std.Io` handles short writes and maps the same positional operation to POSIX
-  and Windows implementations.
+* Raw records for one shard are strided in the source. Each shard task packs a
+  bounded subchunk into its reusable scratch and performs one contiguous
+  positional write; a single record larger than the normal subchunk budget is
+  written directly. There is no per-record syscall or platform-specific vectored
+  path. The extra memory copy is the deliberate cost paid to remove concentrated
+  single-inode writes.
 **2. Read.** Fill the destination memory concurrently, in place from the Python
 side (executed on async threads).
 * Committed records are immutable and `length` is published through an atomic.
-  The fixed `meta.zr` and `data.zr` handles require no rollover fd-table
-  synchronization; record I/O itself stays lock free.
-* Every record is read at the offset its table entry records — the record table
+  The fixed metadata and static data handles require no rollover synchronization.
+* Every record first selects `data_{ID % data_shards}.zr`, then reads the
+  shard-local offset its table entry records — the record table
   is addressed by pure arithmetic, so random access is one pread for the
   location and one for the bytes, with no batching assumptions about layout.
   Each lane reads one location and immediately reads/decompresses that record;
@@ -1241,8 +1281,9 @@ budget; lanes for which the runtime cannot reserve concurrency run inline on the
 calling thread. Python configures the process-level budget as
 `max(physical cores, logical cores * 3 / 4)`, using platform topology where
 available.
-* Lanes receive contiguous blocks rather than a strided subset, keeping each
-  worker's reads and writes sequential.
+* Gather lanes receive contiguous request blocks. Append lanes dynamically claim
+  bounded chunks of strided shard ranges; short per-shard reservation locks keep
+  offsets disjoint while positional writes remain concurrent.
 * Each lane creates one zstd context (`ZSTD_CCtx` to write, `ZSTD_DCtx` to
   read) and reuses it across every record it handles, rather than paying that
   setup per record. The dictionary (`ZSTD_CDict`/`ZSTD_DDict`) is immutable, so
@@ -1252,10 +1293,10 @@ available.
 
 **4. File access.**
 * Metadata: one naturally growing `meta.zr`, containing the fixed Header/schema
-  page and loc table.
-* Payload: one naturally growing `data.zr`, accessed concurrently through
-  `readPositionalAll`/`writePositionalAll`. No path depends on filesystem
-  sparse-file support.
+  /tails page and loc table. It is intentionally not sharded because measured
+  write pressure is in payload I/O; one coordinator writes each loc window.
+* Payload: a static list of naturally growing `data_<shard>.zr` files, accessed through
+  `readPositionalAll`/`writePositionalAll`. No path depends on sparse files.
 
 **Execution model.** Opened readers are immutable, so calls on the same reader may
 gather concurrently. Creator appends are synchronous and native Storage