loaderx 2.0.0__tar.gz → 2.0.1__tar.gz

{loaderx-2.0.0/loaderx.egg-info → loaderx-2.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.0.0
+Version: 2.0.1
 Summary: Rebuildable high-performance record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
@@ -121,8 +121,9 @@ Python defines the exact record schema: ``dtype`` plus the dense ``item_shape``,
 or only ``dtype`` for ragged stores. The MsgPack bytes live opaquely in the
 static page at the front of ``meta.zr``; Zig persists them but never interprets
 them. The schema accepts no user metadata. Python selects the geometry and gives
-the private native engine only the runtime record boundaries it needs. Each Ragged record
-carries its shape in an inline little-endian u64 prefix. One native physical
+the private native engine only the runtime record boundaries it needs. Each
+Ragged record carries a u8 rank followed by inline little-endian u64 dimensions.
+One native physical
 engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
 strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
 iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
@@ -461,29 +462,50 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 
 ### Methodology
 
-The results below are one complete qualitative pass from the same checkout on a
-warm page cache. They are not three-run medians: an unexpected result is traced
-separately instead of being hidden by repeated aggregation. Within each store workload every backend
-receives identical source records and random index plans. Loader backends receive
+The results below are one complete run from the same checkout on a warm page
+cache. They are not three-run medians: an unexpected result is traced separately
+instead of being hidden by repeated aggregation. Within each store workload every
+backend receives identical source records. Correctness and timing use independent
+deterministic `zsampler` IID streams. Loader backends receive
 the same source and seed but use their own shipped samplers, so their exact
 permutations differ. Before gather timing, the store benchmark sweeps every record and
-validates every planned result for exact
+validates a fixed number of IID batches for exact
 dtype, shape, order, and values. `logical write` and `logical gather` divide
 uncompressed NumPy payload bytes by elapsed time; they measure bytes accepted or
 returned by the public API, not physical storage bandwidth. Writable containers
-are created before timing. `logical write` times only append/write/assignment
-calls and stops when the final call returns; close, commit, format finalization
-and Zrecord Header publication happen afterward. No backend requests `fsync`.
-Reusable source preparation is outside that timer; in particular, ``zstd_dict``
-trains its standalone dictionary first and byte-record adapters encode input
-before their timed write calls.
+are created before timing. `logical write` starts when the already-generated
+source enters the backend, includes byte encoding, packing, key construction and
+Arrow array construction, and ends after logical commit/finalization returns. No
+backend requests `fsync`, LMDB `env.sync()`, or another stable-media durability
+operation. Reusable source preparation is outside that timer; in particular,
+``zstd_dict`` trains its standalone dictionary first.
+
+The measured Store paths are explicit:
+
+```text
+write: make_data returns -> writer setup [untimed]
+       -> start -> encode/pack -> append/write -> logical finalize -> return -> stop
+       -> resource-only cleanup [untimed]
+
+read:  open -> full warm sweep -> IID correctness stream(seed) [untimed]
+       -> IID timing stream(seed + 1): sampler.next() [untimed]
+       -> start one gather -> public return -> stop
+       -> destroy returned batch [untimed]
+       -> repeat timed calls until their accumulated time is at least 2 seconds
+```
+
+Logical finalize means Zrecord Header publication, LMDB transaction commit, or
+an Arrow/Parquet footer; none of these paths requests stable-media sync.
 Disk size is allocated blocks, not sparse apparent size. `krecords/s` is gather
 record throughput and `p95` is the 95th-percentile latency of one random gather
 batch. Results are comparable within one workload table, not across payload
 distributions or geometries.
-The finalized output is opened read-only before timing. Every random gather plan
-is timed exactly once; output allocation, reads, decompression and reconstruction
-are included, while open and close are not.
+The finalized output is opened read-only before timing. After the warm sweep and
+correctness stream, an independent IID stream draws fresh indices until timed
+gather calls accumulate at least two seconds. IID sampling is uniform with
+replacement, and sampler time is excluded. Output allocation, reads, decompression and
+reconstruction are included, while open, close and destruction after return are
+not.
 Every backend name states its actual codec; the full default set is required
 rather than silently skipped when a package is missing.
 Dense and Ragged use the same CHW RGB image generator, record count, batch plan
@@ -519,24 +541,24 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 7206 MiB/s | 7395 MiB/s | 51.5 | 5.82 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 30 MiB/s | 7138 MiB/s | 49.7 | 6.05 ms | 15.8 MiB | 22.74x |
-| zrecord-raw | 3117 MiB/s | 9590 MiB/s | 66.8 | 4.70 ms | 358.9 MiB | 1.00x |
-| npy-mmap-raw | 2314 MiB/s | 4152 MiB/s | 28.9 | 10.74 ms | 358.9 MiB | 1.00x |
-| hdf5-raw | 2499 MiB/s | 1810 MiB/s | 12.6 | 25.49 ms | 359.0 MiB | 1.00x |
-| hdf5-gzip | 257 MiB/s | 592 MiB/s | 4.1 | 68.37 ms | 26.1 MiB | 13.73x |
-| lmdb-raw | 4291 MiB/s | 3800 MiB/s | 26.5 | 10.44 ms | 361.4 MiB | 0.99x |
-| arrow-ipc-raw | 3070 MiB/s | 2965 MiB/s | 20.7 | 16.28 ms | 358.9 MiB | 1.00x |
-| arrow-ipc-zstd | 700 MiB/s | 171 MiB/s | 1.2 | 254.38 ms | 22.6 MiB | 15.86x |
-| parquet-raw | 1700 MiB/s | 470 MiB/s | 3.3 | 86.56 ms | 358.9 MiB | 1.00x |
-| parquet-zstd | 613 MiB/s | 150 MiB/s | 1.0 | 289.02 ms | 22.6 MiB | 15.86x |
-| arrayrecord-raw | 2012 MiB/s | 1855 MiB/s | 12.9 | 24.12 ms | 359.2 MiB | 1.00x |
-| arrayrecord-zstd | 1024 MiB/s | 1531 MiB/s | 10.7 | 34.26 ms | 25.1 MiB | 14.32x |
-| tiledb-raw | 692 MiB/s | 668 MiB/s | 4.7 | 59.75 ms | 359.0 MiB | 1.00x |
-| tiledb-zstd | 1253 MiB/s | 1478 MiB/s | 10.3 | 27.34 ms | 26.9 MiB | 13.36x |
-
-At 147 KiB per record, Zrecord-raw reaches 9.4 GiB/s and is 2.3x npy-mmap-raw;
-plain zstd gathers at 7.2 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+| zrecord-zstd | 7091 MiB/s | 11770 MiB/s | 82.0 | 3.97 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 34 MiB/s | 11801 MiB/s | 82.2 | 4.12 ms | 15.8 MiB | 22.74x |
+| zrecord-raw | 2708 MiB/s | 15883 MiB/s | 110.6 | 2.65 ms | 358.9 MiB | 1.00x |
+| npy-mmap-raw | 2055 MiB/s | 4738 MiB/s | 33.0 | 11.00 ms | 358.9 MiB | 1.00x |
+| hdf5-raw | 2360 MiB/s | 1948 MiB/s | 13.6 | 27.31 ms | 359.0 MiB | 1.00x |
+| hdf5-gzip | 281 MiB/s | 641 MiB/s | 4.5 | 65.88 ms | 26.1 MiB | 13.73x |
+| lmdb-raw | 1714 MiB/s | 4375 MiB/s | 30.5 | 10.83 ms | 361.4 MiB | 0.99x |
+| arrow-ipc-raw | 1817 MiB/s | 3480 MiB/s | 24.2 | 14.26 ms | 358.9 MiB | 1.00x |
+| arrow-ipc-zstd | 611 MiB/s | 177 MiB/s | 1.2 | 228.63 ms | 22.6 MiB | 15.86x |
+| parquet-raw | 1341 MiB/s | 487 MiB/s | 3.4 | 87.83 ms | 358.9 MiB | 1.00x |
+| parquet-zstd | 644 MiB/s | 162 MiB/s | 1.1 | 246.21 ms | 22.6 MiB | 15.86x |
+| arrayrecord-raw | 1543 MiB/s | 2107 MiB/s | 14.7 | 21.69 ms | 359.2 MiB | 1.00x |
+| arrayrecord-zstd | 917 MiB/s | 1586 MiB/s | 11.0 | 30.72 ms | 25.1 MiB | 14.32x |
+| tiledb-raw | 712 MiB/s | 688 MiB/s | 4.8 | 58.83 ms | 359.0 MiB | 1.00x |
+| tiledb-zstd | 1572 MiB/s | 1525 MiB/s | 10.6 | 26.80 ms | 26.9 MiB | 13.36x |
+
+At 147 KiB per record, Zrecord-raw reaches 15.5 GiB/s and is 3.4x npy-mmap-raw;
+plain zstd gathers at 11.5 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -554,23 +576,23 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2220 MiB/s | 5666 MiB/s | 34.2 | 8.41 ms | 26.9 MiB | 15.52x |
-| zrecord-zstdict | 28 MiB/s | 5681 MiB/s | 34.3 | 8.55 ms | 17.7 MiB | 23.58x |
-| zrecord-raw | 1460 MiB/s | 6806 MiB/s | 41.1 | 7.58 ms | 417.2 MiB | 1.00x |
-| hdf5-raw | 1610 MiB/s | 1108 MiB/s | 6.7 | 45.80 ms | 418.0 MiB | 1.00x |
-| hdf5-gzip | 229 MiB/s | 114 MiB/s | 0.7 | 415.95 ms | 29.9 MiB | 13.94x |
-| lmdb-raw | 4126 MiB/s | 6837 MiB/s | 41.2 | 8.14 ms | 422.2 MiB | 0.99x |
-| arrow-ipc-raw | 3213 MiB/s | 4251 MiB/s | 25.6 | 12.90 ms | 417.2 MiB | 1.00x |
-| arrow-ipc-zstd | 706 MiB/s | 177 MiB/s | 1.1 | 266.37 ms | 25.9 MiB | 16.10x |
-| parquet-raw | 2219 MiB/s | 505 MiB/s | 3.0 | 92.79 ms | 417.2 MiB | 1.00x |
-| parquet-zstd | 728 MiB/s | 158 MiB/s | 1.0 | 296.60 ms | 25.9 MiB | 16.10x |
-| arrayrecord-raw | 1650 MiB/s | 2571 MiB/s | 15.5 | 18.34 ms | 417.6 MiB | 1.00x |
-| arrayrecord-zstd | 941 MiB/s | 1420 MiB/s | 8.6 | 37.14 ms | 27.2 MiB | 15.31x |
-| tiledb-raw | 443 MiB/s | 49 MiB/s | 0.3 | 920.09 ms | 417.2 MiB | 1.00x |
-| tiledb-zstd | 727 MiB/s | 124 MiB/s | 0.7 | 375.98 ms | 27.0 MiB | 15.46x |
-
-Zrecord-raw and LMDB are effectively tied in this pass; Arrow IPC is
-the strongest raw typed-file alternative. Zrecord-zstd delivers 5.5 GiB/s of
+| zrecord-zstd | 2502 MiB/s | 10071 MiB/s | 60.5 | 5.06 ms | 26.9 MiB | 15.52x |
+| zrecord-zstdict | 36 MiB/s | 10930 MiB/s | 65.6 | 4.62 ms | 17.7 MiB | 23.58x |
+| zrecord-raw | 1661 MiB/s | 13056 MiB/s | 78.4 | 3.84 ms | 417.2 MiB | 1.00x |
+| hdf5-raw | 1578 MiB/s | 1359 MiB/s | 8.2 | 37.26 ms | 418.0 MiB | 1.00x |
+| hdf5-gzip | 243 MiB/s | 130 MiB/s | 0.8 | 356.26 ms | 29.9 MiB | 13.94x |
+| lmdb-raw | 1790 MiB/s | 7412 MiB/s | 44.5 | 7.48 ms | 422.2 MiB | 0.99x |
+| arrow-ipc-raw | 1572 MiB/s | 4070 MiB/s | 24.4 | 14.56 ms | 417.2 MiB | 1.00x |
+| arrow-ipc-zstd | 620 MiB/s | 208 MiB/s | 1.2 | 232.74 ms | 25.9 MiB | 16.10x |
+| parquet-raw | 889 MiB/s | 481 MiB/s | 2.9 | 93.87 ms | 417.2 MiB | 1.00x |
+| parquet-zstd | 498 MiB/s | 184 MiB/s | 1.1 | 251.14 ms | 25.9 MiB | 16.10x |
+| arrayrecord-raw | 1644 MiB/s | 2642 MiB/s | 15.9 | 18.22 ms | 417.6 MiB | 1.00x |
+| arrayrecord-zstd | 829 MiB/s | 1556 MiB/s | 9.3 | 37.43 ms | 27.2 MiB | 15.31x |
+| tiledb-raw | 411 MiB/s | 55 MiB/s | 0.3 | 806.95 ms | 417.2 MiB | 1.00x |
+| tiledb-zstd | 713 MiB/s | 151 MiB/s | 0.9 | 307.58 ms | 27.0 MiB | 15.46x |
+
+Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
+delivers 9.8 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
@@ -593,44 +615,45 @@ combines fragments shorter than 16 tokens, and splits records at 512 tokens into
 
 The Dense workload ignores text boundaries and packs the stream into 200,000
 fixed `int32[512]` records: 2 KiB per record and 390.6 MiB logical payload.
-Each of 100 random batches gathers 256 records.
+Each IID batch gathers 256 records; fresh draws continue until timed gathers
+accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 346 MiB/s | 1681 MiB/s | 860.9 | 0.38 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 41 MiB/s | 1892 MiB/s | 968.8 | 0.33 ms | 153.8 MiB | 2.54x |
-| zrecord-raw | 2477 MiB/s | 6261 MiB/s | 3205.5 | 0.11 ms | 393.7 MiB | 0.99x |
-| npy-mmap-raw | 2024 MiB/s | 7897 MiB/s | 4043.3 | 0.08 ms | 390.6 MiB | 1.00x |
-| lmdb-raw | 1022 MiB/s | 1066 MiB/s | 545.9 | 0.53 ms | 786.3 MiB | 0.50x |
-| arrow-ipc-raw | 2744 MiB/s | 207 MiB/s | 105.8 | 2.60 ms | 390.8 MiB | 1.00x |
-| arrayrecord-raw | 905 MiB/s | 149 MiB/s | 76.4 | 6.33 ms | 401.4 MiB | 0.97x |
-| arrayrecord-zstd | 114 MiB/s | 160 MiB/s | 82.0 | 4.83 ms | 201.3 MiB | 1.94x |
+| zrecord-zstd | 456 MiB/s | 2047 MiB/s | 1047.9 | 0.32 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 54 MiB/s | 2393 MiB/s | 1225.3 | 0.28 ms | 153.8 MiB | 2.54x |
+| zrecord-raw | 2550 MiB/s | 7688 MiB/s | 3936.3 | 0.08 ms | 393.7 MiB | 0.99x |
+| npy-mmap-raw | 2139 MiB/s | 10709 MiB/s | 5482.9 | 0.07 ms | 390.6 MiB | 1.00x |
+| lmdb-raw | 721 MiB/s | 1166 MiB/s | 597.0 | 0.55 ms | 786.3 MiB | 0.50x |
+| arrow-ipc-raw | 2335 MiB/s | 231 MiB/s | 118.2 | 2.53 ms | 390.8 MiB | 1.00x |
+| arrayrecord-raw | 847 MiB/s | 162 MiB/s | 83.0 | 4.98 ms | 401.4 MiB | 0.97x |
+| arrayrecord-zstd | 124 MiB/s | 133 MiB/s | 68.3 | 4.68 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.21 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.94 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-0.86–0.97 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.05–1.23 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
 
 The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28,
 median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record,
-100-batch random plan and every backend must return ordered `list[np.ndarray]`
+random plan and every backend must return ordered `list[np.ndarray]`
 with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 105 MiB/s | 225 MiB/s | 426.5 | 0.76 ms | 67.8 MiB | 1.56x |
-| zrecord-zstdict | 37 MiB/s | 277 MiB/s | 526.3 | 0.63 ms | 49.4 MiB | 2.14x |
-| zrecord-raw | 457 MiB/s | 325 MiB/s | 616.1 | 0.46 ms | 112.0 MiB | 0.95x |
-| lmdb-raw | 887 MiB/s | 106 MiB/s | 200.7 | 1.34 ms | 153.0 MiB | 0.69x |
-| arrow-ipc-raw | 2230 MiB/s | 39 MiB/s | 74.1 | 4.48 ms | 109.9 MiB | 0.96x |
-| arrayrecord-raw | 399 MiB/s | 24 MiB/s | 45.7 | 8.34 ms | 118.8 MiB | 0.89x |
-| arrayrecord-zstd | 59 MiB/s | 27 MiB/s | 51.1 | 7.79 ms | 76.2 MiB | 1.39x |
+| zrecord-zstd | 114 MiB/s | 291 MiB/s | 550.1 | 0.59 ms | 67.5 MiB | 1.57x |
+| zrecord-zstdict | 47 MiB/s | 325 MiB/s | 614.3 | 0.53 ms | 49.2 MiB | 2.15x |
+| zrecord-raw | 571 MiB/s | 352 MiB/s | 665.5 | 0.50 ms | 110.7 MiB | 0.96x |
+| lmdb-raw | 341 MiB/s | 116 MiB/s | 218.9 | 1.32 ms | 153.0 MiB | 0.69x |
+| arrow-ipc-raw | 567 MiB/s | 44 MiB/s | 84.1 | 3.66 ms | 109.9 MiB | 0.96x |
+| arrayrecord-raw | 290 MiB/s | 28 MiB/s | 53.5 | 7.22 ms | 118.8 MiB | 0.89x |
+| arrayrecord-zstd | 61 MiB/s | 34 MiB/s | 64.1 | 5.12 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 427–616 krecords/s with less than 0.8 ms p95;
+Zrecord's three codecs return 550–666 krecords/s with less than 0.6 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -773,13 +796,13 @@ explicit solutions rather than silently changing the default transform.
 **What these numbers do not claim.** Everything runs with a warm page cache: this
 measures the access path, not cold storage or disk. `disk` is allocated
 blocks, and zrecord files grow to their written frontier. `logical write`
-measures append/write calls after writer setup and before close/finalization; it
+measures source adaptation through logical finalization after writer setup; it
 does not benchmark durability, stronger transactional guarantees, or reusable
 dictionary training. Arrow IPC and Parquet use 256-record groups, so random
 batches pay their real group-level read amplification. Ragged Zrecord arrays are
 views into one batch allocation, while most byte-store adapters return
-independent copies. This is a single
-qualitative pass —
+independent copies. This is a single benchmark run, while each Store read path
+accumulates at least two timed seconds —
 the µs-scale sampler timings and the loader `batches/s` fluctuate with box load
 (on these 12 cores the compressed loader trails the raw one by about 25%), so
 treat the absolute numbers as ballpark and the cross-backend margins as the

{loaderx-2.0.0 → loaderx-2.0.1}/README.md

@@ -96,8 +96,9 @@ Python defines the exact record schema: ``dtype`` plus the dense ``item_shape``,
 or only ``dtype`` for ragged stores. The MsgPack bytes live opaquely in the
 static page at the front of ``meta.zr``; Zig persists them but never interprets
 them. The schema accepts no user metadata. Python selects the geometry and gives
-the private native engine only the runtime record boundaries it needs. Each Ragged record
-carries its shape in an inline little-endian u64 prefix. One native physical
+the private native engine only the runtime record boundaries it needs. Each
+Ragged record carries a u8 rank followed by inline little-endian u64 dimensions.
+One native physical
 engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
 strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
 iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
@@ -436,29 +437,50 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 
 ### Methodology
 
-The results below are one complete qualitative pass from the same checkout on a
-warm page cache. They are not three-run medians: an unexpected result is traced
-separately instead of being hidden by repeated aggregation. Within each store workload every backend
-receives identical source records and random index plans. Loader backends receive
+The results below are one complete run from the same checkout on a warm page
+cache. They are not three-run medians: an unexpected result is traced separately
+instead of being hidden by repeated aggregation. Within each store workload every
+backend receives identical source records. Correctness and timing use independent
+deterministic `zsampler` IID streams. Loader backends receive
 the same source and seed but use their own shipped samplers, so their exact
 permutations differ. Before gather timing, the store benchmark sweeps every record and
-validates every planned result for exact
+validates a fixed number of IID batches for exact
 dtype, shape, order, and values. `logical write` and `logical gather` divide
 uncompressed NumPy payload bytes by elapsed time; they measure bytes accepted or
 returned by the public API, not physical storage bandwidth. Writable containers
-are created before timing. `logical write` times only append/write/assignment
-calls and stops when the final call returns; close, commit, format finalization
-and Zrecord Header publication happen afterward. No backend requests `fsync`.
-Reusable source preparation is outside that timer; in particular, ``zstd_dict``
-trains its standalone dictionary first and byte-record adapters encode input
-before their timed write calls.
+are created before timing. `logical write` starts when the already-generated
+source enters the backend, includes byte encoding, packing, key construction and
+Arrow array construction, and ends after logical commit/finalization returns. No
+backend requests `fsync`, LMDB `env.sync()`, or another stable-media durability
+operation. Reusable source preparation is outside that timer; in particular,
+``zstd_dict`` trains its standalone dictionary first.
+
+The measured Store paths are explicit:
+
+```text
+write: make_data returns -> writer setup [untimed]
+       -> start -> encode/pack -> append/write -> logical finalize -> return -> stop
+       -> resource-only cleanup [untimed]
+
+read:  open -> full warm sweep -> IID correctness stream(seed) [untimed]
+       -> IID timing stream(seed + 1): sampler.next() [untimed]
+       -> start one gather -> public return -> stop
+       -> destroy returned batch [untimed]
+       -> repeat timed calls until their accumulated time is at least 2 seconds
+```
+
+Logical finalize means Zrecord Header publication, LMDB transaction commit, or
+an Arrow/Parquet footer; none of these paths requests stable-media sync.
 Disk size is allocated blocks, not sparse apparent size. `krecords/s` is gather
 record throughput and `p95` is the 95th-percentile latency of one random gather
 batch. Results are comparable within one workload table, not across payload
 distributions or geometries.
-The finalized output is opened read-only before timing. Every random gather plan
-is timed exactly once; output allocation, reads, decompression and reconstruction
-are included, while open and close are not.
+The finalized output is opened read-only before timing. After the warm sweep and
+correctness stream, an independent IID stream draws fresh indices until timed
+gather calls accumulate at least two seconds. IID sampling is uniform with
+replacement, and sampler time is excluded. Output allocation, reads, decompression and
+reconstruction are included, while open, close and destruction after return are
+not.
 Every backend name states its actual codec; the full default set is required
 rather than silently skipped when a package is missing.
 Dense and Ragged use the same CHW RGB image generator, record count, batch plan
@@ -494,24 +516,24 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 7206 MiB/s | 7395 MiB/s | 51.5 | 5.82 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 30 MiB/s | 7138 MiB/s | 49.7 | 6.05 ms | 15.8 MiB | 22.74x |
-| zrecord-raw | 3117 MiB/s | 9590 MiB/s | 66.8 | 4.70 ms | 358.9 MiB | 1.00x |
-| npy-mmap-raw | 2314 MiB/s | 4152 MiB/s | 28.9 | 10.74 ms | 358.9 MiB | 1.00x |
-| hdf5-raw | 2499 MiB/s | 1810 MiB/s | 12.6 | 25.49 ms | 359.0 MiB | 1.00x |
-| hdf5-gzip | 257 MiB/s | 592 MiB/s | 4.1 | 68.37 ms | 26.1 MiB | 13.73x |
-| lmdb-raw | 4291 MiB/s | 3800 MiB/s | 26.5 | 10.44 ms | 361.4 MiB | 0.99x |
-| arrow-ipc-raw | 3070 MiB/s | 2965 MiB/s | 20.7 | 16.28 ms | 358.9 MiB | 1.00x |
-| arrow-ipc-zstd | 700 MiB/s | 171 MiB/s | 1.2 | 254.38 ms | 22.6 MiB | 15.86x |
-| parquet-raw | 1700 MiB/s | 470 MiB/s | 3.3 | 86.56 ms | 358.9 MiB | 1.00x |
-| parquet-zstd | 613 MiB/s | 150 MiB/s | 1.0 | 289.02 ms | 22.6 MiB | 15.86x |
-| arrayrecord-raw | 2012 MiB/s | 1855 MiB/s | 12.9 | 24.12 ms | 359.2 MiB | 1.00x |
-| arrayrecord-zstd | 1024 MiB/s | 1531 MiB/s | 10.7 | 34.26 ms | 25.1 MiB | 14.32x |
-| tiledb-raw | 692 MiB/s | 668 MiB/s | 4.7 | 59.75 ms | 359.0 MiB | 1.00x |
-| tiledb-zstd | 1253 MiB/s | 1478 MiB/s | 10.3 | 27.34 ms | 26.9 MiB | 13.36x |
-
-At 147 KiB per record, Zrecord-raw reaches 9.4 GiB/s and is 2.3x npy-mmap-raw;
-plain zstd gathers at 7.2 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+| zrecord-zstd | 7091 MiB/s | 11770 MiB/s | 82.0 | 3.97 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 34 MiB/s | 11801 MiB/s | 82.2 | 4.12 ms | 15.8 MiB | 22.74x |
+| zrecord-raw | 2708 MiB/s | 15883 MiB/s | 110.6 | 2.65 ms | 358.9 MiB | 1.00x |
+| npy-mmap-raw | 2055 MiB/s | 4738 MiB/s | 33.0 | 11.00 ms | 358.9 MiB | 1.00x |
+| hdf5-raw | 2360 MiB/s | 1948 MiB/s | 13.6 | 27.31 ms | 359.0 MiB | 1.00x |
+| hdf5-gzip | 281 MiB/s | 641 MiB/s | 4.5 | 65.88 ms | 26.1 MiB | 13.73x |
+| lmdb-raw | 1714 MiB/s | 4375 MiB/s | 30.5 | 10.83 ms | 361.4 MiB | 0.99x |
+| arrow-ipc-raw | 1817 MiB/s | 3480 MiB/s | 24.2 | 14.26 ms | 358.9 MiB | 1.00x |
+| arrow-ipc-zstd | 611 MiB/s | 177 MiB/s | 1.2 | 228.63 ms | 22.6 MiB | 15.86x |
+| parquet-raw | 1341 MiB/s | 487 MiB/s | 3.4 | 87.83 ms | 358.9 MiB | 1.00x |
+| parquet-zstd | 644 MiB/s | 162 MiB/s | 1.1 | 246.21 ms | 22.6 MiB | 15.86x |
+| arrayrecord-raw | 1543 MiB/s | 2107 MiB/s | 14.7 | 21.69 ms | 359.2 MiB | 1.00x |
+| arrayrecord-zstd | 917 MiB/s | 1586 MiB/s | 11.0 | 30.72 ms | 25.1 MiB | 14.32x |
+| tiledb-raw | 712 MiB/s | 688 MiB/s | 4.8 | 58.83 ms | 359.0 MiB | 1.00x |
+| tiledb-zstd | 1572 MiB/s | 1525 MiB/s | 10.6 | 26.80 ms | 26.9 MiB | 13.36x |
+
+At 147 KiB per record, Zrecord-raw reaches 15.5 GiB/s and is 3.4x npy-mmap-raw;
+plain zstd gathers at 11.5 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -529,23 +551,23 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2220 MiB/s | 5666 MiB/s | 34.2 | 8.41 ms | 26.9 MiB | 15.52x |
-| zrecord-zstdict | 28 MiB/s | 5681 MiB/s | 34.3 | 8.55 ms | 17.7 MiB | 23.58x |
-| zrecord-raw | 1460 MiB/s | 6806 MiB/s | 41.1 | 7.58 ms | 417.2 MiB | 1.00x |
-| hdf5-raw | 1610 MiB/s | 1108 MiB/s | 6.7 | 45.80 ms | 418.0 MiB | 1.00x |
-| hdf5-gzip | 229 MiB/s | 114 MiB/s | 0.7 | 415.95 ms | 29.9 MiB | 13.94x |
-| lmdb-raw | 4126 MiB/s | 6837 MiB/s | 41.2 | 8.14 ms | 422.2 MiB | 0.99x |
-| arrow-ipc-raw | 3213 MiB/s | 4251 MiB/s | 25.6 | 12.90 ms | 417.2 MiB | 1.00x |
-| arrow-ipc-zstd | 706 MiB/s | 177 MiB/s | 1.1 | 266.37 ms | 25.9 MiB | 16.10x |
-| parquet-raw | 2219 MiB/s | 505 MiB/s | 3.0 | 92.79 ms | 417.2 MiB | 1.00x |
-| parquet-zstd | 728 MiB/s | 158 MiB/s | 1.0 | 296.60 ms | 25.9 MiB | 16.10x |
-| arrayrecord-raw | 1650 MiB/s | 2571 MiB/s | 15.5 | 18.34 ms | 417.6 MiB | 1.00x |
-| arrayrecord-zstd | 941 MiB/s | 1420 MiB/s | 8.6 | 37.14 ms | 27.2 MiB | 15.31x |
-| tiledb-raw | 443 MiB/s | 49 MiB/s | 0.3 | 920.09 ms | 417.2 MiB | 1.00x |
-| tiledb-zstd | 727 MiB/s | 124 MiB/s | 0.7 | 375.98 ms | 27.0 MiB | 15.46x |
-
-Zrecord-raw and LMDB are effectively tied in this pass; Arrow IPC is
-the strongest raw typed-file alternative. Zrecord-zstd delivers 5.5 GiB/s of
+| zrecord-zstd | 2502 MiB/s | 10071 MiB/s | 60.5 | 5.06 ms | 26.9 MiB | 15.52x |
+| zrecord-zstdict | 36 MiB/s | 10930 MiB/s | 65.6 | 4.62 ms | 17.7 MiB | 23.58x |
+| zrecord-raw | 1661 MiB/s | 13056 MiB/s | 78.4 | 3.84 ms | 417.2 MiB | 1.00x |
+| hdf5-raw | 1578 MiB/s | 1359 MiB/s | 8.2 | 37.26 ms | 418.0 MiB | 1.00x |
+| hdf5-gzip | 243 MiB/s | 130 MiB/s | 0.8 | 356.26 ms | 29.9 MiB | 13.94x |
+| lmdb-raw | 1790 MiB/s | 7412 MiB/s | 44.5 | 7.48 ms | 422.2 MiB | 0.99x |
+| arrow-ipc-raw | 1572 MiB/s | 4070 MiB/s | 24.4 | 14.56 ms | 417.2 MiB | 1.00x |
+| arrow-ipc-zstd | 620 MiB/s | 208 MiB/s | 1.2 | 232.74 ms | 25.9 MiB | 16.10x |
+| parquet-raw | 889 MiB/s | 481 MiB/s | 2.9 | 93.87 ms | 417.2 MiB | 1.00x |
+| parquet-zstd | 498 MiB/s | 184 MiB/s | 1.1 | 251.14 ms | 25.9 MiB | 16.10x |
+| arrayrecord-raw | 1644 MiB/s | 2642 MiB/s | 15.9 | 18.22 ms | 417.6 MiB | 1.00x |
+| arrayrecord-zstd | 829 MiB/s | 1556 MiB/s | 9.3 | 37.43 ms | 27.2 MiB | 15.31x |
+| tiledb-raw | 411 MiB/s | 55 MiB/s | 0.3 | 806.95 ms | 417.2 MiB | 1.00x |
+| tiledb-zstd | 713 MiB/s | 151 MiB/s | 0.9 | 307.58 ms | 27.0 MiB | 15.46x |
+
+Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
+delivers 9.8 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
@@ -568,44 +590,45 @@ combines fragments shorter than 16 tokens, and splits records at 512 tokens into
 
 The Dense workload ignores text boundaries and packs the stream into 200,000
 fixed `int32[512]` records: 2 KiB per record and 390.6 MiB logical payload.
-Each of 100 random batches gathers 256 records.
+Each IID batch gathers 256 records; fresh draws continue until timed gathers
+accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 346 MiB/s | 1681 MiB/s | 860.9 | 0.38 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 41 MiB/s | 1892 MiB/s | 968.8 | 0.33 ms | 153.8 MiB | 2.54x |
-| zrecord-raw | 2477 MiB/s | 6261 MiB/s | 3205.5 | 0.11 ms | 393.7 MiB | 0.99x |
-| npy-mmap-raw | 2024 MiB/s | 7897 MiB/s | 4043.3 | 0.08 ms | 390.6 MiB | 1.00x |
-| lmdb-raw | 1022 MiB/s | 1066 MiB/s | 545.9 | 0.53 ms | 786.3 MiB | 0.50x |
-| arrow-ipc-raw | 2744 MiB/s | 207 MiB/s | 105.8 | 2.60 ms | 390.8 MiB | 1.00x |
-| arrayrecord-raw | 905 MiB/s | 149 MiB/s | 76.4 | 6.33 ms | 401.4 MiB | 0.97x |
-| arrayrecord-zstd | 114 MiB/s | 160 MiB/s | 82.0 | 4.83 ms | 201.3 MiB | 1.94x |
+| zrecord-zstd | 456 MiB/s | 2047 MiB/s | 1047.9 | 0.32 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 54 MiB/s | 2393 MiB/s | 1225.3 | 0.28 ms | 153.8 MiB | 2.54x |
+| zrecord-raw | 2550 MiB/s | 7688 MiB/s | 3936.3 | 0.08 ms | 393.7 MiB | 0.99x |
+| npy-mmap-raw | 2139 MiB/s | 10709 MiB/s | 5482.9 | 0.07 ms | 390.6 MiB | 1.00x |
+| lmdb-raw | 721 MiB/s | 1166 MiB/s | 597.0 | 0.55 ms | 786.3 MiB | 0.50x |
+| arrow-ipc-raw | 2335 MiB/s | 231 MiB/s | 118.2 | 2.53 ms | 390.8 MiB | 1.00x |
+| arrayrecord-raw | 847 MiB/s | 162 MiB/s | 83.0 | 4.98 ms | 401.4 MiB | 0.97x |
+| arrayrecord-zstd | 124 MiB/s | 133 MiB/s | 68.3 | 4.68 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.21 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.94 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-0.86–0.97 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.05–1.23 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
 
 The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28,
 median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record,
-100-batch random plan and every backend must return ordered `list[np.ndarray]`
+random plan and every backend must return ordered `list[np.ndarray]`
 with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 105 MiB/s | 225 MiB/s | 426.5 | 0.76 ms | 67.8 MiB | 1.56x |
-| zrecord-zstdict | 37 MiB/s | 277 MiB/s | 526.3 | 0.63 ms | 49.4 MiB | 2.14x |
-| zrecord-raw | 457 MiB/s | 325 MiB/s | 616.1 | 0.46 ms | 112.0 MiB | 0.95x |
-| lmdb-raw | 887 MiB/s | 106 MiB/s | 200.7 | 1.34 ms | 153.0 MiB | 0.69x |
-| arrow-ipc-raw | 2230 MiB/s | 39 MiB/s | 74.1 | 4.48 ms | 109.9 MiB | 0.96x |
-| arrayrecord-raw | 399 MiB/s | 24 MiB/s | 45.7 | 8.34 ms | 118.8 MiB | 0.89x |
-| arrayrecord-zstd | 59 MiB/s | 27 MiB/s | 51.1 | 7.79 ms | 76.2 MiB | 1.39x |
+| zrecord-zstd | 114 MiB/s | 291 MiB/s | 550.1 | 0.59 ms | 67.5 MiB | 1.57x |
+| zrecord-zstdict | 47 MiB/s | 325 MiB/s | 614.3 | 0.53 ms | 49.2 MiB | 2.15x |
+| zrecord-raw | 571 MiB/s | 352 MiB/s | 665.5 | 0.50 ms | 110.7 MiB | 0.96x |
+| lmdb-raw | 341 MiB/s | 116 MiB/s | 218.9 | 1.32 ms | 153.0 MiB | 0.69x |
+| arrow-ipc-raw | 567 MiB/s | 44 MiB/s | 84.1 | 3.66 ms | 109.9 MiB | 0.96x |
+| arrayrecord-raw | 290 MiB/s | 28 MiB/s | 53.5 | 7.22 ms | 118.8 MiB | 0.89x |
+| arrayrecord-zstd | 61 MiB/s | 34 MiB/s | 64.1 | 5.12 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 427–616 krecords/s with less than 0.8 ms p95;
+Zrecord's three codecs return 550–666 krecords/s with less than 0.6 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -748,13 +771,13 @@ explicit solutions rather than silently changing the default transform.
 **What these numbers do not claim.** Everything runs with a warm page cache: this
 measures the access path, not cold storage or disk. `disk` is allocated
 blocks, and zrecord files grow to their written frontier. `logical write`
-measures append/write calls after writer setup and before close/finalization; it
+measures source adaptation through logical finalization after writer setup; it
 does not benchmark durability, stronger transactional guarantees, or reusable
 dictionary training. Arrow IPC and Parquet use 256-record groups, so random
 batches pay their real group-level read amplification. Ragged Zrecord arrays are
 views into one batch allocation, while most byte-store adapters return
-independent copies. This is a single
-qualitative pass —
+independent copies. This is a single benchmark run, while each Store read path
+accumulates at least two timed seconds —
 the µs-scale sampler timings and the loader `batches/s` fluctuate with box load
 (on these 12 cores the compressed loader trails the raw one by about 25%), so
 treat the absolute numbers as ballpark and the cross-backend margins as the

{loaderx-2.0.0 → loaderx-2.0.1}/build.zig.zon

@@ -1,6 +1,6 @@
 .{
     .name = .loaderx,
-    .version = "2.0.0",
+    .version = "2.0.1",
     .fingerprint = 0x350591788e45d1d3,
     .dependencies = .{},
     .paths = .{

{loaderx-2.0.0 → loaderx-2.0.1}/loaderx/__init__.py

@@ -15,4 +15,4 @@ boundaries stay visible:
 step-based sampler + prefetch pipeline over them.
 """
 
-__version__ = "2.0.0"
+__version__ = "2.0.1"

{loaderx-2.0.0 → loaderx-2.0.1}/loaderx/zrecord.py

@@ -41,12 +41,9 @@ from ._store import MAX_RECORD_LEN, MAX_SCHEMA_LEN, ZrecordError, _Store
 
 __all__ = ["Dense", "Ragged", "ZrecordError", "DICT_TIERS"]
 
-_RANK_UNPACKER = struct.Struct("<Q")
 _SHAPE_UNPACKERS = tuple(
     struct.Struct("<" + "Q" * rank) for rank in range(9)
 )
-# The on-disk rank is u64; this is the active NumPy runtime's ndarray limit,
-# not a Zrecord format limit.
 _NUMPY_MAX_NDIMS = 64 if int(np.__version__.split(".", 1)[0]) >= 2 else 32
 _MAX_UINT64 = np.iinfo(np.uint64).max
 _MAX_INTP = np.iinfo(np.intp).max
@@ -320,7 +317,7 @@ class Ragged(_Base):
     """Variable-length records over a zrecord container.
 
     ``dtype`` is unified across the whole store (the static schema); each record
-    keeps its own shape in an inline little-endian u64 prefix. Reads
+    keeps its own shape as a u8 rank followed by little-endian u64 dimensions. Reads
     restore every record to exactly the shape it was written with, so a batch
     is a list of the original arrays — ``ds[i]`` is one record.
 
@@ -372,7 +369,7 @@ class Ragged(_Base):
             rank = len(shape)
             packer = packers.get(rank)
             if packer is None:
-                packer = packers[rank] = struct.Struct("<" + "Q" * (rank + 1))
+                packer = packers[rank] = struct.Struct("<B" + "Q" * rank)
             encoded_bytes = packer.size + payload_bytes
             if encoded_bytes > MAX_RECORD_LEN:
                 raise ValueError(
@@ -408,17 +405,17 @@ class Ragged(_Base):
         for offset, length in zip(offsets, lengths):
             start = int(offset)
             length = int(length)
-            if length <= _RANK_UNPACKER.size:
+            if length <= 1:
                 raise ValueError("corrupt ragged record shape header")
-            rank = _RANK_UNPACKER.unpack_from(buf, start)[0]
-            header_bytes = 8 * (rank + 1)
+            rank = int(buf[start])
+            header_bytes = 1 + 8 * rank
             if rank > _NUMPY_MAX_NDIMS or header_bytes >= length:
                 raise ValueError("corrupt ragged record shape header")
             if rank < len(_SHAPE_UNPACKERS):
                 unpacker = _SHAPE_UNPACKERS[rank]
             else:
                 unpacker = struct.Struct("<" + "Q" * rank)
-            shape = unpacker.unpack_from(buf, start + 8)
+            shape = unpacker.unpack_from(buf, start + 1)
             payload_bytes = length - header_bytes
             itemsize = self.dtype.itemsize
             if not itemsize or payload_bytes % itemsize: