PyPI - loaderx - Versions diffs - 2.0.1__tar.gz → 2.0.7__tar.gz - Mend

loaderx 2.0.1tar.gz → 2.0.7tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (107) hide show

{loaderx-2.0.1/loaderx.egg-info → loaderx-2.0.7}/PKG-INFO RENAMED Viewed

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.0.1
-Summary: Rebuildable high-performance record containers
+Version: 2.0.7
+Summary: Rebuildable high-performance ordered record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
 Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
@@ -24,10 +24,11 @@ Requires-Dist: msgpack
 Dynamic: license-file
 # Loaderx
-Zrecord is a rebuildable high-performance on-disk record container built from
-authoritative source data and scripts. ``create`` is append-only, ``close``
-publishes the container Header, and ``open`` is read-only. To change
-content, rebuild it at a new path.
+Zrecord is a rebuildable, typed, ordered record sequence built from authoritative
+source data and scripts. A creator consumes records in append order, ``close``
+publishes one immutable container, and readers support both sequential slicing
+and indexed gather without changing record identity. To change content or order,
+rebuild it at a new path.
 Zrecord is the typed on-disk container; Loaderx is the sampler and prefetch
 loader that consumes Zrecord streams. They currently ship together while both
@@ -54,6 +55,9 @@ loaderx is built around several core principles:
    dense stream stacks into one array per batch; a ragged one comes back as a
    list. Neither is padded, and equal length is never treated as a special case
    of variable length.
+6. **Logical IDs are stable sequence positions.** Native chunks may complete in
+   any physical order, but append input order defines ``0..N-1`` and a published
+   container never deletes, compacts, updates, or renumbers those records.
 ## 设计文档
@@ -149,9 +153,10 @@ ds[0, 5, 2]                          # (3, 2, 4) — shape from the persisted sc
 ds.close()
 ```
-A store is a collection of records, not an ndarray, so ``ds[0, 5, 2]`` is a
-record set — never ``ds[0][5][2]``. A scalar selects one record; indices must be
-in ``0..len(ds)-1``. The ragged example below reads the same way.
+A store is an ordered sequence of records, not an ndarray, so ``ds[0, 5, 2]``
+selects sequence positions 0, 5 and 2 — never ``ds[0][5][2]``. A scalar selects
+one record; indices must be in ``0..len(ds)-1``. The ragged example below reads
+the same way.
 ``Ragged`` is the **ragged** contract for variable-length records. It is a
 separate contract: :class:`Ragged` hands back a list of arrays, so a
@@ -179,6 +184,34 @@ for i, r in enumerate(records):
     padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop
 ```
+### Ordered sequences and build streams
+Logical ID is the stable sequence position. One ``append`` preserves every
+record in its input order; successive calls from one producer extend that
+sequence. Native lanes may reserve and write payload chunks in a different
+physical completion order, but each ``RecordLoc`` is installed in its original
+logical slot, so physical scheduling never changes ``ds[i]``. After ``close``,
+the sequence is immutable: there is no delete, compact, update, or reopen-append
+operation that can renumber it.
+This makes a creator a finite **build stream** and its published result an
+immutable sequence. It is not a live log: readers do not tail a writer, and an
+unbounded producer must choose a finite publication boundary. A large sequence
+can be consumed in bounded ordered batches with ordinary slices:
+```python
+with Dense.open("events") as events:
+    for start in range(0, len(events), 1024):
+        batch = events[start:start + 1024]
+        consume(batch)
+```
+Ragged creators may consume a one-pass iterable, while Dense creators take
+explicit ndarray batches. Concurrent append calls are serialized by the native
+writer lock, but their batch order is the lock-acquisition order rather than
+Python call-start order; sequence builders that require an external temporal
+order should use one producer or order batches before append.
 A ``DataLoader`` dynamically composes a dict of dense and ragged streams.
 Collation is the ``transform`` — a batch dict in, a batch dict out:
@@ -225,8 +258,9 @@ warmup belongs outside build or loader benchmark timing.
 ### Creating containers
-``Dense.create`` and ``Ragged.create`` return append-only writers. ``append`` is
-explicit — one container of records becomes one native batch.
+``Dense.create`` and ``Ragged.create`` return append-only ordered-sequence
+builders. ``append`` is explicit — one input batch extends the logical sequence
+without exposing native physical completion order.
 Dense append is synchronous and borrows an already-contiguous ndarray without a
 snapshot copy. Ragged append consumes its iterable once into one owned packed
 buffer, then completes the native append before returning. Nothing is inferred.
@@ -541,24 +575,24 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 7091 MiB/s | 11770 MiB/s | 82.0 | 3.97 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 34 MiB/s | 11801 MiB/s | 82.2 | 4.12 ms | 15.8 MiB | 22.74x |
-| zrecord-raw | 2708 MiB/s | 15883 MiB/s | 110.6 | 2.65 ms | 358.9 MiB | 1.00x |
-| npy-mmap-raw | 2055 MiB/s | 4738 MiB/s | 33.0 | 11.00 ms | 358.9 MiB | 1.00x |
-| hdf5-raw | 2360 MiB/s | 1948 MiB/s | 13.6 | 27.31 ms | 359.0 MiB | 1.00x |
-| hdf5-gzip | 281 MiB/s | 641 MiB/s | 4.5 | 65.88 ms | 26.1 MiB | 13.73x |
-| lmdb-raw | 1714 MiB/s | 4375 MiB/s | 30.5 | 10.83 ms | 361.4 MiB | 0.99x |
-| arrow-ipc-raw | 1817 MiB/s | 3480 MiB/s | 24.2 | 14.26 ms | 358.9 MiB | 1.00x |
-| arrow-ipc-zstd | 611 MiB/s | 177 MiB/s | 1.2 | 228.63 ms | 22.6 MiB | 15.86x |
-| parquet-raw | 1341 MiB/s | 487 MiB/s | 3.4 | 87.83 ms | 358.9 MiB | 1.00x |
-| parquet-zstd | 644 MiB/s | 162 MiB/s | 1.1 | 246.21 ms | 22.6 MiB | 15.86x |
-| arrayrecord-raw | 1543 MiB/s | 2107 MiB/s | 14.7 | 21.69 ms | 359.2 MiB | 1.00x |
-| arrayrecord-zstd | 917 MiB/s | 1586 MiB/s | 11.0 | 30.72 ms | 25.1 MiB | 14.32x |
-| tiledb-raw | 712 MiB/s | 688 MiB/s | 4.8 | 58.83 ms | 359.0 MiB | 1.00x |
-| tiledb-zstd | 1572 MiB/s | 1525 MiB/s | 10.6 | 26.80 ms | 26.9 MiB | 13.36x |
-At 147 KiB per record, Zrecord-raw reaches 15.5 GiB/s and is 3.4x npy-mmap-raw;
-plain zstd gathers at 11.5 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+| zrecord-zstd | 7998 MiB/s | 10937 MiB/s | 76.2 | 4.37 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 35 MiB/s | 12173 MiB/s | 84.8 | 4.06 ms | 15.8 MiB | 22.74x |
+| zrecord-raw | 2523 MiB/s | 14329 MiB/s | 99.8 | 3.12 ms | 358.9 MiB | 1.00x |
+| npy-mmap-raw | 2004 MiB/s | 4749 MiB/s | 33.1 | 11.08 ms | 358.9 MiB | 1.00x |
+| hdf5-raw | 2350 MiB/s | 1836 MiB/s | 12.8 | 29.68 ms | 359.0 MiB | 1.00x |
+| hdf5-gzip | 277 MiB/s | 657 MiB/s | 4.6 | 62.53 ms | 26.1 MiB | 13.73x |
+| lmdb-raw | 1546 MiB/s | 4034 MiB/s | 28.1 | 11.38 ms | 361.4 MiB | 0.99x |
+| arrow-ipc-raw | 1766 MiB/s | 3413 MiB/s | 23.8 | 15.10 ms | 358.9 MiB | 1.00x |
+| arrow-ipc-zstd | 656 MiB/s | 174 MiB/s | 1.2 | 239.23 ms | 22.6 MiB | 15.86x |
+| parquet-raw | 1174 MiB/s | 435 MiB/s | 3.0 | 95.45 ms | 358.9 MiB | 1.00x |
+| parquet-zstd | 592 MiB/s | 159 MiB/s | 1.1 | 251.37 ms | 22.6 MiB | 15.86x |
+| arrayrecord-raw | 1704 MiB/s | 2066 MiB/s | 14.4 | 20.36 ms | 359.2 MiB | 1.00x |
+| arrayrecord-zstd | 804 MiB/s | 1294 MiB/s | 9.0 | 36.89 ms | 25.1 MiB | 14.32x |
+| tiledb-raw | 743 MiB/s | 630 MiB/s | 4.4 | 66.01 ms | 359.0 MiB | 1.00x |
+| tiledb-zstd | 1498 MiB/s | 1503 MiB/s | 10.5 | 27.82 ms | 26.9 MiB | 13.36x |
+At 147 KiB per record, Zrecord-raw reaches 14.0 GiB/s and is 3.0x npy-mmap-raw;
+plain zstd gathers at 10.7 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -576,23 +610,23 @@ list or a one-dimensional variable-length abstraction is not enough.
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2502 MiB/s | 10071 MiB/s | 60.5 | 5.06 ms | 26.9 MiB | 15.52x |
-| zrecord-zstdict | 36 MiB/s | 10930 MiB/s | 65.6 | 4.62 ms | 17.7 MiB | 23.58x |
-| zrecord-raw | 1661 MiB/s | 13056 MiB/s | 78.4 | 3.84 ms | 417.2 MiB | 1.00x |
-| hdf5-raw | 1578 MiB/s | 1359 MiB/s | 8.2 | 37.26 ms | 418.0 MiB | 1.00x |
-| hdf5-gzip | 243 MiB/s | 130 MiB/s | 0.8 | 356.26 ms | 29.9 MiB | 13.94x |
-| lmdb-raw | 1790 MiB/s | 7412 MiB/s | 44.5 | 7.48 ms | 422.2 MiB | 0.99x |
-| arrow-ipc-raw | 1572 MiB/s | 4070 MiB/s | 24.4 | 14.56 ms | 417.2 MiB | 1.00x |
-| arrow-ipc-zstd | 620 MiB/s | 208 MiB/s | 1.2 | 232.74 ms | 25.9 MiB | 16.10x |
-| parquet-raw | 889 MiB/s | 481 MiB/s | 2.9 | 93.87 ms | 417.2 MiB | 1.00x |
-| parquet-zstd | 498 MiB/s | 184 MiB/s | 1.1 | 251.14 ms | 25.9 MiB | 16.10x |
-| arrayrecord-raw | 1644 MiB/s | 2642 MiB/s | 15.9 | 18.22 ms | 417.6 MiB | 1.00x |
-| arrayrecord-zstd | 829 MiB/s | 1556 MiB/s | 9.3 | 37.43 ms | 27.2 MiB | 15.31x |
-| tiledb-raw | 411 MiB/s | 55 MiB/s | 0.3 | 806.95 ms | 417.2 MiB | 1.00x |
-| tiledb-zstd | 713 MiB/s | 151 MiB/s | 0.9 | 307.58 ms | 27.0 MiB | 15.46x |
+| zrecord-zstd | 2407 MiB/s | 9927 MiB/s | 59.6 | 5.44 ms | 26.9 MiB | 15.52x |
+| zrecord-zstdict | 37 MiB/s | 10814 MiB/s | 65.0 | 4.86 ms | 17.7 MiB | 23.58x |
+| zrecord-raw | 1591 MiB/s | 13032 MiB/s | 78.2 | 3.87 ms | 417.2 MiB | 1.00x |
+| hdf5-raw | 1451 MiB/s | 1059 MiB/s | 6.4 | 44.91 ms | 418.0 MiB | 1.00x |
+| hdf5-gzip | 250 MiB/s | 126 MiB/s | 0.7 | 359.23 ms | 29.9 MiB | 13.94x |
+| lmdb-raw | 1717 MiB/s | 7269 MiB/s | 43.6 | 7.93 ms | 422.2 MiB | 0.99x |
+| arrow-ipc-raw | 1363 MiB/s | 4015 MiB/s | 24.1 | 14.97 ms | 417.2 MiB | 1.00x |
+| arrow-ipc-zstd | 570 MiB/s | 198 MiB/s | 1.2 | 238.70 ms | 25.9 MiB | 16.10x |
+| parquet-raw | 927 MiB/s | 501 MiB/s | 3.0 | 94.62 ms | 417.2 MiB | 1.00x |
+| parquet-zstd | 507 MiB/s | 173 MiB/s | 1.0 | 268.56 ms | 25.9 MiB | 16.10x |
+| arrayrecord-raw | 1650 MiB/s | 2661 MiB/s | 16.0 | 18.19 ms | 417.6 MiB | 1.00x |
+| arrayrecord-zstd | 758 MiB/s | 1499 MiB/s | 9.0 | 39.16 ms | 27.2 MiB | 15.31x |
+| tiledb-raw | 392 MiB/s | 48 MiB/s | 0.3 | 941.09 ms | 417.2 MiB | 1.00x |
+| tiledb-zstd | 675 MiB/s | 138 MiB/s | 0.8 | 328.05 ms | 27.0 MiB | 15.46x |
 Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
-delivers 9.8 GiB/s of
+delivers 9.7 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
@@ -620,19 +654,19 @@ accumulate at least two seconds.
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 456 MiB/s | 2047 MiB/s | 1047.9 | 0.32 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 54 MiB/s | 2393 MiB/s | 1225.3 | 0.28 ms | 153.8 MiB | 2.54x |
-| zrecord-raw | 2550 MiB/s | 7688 MiB/s | 3936.3 | 0.08 ms | 393.7 MiB | 0.99x |
-| npy-mmap-raw | 2139 MiB/s | 10709 MiB/s | 5482.9 | 0.07 ms | 390.6 MiB | 1.00x |
-| lmdb-raw | 721 MiB/s | 1166 MiB/s | 597.0 | 0.55 ms | 786.3 MiB | 0.50x |
-| arrow-ipc-raw | 2335 MiB/s | 231 MiB/s | 118.2 | 2.53 ms | 390.8 MiB | 1.00x |
-| arrayrecord-raw | 847 MiB/s | 162 MiB/s | 83.0 | 4.98 ms | 401.4 MiB | 0.97x |
-| arrayrecord-zstd | 124 MiB/s | 133 MiB/s | 68.3 | 4.68 ms | 201.3 MiB | 1.94x |
+| zrecord-zstd | 1019 MiB/s | 1959 MiB/s | 1002.8 | 0.35 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 52 MiB/s | 2238 MiB/s | 1145.9 | 0.31 ms | 153.8 MiB | 2.54x |
+| zrecord-raw | 2101 MiB/s | 7441 MiB/s | 3810.0 | 0.09 ms | 393.7 MiB | 0.99x |
+| npy-mmap-raw | 1778 MiB/s | 11454 MiB/s | 5864.4 | 0.06 ms | 390.6 MiB | 1.00x |
+| lmdb-raw | 639 MiB/s | 1104 MiB/s | 565.0 | 0.71 ms | 786.3 MiB | 0.50x |
+| arrow-ipc-raw | 2068 MiB/s | 225 MiB/s | 115.0 | 2.81 ms | 390.8 MiB | 1.00x |
+| arrayrecord-raw | 807 MiB/s | 150 MiB/s | 76.7 | 4.99 ms | 401.4 MiB | 0.97x |
+| arrayrecord-zstd | 127 MiB/s | 135 MiB/s | 69.4 | 4.45 ms | 201.3 MiB | 1.94x |
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.94 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.81 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-1.05–1.23 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.00–1.15 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 #### Variable Token Sequences
@@ -644,16 +678,16 @@ with exact `int32` values and original one-dimensional shapes.
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 114 MiB/s | 291 MiB/s | 550.1 | 0.59 ms | 67.5 MiB | 1.57x |
-| zrecord-zstdict | 47 MiB/s | 325 MiB/s | 614.3 | 0.53 ms | 49.2 MiB | 2.15x |
-| zrecord-raw | 571 MiB/s | 352 MiB/s | 665.5 | 0.50 ms | 110.7 MiB | 0.96x |
-| lmdb-raw | 341 MiB/s | 116 MiB/s | 218.9 | 1.32 ms | 153.0 MiB | 0.69x |
-| arrow-ipc-raw | 567 MiB/s | 44 MiB/s | 84.1 | 3.66 ms | 109.9 MiB | 0.96x |
-| arrayrecord-raw | 290 MiB/s | 28 MiB/s | 53.5 | 7.22 ms | 118.8 MiB | 0.89x |
-| arrayrecord-zstd | 61 MiB/s | 34 MiB/s | 64.1 | 5.12 ms | 76.2 MiB | 1.39x |
+| zrecord-zstd | 236 MiB/s | 282 MiB/s | 533.7 | 0.66 ms | 67.5 MiB | 1.57x |
+| zrecord-zstdict | 43 MiB/s | 305 MiB/s | 576.9 | 0.61 ms | 49.2 MiB | 2.15x |
+| zrecord-raw | 486 MiB/s | 359 MiB/s | 679.0 | 0.52 ms | 110.7 MiB | 0.96x |
+| lmdb-raw | 318 MiB/s | 113 MiB/s | 214.3 | 1.48 ms | 153.0 MiB | 0.69x |
+| arrow-ipc-raw | 562 MiB/s | 44 MiB/s | 83.1 | 4.00 ms | 109.9 MiB | 0.96x |
+| arrayrecord-raw | 288 MiB/s | 26 MiB/s | 49.3 | 7.39 ms | 118.8 MiB | 0.89x |
+| arrayrecord-zstd | 62 MiB/s | 33 MiB/s | 61.5 | 5.58 ms | 76.2 MiB | 1.39x |
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 550–666 krecords/s with less than 0.6 ms p95;
+Zrecord's three codecs return 534–679 krecords/s with 0.52–0.66 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
@@ -661,16 +695,16 @@ this pass.
 Index generation on its own, IID (with replacement), 1M index space, against
 NumPy's modern API. The µs-scale figures fluctuate with box load; this pass shows
-an approximately 2.1–2.3x margin across batch sizes.
+a 1.87–7.96x margin across batch sizes.
 | sampler | batch | per batch | vs default_rng |
 |---|---:|---:|---:|
-| numpy default_rng | 256 | 4.6 µs | 1.00x |
-| **zsampler** | 256 | 2.0 µs | **2.34x** |
-| numpy default_rng | 1024 | 5.5 µs | 1.00x |
-| **zsampler** | 1024 | 2.5 µs | **2.25x** |
-| numpy default_rng | 8192 | 20.6 µs | 1.00x |
-| **zsampler** | 8192 | 9.9 µs | **2.09x** |
+| numpy default_rng | 256 | 5.3 µs | 1.00x |
+| **zsampler** | 256 | 0.7 µs | **7.96x** |
+| numpy default_rng | 1024 | 4.7 µs | 1.00x |
+| **zsampler** | 1024 | 1.3 µs | **3.64x** |
+| numpy default_rng | 8192 | 18.0 µs | 1.00x |
+| **zsampler** | 8192 | 9.6 µs | **1.87x** |
 ### End-to-End DataLoader
@@ -700,19 +734,19 @@ and Grain reads ArrayRecord.
 | loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
 |---|---|---|---:|---:|---:|---:|---:|
-| **loaderx** | threads | zrecord-zstd | 147.0 | 15.35 ms | 985 MiB | 986 MiB | 990 MiB |
-| loaderx-raw | threads | zrecord-raw | 210.3 | 13.69 ms | 987 MiB | 987 MiB | 991 MiB |
-| torch | fork | npy-mmap-raw | 100.1 | 32.96 ms | 1748 MiB | 1841 MiB | 6387 MiB |
-| torch-spawn | spawn | npy-mmap-raw | 106.8 | 32.14 ms | 2833 MiB | 2890 MiB | 4869 MiB |
-| grain | processes | arrayrecord-zstd | 40.3 | 109.40 ms | 1834 MiB | 1917 MiB | 2036 MiB |
+| **loaderx** | threads | zrecord-zstd | 167.1 | 14.63 ms | 986 MiB | 986 MiB | 989 MiB |
+| loaderx-raw | threads | zrecord-raw | 215.3 | 13.77 ms | 987 MiB | 987 MiB | 991 MiB |
+| torch | fork | npy-mmap-raw | 109.4 | 32.70 ms | 1783 MiB | 1889 MiB | 6396 MiB |
+| torch-spawn | spawn | npy-mmap-raw | 111.6 | 30.69 ms | 2835 MiB | 2913 MiB | 4880 MiB |
+| grain | processes | arrayrecord-zstd | 46.6 | 93.48 ms | 1847 MiB | 1946 MiB | 2065 MiB |
 At 36.8 MiB per batch the per-batch gather dominates the tiny sampler cost, and
 the transform threads overlap Python-side collation with the next gather. The
 memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
 threads inside one process, so workers share one interpreter, one NumPy runtime
 and one set of gather buffers. With source geometry and entropy held constant,
-raw is 1.43x compressed loaderx; compressed loaderx is 1.47x Torch fork, 1.38x
-Torch spawn and 3.65x Grain, while raw is 2.10x, 1.97x and 5.22x faster.
+raw is 1.29x compressed loaderx; compressed loaderx is 1.53x Torch fork, 1.50x
+Torch spawn and 3.59x Grain, while raw is 1.97x, 1.93x and 4.62x faster.
 Torch's aggregate RSS is high because
 Linux fork mappings are counted repeatedly; it is not a total-memory ratio
 against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
@@ -721,19 +755,19 @@ not copy the full corpus into every worker; a Windows Dataset holding Python
 lists or in-memory arrays would be a different, deliberately harsher workload.
 The Torch-only worker sweep runs each count once. Worker 0 is an in-process
-baseline (145.5–153.8 batches/s and about 1485 MiB peak PSS in the two equivalent
+baseline (154.5 and 163.5 batches/s with 1484/1486 MiB peak PSS in the two equivalent
 rows), so the process-context comparison starts at one worker:
 | workers | fork batches/s | fork peak PSS | fork peak RSS | spawn batches/s | spawn peak PSS | spawn peak RSS |
 |---:|---:|---:|---:|---:|---:|---:|
-| 1 | 42.5 | 1678 MiB | 2483 MiB | 44.9 | 1891 MiB | 2128 MiB |
-| 2 | 69.4 | 1723 MiB | 3792 MiB | 71.8 | 2241 MiB | 3058 MiB |
-| 4 | 104.4 | 1866 MiB | 6365 MiB | 102.0 | 2926 MiB | 4893 MiB |
-| 8 | 117.8 | 2016 MiB | 11320 MiB | 103.6 | 4218 MiB | 8351 MiB |
-Spawn peak PSS grows from 1891 to 4218 MiB as workers rise from one to eight,
-while fork grows from 1678 to 2016 MiB because it retains COW sharing. At eight
-workers spawn uses 2.09x fork's peak PSS and throughput has already flattened.
+| 1 | 42.5 | 1647 MiB | 2483 MiB | 46.3 | 1890 MiB | 2128 MiB |
+| 2 | 68.9 | 1725 MiB | 3796 MiB | 72.3 | 2241 MiB | 3066 MiB |
+| 4 | 106.0 | 1849 MiB | 6385 MiB | 102.7 | 2878 MiB | 4859 MiB |
+| 8 | 114.3 | 2021 MiB | 11334 MiB | 107.8 | 4195 MiB | 8402 MiB |
+Spawn peak PSS grows from 1890 to 4195 MiB as workers rise from one to eight,
+while fork grows from 1647 to 2021 MiB because it retains COW sharing. At eight
+workers spawn uses 2.08x fork's peak PSS and throughput has already flattened.
 This demonstrates no-fork memory pressure; it is not labeled OOM because this
 31 GiB machine completed the run. Fork aggregate RSS grows faster because Linux
 counts shared/COW mappings in every process, so RSS is diagnostic rather than
@@ -749,11 +783,11 @@ no separate no-GIL Store implementation; the Store table above applies to both.
 | loader | GIL Python | GIL + Numba nogil | free-threaded Python | Numba gain | free-threaded gain |
 |---|---:|---:|---:|---:|---:|
-| **loaderx** | 44.5 batches/s | 107.0 batches/s | 73.7 batches/s | **2.40x** | **1.66x** |
-| loaderx-raw | 47.2 batches/s | 123.5 batches/s | 82.1 batches/s | **2.62x** | **1.74x** |
+| **loaderx** | 48.8 batches/s | 107.2 batches/s | 88.0 batches/s | **2.20x** | **1.80x** |
+| loaderx-raw | 51.2 batches/s | 113.1 batches/s | 94.1 batches/s | **2.21x** | **1.84x** |
-Peak PSS for compressed/raw was 1204/1189 MiB with GIL Python,
-1290/1262 MiB with Numba, and 1190/1190 MiB with free-threaded Python. These are
+Peak PSS for compressed/raw was 1193/1202 MiB with GIL Python,
+1291/1314 MiB with Numba, and 1190/1191 MiB with free-threaded Python. These are
 two deployment solutions to the transform bottleneck, not claims that Store
 itself was optimized for either runtime.
@@ -766,11 +800,13 @@ prefix and reconstructs the exact arrays. The speedup is not
 bought with sampling shortcuts: the IID draw is unbiased like NumPy's (Lemire
 with rejection, so uniformity costs nothing over a real index space).
-**The layouts match what a training loader does.** Zrecord is built for random
-record access: Dense gathers fixed-width records directly into one ndarray;
-Ragged restores independently shaped records from inline shape/payload
-entries. Array stores are built primarily for contiguous scans, so a scattered
-batch fights their layout. A dense raw gather fans out across the shared
+**The layouts match what a training loader does.** Zrecord preserves an ordered
+record sequence while supporting indexed access: Dense gathers fixed-width
+records directly into one ndarray; Ragged restores independently shaped records
+from inline shape/payload entries. The benchmark deliberately stresses random
+gather rather than claiming that ordering is absent. Array stores are built
+primarily for contiguous scans, so a scattered batch fights their layout. A
+dense raw gather fans out across the shared
 Executor budget where NumPy fancy indexing is one thread; Ragged instead trades some raw
 specialization for compression and a complete variable-shape persistence model.
@@ -784,8 +820,8 @@ format: in the current Dense structured-vision workload, plain zstd reaches
 **Loader results combine architecture and storage.** loaderx uses threads and
 never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
 torch uses finite shuffled epochs, worker processes and shared-memory handoff.
-Here compressed loaderx is 1.47x Torch fork, 1.38x Torch spawn and 3.65x Grain;
-raw loaderx is 2.10x, 1.97x and 5.22x faster, respectively.
+Here compressed loaderx is 1.53x Torch fork, 1.50x Torch spawn and 3.59x Grain;
+raw loaderx is 1.97x, 1.93x and 4.62x faster, respectively.
 Storage also differs per loader — each reads from what it was
 built for — so the loader table is a different comparison from either store
 table, not a rerun.
@@ -804,7 +840,7 @@ views into one batch allocation, while most byte-store adapters return
 independent copies. This is a single benchmark run, while each Store read path
 accumulates at least two timed seconds —
 the µs-scale sampler timings and the loader `batches/s` fluctuate with box load
-(on these 12 cores the compressed loader trails the raw one by about 25%), so
+(on these 12 cores the compressed loader trails the raw one by about 22%), so
 treat the absolute numbers as ballpark and the cross-backend margins as the
 signal.
@@ -1011,11 +1047,12 @@ license to trust storage or the operating system: native code still validates
     normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only
 defend against bypassing the public Python API do not belong in zrecord.
-1. `RecordEngine` is an unordered physical store made of N records. Records are
-   independent and carry no ordering, so every index and slice operation is
-   equivalent to a gather.
-2. It hands the container layer a dense index space: records are exactly
-   `0..N`. Named streams are composed dynamically by a plain Python dict;
+1. `RecordEngine` stores N logically ordered records. Parallel chunks may finish
+   and occupy `data.zr` in a different physical order, but logical ID is the
+   stable append position and `RecordLoc[ID]` preserves it. Index and slice
+   operations are implemented as ordered gathers over those positions.
+2. It hands the container layer a dense sequence space: records are exactly
+   `0..N-1`. Named streams are composed dynamically by a plain Python dict;
    `DataLoader` validates that the independent containers have equal lengths.
 3. The engine reads and writes byte ranges. Python owns the record schema and
    selects Dense or Ragged geometry; the private ABI receives only the runtime

loaderx 2.0.1__tar.gz → 2.0.7__tar.gz

loaderx 2.0.1tar.gz → 2.0.7tar.gz