loaderx 2.0.7__tar.gz → 2.3.3__tar.gz

{loaderx-2.0.7 → loaderx-2.3.3}/MANIFEST.in

@@ -8,7 +8,7 @@ recursive-include scripts *.py
 include scripts/requirements-bench.txt
 
 # Vendored zstd: the sdist rebuilds with Zig, so it needs the C sources.
-recursive-include vendor *.c *.h
+recursive-include vendor *.c *.h *.S
 include vendor/zstd/LICENSE vendor/zstd/COPYING
 
 exclude loaderx/lib/lib_here

{loaderx-2.0.7/loaderx.egg-info → loaderx-2.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.0.7
+Version: 2.3.3
 Summary: Rebuildable high-performance ordered record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
@@ -121,25 +121,26 @@ reader. Open the result with ``Dense.open``:
     batch = ds[:]
     ds.close()
 
-Python defines the exact record schema: ``dtype`` plus the dense ``item_shape``,
-or only ``dtype`` for ragged stores. The MsgPack bytes live opaquely in the
+Python defines the exact record schema: both geometries store ``dtype`` and
+``ndim``; Dense additionally stores ``item_shape``. The MsgPack bytes live opaquely in the
 static page at the front of ``meta.zr``; Zig persists them but never interprets
 them. The schema accepts no user metadata. Python selects the geometry and gives
 the private native engine only the runtime record boundaries it needs. Each
-Ragged record carries a u8 rank followed by inline little-endian u64 dimensions.
+Ragged record carries exactly ``ndim`` inline little-endian u64 dimensions.
 One native physical
 engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
 strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
 iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
 ``np.frombuffer(raw, dtype=np.uint8)`` and stored in a
-``Ragged.create(path, dtype=np.uint8)`` rather than creating a second
+``Ragged.create(path, dtype=np.uint8, ndim=1)`` rather than creating a second
 public storage API.
 
 ### Records
 One persistent format, two native execution contracts. ``Dense`` is the **dense** contract
 where every record is exactly one row of the recorded ``item_shape``; reads
-are fixed-stride gathers and the batch shape follows from the schema, so no
-per-record metadata is touched:
+allocate a fixed-stride destination whose batch shape follows from the schema.
+Physical records still use the shared ``RecordLoc[logical_id] -> payload``
+pipeline, so compressed completion order never becomes a second Dense layout:
 
 ```python
 import numpy as np
@@ -160,11 +161,11 @@ the same way.
 
 ``Ragged`` is the **ragged** contract for variable-length records. It is a
 separate contract: :class:`Ragged` hands back a list of arrays, so a
-loader never has to carry ``row_splits`` around. ``dtype`` is unified and
-explicit; each record keeps its **own shape**, recorded per record as it is
-written and restored exactly on read — so records may differ in shape
-arbitrarily, and nothing is ever inferred from the source (an iterator can't
-tell you what its later records look like). Scalar ``shape == ()`` is preserved;
+loader never has to carry ``row_splits`` around. ``dtype`` and ``ndim`` are
+unified and explicit; each record keeps its **own dimension lengths**, recorded
+per record and restored exactly on read. Every axis may vary, but every record
+has the schema rank; nothing is inferred from the source. Ragged requires at
+least one axis; scalar records use Dense with ``item_shape=()``.
 zero-byte arrays are rejected because physical records are nonempty. Densifying a list into a dense
 batch is the model's call — a plain numpy loop, wherever you need it:
 
@@ -172,8 +173,8 @@ batch is the model's call — a plain numpy loop, wherever you need it:
 from loaderx.zrecord import Ragged
 
 seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
-with Ragged.create('tokens', np.int32) as rs:
-    rs.append(seqs)                       # dtype explicit; each record keeps its shape
+with Ragged.create('tokens', np.int32, ndim=1) as rs:
+    rs.append(seqs)                       # dtype/rank fixed; lengths remain per-record
 rs = Ragged.open('tokens')
 
 records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
@@ -268,12 +269,12 @@ buffer, then completes the native append before returning. Nothing is inferred.
 ```python
 from loaderx.zrecord import Dense, Ragged
 
-ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))
+ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28), data_shards=4)
 ds.append(images[i:i + 1024])        # synchronous native batch; returns None
 ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
 ds.close()                            # publish before opening
 
-tok = Ragged.create('tokens', dtype=np.int32)
+tok = Ragged.create('tokens', dtype=np.int32, ndim=1)
 tok.append([seq_a, seq_b, seq_c])
 tok.close()
 
@@ -281,6 +282,11 @@ with Dense.open('mnist/x') as ds:
     first_four = ds[:4]               # opened containers are read-only
 ```
 
+`data_shards` is the keyword-only write-parallelism setting. It is persisted in
+the native Header, must be in `1..255`, and defaults to four. Values near the
+writer lane count spread payload writes across independent files; opening a
+completed container discovers the value automatically.
+
 Dense and Ragged append both report native errors in the current call and return
 only after accepting the batch. A writer cannot be read, and a reader cannot be
 appended to. ``close()`` on a writer publishes the container Header;
@@ -288,12 +294,14 @@ appended to. ``close()`` on a writer publishes the container Header;
 ``with`` for scoped lifetimes. Content is never changed in place: rerun the
 authoritative build at a new path, validate it, then switch consumers to it.
 
-The exact schema is declared at creation and encoded by Python as MsgPack. Dense
-schema contains only ``dtype`` and ``item_shape``; Ragged schema contains only
-``dtype``. Structured, subarray, object, and metadata-bearing dtypes are not
+The exact schema is declared at creation and encoded by Python as MsgPack. Both
+schemas contain ``dtype`` and ``ndim``; Dense additionally contains
+``item_shape`` and requires its length to equal ``ndim``. Ragged requires every
+appended record to have that rank, while every dimension length may vary.
+Structured, subarray, object, metadata-bearing, and zero-itemsize dtypes are not
 supported: their semantics do not round-trip through one canonical NumPy dtype
-string. The encoded schema has 4064 bytes available in the fixed 4096-byte
-metadata page. Ragged schema size is effectively fixed; Dense schema size grows
+string. The encoded schema has 2040 bytes available in the fixed 4096-byte
+metadata page. Ragged schema size is fixed; Dense schema size grows
 only with the integer ``item_shape``, so the physical limit is far above any
 practical NumPy array rank.
 Unexpected fields are rejected. ``append`` validates dtype and shape in Python,
@@ -310,7 +318,7 @@ published without one.
 for general-purpose compression — it is fast and the default.
 
 `"zstd_dict"` trains a shared dictionary on a sample of the data before writing
-any record, then compresses every record against it at level 19. The dictionary
+any record, then compresses every record against it at level 15. The dictionary
 captures structure shared across records that per-record compression cannot see —
 a large win for many small, similar records (image tiles, token sequences).
 
@@ -353,7 +361,8 @@ from loaderx.zrecord import Dense, Ragged
 d = train_dict(settled_array, tier="balanced")
 
 # then any new store can install it and append explicitly
-with Ragged.create('tokens', np.int32, codec='zstd_dict', dict_bytes=d) as ds:
+with Ragged.create('tokens', np.int32, ndim=1,
+                   codec='zstd_dict', dict_bytes=d) as ds:
     ds.append(token_generator)
 with Dense.create('data', data.dtype, data.shape[1:],
                        codec='zstd_dict', dict_bytes=d) as ds:
@@ -377,7 +386,8 @@ with Dense.open("src") as s, \
 ```
 
 ``Ragged`` is the same shape: ``s[:]`` returns ``list[np.ndarray]``, which
-is exactly its append input, and ``dtype`` is all the destination needs. The
+is exactly its append input; a destination is created with ``s.dtype`` and
+``s.ndim``. The
 native compression path bounds its own working memory; there is no public chunk
 parameter. ``dst`` must not already hold a store.
 
@@ -403,7 +413,7 @@ with Dense.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
     s.append(joint)
 with Dense.create(root + "/label", label.dtype, label.shape[1:]) as s:
     s.append(label)
-with Ragged.create(root + "/token", np.int32) as s:
+with Ragged.create(root + "/token", np.int32, ndim=1) as s:
     s.append(seqs)
 
 streams = {
@@ -496,9 +506,12 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 
 ### Methodology
 
-The results below are one complete run from the same checkout on a warm page
-cache. They are not three-run medians: an unexpected result is traced separately
-instead of being hidden by repeated aggregation. Within each store workload every
+The comparison matrix began as one complete run on a warm page cache. All
+Zrecord Store rows and the two loaderx identity rows were refreshed on 2.3.0
+with the default `data_shards=4`. Unchanged
+optional Store backends, sampler rows and external loader rows remain from the
+immediately preceding 2.0.9 run. These are not three-run medians: an unexpected result is traced
+separately instead of being hidden by repeated aggregation. Within each store workload every
 backend receives identical source records. Correctness and timing use independent
 deterministic `zsampler` IID streams. Loader backends receive
 the same source and seed but use their own shipped samplers, so their exact
@@ -556,8 +569,8 @@ threads):
 | NUMA      | 1 node |
 | memory    | 31 GiB (not limited by cgroup) |
 | shared memory | 16 GiB `/dev/shm` |
-| OS        | Debian GNU/Linux forky/sid, kernel 7.1.3+deb13-amd64, x86_64 |
-| python    | CPython 3.14.7 (standard GIL build), numpy 2.5.2 |
+| OS        | Debian GNU/Linux forky/sid, kernel 7.1.8+deb13-amd64, x86_64 |
+| python    | CPython 3.14.7 (standard GIL build), numpy 2.4.6 |
 
 The benchmark process sees all 24 threads and is not memory-limited by cgroup.
 The 16 GiB shared-memory mount accommodates the four-worker, 36.8 MiB-batch torch
@@ -575,9 +588,9 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 7998 MiB/s | 10937 MiB/s | 76.2 | 4.37 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 35 MiB/s | 12173 MiB/s | 84.8 | 4.06 ms | 15.8 MiB | 22.74x |
-| zrecord-raw | 2523 MiB/s | 14329 MiB/s | 99.8 | 3.12 ms | 358.9 MiB | 1.00x |
+| zrecord-zstd | 7172 MiB/s | 11081 MiB/s | 77.2 | 4.38 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 104 MiB/s | 12042 MiB/s | 83.9 | 4.09 ms | 16.2 MiB | 22.10x |
+| zrecord-raw | 5721 MiB/s | 14457 MiB/s | 100.7 | 3.16 ms | 358.9 MiB | 1.00x |
 | npy-mmap-raw | 2004 MiB/s | 4749 MiB/s | 33.1 | 11.08 ms | 358.9 MiB | 1.00x |
 | hdf5-raw | 2350 MiB/s | 1836 MiB/s | 12.8 | 29.68 ms | 359.0 MiB | 1.00x |
 | hdf5-gzip | 277 MiB/s | 657 MiB/s | 4.6 | 62.53 ms | 26.1 MiB | 13.73x |
@@ -591,8 +604,8 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 | tiledb-raw | 743 MiB/s | 630 MiB/s | 4.4 | 66.01 ms | 359.0 MiB | 1.00x |
 | tiledb-zstd | 1498 MiB/s | 1503 MiB/s | 10.5 | 27.82 ms | 26.9 MiB | 13.36x |
 
-At 147 KiB per record, Zrecord-raw reaches 14.0 GiB/s and is 3.0x npy-mmap-raw;
-plain zstd gathers at 10.7 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+At 147 KiB per record, Zrecord-raw reaches 14.1 GiB/s and is 3.0x npy-mmap-raw;
+plain zstd gathers at 10.8 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -610,9 +623,9 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2407 MiB/s | 9927 MiB/s | 59.6 | 5.44 ms | 26.9 MiB | 15.52x |
-| zrecord-zstdict | 37 MiB/s | 10814 MiB/s | 65.0 | 4.86 ms | 17.7 MiB | 23.58x |
-| zrecord-raw | 1591 MiB/s | 13032 MiB/s | 78.2 | 3.87 ms | 417.2 MiB | 1.00x |
+| zrecord-zstd | 2220 MiB/s | 9444 MiB/s | 56.7 | 5.67 ms | 26.9 MiB | 15.48x |
+| zrecord-zstdict | 102 MiB/s | 10345 MiB/s | 62.1 | 5.15 ms | 18.2 MiB | 22.97x |
+| zrecord-raw | 2171 MiB/s | 12356 MiB/s | 74.2 | 4.22 ms | 417.2 MiB | 1.00x |
 | hdf5-raw | 1451 MiB/s | 1059 MiB/s | 6.4 | 44.91 ms | 418.0 MiB | 1.00x |
 | hdf5-gzip | 250 MiB/s | 126 MiB/s | 0.7 | 359.23 ms | 29.9 MiB | 13.94x |
 | lmdb-raw | 1717 MiB/s | 7269 MiB/s | 43.6 | 7.93 ms | 422.2 MiB | 0.99x |
@@ -625,16 +638,16 @@ list or a one-dimensional variable-length abstraction is not enough.
 | tiledb-raw | 392 MiB/s | 48 MiB/s | 0.3 | 941.09 ms | 417.2 MiB | 1.00x |
 | tiledb-zstd | 675 MiB/s | 138 MiB/s | 0.8 | 328.05 ms | 27.0 MiB | 15.46x |
 
-Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
-delivers 9.7 GiB/s of
+Zrecord-raw is 1.7x LMDB and 3.1x Arrow IPC in logical gather. Zrecord-zstd
+delivers 9.2 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
 and row-group formats pay read amplification on random records.
 
 The shared generator makes compression ratios directly comparable across
-contracts: Zrecord zstd is 14.69x Dense versus 15.52x Ragged, and zstdict is
-22.74x versus 23.58x. The remaining difference comes from the H/W distribution
+contracts: Zrecord zstd is 14.69x Dense versus 15.48x Ragged, and zstdict is
+22.10x versus 22.97x. The remaining difference comes from the H/W distribution
 and Ragged shape metadata, not a different image entropy model.
 
 ### Small Token Records
@@ -654,19 +667,19 @@ accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 1019 MiB/s | 1959 MiB/s | 1002.8 | 0.35 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 52 MiB/s | 2238 MiB/s | 1145.9 | 0.31 ms | 153.8 MiB | 2.54x |
-| zrecord-raw | 2101 MiB/s | 7441 MiB/s | 3810.0 | 0.09 ms | 393.7 MiB | 0.99x |
-| npy-mmap-raw | 1778 MiB/s | 11454 MiB/s | 5864.4 | 0.06 ms | 390.6 MiB | 1.00x |
+| zrecord-zstd | 989 MiB/s | 2012 MiB/s | 1030.3 | 0.34 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 62 MiB/s | 2079 MiB/s | 1064.6 | 0.32 ms | 154.4 MiB | 2.53x |
+| zrecord-raw | 4542 MiB/s | 6391 MiB/s | 3272.3 | 0.11 ms | 393.7 MiB | 0.99x |
+| npy-mmap-raw | 1587 MiB/s | 9070 MiB/s | 4644.0 | 0.08 ms | 390.6 MiB | 1.00x |
 | lmdb-raw | 639 MiB/s | 1104 MiB/s | 565.0 | 0.71 ms | 786.3 MiB | 0.50x |
 | arrow-ipc-raw | 2068 MiB/s | 225 MiB/s | 115.0 | 2.81 ms | 390.8 MiB | 1.00x |
 | arrayrecord-raw | 807 MiB/s | 150 MiB/s | 76.7 | 4.99 ms | 401.4 MiB | 0.97x |
 | arrayrecord-zstd | 127 MiB/s | 135 MiB/s | 69.4 | 4.45 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.81 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.27 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-1.00–1.15 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.03–1.06 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
@@ -678,16 +691,16 @@ with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 236 MiB/s | 282 MiB/s | 533.7 | 0.66 ms | 67.5 MiB | 1.57x |
-| zrecord-zstdict | 43 MiB/s | 305 MiB/s | 576.9 | 0.61 ms | 49.2 MiB | 2.15x |
-| zrecord-raw | 486 MiB/s | 359 MiB/s | 679.0 | 0.52 ms | 110.7 MiB | 0.96x |
+| zrecord-zstd | 287 MiB/s | 315 MiB/s | 594.6 | 0.60 ms | 67.4 MiB | 1.57x |
+| zrecord-zstdict | 54 MiB/s | 332 MiB/s | 627.6 | 0.58 ms | 49.1 MiB | 2.16x |
+| zrecord-raw | 536 MiB/s | 389 MiB/s | 734.9 | 0.50 ms | 110.5 MiB | 0.96x |
 | lmdb-raw | 318 MiB/s | 113 MiB/s | 214.3 | 1.48 ms | 153.0 MiB | 0.69x |
 | arrow-ipc-raw | 562 MiB/s | 44 MiB/s | 83.1 | 4.00 ms | 109.9 MiB | 0.96x |
 | arrayrecord-raw | 288 MiB/s | 26 MiB/s | 49.3 | 7.39 ms | 118.8 MiB | 0.89x |
 | arrayrecord-zstd | 62 MiB/s | 33 MiB/s | 61.5 | 5.58 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 534–679 krecords/s with 0.52–0.66 ms p95;
+Zrecord's three codecs return 595–735 krecords/s with 0.50–0.60 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -734,8 +747,8 @@ and Grain reads ArrayRecord.
 
 | loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
 |---|---|---|---:|---:|---:|---:|---:|
-| **loaderx** | threads | zrecord-zstd | 167.1 | 14.63 ms | 986 MiB | 986 MiB | 989 MiB |
-| loaderx-raw | threads | zrecord-raw | 215.3 | 13.77 ms | 987 MiB | 987 MiB | 991 MiB |
+| **loaderx** | threads | zrecord-zstd | 171.5 | 12.69 ms | 984 MiB | 985 MiB | 988 MiB |
+| loaderx-raw | threads | zrecord-raw | 200.0 | 12.50 ms | 993 MiB | 993 MiB | 996 MiB |
 | torch | fork | npy-mmap-raw | 109.4 | 32.70 ms | 1783 MiB | 1889 MiB | 6396 MiB |
 | torch-spawn | spawn | npy-mmap-raw | 111.6 | 30.69 ms | 2835 MiB | 2913 MiB | 4880 MiB |
 | grain | processes | arrayrecord-zstd | 46.6 | 93.48 ms | 1847 MiB | 1946 MiB | 2065 MiB |
@@ -745,8 +758,8 @@ the transform threads overlap Python-side collation with the next gather. The
 memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
 threads inside one process, so workers share one interpreter, one NumPy runtime
 and one set of gather buffers. With source geometry and entropy held constant,
-raw is 1.29x compressed loaderx; compressed loaderx is 1.53x Torch fork, 1.50x
-Torch spawn and 3.59x Grain, while raw is 1.97x, 1.93x and 4.62x faster.
+raw is 1.17x compressed loaderx; compressed loaderx is 1.57x Torch fork, 1.54x
+Torch spawn and 3.68x Grain, while raw is 1.83x, 1.79x and 4.29x faster.
 Torch's aggregate RSS is high because
 Linux fork mappings are counted repeatedly; it is not a total-memory ratio
 against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
@@ -815,13 +828,13 @@ specialization for compression and a complete variable-shape persistence model.
 copy are one path, so turning compression on costs part of a margin, not an
 order of magnitude. The ratio is the data, not the
 format: in the current Dense structured-vision workload, plain zstd reaches
-14.69x and the balanced dictionary reaches 22.74x.
+14.69x and the balanced dictionary reaches 22.10x.
 
 **Loader results combine architecture and storage.** loaderx uses threads and
 never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
 torch uses finite shuffled epochs, worker processes and shared-memory handoff.
-Here compressed loaderx is 1.53x Torch fork, 1.50x Torch spawn and 3.59x Grain;
-raw loaderx is 1.97x, 1.93x and 4.62x faster, respectively.
+Here compressed loaderx is 1.57x Torch fork, 1.54x Torch spawn and 3.68x Grain;
+raw loaderx is 1.83x, 1.79x and 4.29x faster, respectively.
 Storage also differs per loader — each reads from what it was
 built for — so the loader table is a different comparison from either store
 table, not a rerun.
@@ -1047,9 +1060,9 @@ license to trust storage or the operating system: native code still validates
     normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only
 defend against bypassing the public Python API do not belong in zrecord.
 
-1. `RecordEngine` stores N logically ordered records. Parallel chunks may finish
-   and occupy `data.zr` in a different physical order, but logical ID is the
-   stable append position and `RecordLoc[ID]` preserves it. Index and slice
+1. `RecordEngine` stores N logically ordered records. Payload record `i` belongs
+   to `data_{i % data_shards}.zr`; logical ID remains the stable append position
+   and `RecordLoc[ID]` preserves its shard-local offset. Index and slice
    operations are implemented as ordered gathers over those positions.
 2. It hands the container layer a dense sequence space: records are exactly
    `0..N-1`. Named streams are composed dynamically by a plain Python dict;
@@ -1059,7 +1072,7 @@ defend against bypassing the public Python API do not belong in zrecord.
    byte geometry. Dense stores persist
    one fixed-width physical record per logical record. Ragged stores
    persist one variable-width physical record per logical record:
-   `[u64le ndim][u64le dims...][payload]`. Shape and payload therefore share one
+   `[u64le dim] * schema.ndim + [payload]`. Shape and payload therefore share one
    location, codec frame, and append publication.
 4. The IO model (`append | read`) is batch-oriented and
    shape-agnostic. Per-call adapters expose record boundaries through a compile-time
@@ -1076,7 +1089,7 @@ defend against bypassing the public Python API do not belong in zrecord.
 |-------|-----------|----------------------------------------|
 |   0   |  raw      | none                                   |
 |   1   |  zstd     | zstd (plain, level 3)                  |
-|   2   |  zstdict  | zstd with a trained dictionary (level 19) |
+|   2   |  zstdict  | zstd with a trained dictionary (level 15) |
 ```
 
 7. Compression is transparent to the client:
@@ -1096,40 +1109,51 @@ defend against bypassing the public Python API do not belong in zrecord.
       dictionary carries the structure shared across records, which per-record
       compression cannot see. On many small, similar records (image tiles, token
       sequences) this is a large win: the current Dense structured-vision set
-      is 14.69x with plain zstd and 22.74x with the balanced dictionary. The dictionary is loaded once on
+      is 14.69x with plain zstd and 22.10x with the balanced dictionary. The dictionary is loaded once on
       open and shared, lock-free, across all reader threads. The dictionary size
       is chosen from the `DICT_TIERS` presets (see Codec notes).
     * A `zstd_dict` store needs its dictionary to read every record; a `raw` or
       `zstd` store rejects an unexpected dictionary as malformed state.
 
 ## Persistence format
-The 2.0 clean break does not change the current on-disk format. That format is
-the current implementation only: there is no compatibility layer, migration,
-version dispatch, checksum, or recovery facility. Zrecord is not the authority
-for irreplaceable data. Keep authoritative source data and reproducible build
-scripts; after an interrupted build, storage failure, implementation change, or
-content change, rebuild a complete container at a new path.
-
-Native storage uses a fixed file set:
+The current format is the settled internal baseline for implementation work: optimizations keep
+the fixed files, Header/schema page, contiguous ``RecordLoc`` table and independent
+record payloads unless the product boundary is deliberately reopened. "Settled"
+does not promise cross-version persistence compatibility: there is no compatibility
+layer, migration, version dispatch, checksum, or recovery facility. Zrecord is not
+the authority for irreplaceable data. Keep authoritative source data and reproducible
+build scripts; after an interrupted build, storage failure, incompatible implementation
+change, or content change, rebuild a complete container at a new path.
+
+Native storage uses one metadata file and a create-time-fixed payload file set:
 ```
 store/
-  ├── meta.zr      4096-byte static Header/schema page + RecordLoc table
-  ├── data.zr      payload stream
+  ├── meta.zr      4096-byte Header/schema/tails page + RecordLoc table
+  ├── data_0.zr    payload records where ID % data_shards == 0
+  ├── ...
+  ├── data_{N-1}.zr final static payload shard
   └── dict.zr      zstd dictionary (only in dict stores)
 ```
 
+`data_shards` is a write-performance parameter in `1..255`, fixed by `create`
+and recovered automatically by `open`. The default is four; practical values are
+usually 2, 4, 8 or 16, near the writer lane count. More files spread positional
+writes across payload inodes but consume one descriptor each. This physical
+striping does not change record IDs, order, codec, or read results.
+
 ### Metadata (meta.zr)
 
 Files are read and written **positionally** — pread/pwrite at computed offsets,
 no mmap. `meta.zr` starts with one fixed 4096-byte static page: a naturally
-aligned 32-byte Header, then the opaque MsgPack schema and unused zero padding.
+aligned 16-byte Header, 255 shard-local u64 tails at bytes 16..2055, then up to
+2040 bytes of opaque MsgPack schema at bytes 2056..4095.
 An array of 16-byte `RecordLoc`s starts at offset 4096. Record `i` is one
 pread/pwrite at `4096 + i * 16`; there is no variable table base, segment
 mapping, or rollover fd table.
 
-**1. Python schema** — bytes `32..32+schema_length` are exactly one immutable
-MsgPack object. Dense stores contain only ``dtype`` and ``item_shape``; Ragged
-stores contain only ``dtype``. Native create persists these bytes together with
+**1. Python schema** — bytes `2056..2056+schema_length` are exactly one immutable
+MsgPack object. Both stores contain ``dtype`` and ``ndim``; Dense additionally
+contains ``item_shape``. Native create persists these bytes together with
 the physical container but does not decode them. Open acquires the native lifetime
 lock before copying the schema to Python for validation, so schema and physical
 metadata are one locked snapshot. Dense record width is derived once from
@@ -1137,30 +1161,36 @@ dtype/item_shape and passed to the native handle as runtime geometry; it is not
 independently persisted as a second authority. There is no format version or
 legacy kind dispatch.
 
-**2. Physical header** — the first 32 bytes of `meta.zr`. The format
+**2. Physical header** — the first 16 bytes of `meta.zr`. The format
 deliberately carries no payload or metadata checksum.
 
 * `codec` is the store's one compression method, stamped at creation and
   immutable — there is no per-record tag anywhere.
 * `length` (u64) is the physical record count; it equals logical length for both
   dense stores and inline ragged stores.
-* `tail_offset` (u64) is the absolute committed frontier in `data.zr`.
-* `schema_length` (u32) is the occupied prefix of the static schema area and
-  must be in `1..4064`.
+* `schema_length` (u16) is the occupied prefix of the static schema area and
+  must be in `1..2040`.
+* `data_shards` (u8) is the static payload file count and must be in `1..255`.
 
 ```zig
 const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
 const Header = extern struct {
     length: u64,
-    tail_offset: u64,
-    schema_length: u32,
-    reserved: [11]u8,
+    schema_length: u16,
+    data_shards: u8,
     codec: u8,
+    reserved: [4]u8,
 };
 ```
 
-**3. Record table** — contiguous 16-byte entries start at offset 4096 in
-`meta.zr` and grow as location runs are written. `offset` is an absolute byte offset in `data.zr`;
+**3. Shard frontiers** — tail slot `s` at `16 + s * 8` is the committed
+end of `data_s.zr`. Unused slots among the 255 fixed u64 entries are zero.
+Open requires every data file to be at least its persisted tail; locations may
+not cross that shard-local frontier.
+
+**4. Record table** — contiguous 16-byte entries start at offset 4096 in
+`meta.zr` and grow as location windows are written. `offset` is local to
+`data_{ID % data_shards}.zr`;
 `phys_length`/`logic_length` are the stored and original sizes. The
 codec is not here: it is the header's, so a record is stored exactly the way the
 store is declared.
@@ -1175,9 +1205,9 @@ const RecordLoc = extern struct {
 
 There is no liveness flag. Every entry below `length` is a record.
 
-**4. No fixed record-count cap.** The table and payload stream grow naturally in
-their fixed files. The practical bounds are the u64 count, supported positional
-file offsets, 2 GiB per record, and disk.
+**5. No fixed record-count cap.** The table and payload streams grow naturally.
+The practical bounds are the u64 count, supported positional file offsets,
+2 GiB per record, descriptor budget, and disk.
 
 ## Executor
 
@@ -1192,34 +1222,36 @@ while `Ragged` supplies offsets for its shape-prefixed records.
 The private ABI turns those inputs into compile-time record sources and
 destinations; the engine has one append and one gather operation. Its shared
 opaque handle remains private and carries no typed-store geometry.
-* Compressed append: workers claim individual records and reuse one operation-local
-  frame buffer per task, reserve physical offsets in completion order through a short frontier
-  lock, and issue positional payload writes in parallel. Logical IDs remain in
-  the loc table, so physical completion order does not change random gather.
-  The caller waits for every payload before publishing the new locations and
-  in-process length. Append does not overwrite the on-disk Header and is
-  intentionally lazy. Writer ``close()`` writes the current Header and closes
-  the files, making the page-cache state available for read-only open.
-  Every compressed frame owns one absolute range in `data.zr`; positional writes extend the file to the
-  current frontier. Python budgets the process-wide executor at three quarters
+* Append validates the complete call before physical I/O, then processes bounded
+  logical windows. Within a window, record positions are divided by
+  `ID % data_shards` and planned as bounded shard-local chunks. Fixed Executor
+  lanes dynamically claim those chunks, encode or pack them, briefly lock only
+  the selected shard's tail reservation, then issue positional payload writes
+  directly. Multiple lanes may write non-overlapping ranges of one shard; the
+  static file set spreads that pressure across inodes without limiting codec
+  concurrency to the shard count. Workers fill disjoint entries in one
+  contiguous location buffer. A payload barrier precedes one contiguous
+  `meta.zr` location write. All windows must succeed before the in-process
+  length and per-shard tails advance. Writer `close()` truncates each data file
+  to its committed tail and publishes the 4096-byte static page.
+* Compressed shard tasks lease process-bounded `ExecutionSlot` scratch and write
+  independent frames in bounded subchunks. Python budgets the process-wide executor at three quarters
   of the logical CPUs available to the process, leaving headroom for packing,
   transforms, and the caller without encoding a platform-specific thread count.
   Each producer configures its CCtx or shared immutable CDict once, then starts
   every independent record frame with `ZSTD_compress2`.
-* Raw append preserves the stronger invariant already supplied by Python: every
-  record in one Dense or packed Ragged batch is a boundary inside one contiguous
-  source buffer. After locating the records, the engine consumes that buffer in
-  plan order and passes each contiguous planned range directly to
-  `writePositionalAll` on `data.zr`. There is no per-record iovec construction,
-  byte-budget flush, payload copy, or platform-specific syscall path. Zig's
-  `std.Io` handles short writes and maps the same positional operation to POSIX
-  and Windows implementations.
+* Raw records for one shard are strided in the source. Each shard task packs a
+  bounded subchunk into its reusable scratch and performs one contiguous
+  positional write; a single record larger than the normal subchunk budget is
+  written directly. There is no per-record syscall or platform-specific vectored
+  path. The extra memory copy is the deliberate cost paid to remove concentrated
+  single-inode writes.
 **2. Read.** Fill the destination memory concurrently, in place from the Python
 side (executed on async threads).
 * Committed records are immutable and `length` is published through an atomic.
-  The fixed `meta.zr` and `data.zr` handles require no rollover fd-table
-  synchronization; record I/O itself stays lock free.
-* Every record is read at the offset its table entry records — the record table
+  The fixed metadata and static data handles require no rollover synchronization.
+* Every record first selects `data_{ID % data_shards}.zr`, then reads the
+  shard-local offset its table entry records — the record table
   is addressed by pure arithmetic, so random access is one pread for the
   location and one for the bytes, with no batching assumptions about layout.
   Each lane reads one location and immediately reads/decompresses that record;
@@ -1231,8 +1263,9 @@ budget; lanes for which the runtime cannot reserve concurrency run inline on the
 calling thread. Python configures the process-level budget as
 `max(physical cores, logical cores * 3 / 4)`, using platform topology where
 available.
-* Lanes receive contiguous blocks rather than a strided subset, keeping each
-  worker's reads and writes sequential.
+* Gather lanes receive contiguous request blocks. Append lanes dynamically claim
+  bounded chunks of strided shard ranges; short per-shard reservation locks keep
+  offsets disjoint while positional writes remain concurrent.
 * Each lane creates one zstd context (`ZSTD_CCtx` to write, `ZSTD_DCtx` to
   read) and reuses it across every record it handles, rather than paying that
   setup per record. The dictionary (`ZSTD_CDict`/`ZSTD_DDict`) is immutable, so
@@ -1242,10 +1275,10 @@ available.
 
 **4. File access.**
 * Metadata: one naturally growing `meta.zr`, containing the fixed Header/schema
-  page and loc table.
-* Payload: one naturally growing `data.zr`, accessed concurrently through
-  `readPositionalAll`/`writePositionalAll`. No path depends on filesystem
-  sparse-file support.
+  /tails page and loc table. It is intentionally not sharded because measured
+  write pressure is in payload I/O; one coordinator writes each loc window.
+* Payload: a static list of naturally growing `data_<shard>.zr` files, accessed through
+  `readPositionalAll`/`writePositionalAll`. No path depends on sparse files.
 
 **Execution model.** Opened readers are immutable, so calls on the same reader may
 gather concurrently. Creator appends are synchronous and native Storage