loaderx 2.2.1__tar.gz → 2.3.3__tar.gz

{loaderx-2.2.1/loaderx.egg-info → loaderx-2.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.2.1
+Version: 2.3.3
 Summary: Rebuildable high-performance ordered record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
@@ -269,7 +269,7 @@ buffer, then completes the native append before returning. Nothing is inferred.
 ```python
 from loaderx.zrecord import Dense, Ragged
 
-ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))
+ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28), data_shards=4)
 ds.append(images[i:i + 1024])        # synchronous native batch; returns None
 ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
 ds.close()                            # publish before opening
@@ -282,6 +282,11 @@ with Dense.open('mnist/x') as ds:
     first_four = ds[:4]               # opened containers are read-only
 ```
 
+`data_shards` is the keyword-only write-parallelism setting. It is persisted in
+the native Header, must be in `1..255`, and defaults to four. Values near the
+writer lane count spread payload writes across independent files; opening a
+completed container discovers the value automatically.
+
 Dense and Ragged append both report native errors in the current call and return
 only after accepting the batch. A writer cannot be read, and a reader cannot be
 appended to. ``close()`` on a writer publishes the container Header;
@@ -295,7 +300,7 @@ schemas contain ``dtype`` and ``ndim``; Dense additionally contains
 appended record to have that rank, while every dimension length may vary.
 Structured, subarray, object, metadata-bearing, and zero-itemsize dtypes are not
 supported: their semantics do not round-trip through one canonical NumPy dtype
-string. The encoded schema has 4064 bytes available in the fixed 4096-byte
+string. The encoded schema has 2040 bytes available in the fixed 4096-byte
 metadata page. Ragged schema size is fixed; Dense schema size grows
 only with the integer ``item_shape``, so the physical limit is far above any
 practical NumPy array rank.
@@ -502,8 +507,8 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 ### Methodology
 
 The comparison matrix began as one complete run on a warm page cache. All
-Zrecord Store rows and the two loaderx identity rows were refreshed on 2.2.0
-after the libzstd 1.5.7 and runtime-dispatched Huffman kernel update. Unchanged
+Zrecord Store rows and the two loaderx identity rows were refreshed on 2.3.0
+with the default `data_shards=4`. Unchanged
 optional Store backends, sampler rows and external loader rows remain from the
 immediately preceding 2.0.9 run. These are not three-run medians: an unexpected result is traced
 separately instead of being hidden by repeated aggregation. Within each store workload every
@@ -583,9 +588,9 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 6796 MiB/s | 10816 MiB/s | 75.3 | 4.09 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 102 MiB/s | 11844 MiB/s | 82.5 | 3.61 ms | 16.2 MiB | 22.10x |
-| zrecord-raw | 2801 MiB/s | 14655 MiB/s | 102.1 | 2.92 ms | 358.9 MiB | 1.00x |
+| zrecord-zstd | 7172 MiB/s | 11081 MiB/s | 77.2 | 4.38 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 104 MiB/s | 12042 MiB/s | 83.9 | 4.09 ms | 16.2 MiB | 22.10x |
+| zrecord-raw | 5721 MiB/s | 14457 MiB/s | 100.7 | 3.16 ms | 358.9 MiB | 1.00x |
 | npy-mmap-raw | 2004 MiB/s | 4749 MiB/s | 33.1 | 11.08 ms | 358.9 MiB | 1.00x |
 | hdf5-raw | 2350 MiB/s | 1836 MiB/s | 12.8 | 29.68 ms | 359.0 MiB | 1.00x |
 | hdf5-gzip | 277 MiB/s | 657 MiB/s | 4.6 | 62.53 ms | 26.1 MiB | 13.73x |
@@ -599,8 +604,8 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 | tiledb-raw | 743 MiB/s | 630 MiB/s | 4.4 | 66.01 ms | 359.0 MiB | 1.00x |
 | tiledb-zstd | 1498 MiB/s | 1503 MiB/s | 10.5 | 27.82 ms | 26.9 MiB | 13.36x |
 
-At 147 KiB per record, Zrecord-raw reaches 14.4 GiB/s and is 3.1x npy-mmap-raw;
-plain zstd gathers at 10.6 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+At 147 KiB per record, Zrecord-raw reaches 14.1 GiB/s and is 3.0x npy-mmap-raw;
+plain zstd gathers at 10.8 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -618,9 +623,9 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2339 MiB/s | 9507 MiB/s | 57.1 | 5.16 ms | 26.9 MiB | 15.49x |
-| zrecord-zstdict | 102 MiB/s | 10336 MiB/s | 62.1 | 4.85 ms | 18.2 MiB | 22.98x |
-| zrecord-raw | 1420 MiB/s | 12929 MiB/s | 77.6 | 3.86 ms | 417.2 MiB | 1.00x |
+| zrecord-zstd | 2220 MiB/s | 9444 MiB/s | 56.7 | 5.67 ms | 26.9 MiB | 15.48x |
+| zrecord-zstdict | 102 MiB/s | 10345 MiB/s | 62.1 | 5.15 ms | 18.2 MiB | 22.97x |
+| zrecord-raw | 2171 MiB/s | 12356 MiB/s | 74.2 | 4.22 ms | 417.2 MiB | 1.00x |
 | hdf5-raw | 1451 MiB/s | 1059 MiB/s | 6.4 | 44.91 ms | 418.0 MiB | 1.00x |
 | hdf5-gzip | 250 MiB/s | 126 MiB/s | 0.7 | 359.23 ms | 29.9 MiB | 13.94x |
 | lmdb-raw | 1717 MiB/s | 7269 MiB/s | 43.6 | 7.93 ms | 422.2 MiB | 0.99x |
@@ -633,16 +638,16 @@ list or a one-dimensional variable-length abstraction is not enough.
 | tiledb-raw | 392 MiB/s | 48 MiB/s | 0.3 | 941.09 ms | 417.2 MiB | 1.00x |
 | tiledb-zstd | 675 MiB/s | 138 MiB/s | 0.8 | 328.05 ms | 27.0 MiB | 15.46x |
 
-Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
-delivers 9.3 GiB/s of
+Zrecord-raw is 1.7x LMDB and 3.1x Arrow IPC in logical gather. Zrecord-zstd
+delivers 9.2 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
 and row-group formats pay read amplification on random records.
 
 The shared generator makes compression ratios directly comparable across
-contracts: Zrecord zstd is 14.69x Dense versus 15.49x Ragged, and zstdict is
-22.10x versus 22.98x. The remaining difference comes from the H/W distribution
+contracts: Zrecord zstd is 14.69x Dense versus 15.48x Ragged, and zstdict is
+22.10x versus 22.97x. The remaining difference comes from the H/W distribution
 and Ragged shape metadata, not a different image entropy model.
 
 ### Small Token Records
@@ -662,9 +667,9 @@ accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 1103 MiB/s | 2083 MiB/s | 1066.3 | 0.33 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 70 MiB/s | 2160 MiB/s | 1105.7 | 0.32 ms | 154.4 MiB | 2.53x |
-| zrecord-raw | 2178 MiB/s | 6862 MiB/s | 3513.2 | 0.10 ms | 393.7 MiB | 0.99x |
+| zrecord-zstd | 989 MiB/s | 2012 MiB/s | 1030.3 | 0.34 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 62 MiB/s | 2079 MiB/s | 1064.6 | 0.32 ms | 154.4 MiB | 2.53x |
+| zrecord-raw | 4542 MiB/s | 6391 MiB/s | 3272.3 | 0.11 ms | 393.7 MiB | 0.99x |
 | npy-mmap-raw | 1587 MiB/s | 9070 MiB/s | 4644.0 | 0.08 ms | 390.6 MiB | 1.00x |
 | lmdb-raw | 639 MiB/s | 1104 MiB/s | 565.0 | 0.71 ms | 786.3 MiB | 0.50x |
 | arrow-ipc-raw | 2068 MiB/s | 225 MiB/s | 115.0 | 2.81 ms | 390.8 MiB | 1.00x |
@@ -672,9 +677,9 @@ accumulate at least two seconds.
 | arrayrecord-zstd | 127 MiB/s | 135 MiB/s | 69.4 | 4.45 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.51 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.27 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-1.07–1.11 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.03–1.06 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
@@ -686,16 +691,16 @@ with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 271 MiB/s | 336 MiB/s | 634.2 | 0.54 ms | 67.4 MiB | 1.57x |
-| zrecord-zstdict | 58 MiB/s | 364 MiB/s | 688.2 | 0.50 ms | 49.1 MiB | 2.16x |
-| zrecord-raw | 546 MiB/s | 420 MiB/s | 794.2 | 0.44 ms | 110.5 MiB | 0.96x |
+| zrecord-zstd | 287 MiB/s | 315 MiB/s | 594.6 | 0.60 ms | 67.4 MiB | 1.57x |
+| zrecord-zstdict | 54 MiB/s | 332 MiB/s | 627.6 | 0.58 ms | 49.1 MiB | 2.16x |
+| zrecord-raw | 536 MiB/s | 389 MiB/s | 734.9 | 0.50 ms | 110.5 MiB | 0.96x |
 | lmdb-raw | 318 MiB/s | 113 MiB/s | 214.3 | 1.48 ms | 153.0 MiB | 0.69x |
 | arrow-ipc-raw | 562 MiB/s | 44 MiB/s | 83.1 | 4.00 ms | 109.9 MiB | 0.96x |
 | arrayrecord-raw | 288 MiB/s | 26 MiB/s | 49.3 | 7.39 ms | 118.8 MiB | 0.89x |
 | arrayrecord-zstd | 62 MiB/s | 33 MiB/s | 61.5 | 5.58 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 634–794 krecords/s with 0.44–0.54 ms p95;
+Zrecord's three codecs return 595–735 krecords/s with 0.50–0.60 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -742,8 +747,8 @@ and Grain reads ArrayRecord.
 
 | loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
 |---|---|---|---:|---:|---:|---:|---:|
-| **loaderx** | threads | zrecord-zstd | 166.3 | 15.95 ms | 985 MiB | 986 MiB | 988 MiB |
-| loaderx-raw | threads | zrecord-raw | 208.4 | 13.56 ms | 987 MiB | 987 MiB | 989 MiB |
+| **loaderx** | threads | zrecord-zstd | 171.5 | 12.69 ms | 984 MiB | 985 MiB | 988 MiB |
+| loaderx-raw | threads | zrecord-raw | 200.0 | 12.50 ms | 993 MiB | 993 MiB | 996 MiB |
 | torch | fork | npy-mmap-raw | 109.4 | 32.70 ms | 1783 MiB | 1889 MiB | 6396 MiB |
 | torch-spawn | spawn | npy-mmap-raw | 111.6 | 30.69 ms | 2835 MiB | 2913 MiB | 4880 MiB |
 | grain | processes | arrayrecord-zstd | 46.6 | 93.48 ms | 1847 MiB | 1946 MiB | 2065 MiB |
@@ -753,8 +758,8 @@ the transform threads overlap Python-side collation with the next gather. The
 memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
 threads inside one process, so workers share one interpreter, one NumPy runtime
 and one set of gather buffers. With source geometry and entropy held constant,
-raw is 1.25x compressed loaderx; compressed loaderx is 1.52x Torch fork, 1.49x
-Torch spawn and 3.57x Grain, while raw is 1.90x, 1.87x and 4.47x faster.
+raw is 1.17x compressed loaderx; compressed loaderx is 1.57x Torch fork, 1.54x
+Torch spawn and 3.68x Grain, while raw is 1.83x, 1.79x and 4.29x faster.
 Torch's aggregate RSS is high because
 Linux fork mappings are counted repeatedly; it is not a total-memory ratio
 against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
@@ -828,8 +833,8 @@ format: in the current Dense structured-vision workload, plain zstd reaches
 **Loader results combine architecture and storage.** loaderx uses threads and
 never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
 torch uses finite shuffled epochs, worker processes and shared-memory handoff.
-Here compressed loaderx is 1.52x Torch fork, 1.49x Torch spawn and 3.57x Grain;
-raw loaderx is 1.90x, 1.87x and 4.47x faster, respectively.
+Here compressed loaderx is 1.57x Torch fork, 1.54x Torch spawn and 3.68x Grain;
+raw loaderx is 1.83x, 1.79x and 4.29x faster, respectively.
 Storage also differs per loader — each reads from what it was
 built for — so the loader table is a different comparison from either store
 table, not a rerun.
@@ -1055,9 +1060,9 @@ license to trust storage or the operating system: native code still validates
     normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only
 defend against bypassing the public Python API do not belong in zrecord.
 
-1. `RecordEngine` stores N logically ordered records. Parallel chunks may finish
-   and occupy `data.zr` in a different physical order, but logical ID is the
-   stable append position and `RecordLoc[ID]` preserves it. Index and slice
+1. `RecordEngine` stores N logically ordered records. Payload record `i` belongs
+   to `data_{i % data_shards}.zr`; logical ID remains the stable append position
+   and `RecordLoc[ID]` preserves its shard-local offset. Index and slice
    operations are implemented as ordered gathers over those positions.
 2. It hands the container layer a dense sequence space: records are exactly
    `0..N-1`. Named streams are composed dynamically by a plain Python dict;
@@ -1120,24 +1125,33 @@ the authority for irreplaceable data. Keep authoritative source data and reprodu
 build scripts; after an interrupted build, storage failure, incompatible implementation
 change, or content change, rebuild a complete container at a new path.
 
-Native storage uses a fixed file set:
+Native storage uses one metadata file and a create-time-fixed payload file set:
 ```
 store/
-  ├── meta.zr      4096-byte static Header/schema page + RecordLoc table
-  ├── data.zr      payload stream
+  ├── meta.zr      4096-byte Header/schema/tails page + RecordLoc table
+  ├── data_0.zr    payload records where ID % data_shards == 0
+  ├── ...
+  ├── data_{N-1}.zr final static payload shard
   └── dict.zr      zstd dictionary (only in dict stores)
 ```
 
+`data_shards` is a write-performance parameter in `1..255`, fixed by `create`
+and recovered automatically by `open`. The default is four; practical values are
+usually 2, 4, 8 or 16, near the writer lane count. More files spread positional
+writes across payload inodes but consume one descriptor each. This physical
+striping does not change record IDs, order, codec, or read results.
+
 ### Metadata (meta.zr)
 
 Files are read and written **positionally** — pread/pwrite at computed offsets,
 no mmap. `meta.zr` starts with one fixed 4096-byte static page: a naturally
-aligned 32-byte Header, then the opaque MsgPack schema and unused zero padding.
+aligned 16-byte Header, 255 shard-local u64 tails at bytes 16..2055, then up to
+2040 bytes of opaque MsgPack schema at bytes 2056..4095.
 An array of 16-byte `RecordLoc`s starts at offset 4096. Record `i` is one
 pread/pwrite at `4096 + i * 16`; there is no variable table base, segment
 mapping, or rollover fd table.
 
-**1. Python schema** — bytes `32..32+schema_length` are exactly one immutable
+**1. Python schema** — bytes `2056..2056+schema_length` are exactly one immutable
 MsgPack object. Both stores contain ``dtype`` and ``ndim``; Dense additionally
 contains ``item_shape``. Native create persists these bytes together with
 the physical container but does not decode them. Open acquires the native lifetime
@@ -1147,30 +1161,36 @@ dtype/item_shape and passed to the native handle as runtime geometry; it is not
 independently persisted as a second authority. There is no format version or
 legacy kind dispatch.
 
-**2. Physical header** — the first 32 bytes of `meta.zr`. The format
+**2. Physical header** — the first 16 bytes of `meta.zr`. The format
 deliberately carries no payload or metadata checksum.
 
 * `codec` is the store's one compression method, stamped at creation and
   immutable — there is no per-record tag anywhere.
 * `length` (u64) is the physical record count; it equals logical length for both
   dense stores and inline ragged stores.
-* `tail_offset` (u64) is the absolute committed frontier in `data.zr`.
-* `schema_length` (u32) is the occupied prefix of the static schema area and
-  must be in `1..4064`.
+* `schema_length` (u16) is the occupied prefix of the static schema area and
+  must be in `1..2040`.
+* `data_shards` (u8) is the static payload file count and must be in `1..255`.
 
 ```zig
 const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
 const Header = extern struct {
     length: u64,
-    tail_offset: u64,
-    schema_length: u32,
-    reserved: [11]u8,
+    schema_length: u16,
+    data_shards: u8,
     codec: u8,
+    reserved: [4]u8,
 };
 ```
 
-**3. Record table** — contiguous 16-byte entries start at offset 4096 in
-`meta.zr` and grow as location runs are written. `offset` is an absolute byte offset in `data.zr`;
+**3. Shard frontiers** — tail slot `s` at `16 + s * 8` is the committed
+end of `data_s.zr`. Unused slots among the 255 fixed u64 entries are zero.
+Open requires every data file to be at least its persisted tail; locations may
+not cross that shard-local frontier.
+
+**4. Record table** — contiguous 16-byte entries start at offset 4096 in
+`meta.zr` and grow as location windows are written. `offset` is local to
+`data_{ID % data_shards}.zr`;
 `phys_length`/`logic_length` are the stored and original sizes. The
 codec is not here: it is the header's, so a record is stored exactly the way the
 store is declared.
@@ -1185,9 +1205,9 @@ const RecordLoc = extern struct {
 
 There is no liveness flag. Every entry below `length` is a record.
 
-**4. No fixed record-count cap.** The table and payload stream grow naturally in
-their fixed files. The practical bounds are the u64 count, supported positional
-file offsets, 2 GiB per record, and disk.
+**5. No fixed record-count cap.** The table and payload streams grow naturally.
+The practical bounds are the u64 count, supported positional file offsets,
+2 GiB per record, descriptor budget, and disk.
 
 ## Executor
 
@@ -1202,34 +1222,36 @@ while `Ragged` supplies offsets for its shape-prefixed records.
 The private ABI turns those inputs into compile-time record sources and
 destinations; the engine has one append and one gather operation. Its shared
 opaque handle remains private and carries no typed-store geometry.
-* Compressed append: workers claim record chunks and lease process-bounded
-  ``ExecutionSlot`` scratch, reserve physical offsets in completion order through a short frontier
-  lock, and issue positional payload writes in parallel. Logical IDs remain in
-  the loc table, so physical completion order does not change random gather.
-  The caller waits for every payload before publishing the new locations and
-  in-process length. Append does not overwrite the on-disk Header and is
-  intentionally lazy. Writer ``close()`` writes the current Header and closes
-  the files, making the page-cache state available for read-only open.
-  Every compressed frame owns one absolute range in `data.zr`; positional writes extend the file to the
-  current frontier. Python budgets the process-wide executor at three quarters
+* Append validates the complete call before physical I/O, then processes bounded
+  logical windows. Within a window, record positions are divided by
+  `ID % data_shards` and planned as bounded shard-local chunks. Fixed Executor
+  lanes dynamically claim those chunks, encode or pack them, briefly lock only
+  the selected shard's tail reservation, then issue positional payload writes
+  directly. Multiple lanes may write non-overlapping ranges of one shard; the
+  static file set spreads that pressure across inodes without limiting codec
+  concurrency to the shard count. Workers fill disjoint entries in one
+  contiguous location buffer. A payload barrier precedes one contiguous
+  `meta.zr` location write. All windows must succeed before the in-process
+  length and per-shard tails advance. Writer `close()` truncates each data file
+  to its committed tail and publishes the 4096-byte static page.
+* Compressed shard tasks lease process-bounded `ExecutionSlot` scratch and write
+  independent frames in bounded subchunks. Python budgets the process-wide executor at three quarters
   of the logical CPUs available to the process, leaving headroom for packing,
   transforms, and the caller without encoding a platform-specific thread count.
   Each producer configures its CCtx or shared immutable CDict once, then starts
   every independent record frame with `ZSTD_compress2`.
-* Raw append preserves the stronger invariant already supplied by Python: every
-  record in one Dense or packed Ragged batch is a boundary inside one contiguous
-  source buffer. After locating the records, the engine consumes that buffer in
-  plan order and passes each contiguous planned range directly to
-  `writePositionalAll` on `data.zr`. There is no per-record iovec construction,
-  byte-budget flush, payload copy, or platform-specific syscall path. Zig's
-  `std.Io` handles short writes and maps the same positional operation to POSIX
-  and Windows implementations.
+* Raw records for one shard are strided in the source. Each shard task packs a
+  bounded subchunk into its reusable scratch and performs one contiguous
+  positional write; a single record larger than the normal subchunk budget is
+  written directly. There is no per-record syscall or platform-specific vectored
+  path. The extra memory copy is the deliberate cost paid to remove concentrated
+  single-inode writes.
 **2. Read.** Fill the destination memory concurrently, in place from the Python
 side (executed on async threads).
 * Committed records are immutable and `length` is published through an atomic.
-  The fixed `meta.zr` and `data.zr` handles require no rollover fd-table
-  synchronization; record I/O itself stays lock free.
-* Every record is read at the offset its table entry records — the record table
+  The fixed metadata and static data handles require no rollover synchronization.
+* Every record first selects `data_{ID % data_shards}.zr`, then reads the
+  shard-local offset its table entry records — the record table
   is addressed by pure arithmetic, so random access is one pread for the
   location and one for the bytes, with no batching assumptions about layout.
   Each lane reads one location and immediately reads/decompresses that record;
@@ -1241,8 +1263,9 @@ budget; lanes for which the runtime cannot reserve concurrency run inline on the
 calling thread. Python configures the process-level budget as
 `max(physical cores, logical cores * 3 / 4)`, using platform topology where
 available.
-* Lanes receive contiguous blocks rather than a strided subset, keeping each
-  worker's reads and writes sequential.
+* Gather lanes receive contiguous request blocks. Append lanes dynamically claim
+  bounded chunks of strided shard ranges; short per-shard reservation locks keep
+  offsets disjoint while positional writes remain concurrent.
 * Each lane creates one zstd context (`ZSTD_CCtx` to write, `ZSTD_DCtx` to
   read) and reuses it across every record it handles, rather than paying that
   setup per record. The dictionary (`ZSTD_CDict`/`ZSTD_DDict`) is immutable, so
@@ -1252,10 +1275,10 @@ available.
 
 **4. File access.**
 * Metadata: one naturally growing `meta.zr`, containing the fixed Header/schema
-  page and loc table.
-* Payload: one naturally growing `data.zr`, accessed concurrently through
-  `readPositionalAll`/`writePositionalAll`. No path depends on filesystem
-  sparse-file support.
+  /tails page and loc table. It is intentionally not sharded because measured
+  write pressure is in payload I/O; one coordinator writes each loc window.
+* Payload: a static list of naturally growing `data_<shard>.zr` files, accessed through
+  `readPositionalAll`/`writePositionalAll`. No path depends on sparse files.
 
 **Execution model.** Opened readers are immutable, so calls on the same reader may
 gather concurrently. Creator appends are synchronous and native Storage