loaderx 2.0.7__tar.gz → 2.2.1__tar.gz

{loaderx-2.0.7 → loaderx-2.2.1}/MANIFEST.in

@@ -8,7 +8,7 @@ recursive-include scripts *.py
 include scripts/requirements-bench.txt
 
 # Vendored zstd: the sdist rebuilds with Zig, so it needs the C sources.
-recursive-include vendor *.c *.h
+recursive-include vendor *.c *.h *.S
 include vendor/zstd/LICENSE vendor/zstd/COPYING
 
 exclude loaderx/lib/lib_here

{loaderx-2.0.7/loaderx.egg-info → loaderx-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 2.0.7
+Version: 2.2.1
 Summary: Rebuildable high-performance ordered record containers
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License-Expression: MIT
@@ -121,25 +121,26 @@ reader. Open the result with ``Dense.open``:
     batch = ds[:]
     ds.close()
 
-Python defines the exact record schema: ``dtype`` plus the dense ``item_shape``,
-or only ``dtype`` for ragged stores. The MsgPack bytes live opaquely in the
+Python defines the exact record schema: both geometries store ``dtype`` and
+``ndim``; Dense additionally stores ``item_shape``. The MsgPack bytes live opaquely in the
 static page at the front of ``meta.zr``; Zig persists them but never interprets
 them. The schema accepts no user metadata. Python selects the geometry and gives
 the private native engine only the runtime record boundaries it needs. Each
-Ragged record carries a u8 rank followed by inline little-endian u64 dimensions.
+Ragged record carries exactly ``ndim`` inline little-endian u64 dimensions.
 One native physical
 engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
 strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
 iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
 ``np.frombuffer(raw, dtype=np.uint8)`` and stored in a
-``Ragged.create(path, dtype=np.uint8)`` rather than creating a second
+``Ragged.create(path, dtype=np.uint8, ndim=1)`` rather than creating a second
 public storage API.
 
 ### Records
 One persistent format, two native execution contracts. ``Dense`` is the **dense** contract
 where every record is exactly one row of the recorded ``item_shape``; reads
-are fixed-stride gathers and the batch shape follows from the schema, so no
-per-record metadata is touched:
+allocate a fixed-stride destination whose batch shape follows from the schema.
+Physical records still use the shared ``RecordLoc[logical_id] -> payload``
+pipeline, so compressed completion order never becomes a second Dense layout:
 
 ```python
 import numpy as np
@@ -160,11 +161,11 @@ the same way.
 
 ``Ragged`` is the **ragged** contract for variable-length records. It is a
 separate contract: :class:`Ragged` hands back a list of arrays, so a
-loader never has to carry ``row_splits`` around. ``dtype`` is unified and
-explicit; each record keeps its **own shape**, recorded per record as it is
-written and restored exactly on read — so records may differ in shape
-arbitrarily, and nothing is ever inferred from the source (an iterator can't
-tell you what its later records look like). Scalar ``shape == ()`` is preserved;
+loader never has to carry ``row_splits`` around. ``dtype`` and ``ndim`` are
+unified and explicit; each record keeps its **own dimension lengths**, recorded
+per record and restored exactly on read. Every axis may vary, but every record
+has the schema rank; nothing is inferred from the source. Ragged requires at
+least one axis; scalar records use Dense with ``item_shape=()``.
 zero-byte arrays are rejected because physical records are nonempty. Densifying a list into a dense
 batch is the model's call — a plain numpy loop, wherever you need it:
 
@@ -172,8 +173,8 @@ batch is the model's call — a plain numpy loop, wherever you need it:
 from loaderx.zrecord import Ragged
 
 seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
-with Ragged.create('tokens', np.int32) as rs:
-    rs.append(seqs)                       # dtype explicit; each record keeps its shape
+with Ragged.create('tokens', np.int32, ndim=1) as rs:
+    rs.append(seqs)                       # dtype/rank fixed; lengths remain per-record
 rs = Ragged.open('tokens')
 
 records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
@@ -273,7 +274,7 @@ ds.append(images[i:i + 1024])        # synchronous native batch; returns None
 ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
 ds.close()                            # publish before opening
 
-tok = Ragged.create('tokens', dtype=np.int32)
+tok = Ragged.create('tokens', dtype=np.int32, ndim=1)
 tok.append([seq_a, seq_b, seq_c])
 tok.close()
 
@@ -288,12 +289,14 @@ appended to. ``close()`` on a writer publishes the container Header;
 ``with`` for scoped lifetimes. Content is never changed in place: rerun the
 authoritative build at a new path, validate it, then switch consumers to it.
 
-The exact schema is declared at creation and encoded by Python as MsgPack. Dense
-schema contains only ``dtype`` and ``item_shape``; Ragged schema contains only
-``dtype``. Structured, subarray, object, and metadata-bearing dtypes are not
+The exact schema is declared at creation and encoded by Python as MsgPack. Both
+schemas contain ``dtype`` and ``ndim``; Dense additionally contains
+``item_shape`` and requires its length to equal ``ndim``. Ragged requires every
+appended record to have that rank, while every dimension length may vary.
+Structured, subarray, object, metadata-bearing, and zero-itemsize dtypes are not
 supported: their semantics do not round-trip through one canonical NumPy dtype
 string. The encoded schema has 4064 bytes available in the fixed 4096-byte
-metadata page. Ragged schema size is effectively fixed; Dense schema size grows
+metadata page. Ragged schema size is fixed; Dense schema size grows
 only with the integer ``item_shape``, so the physical limit is far above any
 practical NumPy array rank.
 Unexpected fields are rejected. ``append`` validates dtype and shape in Python,
@@ -310,7 +313,7 @@ published without one.
 for general-purpose compression — it is fast and the default.
 
 `"zstd_dict"` trains a shared dictionary on a sample of the data before writing
-any record, then compresses every record against it at level 19. The dictionary
+any record, then compresses every record against it at level 15. The dictionary
 captures structure shared across records that per-record compression cannot see —
 a large win for many small, similar records (image tiles, token sequences).
 
@@ -353,7 +356,8 @@ from loaderx.zrecord import Dense, Ragged
 d = train_dict(settled_array, tier="balanced")
 
 # then any new store can install it and append explicitly
-with Ragged.create('tokens', np.int32, codec='zstd_dict', dict_bytes=d) as ds:
+with Ragged.create('tokens', np.int32, ndim=1,
+                   codec='zstd_dict', dict_bytes=d) as ds:
     ds.append(token_generator)
 with Dense.create('data', data.dtype, data.shape[1:],
                        codec='zstd_dict', dict_bytes=d) as ds:
@@ -377,7 +381,8 @@ with Dense.open("src") as s, \
 ```
 
 ``Ragged`` is the same shape: ``s[:]`` returns ``list[np.ndarray]``, which
-is exactly its append input, and ``dtype`` is all the destination needs. The
+is exactly its append input; a destination is created with ``s.dtype`` and
+``s.ndim``. The
 native compression path bounds its own working memory; there is no public chunk
 parameter. ``dst`` must not already hold a store.
 
@@ -403,7 +408,7 @@ with Dense.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
     s.append(joint)
 with Dense.create(root + "/label", label.dtype, label.shape[1:]) as s:
     s.append(label)
-with Ragged.create(root + "/token", np.int32) as s:
+with Ragged.create(root + "/token", np.int32, ndim=1) as s:
     s.append(seqs)
 
 streams = {
@@ -496,9 +501,12 @@ so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.
 
 ### Methodology
 
-The results below are one complete run from the same checkout on a warm page
-cache. They are not three-run medians: an unexpected result is traced separately
-instead of being hidden by repeated aggregation. Within each store workload every
+The comparison matrix began as one complete run on a warm page cache. All
+Zrecord Store rows and the two loaderx identity rows were refreshed on 2.2.0
+after the libzstd 1.5.7 and runtime-dispatched Huffman kernel update. Unchanged
+optional Store backends, sampler rows and external loader rows remain from the
+immediately preceding 2.0.9 run. These are not three-run medians: an unexpected result is traced
+separately instead of being hidden by repeated aggregation. Within each store workload every
 backend receives identical source records. Correctness and timing use independent
 deterministic `zsampler` IID streams. Loader backends receive
 the same source and seed but use their own shipped samplers, so their exact
@@ -556,8 +564,8 @@ threads):
 | NUMA      | 1 node |
 | memory    | 31 GiB (not limited by cgroup) |
 | shared memory | 16 GiB `/dev/shm` |
-| OS        | Debian GNU/Linux forky/sid, kernel 7.1.3+deb13-amd64, x86_64 |
-| python    | CPython 3.14.7 (standard GIL build), numpy 2.5.2 |
+| OS        | Debian GNU/Linux forky/sid, kernel 7.1.8+deb13-amd64, x86_64 |
+| python    | CPython 3.14.7 (standard GIL build), numpy 2.4.6 |
 
 The benchmark process sees all 24 threads and is not memory-limited by cgroup.
 The 16 GiB shared-memory mount accommodates the four-worker, 36.8 MiB-batch torch
@@ -575,9 +583,9 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 7998 MiB/s | 10937 MiB/s | 76.2 | 4.37 ms | 24.4 MiB | 14.69x |
-| zrecord-zstdict | 35 MiB/s | 12173 MiB/s | 84.8 | 4.06 ms | 15.8 MiB | 22.74x |
-| zrecord-raw | 2523 MiB/s | 14329 MiB/s | 99.8 | 3.12 ms | 358.9 MiB | 1.00x |
+| zrecord-zstd | 6796 MiB/s | 10816 MiB/s | 75.3 | 4.09 ms | 24.4 MiB | 14.69x |
+| zrecord-zstdict | 102 MiB/s | 11844 MiB/s | 82.5 | 3.61 ms | 16.2 MiB | 22.10x |
+| zrecord-raw | 2801 MiB/s | 14655 MiB/s | 102.1 | 2.92 ms | 358.9 MiB | 1.00x |
 | npy-mmap-raw | 2004 MiB/s | 4749 MiB/s | 33.1 | 11.08 ms | 358.9 MiB | 1.00x |
 | hdf5-raw | 2350 MiB/s | 1836 MiB/s | 12.8 | 29.68 ms | 359.0 MiB | 1.00x |
 | hdf5-gzip | 277 MiB/s | 657 MiB/s | 4.6 | 62.53 ms | 26.1 MiB | 13.73x |
@@ -591,8 +599,8 @@ Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:
 | tiledb-raw | 743 MiB/s | 630 MiB/s | 4.4 | 66.01 ms | 359.0 MiB | 1.00x |
 | tiledb-zstd | 1498 MiB/s | 1503 MiB/s | 10.5 | 27.82 ms | 26.9 MiB | 13.36x |
 
-At 147 KiB per record, Zrecord-raw reaches 14.0 GiB/s and is 3.0x npy-mmap-raw;
-plain zstd gathers at 10.7 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
+At 147 KiB per record, Zrecord-raw reaches 14.4 GiB/s and is 3.1x npy-mmap-raw;
+plain zstd gathers at 10.6 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
 IPC are competitive raw record
 stores, while codecs tied to whole IPC batches or Parquet row groups pay read
 amplification on random gathers. Dense demonstrates that
@@ -610,9 +618,9 @@ list or a one-dimensional variable-length abstraction is not enough.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 2407 MiB/s | 9927 MiB/s | 59.6 | 5.44 ms | 26.9 MiB | 15.52x |
-| zrecord-zstdict | 37 MiB/s | 10814 MiB/s | 65.0 | 4.86 ms | 17.7 MiB | 23.58x |
-| zrecord-raw | 1591 MiB/s | 13032 MiB/s | 78.2 | 3.87 ms | 417.2 MiB | 1.00x |
+| zrecord-zstd | 2339 MiB/s | 9507 MiB/s | 57.1 | 5.16 ms | 26.9 MiB | 15.49x |
+| zrecord-zstdict | 102 MiB/s | 10336 MiB/s | 62.1 | 4.85 ms | 18.2 MiB | 22.98x |
+| zrecord-raw | 1420 MiB/s | 12929 MiB/s | 77.6 | 3.86 ms | 417.2 MiB | 1.00x |
 | hdf5-raw | 1451 MiB/s | 1059 MiB/s | 6.4 | 44.91 ms | 418.0 MiB | 1.00x |
 | hdf5-gzip | 250 MiB/s | 126 MiB/s | 0.7 | 359.23 ms | 29.9 MiB | 13.94x |
 | lmdb-raw | 1717 MiB/s | 7269 MiB/s | 43.6 | 7.93 ms | 422.2 MiB | 0.99x |
@@ -626,15 +634,15 @@ list or a one-dimensional variable-length abstraction is not enough.
 | tiledb-zstd | 675 MiB/s | 138 MiB/s | 0.8 | 328.05 ms | 27.0 MiB | 15.46x |
 
 Zrecord-raw is 1.8x LMDB and 3.2x Arrow IPC in logical gather. Zrecord-zstd
-delivers 9.7 GiB/s of
+delivers 9.3 GiB/s of
 logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
 ArrayRecord and TileDB
 show the same framework/codec tradeoffs in both tables; compressed batch, chunk
 and row-group formats pay read amplification on random records.
 
 The shared generator makes compression ratios directly comparable across
-contracts: Zrecord zstd is 14.69x Dense versus 15.52x Ragged, and zstdict is
-22.74x versus 23.58x. The remaining difference comes from the H/W distribution
+contracts: Zrecord zstd is 14.69x Dense versus 15.49x Ragged, and zstdict is
+22.10x versus 22.98x. The remaining difference comes from the H/W distribution
 and Ragged shape metadata, not a different image entropy model.
 
 ### Small Token Records
@@ -654,19 +662,19 @@ accumulate at least two seconds.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 1019 MiB/s | 1959 MiB/s | 1002.8 | 0.35 ms | 193.2 MiB | 2.02x |
-| zrecord-zstdict | 52 MiB/s | 2238 MiB/s | 1145.9 | 0.31 ms | 153.8 MiB | 2.54x |
-| zrecord-raw | 2101 MiB/s | 7441 MiB/s | 3810.0 | 0.09 ms | 393.7 MiB | 0.99x |
-| npy-mmap-raw | 1778 MiB/s | 11454 MiB/s | 5864.4 | 0.06 ms | 390.6 MiB | 1.00x |
+| zrecord-zstd | 1103 MiB/s | 2083 MiB/s | 1066.3 | 0.33 ms | 193.2 MiB | 2.02x |
+| zrecord-zstdict | 70 MiB/s | 2160 MiB/s | 1105.7 | 0.32 ms | 154.4 MiB | 2.53x |
+| zrecord-raw | 2178 MiB/s | 6862 MiB/s | 3513.2 | 0.10 ms | 393.7 MiB | 0.99x |
+| npy-mmap-raw | 1587 MiB/s | 9070 MiB/s | 4644.0 | 0.08 ms | 390.6 MiB | 1.00x |
 | lmdb-raw | 639 MiB/s | 1104 MiB/s | 565.0 | 0.71 ms | 786.3 MiB | 0.50x |
 | arrow-ipc-raw | 2068 MiB/s | 225 MiB/s | 115.0 | 2.81 ms | 390.8 MiB | 1.00x |
 | arrayrecord-raw | 807 MiB/s | 150 MiB/s | 76.7 | 4.99 ms | 401.4 MiB | 0.97x |
 | arrayrecord-zstd | 127 MiB/s | 135 MiB/s | 69.4 | 4.45 ms | 201.3 MiB | 1.94x |
 
 The contiguous NumPy baseline is strongest when the whole corpus is one fixed
-typed matrix. Zrecord-raw reaches 3.81 Mrecords/s while retaining independent
+typed matrix. Zrecord-raw reaches 3.51 Mrecords/s while retaining independent
 record semantics; the per-record zstd codecs halve disk and still return
-1.00–1.15 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
+1.07–1.11 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
 and disk.
 
 #### Variable Token Sequences
@@ -678,16 +686,16 @@ with exact `int32` values and original one-dimensional shapes.
 
 | backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
 |---|---:|---:|---:|---:|---:|---:|
-| zrecord-zstd | 236 MiB/s | 282 MiB/s | 533.7 | 0.66 ms | 67.5 MiB | 1.57x |
-| zrecord-zstdict | 43 MiB/s | 305 MiB/s | 576.9 | 0.61 ms | 49.2 MiB | 2.15x |
-| zrecord-raw | 486 MiB/s | 359 MiB/s | 679.0 | 0.52 ms | 110.7 MiB | 0.96x |
+| zrecord-zstd | 271 MiB/s | 336 MiB/s | 634.2 | 0.54 ms | 67.4 MiB | 1.57x |
+| zrecord-zstdict | 58 MiB/s | 364 MiB/s | 688.2 | 0.50 ms | 49.1 MiB | 2.16x |
+| zrecord-raw | 546 MiB/s | 420 MiB/s | 794.2 | 0.44 ms | 110.5 MiB | 0.96x |
 | lmdb-raw | 318 MiB/s | 113 MiB/s | 214.3 | 1.48 ms | 153.0 MiB | 0.69x |
 | arrow-ipc-raw | 562 MiB/s | 44 MiB/s | 83.1 | 4.00 ms | 109.9 MiB | 0.96x |
 | arrayrecord-raw | 288 MiB/s | 26 MiB/s | 49.3 | 7.39 ms | 118.8 MiB | 0.89x |
 | arrayrecord-zstd | 62 MiB/s | 33 MiB/s | 61.5 | 5.58 ms | 76.2 MiB | 1.39x |
 
 Here the record contract, not bulk byte bandwidth, is the useful scale.
-Zrecord's three codecs return 534–679 krecords/s with 0.52–0.66 ms p95;
+Zrecord's three codecs return 634–794 krecords/s with 0.44–0.54 ms p95;
 the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
 this pass.
 
@@ -734,8 +742,8 @@ and Grain reads ArrayRecord.
 
 | loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
 |---|---|---|---:|---:|---:|---:|---:|
-| **loaderx** | threads | zrecord-zstd | 167.1 | 14.63 ms | 986 MiB | 986 MiB | 989 MiB |
-| loaderx-raw | threads | zrecord-raw | 215.3 | 13.77 ms | 987 MiB | 987 MiB | 991 MiB |
+| **loaderx** | threads | zrecord-zstd | 166.3 | 15.95 ms | 985 MiB | 986 MiB | 988 MiB |
+| loaderx-raw | threads | zrecord-raw | 208.4 | 13.56 ms | 987 MiB | 987 MiB | 989 MiB |
 | torch | fork | npy-mmap-raw | 109.4 | 32.70 ms | 1783 MiB | 1889 MiB | 6396 MiB |
 | torch-spawn | spawn | npy-mmap-raw | 111.6 | 30.69 ms | 2835 MiB | 2913 MiB | 4880 MiB |
 | grain | processes | arrayrecord-zstd | 46.6 | 93.48 ms | 1847 MiB | 1946 MiB | 2065 MiB |
@@ -745,8 +753,8 @@ the transform threads overlap Python-side collation with the next gather. The
 memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
 threads inside one process, so workers share one interpreter, one NumPy runtime
 and one set of gather buffers. With source geometry and entropy held constant,
-raw is 1.29x compressed loaderx; compressed loaderx is 1.53x Torch fork, 1.50x
-Torch spawn and 3.59x Grain, while raw is 1.97x, 1.93x and 4.62x faster.
+raw is 1.25x compressed loaderx; compressed loaderx is 1.52x Torch fork, 1.49x
+Torch spawn and 3.57x Grain, while raw is 1.90x, 1.87x and 4.47x faster.
 Torch's aggregate RSS is high because
 Linux fork mappings are counted repeatedly; it is not a total-memory ratio
 against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
@@ -815,13 +823,13 @@ specialization for compression and a complete variable-shape persistence model.
 copy are one path, so turning compression on costs part of a margin, not an
 order of magnitude. The ratio is the data, not the
 format: in the current Dense structured-vision workload, plain zstd reaches
-14.69x and the balanced dictionary reaches 22.74x.
+14.69x and the balanced dictionary reaches 22.10x.
 
 **Loader results combine architecture and storage.** loaderx uses threads and
 never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
 torch uses finite shuffled epochs, worker processes and shared-memory handoff.
-Here compressed loaderx is 1.53x Torch fork, 1.50x Torch spawn and 3.59x Grain;
-raw loaderx is 1.97x, 1.93x and 4.62x faster, respectively.
+Here compressed loaderx is 1.52x Torch fork, 1.49x Torch spawn and 3.57x Grain;
+raw loaderx is 1.90x, 1.87x and 4.47x faster, respectively.
 Storage also differs per loader — each reads from what it was
 built for — so the loader table is a different comparison from either store
 table, not a rerun.
@@ -1059,7 +1067,7 @@ defend against bypassing the public Python API do not belong in zrecord.
    byte geometry. Dense stores persist
    one fixed-width physical record per logical record. Ragged stores
    persist one variable-width physical record per logical record:
-   `[u64le ndim][u64le dims...][payload]`. Shape and payload therefore share one
+   `[u64le dim] * schema.ndim + [payload]`. Shape and payload therefore share one
    location, codec frame, and append publication.
 4. The IO model (`append | read`) is batch-oriented and
    shape-agnostic. Per-call adapters expose record boundaries through a compile-time
@@ -1076,7 +1084,7 @@ defend against bypassing the public Python API do not belong in zrecord.
 |-------|-----------|----------------------------------------|
 |   0   |  raw      | none                                   |
 |   1   |  zstd     | zstd (plain, level 3)                  |
-|   2   |  zstdict  | zstd with a trained dictionary (level 19) |
+|   2   |  zstdict  | zstd with a trained dictionary (level 15) |
 ```
 
 7. Compression is transparent to the client:
@@ -1096,19 +1104,21 @@ defend against bypassing the public Python API do not belong in zrecord.
       dictionary carries the structure shared across records, which per-record
       compression cannot see. On many small, similar records (image tiles, token
       sequences) this is a large win: the current Dense structured-vision set
-      is 14.69x with plain zstd and 22.74x with the balanced dictionary. The dictionary is loaded once on
+      is 14.69x with plain zstd and 22.10x with the balanced dictionary. The dictionary is loaded once on
       open and shared, lock-free, across all reader threads. The dictionary size
       is chosen from the `DICT_TIERS` presets (see Codec notes).
     * A `zstd_dict` store needs its dictionary to read every record; a `raw` or
       `zstd` store rejects an unexpected dictionary as malformed state.
 
 ## Persistence format
-The 2.0 clean break does not change the current on-disk format. That format is
-the current implementation only: there is no compatibility layer, migration,
-version dispatch, checksum, or recovery facility. Zrecord is not the authority
-for irreplaceable data. Keep authoritative source data and reproducible build
-scripts; after an interrupted build, storage failure, implementation change, or
-content change, rebuild a complete container at a new path.
+The current format is the settled internal baseline for implementation work: optimizations keep
+the fixed files, Header/schema page, contiguous ``RecordLoc`` table and independent
+record payloads unless the product boundary is deliberately reopened. "Settled"
+does not promise cross-version persistence compatibility: there is no compatibility
+layer, migration, version dispatch, checksum, or recovery facility. Zrecord is not
+the authority for irreplaceable data. Keep authoritative source data and reproducible
+build scripts; after an interrupted build, storage failure, incompatible implementation
+change, or content change, rebuild a complete container at a new path.
 
 Native storage uses a fixed file set:
 ```
@@ -1128,8 +1138,8 @@ pread/pwrite at `4096 + i * 16`; there is no variable table base, segment
 mapping, or rollover fd table.
 
 **1. Python schema** — bytes `32..32+schema_length` are exactly one immutable
-MsgPack object. Dense stores contain only ``dtype`` and ``item_shape``; Ragged
-stores contain only ``dtype``. Native create persists these bytes together with
+MsgPack object. Both stores contain ``dtype`` and ``ndim``; Dense additionally
+contains ``item_shape``. Native create persists these bytes together with
 the physical container but does not decode them. Open acquires the native lifetime
 lock before copying the schema to Python for validation, so schema and physical
 metadata are one locked snapshot. Dense record width is derived once from
@@ -1192,8 +1202,8 @@ while `Ragged` supplies offsets for its shape-prefixed records.
 The private ABI turns those inputs into compile-time record sources and
 destinations; the engine has one append and one gather operation. Its shared
 opaque handle remains private and carries no typed-store geometry.
-* Compressed append: workers claim individual records and reuse one operation-local
-  frame buffer per task, reserve physical offsets in completion order through a short frontier
+* Compressed append: workers claim record chunks and lease process-bounded
+  ``ExecutionSlot`` scratch, reserve physical offsets in completion order through a short frontier
   lock, and issue positional payload writes in parallel. Logical IDs remain in
   the loc table, so physical completion order does not change random gather.
   The caller waits for every payload before publishing the new locations and