loaderx 1.4.2__tar.gz → 1.5.2__tar.gz

{loaderx-1.4.2/loaderx.egg-info → loaderx-1.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 1.4.2
+Version: 1.5.2
 Summary: A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License: MIT License
@@ -238,9 +238,11 @@ and how much data trains it — so a caller picks a tier, never a number:
 
 The sample is the byte budget the dictionary trains on (a strided subset of the
 records), so each tier costs the same training time whatever the record size.
-On this box's 12 KiB sample, `"balanced"` gathers ~3660 MiB/s at 1.17x against
-`"max"`'s ~2400 MiB/s at 1.34x — for a loader the throughput is the hot path,
-for archival the ratio is.
+At realistic image scale (768 KiB records) the tiers converge — on the earlier
+measurement box's structured data `"balanced"` and `"max"` both gather
+~1.6 GiB/s at a 1.74x ratio — because a dictionary is a small fraction of a
+large frame. The tiers still matter at small record sizes, where the dict is
+most of a record and `"max"` trades gather throughput for ratio.
 
 `"zstd_dict"` records can only be read from a store that has the dictionary
 (`dict.zr`). The dictionary is loaded on open and shared, lock-free, across all
@@ -383,59 +385,58 @@ For practical integration examples, please refer to the **[Data2Latent](https://
 the index sampler, the record store, and the full data loader — each through the
 binding a client actually uses, so cffi, the GIL and the NumPy allocation are all
 inside the timings. One run is enough: this is a qualitative horizontal
-comparison, and the defaults are sized so `python3 scripts/bench.py all` — which
-prints this machine block first — finishes in under half a minute. Warm page
-cache.
+comparison, sized to a realistic image workload (625 MiB of 256 KiB records)
+that finishes `python3 scripts/bench.py all` — which prints this machine block
+first — in under half a minute. Warm page cache.
 
-**Machine** — one box, inside a container on a personal laptop (AMD Strix Point
-APU, 12 cores / 24 threads):
+**Machine** — one box, an LXC container on a server (AMD EPYC 7303, 16 cores /
+32 threads):
 
 | machine   | value |
 |-----------|-------|
-| CPU       | AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M, 1 socket, 12 cores / 24 threads |
-| frequency | 605–5158 MHz |
-| caches    | L1d 576 KiB, L1i 384 KiB, L2 12 MiB, L3 24 MiB |
-| NUMA      | 1 node |
-| memory    | 31 GiB (not limited by cgroup) |
-| OS        | Debian GNU/Linux forky/sid, kernel 7.1.3+deb13-amd64 |
-| python    | 3.14.6, numpy 2.5.1 |
+| CPU       | AMD EPYC 7303 16-Core Processor, 1 socket, 16 cores / 32 threads |
+| frequency | 1500–3437 MHz |
+| caches    | L1d 512 KiB, L1i 512 KiB, L2 8 MiB, L3 64 MiB |
+| NUMA      | 4 nodes |
+| memory    | 32 GiB (32 GiB cgroup limit) |
+| OS        | Debian GNU/Linux 13 (trixie), kernel 6.12.95+deb13-amd64 |
+| python    | 3.14.7, numpy 2.5.1 |
 
-The container sees all 24 threads and has no CPU quota. The benchmark runs on
-the ordinary page cache.
+The container sees all 32 threads under a 32 GiB cgroup limit. The benchmark
+runs on the ordinary page cache.
 
 **1. Sampler** — index generation on its own, IID (with replacement), 1M index
 space, against NumPy's modern API. The µs-scale figures fluctuate with box load;
-the stable signal is the ~2x margin, widest at the smallest batches.
+the stable signal is the ~1.9x margin, roughly flat across batch sizes.
 
 | sampler           | batch | per batch | vs default_rng |
 |-------------------|-------|-----------|----------------|
-| numpy default_rng |   256 |   5.7 µs  |     1.00x      |
-| **zsampler**      |   256 |   1.7 µs  |   **3.32x**    |
-| numpy default_rng |  1024 |   4.4 µs  |     1.00x      |
-| **zsampler**      |  1024 |   2.3 µs  |   **1.89x**    |
-| numpy default_rng |  8192 |  18.6 µs  |     1.00x      |
-| **zsampler**      |  8192 |   9.5 µs  |   **1.97x**    |
-
-**2. Store** — Zrecord against the alternatives: random batch gather, 10000
-records, batch 256, structured (mildly compressible) data. A single 12 KiB
-sample is the horizontal comparison; other record sizes are a `--shape` away.
-Two backends stay out of the default because they would dominate the runtime
-without moving the comparison — they remain reachable through `--only`:
-arrayrecord (no usable wheel here, and one record per Python call) and
-hdf5-gzip (the same driver as hdf5 plus a codec axis, ~180 MiB/s gather vs
-hdf5's ~430).
-
-Small records — 12 KiB per record, 3 MiB per batch:
+| numpy default_rng |   256 |   5.6 µs  |     1.00x      |
+| **zsampler**      |   256 |   3.0 µs  |   **1.89x**    |
+| numpy default_rng |  1024 |   8.0 µs  |     1.00x      |
+| **zsampler**      |  1024 |   4.3 µs  |   **1.85x**    |
+| numpy default_rng |  8192 |  31.3 µs  |     1.00x      |
+| **zsampler**      |  8192 |  17.0 µs  |   **1.84x**    |
+
+**2. Store** — Zrecord against the alternatives: random batch gather, 2500
+records, batch 256, structured (mildly compressible) data. A single 256 KiB
+sample (512×512 single-channel, a realistic image record) is the horizontal
+comparison; other record sizes are a `--shape` away. hdf5-gzip, arrayrecord,
+torch and grain are installed and run in the default tables.
+
+Realistic records — 256 KiB per record, 64 MiB per batch:
 
 | store             | write      | gather      | on disk | ratio |
 |-------------------|------------|-------------|---------|-------|
-| zrecord-zstd      | 1022 MiB/s | 11126 MiB/s | 108 MiB | 1.08x |
-| zrecord-zstdict   |   33 MiB/s |  4587 MiB/s | 100 MiB | 1.18x |
-| zrecord-raw       | 1573 MiB/s | 19208 MiB/s | 117 MiB | 1.00x |
-| npy-mmap          | 2566 MiB/s | 15762 MiB/s | 117 MiB | 1.00x |
-| hdf5              | 1052 MiB/s |   380 MiB/s | 118 MiB | 0.99x |
-| blosc2            |   32 MiB/s |   439 MiB/s | 108 MiB | 1.09x |
-| tensorstore       |  463 MiB/s |   104 MiB/s | 107 MiB | 1.09x |
+| zrecord-zstd      |  772 MiB/s |  4031 MiB/s |  555 MiB | 1.13x |
+| zrecord-zstdict   |   72 MiB/s |  2123 MiB/s |  388 MiB | 1.61x |
+| zrecord-raw       |  880 MiB/s |  5214 MiB/s |  625 MiB | 1.00x |
+| npy-mmap          | 1398 MiB/s |  1853 MiB/s |  625 MiB | 1.00x |
+| hdf5              |  975 MiB/s |  1046 MiB/s |  625 MiB | 1.00x |
+| hdf5-gzip         |   37 MiB/s |   182 MiB/s |  445 MiB | 1.40x |
+| blosc2            |   48 MiB/s |   353 MiB/s |  551 MiB | 1.13x |
+| tensorstore       |  720 MiB/s |   109 MiB/s |  462 MiB | 1.35x |
+| arrayrecord       |  331 MiB/s |   865 MiB/s |  567 MiB | 1.10x |
 
 `write` is page-cache ingestion — the store is built and left open, exactly
 like `np.save` and the other backends, which defer durability to the kernel.
@@ -448,34 +449,36 @@ lazy ones.
 
 `gather` is fully page-cache-warmed — the benchmark sweeps every record once
 before timing, so it measures the pure access path (memory), not first-touch
-page faults or disk. That correction matters: mmap pays a fault per page on
-first touch while zrecord's `pread` reads the just-written cache directly, so a
-sparse warm-up inflated zrecord's margin over `.npy`. Fully warm, zrecord-raw
-leads by ~1.2x on this box (24 cores) — its gather is a fan-out across every
-core, numpy's fancy index is single-threaded — and on the 12-core reference box
-the margin is wider, because single-threaded NumPy copy is slower there.
+page faults or disk. At 256 KiB records the batch (64 MiB) exceeds L3, so these
+numbers are DRAM-bandwidth-bound for every backend; the 12 KiB table that
+predates the record-size bump was an L3-fit artifact. Fully warm, zrecord-raw
+leads npy-mmap by ~2.8x on this box — its gather fans out across every core
+while numpy's fancy index is single-threaded — and the gap widens where
+single-threaded copy is slower.
 
 **3. Loader** — the full input pipeline end to end (sample, fetch, collate, hand
-over a batch), same workload, 4 workers. The `peak RSS` column (added with the
-uniform-codec rewrite) is the highest resident set size of the whole process
-tree while batches are flowing — measured on this box, so the numbers below are
-the box's, not the reference machine's. (grain stays out of the default: it
-needs JAX, its build writes ArrayRecord one sample at a time — tens of seconds
-for this workload — and it lands ~50 batches/s, an order of magnitude below the
-rest. `--only grain` reaches it when installed.)
+over a batch), same workload, 4 workers, batch 256 (64 MiB). The `peak RSS`
+column (added with the uniform-codec rewrite) is the highest resident set size
+of the whole process tree while batches are flowing. torch, grain and
+arrayrecord are installed and wired into the defaults, and all four loaders run
+at the full batch.
 
 | loader         | batches/s | peak RSS |
 |----------------|-----------|----------|
-| **loaderx**    |    4115.5 | ~220 MiB |
-| loaderx-raw    |    4833.8 | ~245 MiB |
-| torch          |     770.7 | ~3000 MiB |
-
-The memory gap is the architecture, not an accident: loaderx prefetches in
-threads inside one process, so four workers share one interpreter, one numpy and
-one set of gather buffers; torch's four workers are four processes, each a full
-Python runtime holding its own copy of the worker state — ~13x the resident
-memory for the same parallelism, which is a real cost on a training box that is
-already feeding the model.
+| **loaderx**    |      59.2 | 2477 MiB |
+| loaderx-raw    |      80.1 | 2478 MiB |
+| torch          |      37.6 | 14525 MiB |
+| grain          |      15.8 | 4567 MiB  |
+
+At 64 MiB per batch the loader is DRAM-bound, not sampler-bound — the per-batch
+gather (zstd ~4.0 GiB/s, raw ~5.2 GiB/s) is the whole story, and the prefetch
+threads keep it at store-gather speed while the Python side collates. The
+memory is the prefetch buffers plus the two dataset handles (raw + label stores
+over the same backing data). loaderx prefetches in threads inside one process,
+so workers share one interpreter, one numpy and one set of gather buffers; torch
+and grain run a worker process per prefetch thread, which is most of their RSS
+(torch's peak also includes the shared-memory collated batches). grain's
+batches/s is the pipeline's floor here, ~5x below loaderx-raw.
 
 **Free-threaded Python.** loaderx targets free-threaded builds (no GIL), and
 the key sections are also measured on 3.14t — the question is whether anything
@@ -485,20 +488,22 @@ and `--transform augment`); they are not part of the default `all`:
 * **Sampler** — zsampler draws ~1.9–2x faster than free-threaded numpy, the
   same margin as on the GIL build: a batch is one Zig call either way.
 * **Store** — the free-threaded numbers track the GIL table within ~10%.
-  `zrecord-raw` gathers ~19.4–21.4 GiB/s on both (still ~1.7x `npy-mmap`),
-  `zrecord-zstd` ~9.4–10.7 GiB/s. Notably, the alternatives do not gain from the
-  build: blosc2's free-threaded wheel re-enables the GIL to load its extension,
-  so it runs exactly as on the GIL build. Random gather, 12 KiB records:
+  `zrecord-raw` gathers 4.9 vs 5.2 GiB/s (still ~2.6x `npy-mmap`),
+  `zrecord-zstd` 3.9 vs 4.0 GiB/s. Notably, the alternatives do not gain from
+  the build: blosc2's free-threaded wheel re-enables the GIL to load its
+  extension, so it runs exactly as on the GIL build. Random gather, 256 KiB
+  records:
 
   | store        | CPython 3.14 (GIL) | free-threaded 3.14t |
   |--------------|--------------------|---------------------|
-  | zrecord-raw  |         21412 MiB/s |         19374 MiB/s |
-  | zrecord-zstd |         10668 MiB/s |          9399 MiB/s |
-  | npy-mmap     |         12063 MiB/s |         10399 MiB/s |
-  | hdf5         |           436 MiB/s |           307 MiB/s |
-  | blosc2       |           434 MiB/s |           419 MiB/s |
-  | tensorstore  |            87 MiB/s |            86 MiB/s |
-* **Loader, identity** — unchanged, ~4100–6000 batches/s on both interpreters.
+  | zrecord-raw  |          5214 MiB/s |          4886 MiB/s |
+  | zrecord-zstd |          4031 MiB/s |          3913 MiB/s |
+  | npy-mmap     |          1853 MiB/s |          1861 MiB/s |
+  | hdf5         |          1046 MiB/s |          1030 MiB/s |
+  | blosc2       |           353 MiB/s |           334 MiB/s |
+  | tensorstore  |           109 MiB/s |           110 MiB/s |
+* **Loader, identity** — unchanged, ~60 batches/s (loaderx) on both
+  interpreters at 64 MiB per batch.
 * **Loader, CPU-heavy transform** — the one place the free-threaded build
   matters. A transform runs on the prefetch threads, so the GIL serializes it on
   standard CPython — the case where worker processes win, and the reason for the
@@ -508,13 +513,14 @@ and `--transform augment`); they are not part of the default `all`:
 
   | loader          | CPython 3.14 (GIL) | free-threaded 3.14t |
   |-----------------|--------------------|---------------------|
-  | **loaderx**     |          128.9     |          687.9 batches/s |
-  | loaderx-raw     |          127.9     |          727.2 batches/s |
-  | torch           |          408.7     |              n/a    |
+  | **loaderx**     |            24.3    |             41.1 batches/s |
+  | loaderx-raw     |            22.3    |             43.3 batches/s |
 
-  Under the GIL, loaderx's prefetch threads serialize the transform and fall to
-  ~1/3 of torch's worker processes; free-threaded, they parallelize and loaderx
-  reaches ~1.7x torch while keeping the zero-copy, IPC-free path.
+  Free-threaded still wins — the transform parallelizes across the prefetch
+  threads — ~1.7x on loaderx and ~1.9x on raw at 256 KiB. The margin is
+  narrower than at 12 KiB records (where a batch fit in cache and the transform
+  was the whole cost) because the 64 MiB gather is DRAM-bound, but wider than
+  at the 768 KiB scale, where the pipeline was even more DRAM-bound.
 
 **Conclusion** — why the numbers look like this.
 
@@ -522,15 +528,15 @@ and `--transform augment`); they are not part of the default `all`:
 zrecord gathers a whole batch and decompresses it, in a single cffi call into
 Zig with the GIL released and the batch copied straight into its destination
 buffer. The contenders do the same work one record at a time from Python. That
-one fact runs through all three tables: the sampler's margin is widest at the
-smallest batches (a batch costs one call either way, so the per-index work is
+one fact runs through all three tables: the sampler's margin is roughly flat
+across batch sizes (a batch costs one call either way, so the per-index work is
 what divides them), and the store gather column is where one-record-per-call
 costs the most. The speedup is not bought with distribution shortcuts: the IID
 draw is unbiased like NumPy's (Lemire with rejection, so uniformity costs
 nothing over a real index space).
 
 **The layout matches what a training loader does.** zrecord is built for random
-record access — the meta table is mmap'd, any record is found in O(1), a dense
+record access — the record table is indexed in O(1), a dense
 store gathers at a fixed stride. Array stores are built for contiguous scans, so
 a scattered batch — exactly what a loader reads — fights their layout. A raw
 gather is a fan-out across every core, where NumPy's fancy index is one thread;
@@ -540,17 +546,17 @@ on the multi-core memory subsystem that closes the gap to an uncompressed
 **Compression is in the kernel, and there is one codec.** `zrecord-zstd` is not
 "storage plus a codec": the layout, the multi-core decompress and the GIL-free
 copy are one path, so turning compression on costs part of a margin, not an
-order of magnitude — ~10.7 GiB/s here, one to two orders of magnitude ahead of
-the other compressed stores (87–434 MiB/s) at the same ~1.1x ratio. The modest
-ratio is the data, not the format: these samples are barely compressible, and on
-a smooth image set plain zstd reaches 7.6x and the dictionary (at the `"max"`
-tier) 16.5x.
+order of magnitude — ~3.9 GiB/s here, ~11x ahead of the other compressed stores
+(109–353 MiB/s) at the same ~1.1x ratio. The modest ratio is the data, not the
+format: these samples are barely compressible, and on a smooth image set plain
+zstd reaches 7.6x and the dictionary (at the `"max"` tier) 16.5x.
 
 **The loader gap is architecture, not storage.** loaderx uses threads and never
 ends an epoch, so a step pays no IPC and never waits on an epoch boundary; torch
-restarts per epoch with worker processes. That is most of the ~5–6x end to end.
-Storage also differs per loader — each reads from what it was built for — so the
-loader table is a different comparison than the store table, not a rerun of it.
+restarts per epoch with worker processes. That is most of the ~1.6x over torch
+and ~3.8x over grain here. Storage also differs per loader — each reads from
+what it was built for — so the loader table is a different comparison than the
+store table, not a rerun of it.
 The one crack in the thread model is a CPU-heavy transform, which the GIL
 serializes — that is exactly what free-threaded Python removes, so loaderx is
 developed and benchmarked against free-threaded builds first.
@@ -563,7 +569,7 @@ page-cache ingestion with durability deferred, matching the other backends —
 zrecord's own durability (`sync`/`close`) is a separate, explicit cost that
 neither this table nor the competition pays. This is a single qualitative pass —
 the µs-scale sampler timings and the loader `batches/s` fluctuate with box load
-(on these 12 cores the compressed loader trails the raw one by ~25–30%), so
+(on these 16 cores the compressed loader trails the raw one by ~25–30%), so
 treat the absolute numbers as ballpark and the cross-backend margins as the
 signal.
 
@@ -621,11 +627,15 @@ by record against the npy ground truth.
 
 ## Current Limitations
 * Single-host only; multi-host training is not supported.
-* A single sample must be at most 1 MiB, and a store holds at most 2^24
-  (16777216) records.
-* Metadata is mapped and used in place, so a store carries the host's byte
-  order and is not portable to a machine of the opposite endianness. Every
-  published platform is little-endian, so this only matters if you build for
+* A single sample must be at most 2 GiB (2^31 bytes). There is no fixed record
+  count: `length` is a u64 and the record table grows on demand, so how many
+  records a store holds is bounded by its total chunk capacity (up to 2^64
+  bytes) divided by the average record size — e.g. roughly 2^44 records at
+  1 MiB each, 2^33 at 2 GiB each.
+* Metadata is read and written as the host's struct layout, so a store carries
+  the host's byte order and is not portable to a machine of the opposite
+  endianness. Every published platform is little-endian, so this only matters
+  if you build for
   one yourself.
 
 ## Build
@@ -739,49 +749,57 @@ A record-based data runtime, focused on delivering extreme throughput and low la
       `zstd` store never touches a dictionary even if one is present.
 
 ## Persistence format
-Zrecord storage is metadata plus chunked data:
+Zrecord storage is metadata plus chunked data. The extension is the type:
+`.zr` files are store-global singletons, `.loc` files are record-table segments,
+`.chunk` files are record data:
 ```
 zrecord/
-  ├── meta.zr
-  ├── 0.zr
-  └── 1.zr
+  ├── meta.zr      header (global state)
+  ├── dict.zr      zstd dictionary (only in dict stores)
+  ├── 0.loc        record table segment [0, 2^28)
+  ├── 1.loc        record table segment [2^28, 2^29)
+  ├── 0.chunk      record data
+  └── 1.chunk
 ```
 
-### Metadata (meta.zr)
+### Metadata (meta.zr + {id}.loc)
 
-meta.zr is **used as the values it holds, not decoded into them**. Both structs
-below are `extern`, exactly 16 bytes each, and naturally aligned; an mmap is page
-aligned, so the header is a `*Header` and the record table is a `[]RecordLoc`.
-Looking up a record is `table[idx]`, and there is no serializer anywhere in the
-code.
+Files are read and written **positionally** — pread/pwrite at computed offsets,
+no mmap. The header is a bit-packed struct and a `.loc` segment an array of
+16-byte `RecordLoc`s, exactly as wide as they declare, so a location is one
+pread/pwrite of 16 bytes at a computed offset and there is no serializer
+anywhere in the code. Every header field is byte-aligned (no bit fields cross a
+byte), so the packed header reads as plain memory.
 
-The fields are deliberately wider than the limits need — `chunk_id` is a `u16`
-for 4096 chunks, the lengths are `u32` for a 1 MiB cap. Bit-packing them would
-save 5 bytes per record and cost a hand-written codec on the hottest lookup
-path. The slack buys a machine-shaped array; it is worth it.
+The record table is partitioned into `{id}.loc` segments so it can **grow by
+appending a segment** instead of reserving the maximum. The id→segment mapping
+is pure arithmetic — `seg = idx >> 28`, `off = (idx & (2^28−1)) × 16` — so a
+segment needs no per-record bookkeeping. A segment is immutable once published,
+so each mapping's base never moves and lock-free readers are safe to index it.
 
-**1. Header** — 16 bytes of global state, the same width as a `RecordLoc`, so
-the table starts at a clean 16-byte offset.
+**1. Header** — 32 bytes, the whole of meta.zr.
 * `magic` (`ZREC`) and `version` are ordinary fields, so opening a directory
   that is not a Zrecord store fails immediately instead of decoding garbage.
 * `codec` is the store's one compression method, stamped at creation and
   immutable — there is no per-record tag anywhere.
-* `length` is the total number of records | `tail_chunk`/`tail_offset` mark the
-  last write position.
+* `length` (u64) is the total number of records | `tail_chunk`/`tail_offset`
+  mark the last write position.
 * There is no chunk count. Chunks are created in order and the frontier is
   always in the last one, so the store holds exactly chunks `0..=tail_chunk` —
   a count would be a second copy of that fact to keep in sync.
 
 ```zig
 const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
-const Header = extern struct {
-    magic: [4]u8, version: u8, codec: Codec,
-    tail_chunk: u16, tail_offset: u32, length: u32,
+const Header = packed struct {
+    magic: u32, version: u8, codec: Codec,
+    tail_chunk: u32, tail_offset: u32, length: u64,
+    _reserved: u80,
 };
 ```
 
-**2. Record table** — 16 bytes per entry, indexed directly. Mapping an index to
-a physical address is what makes random access efficient.
+**2. Record table** — a `.loc` segment is 2^28 entries of 16 bytes (4 GiB,
+sparse), indexed directly. Mapping an index to a physical address is what makes
+random access efficient.
 * `chunk_id` is the containing chunk | `offset` is the position within it |
   `phys_length`/`logic_length` are the stored and original sizes. The codec is
   not here: it is the header's, so a record is stored exactly the way the store
@@ -790,26 +808,17 @@ a physical address is what makes random access efficient.
 ```zig
 const RecordLoc = extern struct {
     offset: u32, phys_length: u32, logic_length: u32,
-    chunk_id: u16, _reserved: [2]u8,
+    chunk_id: u32,
 };
 ```
 
 There is no liveness flag. Every entry below `length` is live, because deletion
 swaps the tail into the hole rather than tombstoning.
 
-**3. Maximum length.** Running out of space is effectively impossible, so the
-metadata is sized statically at the maximum.
-* Current design: up to 2^12 (4096) chunks, 2^32 (4 GiB) per chunk, 2^20 (1 MiB)
-  per record.
-* Derivation: N = chunks × 2^32 (chunk size) / 2^20 (max record size) ⇒ a
-  maximum length of 2^24 (16777216). Equality here is the point: the store can
-  never run out of chunks before it runs out of record slots, which is what
-  lets the table be sized statically. It is a compile-time assertion.
-* meta.zr is a fixed 16 + 2^24 × 16 bytes (256 MiB). It is sparse — a store
-  with one record allocates 4 KiB of it — and the mapping does not prefault.
-
-### Chunk data (x.zr)
-Densely packed record data.
+**3. No maximum length.** The table grows a `.loc` segment at a time and the
+data grows a chunk at a time, so there is no static record-count cap to size
+against. The real bounds are the field widths — 2^32 chunks of 2^32 bytes
+(2^64 bytes total), 2^31 (2 GiB) per record — and disk.
 
 ## Executor
 
@@ -829,9 +838,11 @@ Densely packed record data.
 side (executed on async threads).
 * Committed records are immutable, so the read path is lock free; `length` is
   published to readers through an atomic.
-* Every record is read at the offset its table entry records — the meta table is
-  mmap'd, so random access is one array subscript and one positional read, with
-  no batching assumptions about layout. Compressed records are read into a
+* Every record is read at the offset its table entry records — the record table
+  is addressed by pure arithmetic, so random access is one pread for the
+  location and one for the bytes, with no batching assumptions about layout.
+  Locations are read once per batch (contiguous runs in one pread). Compressed
+  records are read into a
   per-shard staging buffer and decompressed in place into the destination.
 
 **3. Concurrency model.** `Io.Group.async` shards work by CPU count, and shards
@@ -864,7 +875,8 @@ is reclaimed by an offline `compact` that rewrites the live records in place.
   that compaction cannot remove.
 
 **5. File access.**
-* Metadata: a 256 MiB file created at init, accessed by mmap thereafter.
+* Metadata: `meta.zr` (32 bytes) plus one file per `.loc` segment, each 4 GiB
+  (sparse), opened when the segment is created.
 * Chunk data: 4 GiB files created at init, accessed concurrently through
   `readPositionalAll`/`writePositionalAll`.