loaderx 0.5.0__tar.gz → 0.5.4__tar.gz

{loaderx-0.5.0/loaderx.egg-info → loaderx-0.5.4}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 0.5.0
-Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Version: 0.5.4
+Summary: A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License: MIT License
         
@@ -45,7 +45,7 @@ Requires-Dist: cffi
 Dynamic: license-file
 
 # Loaderx
-A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines.
+A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 
 ```
 pip install loaderx
@@ -115,6 +115,93 @@ run them while a `DataLoader` is iterating the dataset.
 
 For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository
 
+## Benchmarks
+
+`scripts/bench.py` measures loaderx against the alternatives in three layers —
+the index sampler, the record store, and the full data loader — each through the
+binding a client actually uses, so cffi, the GIL and the NumPy allocation are all
+inside the timings. The numbers below are one run on a 24-core machine, Python
+3.14, warm page cache. Reproduce with `python3 scripts/bench.py all`.
+
+**1. Sampler** — index generation on its own, IID (with replacement), 1M index
+space, against both NumPy random APIs.
+
+| sampler           | batch | per batch | indices   | vs default_rng |
+|-------------------|-------|-----------|-----------|----------------|
+| numpy randint     |   256 |   5.9 µs  |   44 M/s  |     0.64x      |
+| numpy default_rng |   256 |   3.7 µs  |   69 M/s  |     1.00x      |
+| **zsampler**      |   256 |   1.9 µs  |  138 M/s  |   **2.01x**    |
+| numpy randint     |  1024 |   7.2 µs  |  142 M/s  |     0.83x      |
+| numpy default_rng |  1024 |   6.0 µs  |  171 M/s  |     1.00x      |
+| **zsampler**      |  1024 |   3.0 µs  |  340 M/s  |   **1.99x**    |
+| numpy randint     |  8192 |  25.6 µs  |  320 M/s  |     0.65x      |
+| numpy default_rng |  8192 |  16.6 µs  |  493 M/s  |     1.00x      |
+| **zsampler**      |  8192 |   7.8 µs  | 1048 M/s  |   **2.13x**    |
+
+Zsampler draws indices about 2x faster than NumPy's faster API — more against the
+legacy `randint` most code still calls — and the lead is widest at the small
+batches a loader actually draws, where a batch is one cffi call either way and
+NumPy's vectorised fill has nothing to amortise. The one concession: zsampler's
+IID draw is modulo-biased where NumPy's is not, ~1e-13 over a u64 draw into a
+million-record space, which is below anything training will notice.
+
+**2. Store** — random batch gather, 20000 records × 12 KiB `uint8(3,64,64)`,
+batch 256 × 200, structured (mildly compressible) data.
+
+| store         | write      | gather      | per batch  | on disk | ratio |
+|---------------|------------|-------------|------------|---------|-------|
+| loaderx-flate |  362 MiB/s |  1671 MiB/s |   1.80 ms  | 209 MiB | 1.12x |
+| loaderx-raw   |  784 MiB/s | 21997 MiB/s |   0.14 ms  | 235 MiB | 1.00x |
+| npy-mmap      | 3064 MiB/s | 13692 MiB/s |   0.22 ms  | 234 MiB | 1.00x |
+| arrayrecord   |  374 MiB/s |   847 MiB/s |   3.54 ms  | 217 MiB | 1.08x |
+| hdf5          | 1327 MiB/s |   511 MiB/s |   5.87 ms  | 236 MiB | 0.99x |
+| hdf5-gzip     |   83 MiB/s |   206 MiB/s |  14.57 ms  | 211 MiB | 1.11x |
+| lmdb          |  677 MiB/s |  5337 MiB/s |   0.56 ms  | 313 MiB | 0.75x |
+| lmdb-zlib     |   71 MiB/s |   336 MiB/s |   8.93 ms  | 235 MiB | 1.00x |
+| blosc2        |   32 MiB/s |   226 MiB/s |  13.27 ms  | 216 MiB | 1.09x |
+| zarr          |   83 MiB/s |    28 MiB/s | 107.36 ms  | 217 MiB | 1.08x |
+| tensorstore   |  531 MiB/s |    98 MiB/s |  30.69 ms  | 214 MiB | 1.09x |
+
+The gather column is what the store is built for. loaderx-raw returns a scattered
+batch faster than everything here, a memory-mapped `.npy` included, because it
+copies straight into the destination across all cores with the GIL released;
+loaderx-flate keeps most of that speed *while* decompressing, and still gathers
+several times faster than any other compressed store (hdf5-gzip, lmdb-zlib,
+blosc2, zarr, tensorstore). The array stores lag on gather because a scattered
+single-record read fights a layout meant for contiguous slabs — which is exactly
+the access pattern a training loader has. Where loaderx does not lead is write: it
+pays compression and its own layout up front, a one-time cost it does not chase.
+Compression is modest (1.12x) only because this data is barely compressible; the
+row's point is that turning it on costs almost nothing on read.
+
+The reads are not all single-threaded: loaderx and blosc2 decompress across cores,
+the rest one record at a time. Zarr and TensorStore are sharded so `on disk`
+reflects compression rather than per-file block rounding, and TensorStore is given
+gzip (its zarr3 default is uncompressed). `on disk` is allocated blocks, which is
+why LMDB's preallocated map lands below 1.00x.
+
+**3. Loader** — the full input pipeline end to end (sample, fetch, collate, hand
+over a batch), same workload, 4 workers. Each loader reads from what it is built
+for: loaderx from Zrecord, Grain from ArrayRecord, torch from a memory-mapped
+`.npy`, jax-dataloader from an in-memory array.
+
+| loader         | batches/s | per batch | samples/s | throughput  |
+|----------------|-----------|-----------|-----------|-------------|
+| **loaderx**    |     724.2 |  1.38 ms  |   185,406 |  2173 MiB/s |
+| loaderx-raw    |    5271.0 |  0.19 ms  | 1,349,378 | 15813 MiB/s |
+| grain          |      67.3 | 14.85 ms  |    17,240 |   202 MiB/s |
+| torch          |     832.2 |  1.20 ms  |   213,038 |  2497 MiB/s |
+| jax-dataloader |    3284.6 |  0.30 ms  |   840,864 |  9854 MiB/s |
+
+End to end, loaderx-raw serves batches several times faster than torch or Grain
+and ahead of jax-dataloader — and it does so from disk-backed storage, where
+jax-dataloader's number is an in-memory ceiling with no storage path at all. The
+default loaderx trades some of that for on-disk compression and still lands in
+torch's range while reading compressed records. loaderx and jax use threads;
+torch and Grain use worker processes that escape the GIL but pay IPC per batch.
+loaderx never ends an epoch, so no step waits on a boundary; the others restart
+per epoch, a cost that is in their numbers as it would be in training.
+
 ## Current Limitations
 * Single-host only; multi-host training is not supported.
 * A single sample must be at most 1 MiB, and a store holds at most 2^24
@@ -128,18 +215,14 @@ For practical integration examples, please refer to the **[Data2Latent](https://
 ```
 zig build                       # host shared objects, into loaderx/lib/
 zig build test                  # Zrecord suite, in both Debug and ReleaseFast
-zig build bench                 # append/gather throughput
 python3 tests/test_loaderx.py   # Python layer, against whichever build is importable
+python3 scripts/bench.py        # throughput, against other stores
 ```
 
-Two things about the shared objects are deliberate rather than incidental:
-
-* **libc is linked.** They are `dlopen`ed into CPython, and without libc they
-  export an unresolved `__tls_get_addr`, so Zig's thread-locals never bind and
-  `std.Thread` deadlocks on the first batch.
-* **Tests run in ReleaseFast too.** The shipped libraries are ReleaseFast, and a
-  codegen difference there has already produced a release-blocking bug that
-  Debug did not have. Test the configuration that ships.
+The Zig side is tested for behaviour only; throughput is measured from Python,
+through the binding a client actually uses. `scripts/bench.py` runs in three
+layers (`sampler`, `store`, `loader`, or `all`) and skips contenders that are not
+installed. See [Benchmarks](#benchmarks).
 
 ### Publishing
 Zig cross-compiles every target from one machine, so releases need no CI matrix:

{loaderx-0.5.0 → loaderx-0.5.4}/README.md

@@ -1,5 +1,5 @@
 # Loaderx
-A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines.
+A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 
 ```
 pip install loaderx
@@ -69,6 +69,93 @@ run them while a `DataLoader` is iterating the dataset.
 
 For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository
 
+## Benchmarks
+
+`scripts/bench.py` measures loaderx against the alternatives in three layers —
+the index sampler, the record store, and the full data loader — each through the
+binding a client actually uses, so cffi, the GIL and the NumPy allocation are all
+inside the timings. The numbers below are one run on a 24-core machine, Python
+3.14, warm page cache. Reproduce with `python3 scripts/bench.py all`.
+
+**1. Sampler** — index generation on its own, IID (with replacement), 1M index
+space, against both NumPy random APIs.
+
+| sampler           | batch | per batch | indices   | vs default_rng |
+|-------------------|-------|-----------|-----------|----------------|
+| numpy randint     |   256 |   5.9 µs  |   44 M/s  |     0.64x      |
+| numpy default_rng |   256 |   3.7 µs  |   69 M/s  |     1.00x      |
+| **zsampler**      |   256 |   1.9 µs  |  138 M/s  |   **2.01x**    |
+| numpy randint     |  1024 |   7.2 µs  |  142 M/s  |     0.83x      |
+| numpy default_rng |  1024 |   6.0 µs  |  171 M/s  |     1.00x      |
+| **zsampler**      |  1024 |   3.0 µs  |  340 M/s  |   **1.99x**    |
+| numpy randint     |  8192 |  25.6 µs  |  320 M/s  |     0.65x      |
+| numpy default_rng |  8192 |  16.6 µs  |  493 M/s  |     1.00x      |
+| **zsampler**      |  8192 |   7.8 µs  | 1048 M/s  |   **2.13x**    |
+
+Zsampler draws indices about 2x faster than NumPy's faster API — more against the
+legacy `randint` most code still calls — and the lead is widest at the small
+batches a loader actually draws, where a batch is one cffi call either way and
+NumPy's vectorised fill has nothing to amortise. The one concession: zsampler's
+IID draw is modulo-biased where NumPy's is not, ~1e-13 over a u64 draw into a
+million-record space, which is below anything training will notice.
+
+**2. Store** — random batch gather, 20000 records × 12 KiB `uint8(3,64,64)`,
+batch 256 × 200, structured (mildly compressible) data.
+
+| store         | write      | gather      | per batch  | on disk | ratio |
+|---------------|------------|-------------|------------|---------|-------|
+| loaderx-flate |  362 MiB/s |  1671 MiB/s |   1.80 ms  | 209 MiB | 1.12x |
+| loaderx-raw   |  784 MiB/s | 21997 MiB/s |   0.14 ms  | 235 MiB | 1.00x |
+| npy-mmap      | 3064 MiB/s | 13692 MiB/s |   0.22 ms  | 234 MiB | 1.00x |
+| arrayrecord   |  374 MiB/s |   847 MiB/s |   3.54 ms  | 217 MiB | 1.08x |
+| hdf5          | 1327 MiB/s |   511 MiB/s |   5.87 ms  | 236 MiB | 0.99x |
+| hdf5-gzip     |   83 MiB/s |   206 MiB/s |  14.57 ms  | 211 MiB | 1.11x |
+| lmdb          |  677 MiB/s |  5337 MiB/s |   0.56 ms  | 313 MiB | 0.75x |
+| lmdb-zlib     |   71 MiB/s |   336 MiB/s |   8.93 ms  | 235 MiB | 1.00x |
+| blosc2        |   32 MiB/s |   226 MiB/s |  13.27 ms  | 216 MiB | 1.09x |
+| zarr          |   83 MiB/s |    28 MiB/s | 107.36 ms  | 217 MiB | 1.08x |
+| tensorstore   |  531 MiB/s |    98 MiB/s |  30.69 ms  | 214 MiB | 1.09x |
+
+The gather column is what the store is built for. loaderx-raw returns a scattered
+batch faster than everything here, a memory-mapped `.npy` included, because it
+copies straight into the destination across all cores with the GIL released;
+loaderx-flate keeps most of that speed *while* decompressing, and still gathers
+several times faster than any other compressed store (hdf5-gzip, lmdb-zlib,
+blosc2, zarr, tensorstore). The array stores lag on gather because a scattered
+single-record read fights a layout meant for contiguous slabs — which is exactly
+the access pattern a training loader has. Where loaderx does not lead is write: it
+pays compression and its own layout up front, a one-time cost it does not chase.
+Compression is modest (1.12x) only because this data is barely compressible; the
+row's point is that turning it on costs almost nothing on read.
+
+The reads are not all single-threaded: loaderx and blosc2 decompress across cores,
+the rest one record at a time. Zarr and TensorStore are sharded so `on disk`
+reflects compression rather than per-file block rounding, and TensorStore is given
+gzip (its zarr3 default is uncompressed). `on disk` is allocated blocks, which is
+why LMDB's preallocated map lands below 1.00x.
+
+**3. Loader** — the full input pipeline end to end (sample, fetch, collate, hand
+over a batch), same workload, 4 workers. Each loader reads from what it is built
+for: loaderx from Zrecord, Grain from ArrayRecord, torch from a memory-mapped
+`.npy`, jax-dataloader from an in-memory array.
+
+| loader         | batches/s | per batch | samples/s | throughput  |
+|----------------|-----------|-----------|-----------|-------------|
+| **loaderx**    |     724.2 |  1.38 ms  |   185,406 |  2173 MiB/s |
+| loaderx-raw    |    5271.0 |  0.19 ms  | 1,349,378 | 15813 MiB/s |
+| grain          |      67.3 | 14.85 ms  |    17,240 |   202 MiB/s |
+| torch          |     832.2 |  1.20 ms  |   213,038 |  2497 MiB/s |
+| jax-dataloader |    3284.6 |  0.30 ms  |   840,864 |  9854 MiB/s |
+
+End to end, loaderx-raw serves batches several times faster than torch or Grain
+and ahead of jax-dataloader — and it does so from disk-backed storage, where
+jax-dataloader's number is an in-memory ceiling with no storage path at all. The
+default loaderx trades some of that for on-disk compression and still lands in
+torch's range while reading compressed records. loaderx and jax use threads;
+torch and Grain use worker processes that escape the GIL but pay IPC per batch.
+loaderx never ends an epoch, so no step waits on a boundary; the others restart
+per epoch, a cost that is in their numbers as it would be in training.
+
 ## Current Limitations
 * Single-host only; multi-host training is not supported.
 * A single sample must be at most 1 MiB, and a store holds at most 2^24
@@ -82,18 +169,14 @@ For practical integration examples, please refer to the **[Data2Latent](https://
 ```
 zig build                       # host shared objects, into loaderx/lib/
 zig build test                  # Zrecord suite, in both Debug and ReleaseFast
-zig build bench                 # append/gather throughput
 python3 tests/test_loaderx.py   # Python layer, against whichever build is importable
+python3 scripts/bench.py        # throughput, against other stores
 ```
 
-Two things about the shared objects are deliberate rather than incidental:
-
-* **libc is linked.** They are `dlopen`ed into CPython, and without libc they
-  export an unresolved `__tls_get_addr`, so Zig's thread-locals never bind and
-  `std.Thread` deadlocks on the first batch.
-* **Tests run in ReleaseFast too.** The shipped libraries are ReleaseFast, and a
-  codegen difference there has already produced a release-blocking bug that
-  Debug did not have. Test the configuration that ships.
+The Zig side is tested for behaviour only; throughput is measured from Python,
+through the binding a client actually uses. `scripts/bench.py` runs in three
+layers (`sampler`, `store`, `loader`, or `all`) and skips contenders that are not
+installed. See [Benchmarks](#benchmarks).
 
 ### Publishing
 Zig cross-compiles every target from one machine, so releases need no CI matrix:

{loaderx-0.5.0 → loaderx-0.5.4}/build.zig

@@ -86,34 +86,24 @@ pub fn build(b: *std.Build) void {
         }
     }
 
-    // ---- host-only development tools ----
+    // ---- tests ----
     //
     // Pinned to the host rather than `target`: a run step for a foreign binary
     // cannot be configured on a machine that has no emulator for it, which
     // would break `zig build -Dtarget=...` outright.
-    const host = b.graph.host;
-
-    const bench = b.addExecutable(.{
-        .name = "bench",
-        .root_module = b.createModule(.{
-            .root_source_file = b.path("src/bench.zig"),
-            .target = host,
-            .optimize = optimize,
-            .link_libc = true,
-        }),
-    });
-    b.step("bench", "Benchmark zrecord append/gather throughput")
-        .dependOn(&b.addRunArtifact(bench).step);
-
-    // Run the suite at both ends of the optimizer. Debug alone is not enough:
-    // codegen differences around packed-struct stores have already produced a
-    // bug that only existed in the shipped ReleaseFast library.
+    //
+    // Run at both ends of the optimizer. Debug alone is not enough: codegen
+    // differences have already produced a bug that only existed in the shipped
+    // ReleaseFast library.
+    //
+    // Throughput is not measured here — it is measured through the binding that
+    // clients actually use, by `scripts/bench.py`.
     const test_step = b.step("test", "Run zrecord tests");
     for ([_]std.builtin.OptimizeMode{ .Debug, .ReleaseFast }) |mode| {
         const tests = b.addTest(.{
             .root_module = b.createModule(.{
                 .root_source_file = b.path("src/zrecord.zig"),
-                .target = host,
+                .target = b.graph.host,
                 .optimize = mode,
                 .link_libc = true,
             }),

{loaderx-0.5.0 → loaderx-0.5.4}/loaderx/__init__.py

@@ -2,4 +2,4 @@ from ._zrecord import Zrecord, ZrecordError
 from .dataset import Dataset
 from .dataloader import DataLoader
 
-__version__ = "0.5.0"
+__version__ = "0.5.4"

{loaderx-0.5.0 → loaderx-0.5.4}/loaderx/_sampler.py

@@ -1,5 +1,3 @@
-import time
-
 import numpy as np
 from cffi import FFI
 
@@ -54,44 +52,4 @@ class Sampler:
     def __del__(self):
         if hasattr(self, "sampler") and self.sampler != ffi.NULL:
             lib.cdeinit(self.sampler)
-            self.sampler = ffi.NULL
-
-def run():
-    from itertools import islice
-    length = 10
-    batch_size = 4
-    n_steps = 5
-
-    print("=== Sampler SEQUENTIAL ===")
-    sampler = Sampler(length, batch_size, Sampler.Mode.SEQUENTIAL)
-    for indices in islice(sampler, n_steps):
-        print(indices)
-
-    print("=== Sampler IID ===")
-    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
-    for indices in islice(sampler, n_steps):
-        print(indices)
-
-# Benchmark, zig-sampler speeder 3.66x
-def bench():
-    length = 1_000_000
-    batch_size = 8192
-    n_steps = 10_000
-
-    print("=== NumPy IID ===")
-    t0 = time.time()
-    batch = np.zeros(batch_size, dtype=np.uint64)
-    for _ in range(n_steps):
-        batch[:] = np.random.randint(0, length, size=batch_size, dtype=np.uint64)
-    print(f"{(time.time() - t0)*1000:.2f} ms")
-
-    print("=== Sampler IID ===")
-    t0 = time.time()
-    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
-    for _ in range(n_steps):
-        sampler.next()
-    print(f"{(time.time() - t0)*1000:.2f} ms")
-
-if __name__ == "__main__":
-    run()
-    bench()
+            self.sampler = ffi.NULL

{loaderx-0.5.0 → loaderx-0.5.4}/loaderx/dataloader.py

@@ -1,6 +1,6 @@
 import threading
 import numpy as np
-from queue import Queue
+from queue import Queue, Empty, Full
 
 from ._sampler import Sampler
 
@@ -42,12 +42,29 @@ class DataLoader:
             thread.daemon = True
             thread.start()
 
+    # Every blocking queue op polls stop_signal on a timeout instead of waiting
+    # forever. A plain get()/put() can outlive close(): once the sampler stops,
+    # a prefetch worker blocked on an empty index queue — or on a full batch
+    # queue with no consumer — never wakes, and join() hangs. Benchmarks build
+    # and tear down loaders in a loop, so this path is always hit.
+    _POLL = 0.1
+
+    def _put(self, queue, item):
+        """Put `item`, returning False if close() fired before it landed."""
+        while not self.stop_signal.is_set():
+            try:
+                queue.put(item, timeout=self._POLL)
+                return True
+            except Full:
+                continue
+        return False
+
     def _sampler(self):
         """
         Sample indices from the dataset and put them into the index queue.
         """
         while not self.stop_signal.is_set():
-            self.indices.put(self.sampler.next())
+            self._put(self.indices, self.sampler.next())
 
     def _prefetch(self, transform):
         """
@@ -55,9 +72,12 @@ class DataLoader:
         in the index queue and put them into the batches queue.
         """
         while not self.stop_signal.is_set():
-            idxs = self.indices.get()
+            try:
+                idxs = self.indices.get(timeout=self._POLL)
+            except Empty:
+                continue
             data, label = transform((self.dataset.__getitems__(idxs), self.labelset.__getitems__(idxs)))
-            self.batches.put({'data': data, 'label': label})
+            self._put(self.batches, {'data': data, 'label': label})
 
     def __len__(self):
         """

{loaderx-0.5.0 → loaderx-0.5.4/loaderx.egg-info}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 0.5.0
-Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Version: 0.5.4
+Summary: A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License: MIT License
         
@@ -45,7 +45,7 @@ Requires-Dist: cffi
 Dynamic: license-file
 
 # Loaderx
-A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines.
+A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines
 
 ```
 pip install loaderx
@@ -115,6 +115,93 @@ run them while a `DataLoader` is iterating the dataset.
 
 For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository
 
+## Benchmarks
+
+`scripts/bench.py` measures loaderx against the alternatives in three layers —
+the index sampler, the record store, and the full data loader — each through the
+binding a client actually uses, so cffi, the GIL and the NumPy allocation are all
+inside the timings. The numbers below are one run on a 24-core machine, Python
+3.14, warm page cache. Reproduce with `python3 scripts/bench.py all`.
+
+**1. Sampler** — index generation on its own, IID (with replacement), 1M index
+space, against both NumPy random APIs.
+
+| sampler           | batch | per batch | indices   | vs default_rng |
+|-------------------|-------|-----------|-----------|----------------|
+| numpy randint     |   256 |   5.9 µs  |   44 M/s  |     0.64x      |
+| numpy default_rng |   256 |   3.7 µs  |   69 M/s  |     1.00x      |
+| **zsampler**      |   256 |   1.9 µs  |  138 M/s  |   **2.01x**    |
+| numpy randint     |  1024 |   7.2 µs  |  142 M/s  |     0.83x      |
+| numpy default_rng |  1024 |   6.0 µs  |  171 M/s  |     1.00x      |
+| **zsampler**      |  1024 |   3.0 µs  |  340 M/s  |   **1.99x**    |
+| numpy randint     |  8192 |  25.6 µs  |  320 M/s  |     0.65x      |
+| numpy default_rng |  8192 |  16.6 µs  |  493 M/s  |     1.00x      |
+| **zsampler**      |  8192 |   7.8 µs  | 1048 M/s  |   **2.13x**    |
+
+Zsampler draws indices about 2x faster than NumPy's faster API — more against the
+legacy `randint` most code still calls — and the lead is widest at the small
+batches a loader actually draws, where a batch is one cffi call either way and
+NumPy's vectorised fill has nothing to amortise. The one concession: zsampler's
+IID draw is modulo-biased where NumPy's is not, ~1e-13 over a u64 draw into a
+million-record space, which is below anything training will notice.
+
+**2. Store** — random batch gather, 20000 records × 12 KiB `uint8(3,64,64)`,
+batch 256 × 200, structured (mildly compressible) data.
+
+| store         | write      | gather      | per batch  | on disk | ratio |
+|---------------|------------|-------------|------------|---------|-------|
+| loaderx-flate |  362 MiB/s |  1671 MiB/s |   1.80 ms  | 209 MiB | 1.12x |
+| loaderx-raw   |  784 MiB/s | 21997 MiB/s |   0.14 ms  | 235 MiB | 1.00x |
+| npy-mmap      | 3064 MiB/s | 13692 MiB/s |   0.22 ms  | 234 MiB | 1.00x |
+| arrayrecord   |  374 MiB/s |   847 MiB/s |   3.54 ms  | 217 MiB | 1.08x |
+| hdf5          | 1327 MiB/s |   511 MiB/s |   5.87 ms  | 236 MiB | 0.99x |
+| hdf5-gzip     |   83 MiB/s |   206 MiB/s |  14.57 ms  | 211 MiB | 1.11x |
+| lmdb          |  677 MiB/s |  5337 MiB/s |   0.56 ms  | 313 MiB | 0.75x |
+| lmdb-zlib     |   71 MiB/s |   336 MiB/s |   8.93 ms  | 235 MiB | 1.00x |
+| blosc2        |   32 MiB/s |   226 MiB/s |  13.27 ms  | 216 MiB | 1.09x |
+| zarr          |   83 MiB/s |    28 MiB/s | 107.36 ms  | 217 MiB | 1.08x |
+| tensorstore   |  531 MiB/s |    98 MiB/s |  30.69 ms  | 214 MiB | 1.09x |
+
+The gather column is what the store is built for. loaderx-raw returns a scattered
+batch faster than everything here, a memory-mapped `.npy` included, because it
+copies straight into the destination across all cores with the GIL released;
+loaderx-flate keeps most of that speed *while* decompressing, and still gathers
+several times faster than any other compressed store (hdf5-gzip, lmdb-zlib,
+blosc2, zarr, tensorstore). The array stores lag on gather because a scattered
+single-record read fights a layout meant for contiguous slabs — which is exactly
+the access pattern a training loader has. Where loaderx does not lead is write: it
+pays compression and its own layout up front, a one-time cost it does not chase.
+Compression is modest (1.12x) only because this data is barely compressible; the
+row's point is that turning it on costs almost nothing on read.
+
+The reads are not all single-threaded: loaderx and blosc2 decompress across cores,
+the rest one record at a time. Zarr and TensorStore are sharded so `on disk`
+reflects compression rather than per-file block rounding, and TensorStore is given
+gzip (its zarr3 default is uncompressed). `on disk` is allocated blocks, which is
+why LMDB's preallocated map lands below 1.00x.
+
+**3. Loader** — the full input pipeline end to end (sample, fetch, collate, hand
+over a batch), same workload, 4 workers. Each loader reads from what it is built
+for: loaderx from Zrecord, Grain from ArrayRecord, torch from a memory-mapped
+`.npy`, jax-dataloader from an in-memory array.
+
+| loader         | batches/s | per batch | samples/s | throughput  |
+|----------------|-----------|-----------|-----------|-------------|
+| **loaderx**    |     724.2 |  1.38 ms  |   185,406 |  2173 MiB/s |
+| loaderx-raw    |    5271.0 |  0.19 ms  | 1,349,378 | 15813 MiB/s |
+| grain          |      67.3 | 14.85 ms  |    17,240 |   202 MiB/s |
+| torch          |     832.2 |  1.20 ms  |   213,038 |  2497 MiB/s |
+| jax-dataloader |    3284.6 |  0.30 ms  |   840,864 |  9854 MiB/s |
+
+End to end, loaderx-raw serves batches several times faster than torch or Grain
+and ahead of jax-dataloader — and it does so from disk-backed storage, where
+jax-dataloader's number is an in-memory ceiling with no storage path at all. The
+default loaderx trades some of that for on-disk compression and still lands in
+torch's range while reading compressed records. loaderx and jax use threads;
+torch and Grain use worker processes that escape the GIL but pay IPC per batch.
+loaderx never ends an epoch, so no step waits on a boundary; the others restart
+per epoch, a cost that is in their numbers as it would be in training.
+
 ## Current Limitations
 * Single-host only; multi-host training is not supported.
 * A single sample must be at most 1 MiB, and a store holds at most 2^24
@@ -128,18 +215,14 @@ For practical integration examples, please refer to the **[Data2Latent](https://
 ```
 zig build                       # host shared objects, into loaderx/lib/
 zig build test                  # Zrecord suite, in both Debug and ReleaseFast
-zig build bench                 # append/gather throughput
 python3 tests/test_loaderx.py   # Python layer, against whichever build is importable
+python3 scripts/bench.py        # throughput, against other stores
 ```
 
-Two things about the shared objects are deliberate rather than incidental:
-
-* **libc is linked.** They are `dlopen`ed into CPython, and without libc they
-  export an unresolved `__tls_get_addr`, so Zig's thread-locals never bind and
-  `std.Thread` deadlocks on the first batch.
-* **Tests run in ReleaseFast too.** The shipped libraries are ReleaseFast, and a
-  codegen difference there has already produced a release-blocking bug that
-  Debug did not have. Test the configuration that ships.
+The Zig side is tested for behaviour only; throughput is measured from Python,
+through the binding a client actually uses. `scripts/bench.py` runs in three
+layers (`sampler`, `store`, `loader`, or `all`) and skips contenders that are not
+installed. See [Benchmarks](#benchmarks).
 
 ### Publishing
 Zig cross-compiles every target from one machine, so releases need no CI matrix:

{loaderx-0.5.0 → loaderx-0.5.4}/loaderx.egg-info/SOURCES.txt

@@ -16,8 +16,8 @@ loaderx.egg-info/SOURCES.txt
 loaderx.egg-info/dependency_links.txt
 loaderx.egg-info/requires.txt
 loaderx.egg-info/top_level.txt
+scripts/bench.py
 scripts/build_wheels.py
-src/bench.zig
 src/zrecord.zig
 src/zsampler.zig
 tests/test_loaderx.py

{loaderx-0.5.0 → loaderx-0.5.4}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "loaderx"
 dynamic = ["version"]
-description = "A record-based data runtime, focused on delivering extreme throughput and low latency"
+description = "A compact, high-performance persistent record store with zero-copy batch gathering and transparent per-record compression, designed for single-machine AI training and serving pipelines"
 readme = "README.md"
 # The bindings use cffi in ABI mode, so nothing links against the CPython ABI
 # and the wheels are py3-none-<platform>. No version-specific constraint applies.