loaderx 0.2.1__tar.gz → 0.4.1__tar.gz

loaderx-0.4.1/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: loaderx
+Version: 0.4.1
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Author-email: Ben0i0d <ben0i0d@foxmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 EOELAB AI Research
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
+Keywords: flax,python,dataloader
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: blosc2
+Requires-Dist: cffi
+Dynamic: license-file
+
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.4.1/README.md

@@ -0,0 +1,54 @@
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.4.1/loaderx/__init__.py

@@ -0,0 +1,4 @@
+from .dataset import Dataset
+from .dataloader import DataLoader
+
+__version__ = "0.4.1"

loaderx-0.4.1/loaderx/_sampler.py

@@ -0,0 +1,97 @@
+import time
+from pathlib import Path
+
+import numpy as np
+from cffi import FFI
+
+ffi = FFI()
+
+ffi.cdef("""
+typedef struct Sampler Sampler;
+
+Sampler* cinit(
+    uint64_t length,
+    uint64_t batch_size,
+    uint8_t mode,
+    uint64_t seed
+);
+
+void cdeinit(Sampler* s);
+
+void cnext(Sampler* s, uint64_t* indices_ptr, size_t indices_len);
+""")
+
+lib_path = Path(__file__).resolve().parent / "lib" / "libsampler.so"
+lib = ffi.dlopen(str(lib_path))
+
+class Sampler:
+    class Mode:
+        SEQUENTIAL = 0
+        IID = 1
+
+    def __init__(self, length: int, batch_size: int, mode: int, seed: int = 42):
+        # sampler
+        self.sampler = lib.cinit(length, batch_size, mode, seed)
+        if self.sampler == ffi.NULL:
+            raise MemoryError("Failed to create sampler")
+        # indices
+        self.indices = np.zeros(batch_size, dtype=np.uint64)
+
+    # step
+    def next(self):
+        ptr = ffi.cast("uint64_t *", self.indices.ctypes.data)
+        lib.cnext(self.sampler, ptr, self.indices.size)
+        # return copy to avoid loss
+        return self.indices.copy()
+
+    # iterator
+    def __iter__(self):
+        return self
+    def __next__(self):
+        return self.next()
+
+    # clean
+    def __del__(self):
+        if hasattr(self, "sampler") and self.sampler != ffi.NULL:
+            lib.cdeinit(self.sampler)
+            self.sampler = ffi.NULL
+
+def run():
+    from itertools import islice
+    length = 10
+    batch_size = 4
+    n_steps = 5
+
+    print("=== Sampler SEQUENTIAL ===")
+    sampler = Sampler(length, batch_size, Sampler.Mode.SEQUENTIAL)
+    for indices in islice(sampler, n_steps):
+        print(indices)
+
+    print("=== Sampler IID ===")
+    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
+    for indices in islice(sampler, n_steps):
+        print(indices)
+
+# Benchmark, zig-sampler speeder 3.66x
+def bench():
+    length = 1_000_000
+    batch_size = 8192
+    n_steps = 10_000
+
+    print("=== NumPy IID ===")
+    t0 = time.time()
+    batch = np.zeros(batch_size, dtype=np.uint64)
+    for _ in range(n_steps):
+        batch[:] = np.random.randint(0, length, size=batch_size, dtype=np.uint64)
+    print(f"{(time.time() - t0)*1000:.2f} ms")
+
+    print("=== Sampler IID ===")
+    t0 = time.time()
+    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
+    for _ in range(n_steps):
+        sampler.next()
+    print(f"{(time.time() - t0)*1000:.2f} ms")
+
+if __name__ == "__main__":
+    run()
+    bench()

{loaderx-0.2.1 → loaderx-0.4.1}/loaderx/dataloader.py

@@ -47,8 +47,7 @@ class DataLoader:
         Sample indices from the dataset and put them into the index queue.
         """
         while not self.stop_signal.is_set():
-            self.sampler.next()   
-            self.indices.put(self.sampler.batch_indices.copy())
+            self.indices.put(self.sampler.next())
 
     def _prefetch(self, transform):
         """
@@ -94,9 +93,6 @@ class DataLoader:
         
         for thread in self.threads:
             thread.join()
-        
-        self.dataset.close()
-        self.labelset.close()
 
     def __del__(self):
         self.close()

loaderx-0.4.1/loaderx/dataset.py

@@ -0,0 +1,39 @@
+import blosc2
+import numpy as np
+
+class Dataset:
+    def __init__(self, path):
+        """
+        Initialize a dataset.
+        """
+        self._array = blosc2.open(path)
+    def __getitem__(self, idx):
+        """
+        Get the item at index idx from the dataset.
+
+        Args:
+            idx (int): Index of the item to retrieve.
+
+        Returns:
+            np.ndarray: The item at index idx from the dataset.
+        """
+        return self._array[idx]
+    def __len__(self):
+        """
+        Return the number of samples in the dataset.
+
+        Returns:
+            int: The number of samples in the dataset.
+        """
+        return len(self._array)
+    def __getitems__(self, idxs):
+        """
+        Get the items at index idxs from the dataset.
+
+        Args:
+            idxs (list of int): indices of the items to retrieve.
+
+        Returns:
+            np.ndarray: The items at index idxs from the dataset.
+        """
+        return self._array[idxs]

loaderx-0.4.1/loaderx.egg-info/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: loaderx
+Version: 0.4.1
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Author-email: Ben0i0d <ben0i0d@foxmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 EOELAB AI Research
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
+Keywords: flax,python,dataloader
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: blosc2
+Requires-Dist: cffi
+Dynamic: license-file
+
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

{loaderx-0.2.1 → loaderx-0.4.1}/loaderx.egg-info/SOURCES.txt

@@ -5,7 +5,6 @@ loaderx/__init__.py
 loaderx/_sampler.py
 loaderx/dataloader.py
 loaderx/dataset.py
-loaderx/utils.py
 loaderx.egg-info/PKG-INFO
 loaderx.egg-info/SOURCES.txt
 loaderx.egg-info/dependency_links.txt

loaderx-0.4.1/loaderx.egg-info/requires.txt

@@ -0,0 +1,3 @@
+numpy
+blosc2
+cffi

{loaderx-0.2.1 → loaderx-0.4.1}/pyproject.toml

@@ -7,7 +7,7 @@ name = "loaderx"
 dynamic = ["version"]
 description = "A record-based data runtime, focused on delivering extreme throughput and low latency"
 readme = "README.md"
-requires-python = ">=3.10"
+requires-python = "==3.13"
 authors = [
     {name = "Ben0i0d", email = "ben0i0d@foxmail.com"},
 ]
@@ -19,7 +19,7 @@ classifiers = [
     "License :: OSI Approved :: MIT License",
     "Programming Language :: Python :: 3",
 ]
-dependencies = ["numpy", "array-record", "tqdm"]
+dependencies = ["numpy", "blosc2", "cffi"]
 
 [tool.setuptools]
 packages = ["loaderx"]