loaderx 0.2.0__tar.gz → 0.3.0__tar.gz

{loaderx-0.2.0 → loaderx-0.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 0.2.0
-Summary: Minimal data loader for Flax
+Version: 0.3.0
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License: MIT License
         
@@ -25,16 +25,16 @@ License: MIT License
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
         
-Project-URL: Homepage, https://github.com/eoeair/loaderx
-Project-URL: Documentation, https://github.com/eoeair/loaderx
-Project-URL: Source, https://github.com/eoeair/loaderx
-Project-URL: Bug Tracker, https://github.com/eoeair/loaderx/issues
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
 Keywords: flax,python,dataloader
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.10
+Requires-Python: ==3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
@@ -45,19 +45,16 @@ Dynamic: license-file
 # loaderx
 A record-based data runtime, focused on delivering extreme throughput and low latency
 
-**Important Reminder**: Only support Linux_amd64. This is because we use a dynamic library built with Zig (we build it on Alpine and test it on Debian).
+**Only Python3.13_Linux_amd64**
 
 ## Sampler
 a high-performance sampler implemented in Zig
-### Build
-zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/libsampler.so
 ### Index Generator
 1. Sequential generation: indices are produced by traversing the index space in order.
     * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
 2. Random generation: indices are sampled randomly from the index space.
     * Global random: a set of samples is drawn randomly from the entire index space.
 
-
 ## zrecord
 基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
 
@@ -85,27 +82,24 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 | flate | Deflate | 
 ```
 3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u16, offset : u40, physical_length : u24], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u16，表示具体chunk
-        * offset为u40, 表示具体chunk内的偏移
-        * physical_length为u24，表示record数据大小
-        * 为了字节对齐和性能，offset项建议选择 8 的整数倍字节宽
+    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
+        * chunk_id为u32，表示具体chunk
+        * offset为u64, 表示具体chunk内的偏移
+        * physical_length为u32，表示record数据大小
+        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
+        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
         * 不同位宽表示的数值范围
         ```
         u16 → 2^16 - 1 （65535）
-        u24 → 2^24 - 1 （16777215）
         u32 → 2^32 - 1 （4.29e9）
         u64 → 2^64 - 1 （1.84e19）
         ```
         * 不同位宽表示的数据大小
         ```
         u16 → 64 KiB
-        u24 → 16 MiB
         u32 → 4 GiB
-        u40 → 1 TiB
         u64 → 16 EiB
         ```
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
     ```
     outside idx (global)
     ↓
@@ -118,10 +112,9 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
         * meta.json：描述全局元数据，包括全局参数与Field参数
         ```
         {
-        "version": 1,
         "length": 65536,
-        "chunks": {"num": 12, "size": 8192, "unfull": [8, 9, 10, 11]},
-        "fields": [{"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"}]
+        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
+        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
         }
         ```
         * Field_offset.zr：索引表
@@ -137,8 +130,8 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 ```
 dataset
   ├── meta.json
-  ├── A_offset.zr
-  ├── B_offset.zr
+  ├── A.offset
+  ├── B.offset
   ├── chunk
   │    ├── 0.zr
   └──└── 1.zr
@@ -168,6 +161,9 @@ dataset
     1. mmap：维护全局offset、chunk文件的handle
         * 初始化阶段注册,关闭前释放
         * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
+        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
+            * 扩容offset表，一次增长 1M项（16MiB）
+            * 扩容chunk文件，一次增长 1GiB
     2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
         * 工作线程完成后注册
         * 必须调用函数来手动释放，释放时batch压入GC队列
@@ -175,9 +171,12 @@ dataset
 #### 在线任务
 
 1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：追加一个数据，record追加到chunk末尾，offset表追加条目，全局长度增加，如果超过chunk_size,则新建chunk
+    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
+        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
+        * 写入越界：捕捉错误，并对chunk扩容
+        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
     2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，全局长度减小（越界访问属于UB行为）
+    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
 
 2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
     1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy

{loaderx-0.2.0 → loaderx-0.3.0}/README.md

@@ -1,19 +1,16 @@
 # loaderx
 A record-based data runtime, focused on delivering extreme throughput and low latency
 
-**Important Reminder**: Only support Linux_amd64. This is because we use a dynamic library built with Zig (we build it on Alpine and test it on Debian).
+**Only Python3.13_Linux_amd64**
 
 ## Sampler
 a high-performance sampler implemented in Zig
-### Build
-zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/libsampler.so
 ### Index Generator
 1. Sequential generation: indices are produced by traversing the index space in order.
     * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
 2. Random generation: indices are sampled randomly from the index space.
     * Global random: a set of samples is drawn randomly from the entire index space.
 
-
 ## zrecord
 基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
 
@@ -41,27 +38,24 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 | flate | Deflate | 
 ```
 3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u16, offset : u40, physical_length : u24], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u16，表示具体chunk
-        * offset为u40, 表示具体chunk内的偏移
-        * physical_length为u24，表示record数据大小
-        * 为了字节对齐和性能，offset项建议选择 8 的整数倍字节宽
+    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
+        * chunk_id为u32，表示具体chunk
+        * offset为u64, 表示具体chunk内的偏移
+        * physical_length为u32，表示record数据大小
+        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
+        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
         * 不同位宽表示的数值范围
         ```
         u16 → 2^16 - 1 （65535）
-        u24 → 2^24 - 1 （16777215）
         u32 → 2^32 - 1 （4.29e9）
         u64 → 2^64 - 1 （1.84e19）
         ```
         * 不同位宽表示的数据大小
         ```
         u16 → 64 KiB
-        u24 → 16 MiB
         u32 → 4 GiB
-        u40 → 1 TiB
         u64 → 16 EiB
         ```
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
     ```
     outside idx (global)
     ↓
@@ -74,10 +68,9 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
         * meta.json：描述全局元数据，包括全局参数与Field参数
         ```
         {
-        "version": 1,
         "length": 65536,
-        "chunks": {"num": 12, "size": 8192, "unfull": [8, 9, 10, 11]},
-        "fields": [{"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"}]
+        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
+        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
         }
         ```
         * Field_offset.zr：索引表
@@ -93,8 +86,8 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 ```
 dataset
   ├── meta.json
-  ├── A_offset.zr
-  ├── B_offset.zr
+  ├── A.offset
+  ├── B.offset
   ├── chunk
   │    ├── 0.zr
   └──└── 1.zr
@@ -124,6 +117,9 @@ dataset
     1. mmap：维护全局offset、chunk文件的handle
         * 初始化阶段注册,关闭前释放
         * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
+        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
+            * 扩容offset表，一次增长 1M项（16MiB）
+            * 扩容chunk文件，一次增长 1GiB
     2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
         * 工作线程完成后注册
         * 必须调用函数来手动释放，释放时batch压入GC队列
@@ -131,9 +127,12 @@ dataset
 #### 在线任务
 
 1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：追加一个数据，record追加到chunk末尾，offset表追加条目，全局长度增加，如果超过chunk_size,则新建chunk
+    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
+        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
+        * 写入越界：捕捉错误，并对chunk扩容
+        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
     2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，全局长度减小（越界访问属于UB行为）
+    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
 
 2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
     1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy

{loaderx-0.2.0 → loaderx-0.3.0}/loaderx/__init__.py

@@ -2,4 +2,4 @@ from .dataset import BaseDataset, NPDataset, ARDataset
 from .dataloader import DataLoader
 from .utils import converter
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

loaderx-0.3.0/loaderx/_sampler.py

@@ -0,0 +1,69 @@
+import os
+import time
+import ctypes
+import numpy as np
+from typing import Union
+
+from lib import libsampler
+
+class Sampler:
+    class Mode:
+        SEQUENTIAL = 0
+        IID = 1
+
+    def __init__(self, length: int, batch_size: int, mode: int, seed: int = 42):
+        # sampler
+        self.sampler = libsampler.Sampler.init(length, batch_size, mode, seed)
+        # indices
+        self.indices = np.zeros(batch_size, dtype=np.uint64)
+
+    # step
+    def next(self):
+        self.sampler.next(self.indices)
+
+    # iterator
+    def __iter__(self):
+        return self
+    def __next__(self):
+        self.next()
+        return self.indices
+
+def run():
+    from itertools import islice
+    length = 10
+    batch_size = 4
+    n_steps = 5
+
+    print("=== Sampler SEQUENTIAL ===")
+    sampler = Sampler(length, batch_size, Sampler.Mode.SEQUENTIAL)
+    for indices in islice(sampler, n_steps):
+        print(indices)
+
+    print("=== Sampler IID ===")
+    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
+    for indices in islice(sampler, n_steps):
+        print(indices)
+
+# Benchmark, zig-sampler speeder 2.2x
+def bench():
+    length = 1_000_000
+    batch_size = 8192
+    n_steps = 10_000
+
+    print("=== NumPy IID ===")
+    t0 = time.time()
+    batch = np.zeros(batch_size, dtype=np.uint64)
+    for _ in range(n_steps):
+        batch[:] = np.random.randint(0, length, size=batch_size, dtype=np.uint64)
+    print(f"{(time.time() - t0)*1000:.2f} ms")
+
+    print("=== Sampler IID ===")
+    t0 = time.time()
+    sampler = Sampler(length, batch_size, Sampler.Mode.IID)
+    for _ in range(n_steps):
+        sampler.next()
+    print(f"{(time.time() - t0)*1000:.2f} ms")
+
+if __name__ == "__main__":
+    run()
+    bench()

{loaderx-0.2.0 → loaderx-0.3.0}/loaderx/dataloader.py

@@ -26,12 +26,10 @@ class DataLoader:
         if len(dataset) != len(labelset):
             raise ValueError("dataset and labelset must have the same length")
 
-        self.rng = np.random.default_rng(seed)
-
         self.indices = Queue(maxsize=prefetch_size)
         self.batches = Queue(maxsize=prefetch_size)
 
-        self.sampler = Sampler(len(dataset), batch_size, mode)
+        self.sampler = Sampler(len(dataset), batch_size, mode, seed)
         
         self.stop_signal = threading.Event()
 
@@ -49,8 +47,8 @@ class DataLoader:
         Sample indices from the dataset and put them into the index queue.
         """
         while not self.stop_signal.is_set():
-            self.sampler.next()   
-            self.indices.put(self.sampler.batch_indices.copy())
+            self.sampler.next()
+            self.indices.put(self.sampler.indices.copy())
 
     def _prefetch(self, transform):
         """
@@ -73,7 +71,7 @@ class DataLoader:
         return self
     def __next__(self):
         # debug: monitor bottlenecks
-        # print(self.indices.qsize(), self.rawes.qsize(), self.batches.qsize())
+        # print(self.indices.qsize(), self.batches.qsize())
         return self.batches.get()
     
     # statement

{loaderx-0.2.0 → loaderx-0.3.0}/loaderx.egg-info/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: loaderx
-Version: 0.2.0
-Summary: Minimal data loader for Flax
+Version: 0.3.0
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
 Author-email: Ben0i0d <ben0i0d@foxmail.com>
 License: MIT License
         
@@ -25,16 +25,16 @@ License: MIT License
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
         
-Project-URL: Homepage, https://github.com/eoeair/loaderx
-Project-URL: Documentation, https://github.com/eoeair/loaderx
-Project-URL: Source, https://github.com/eoeair/loaderx
-Project-URL: Bug Tracker, https://github.com/eoeair/loaderx/issues
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
 Keywords: flax,python,dataloader
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.10
+Requires-Python: ==3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
@@ -45,19 +45,16 @@ Dynamic: license-file
 # loaderx
 A record-based data runtime, focused on delivering extreme throughput and low latency
 
-**Important Reminder**: Only support Linux_amd64. This is because we use a dynamic library built with Zig (we build it on Alpine and test it on Debian).
+**Only Python3.13_Linux_amd64**
 
 ## Sampler
 a high-performance sampler implemented in Zig
-### Build
-zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/libsampler.so
 ### Index Generator
 1. Sequential generation: indices are produced by traversing the index space in order.
     * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
 2. Random generation: indices are sampled randomly from the index space.
     * Global random: a set of samples is drawn randomly from the entire index space.
 
-
 ## zrecord
 基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
 
@@ -85,27 +82,24 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 | flate | Deflate | 
 ```
 3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u16, offset : u40, physical_length : u24], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u16，表示具体chunk
-        * offset为u40, 表示具体chunk内的偏移
-        * physical_length为u24，表示record数据大小
-        * 为了字节对齐和性能，offset项建议选择 8 的整数倍字节宽
+    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
+        * chunk_id为u32，表示具体chunk
+        * offset为u64, 表示具体chunk内的偏移
+        * physical_length为u32，表示record数据大小
+        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
+        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
         * 不同位宽表示的数值范围
         ```
         u16 → 2^16 - 1 （65535）
-        u24 → 2^24 - 1 （16777215）
         u32 → 2^32 - 1 （4.29e9）
         u64 → 2^64 - 1 （1.84e19）
         ```
         * 不同位宽表示的数据大小
         ```
         u16 → 64 KiB
-        u24 → 16 MiB
         u32 → 4 GiB
-        u40 → 1 TiB
         u64 → 16 EiB
         ```
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
     ```
     outside idx (global)
     ↓
@@ -118,10 +112,9 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
         * meta.json：描述全局元数据，包括全局参数与Field参数
         ```
         {
-        "version": 1,
         "length": 65536,
-        "chunks": {"num": 12, "size": 8192, "unfull": [8, 9, 10, 11]},
-        "fields": [{"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"}]
+        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
+        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
         }
         ```
         * Field_offset.zr：索引表
@@ -137,8 +130,8 @@ zig build-lib -dynamic -OReleaseFast sampler/root.zig -femit-bin=loaderx/lib/lib
 ```
 dataset
   ├── meta.json
-  ├── A_offset.zr
-  ├── B_offset.zr
+  ├── A.offset
+  ├── B.offset
   ├── chunk
   │    ├── 0.zr
   └──└── 1.zr
@@ -168,6 +161,9 @@ dataset
     1. mmap：维护全局offset、chunk文件的handle
         * 初始化阶段注册,关闭前释放
         * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
+        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
+            * 扩容offset表，一次增长 1M项（16MiB）
+            * 扩容chunk文件，一次增长 1GiB
     2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
         * 工作线程完成后注册
         * 必须调用函数来手动释放，释放时batch压入GC队列
@@ -175,9 +171,12 @@ dataset
 #### 在线任务
 
 1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：追加一个数据，record追加到chunk末尾，offset表追加条目，全局长度增加，如果超过chunk_size,则新建chunk
+    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
+        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
+        * 写入越界：捕捉错误，并对chunk扩容
+        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
     2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，全局长度减小（越界访问属于UB行为）
+    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
 
 2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
     1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy

{loaderx-0.2.0 → loaderx-0.3.0}/pyproject.toml

@@ -5,9 +5,9 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "loaderx"
 dynamic = ["version"]
-description = "Minimal data loader for Flax"
+description = "A record-based data runtime, focused on delivering extreme throughput and low latency"
 readme = "README.md"
-requires-python = ">=3.10"
+requires-python = "==3.13"
 authors = [
     {name = "Ben0i0d", email = "ben0i0d@foxmail.com"},
 ]
@@ -31,7 +31,7 @@ packages = ["loaderx"]
 version = {attr = "loaderx.__version__"}
 
 [project.urls]
-"Homepage" = "https://github.com/eoeair/loaderx"
-"Documentation" = "https://github.com/eoeair/loaderx"
-"Source" = "https://github.com/eoeair/loaderx"
-"Bug Tracker" = "https://github.com/eoeair/loaderx/issues"
+"Homepage" = "https://codeberg.org/eoelab/loaderx"
+"Documentation" = "https://codeberg.org/eoelab/loaderx"
+"Source" = "https://codeberg.org/eoelab/loaderx"
+"Bug Tracker" = "https://codeberg.org/eoelab/loaderx"

loaderx-0.2.0/loaderx/_sampler.py

@@ -1,99 +0,0 @@
-import os
-import time
-import ctypes
-import numpy as np
-from typing import Union
-
-class Sampler:
-    class Mode:
-        SEQUENTIAL = 0
-        IID = 1
-    # lazy import
-    _lib = None
-    @classmethod
-    def _load_lib(cls):
-        if cls._lib is None:
-            # linux/windows/macos
-            names = ['lib/libsampler.so', 'lib/sampler.dll', 'lib/libsampler.dylib']
-            lib_paths = [os.path.join(os.path.dirname(__file__), name) for name in names]
-
-            for path in lib_paths:
-                if os.path.exists(path):
-                        cls._lib = ctypes.CDLL(path)
-                        cls._setup_function_signatures()
-        return cls._lib
-    
-    @classmethod
-    def _setup_function_signatures(cls):
-        lib = cls._lib
-        
-        # init
-        lib.init.argtypes = [
-            ctypes.c_uint64,  # length
-            ctypes.c_uint32,  # batch_size
-            ctypes.c_uint8,   # mode
-            ctypes.c_uint64   # seed
-        ]
-        lib.init.restype = ctypes.c_void_p
-        
-        # deinit
-        lib.deinit.argtypes = [ctypes.c_void_p] # sampler
-        lib.deinit.restype = None
-        
-        # next
-        lib.next.argtypes = [
-            ctypes.c_void_p,                     # sampler
-            ctypes.POINTER(ctypes.c_uint64)      # batch_indices
-        ]
-        lib.next.restype = None
-
-    def __init__(self, length: int, batch_size: int, mode: int, seed: int = 42):
-        # import lib
-        self.lib = self._load_lib()
-
-        # init sampler
-        self.ptr = self.lib.init(length, batch_size, mode, seed)
-        if not self.ptr:
-            raise RuntimeError(f"Failed to create sampler with params: length={length}, batch_size={batch_size}, mode={mode}, seed={seed}")
-        
-        # batch_indices
-        self.batch_indices = np.zeros(batch_size, dtype=np.uint64)
-
-    # step
-    def next(self):
-        self.lib.next(self.ptr, self.batch_indices.ctypes.data_as(ctypes.POINTER(ctypes.c_uint64)))
-
-    # iterator
-    def __iter__(self):
-        return self
-    def __next__(self):
-        self.next()
-        return self.batch_indices
-
-    # close
-    def deinit(self):
-        self.lib.deinit(self.ptr)
-    def __del__(self):
-        if hasattr(self, 'ptr') and self.ptr:
-            self.deinit()
-
-# Benchmark, zig-sampler is better
-if __name__ == "__main__":
-    length = 1_000_000
-    batch_size = 8192
-    n_steps = 10_000
-
-    print("=== Python NumPy IID ===")
-    t0 = time.time()
-    batch = np.zeros(batch_size, dtype=np.uint64)
-    for _ in range(n_steps):
-        batch[:] = np.random.randint(0, length, size=batch_size, dtype=np.uint64)
-    print(f"{(time.time() - t0)*1000:.2f} ms")
-
-    print("=== Sampler IID ===")
-    t0 = time.time()
-    s = Sampler(length, batch_size, Sampler.Mode.IID)
-    for _ in range(n_steps):
-        s.next()
-    print(f"{(time.time() - t0)*1000:.2f} ms")
-    s.deinit()