loaderx 0.3.0__tar.gz → 0.4.1__tar.gz

loaderx-0.4.1/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: loaderx
+Version: 0.4.1
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Author-email: Ben0i0d <ben0i0d@foxmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 EOELAB AI Research
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
+Keywords: flax,python,dataloader
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: blosc2
+Requires-Dist: cffi
+Dynamic: license-file
+
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.4.1/README.md

@@ -0,0 +1,54 @@
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.4.1/loaderx/__init__.py

@@ -0,0 +1,4 @@
+from .dataset import Dataset
+from .dataloader import DataLoader
+
+__version__ = "0.4.1"

{loaderx-0.3.0 → loaderx-0.4.1}/loaderx/_sampler.py

@@ -1,10 +1,28 @@
-import os
 import time
-import ctypes
+from pathlib import Path
+
 import numpy as np
-from typing import Union
+from cffi import FFI
+
+ffi = FFI()
+
+ffi.cdef("""
+typedef struct Sampler Sampler;
+
+Sampler* cinit(
+    uint64_t length,
+    uint64_t batch_size,
+    uint8_t mode,
+    uint64_t seed
+);
 
-from lib import libsampler
+void cdeinit(Sampler* s);
+
+void cnext(Sampler* s, uint64_t* indices_ptr, size_t indices_len);
+""")
+
+lib_path = Path(__file__).resolve().parent / "lib" / "libsampler.so"
+lib = ffi.dlopen(str(lib_path))
 
 class Sampler:
     class Mode:
@@ -13,20 +31,30 @@ class Sampler:
 
     def __init__(self, length: int, batch_size: int, mode: int, seed: int = 42):
         # sampler
-        self.sampler = libsampler.Sampler.init(length, batch_size, mode, seed)
+        self.sampler = lib.cinit(length, batch_size, mode, seed)
+        if self.sampler == ffi.NULL:
+            raise MemoryError("Failed to create sampler")
         # indices
         self.indices = np.zeros(batch_size, dtype=np.uint64)
 
     # step
     def next(self):
-        self.sampler.next(self.indices)
+        ptr = ffi.cast("uint64_t *", self.indices.ctypes.data)
+        lib.cnext(self.sampler, ptr, self.indices.size)
+        # return copy to avoid loss
+        return self.indices.copy()
 
     # iterator
     def __iter__(self):
         return self
     def __next__(self):
-        self.next()
-        return self.indices
+        return self.next()
+
+    # clean
+    def __del__(self):
+        if hasattr(self, "sampler") and self.sampler != ffi.NULL:
+            lib.cdeinit(self.sampler)
+            self.sampler = ffi.NULL
 
 def run():
     from itertools import islice
@@ -44,7 +72,7 @@ def run():
     for indices in islice(sampler, n_steps):
         print(indices)
 
-# Benchmark, zig-sampler speeder 2.2x
+# Benchmark, zig-sampler speeder 3.66x
 def bench():
     length = 1_000_000
     batch_size = 8192

{loaderx-0.3.0 → loaderx-0.4.1}/loaderx/dataloader.py

@@ -47,8 +47,7 @@ class DataLoader:
         Sample indices from the dataset and put them into the index queue.
         """
         while not self.stop_signal.is_set():
-            self.sampler.next()
-            self.indices.put(self.sampler.indices.copy())
+            self.indices.put(self.sampler.next())
 
     def _prefetch(self, transform):
         """
@@ -94,9 +93,6 @@ class DataLoader:
         
         for thread in self.threads:
             thread.join()
-        
-        self.dataset.close()
-        self.labelset.close()
 
     def __del__(self):
         self.close()

loaderx-0.4.1/loaderx/dataset.py

@@ -0,0 +1,39 @@
+import blosc2
+import numpy as np
+
+class Dataset:
+    def __init__(self, path):
+        """
+        Initialize a dataset.
+        """
+        self._array = blosc2.open(path)
+    def __getitem__(self, idx):
+        """
+        Get the item at index idx from the dataset.
+
+        Args:
+            idx (int): Index of the item to retrieve.
+
+        Returns:
+            np.ndarray: The item at index idx from the dataset.
+        """
+        return self._array[idx]
+    def __len__(self):
+        """
+        Return the number of samples in the dataset.
+
+        Returns:
+            int: The number of samples in the dataset.
+        """
+        return len(self._array)
+    def __getitems__(self, idxs):
+        """
+        Get the items at index idxs from the dataset.
+
+        Args:
+            idxs (list of int): indices of the items to retrieve.
+
+        Returns:
+            np.ndarray: The items at index idxs from the dataset.
+        """
+        return self._array[idxs]

loaderx-0.4.1/loaderx.egg-info/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: loaderx
+Version: 0.4.1
+Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
+Author-email: Ben0i0d <ben0i0d@foxmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 EOELAB AI Research
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
+Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
+Project-URL: Source, https://codeberg.org/eoelab/loaderx
+Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
+Keywords: flax,python,dataloader
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: ==3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: blosc2
+Requires-Dist: cffi
+Dynamic: license-file
+
+# Loaderx
+A compact and high-performance single-machine data loader designed for JAX/Flax.
+
+**Only Linux_amd64**
+## Design Philosophy
+
+loaderx is built around several core principles:
+
+1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
+2. A strong focus on single-machine training workflows.
+3. We implement based on NumPy semantics, supporting Blosc2 backends.
+4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
+
+## Zsampler
+Index Generator: a high-performance sampler implemented in Zig
+
+1. Sequential generation: indices are produced by traversing the index space in order.
+    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
+2. Random generation: indices are sampled randomly from the index space.
+    * Global random: a set of samples is drawn randomly from the entire index space.
+
+## Convert a NumPy tensor to Blosc2
+```
+import blosc2
+import numpy as np
+
+np_arr = np.load('arr.npy',mmap_mode='r')
+b2_arr = blosc2.asarray(np_arr, urlpath="arr.b2nd", mode="w")
+```
+Then, you can use `b2_arr` or load as `b2_arr = blosc2.open("arr.b2nd")`
+
+## Current Limitations
+Currently, loaderx only supports single-host environments and does not yet support multi-host training.
+
+## Quick Start
+```
+from loaderx import Dataset, DataLoader
+
+dataset = Dataset('train_data.b2nd')
+labelset = Dataset('train_label.b2nd')
+
+loader = DataLoader(dataset, labelset)
+
+for i, batch in enumerate(loader):
+    if i >= 256:
+        break
+
+print(batch['data'].shape)
+print(batch['label'].shape)
+```
+
+### Integrating with JAX/Flax
+
+For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

{loaderx-0.3.0 → loaderx-0.4.1}/loaderx.egg-info/SOURCES.txt

@@ -5,7 +5,6 @@ loaderx/__init__.py
 loaderx/_sampler.py
 loaderx/dataloader.py
 loaderx/dataset.py
-loaderx/utils.py
 loaderx.egg-info/PKG-INFO
 loaderx.egg-info/SOURCES.txt
 loaderx.egg-info/dependency_links.txt

loaderx-0.4.1/loaderx.egg-info/requires.txt

@@ -0,0 +1,3 @@
+numpy
+blosc2
+cffi

{loaderx-0.3.0 → loaderx-0.4.1}/pyproject.toml

@@ -19,7 +19,7 @@ classifiers = [
     "License :: OSI Approved :: MIT License",
     "Programming Language :: Python :: 3",
 ]
-dependencies = ["numpy", "array-record", "tqdm"]
+dependencies = ["numpy", "blosc2", "cffi"]
 
 [tool.setuptools]
 packages = ["loaderx"]

loaderx-0.3.0/PKG-INFO

@@ -1,260 +0,0 @@
-Metadata-Version: 2.4
-Name: loaderx
-Version: 0.3.0
-Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
-Author-email: Ben0i0d <ben0i0d@foxmail.com>
-License: MIT License
-        
-        Copyright (c) 2025 EOELAB AI Research
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-        
-Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
-Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
-Project-URL: Source, https://codeberg.org/eoelab/loaderx
-Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
-Keywords: flax,python,dataloader
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Requires-Python: ==3.13
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy
-Requires-Dist: array-record
-Requires-Dist: tqdm
-Dynamic: license-file
-
-# loaderx
-A record-based data runtime, focused on delivering extreme throughput and low latency
-
-**Only Python3.13_Linux_amd64**
-
-## Sampler
-a high-performance sampler implemented in Zig
-### Index Generator
-1. Sequential generation: indices are produced by traversing the index space in order.
-    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
-2. Random generation: indices are sampled randomly from the index space.
-    * Global random: a set of samples is drawn randomly from the entire index space.
-
-## zrecord
-基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
-
-1. ZRecord基于 record 思想设计，我们不假定record间存在顺序关系，对于顺序语义应在record内部表达或建立timestamp
-2. record是相互独立的逻辑记录，仅在 Python 侧提供与 NumPy 兼容的 array 接口投影
-3. zrecord是无类型的，数据只是Bytes，类型解析由上层完成
-4. chunk只是存储管理单元，提高并发性能，不具备语义
-
-### 存储形式
-1. ZRecord 仅包含一个多流数据集，长度为 N（N 个 record）
-    * 索引空间为 [0, N)，所有索引与切块语义等价于 gather 操作
-    * 多流，类似 `set['A'], set['B']`, 但必须等长
-    ```
-    Dataset:
-    record_id = 0..N-1
-
-    Field A: bytes / tensor / audio / text
-    Field B: bytes / label / meta
-    ```
-2. 为降低存储体积，支持粒度为record级的透明压缩，以下是可选的压缩方法
-```
-| 方法  | 算法    |
-| ----- | ------- |
-| raw   | 无压缩  | 
-| flate | Deflate | 
-```
-3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u32，表示具体chunk
-        * offset为u64, 表示具体chunk内的偏移
-        * physical_length为u32，表示record数据大小
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
-        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
-        * 不同位宽表示的数值范围
-        ```
-        u16 → 2^16 - 1 （65535）
-        u32 → 2^32 - 1 （4.29e9）
-        u64 → 2^64 - 1 （1.84e19）
-        ```
-        * 不同位宽表示的数据大小
-        ```
-        u16 → 64 KiB
-        u32 → 4 GiB
-        u64 → 16 EiB
-        ```
-    ```
-    outside idx (global)
-    ↓
-    ↓offset[idx]
-    ↓
-    chunk_id, offset, physical_length
-    ```
-4. ZRecord的存储是元数据+分块的格式
-    1. 元数据
-        * meta.json：描述全局元数据，包括全局参数与Field参数
-        ```
-        {
-        "length": 65536,
-        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
-        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
-        }
-        ```
-        * Field_offset.zr：索引表
-        ```
-        0 0 255
-        0 256 511
-        ...
-        ```
-    2. 分块数据
-        * chunk/x.zr: 分块的chunk数据,chunk不区分数据所属
-
-存储格式如下
-```
-dataset
-  ├── meta.json
-  ├── A.offset
-  ├── B.offset
-  ├── chunk
-  │    ├── 0.zr
-  └──└── 1.zr
-```
-
-### 执行器
-
-负责维护任务队列，工作线程，handle
-
-1. 任务队列：
-    * write_task：写任务队列，压入[ops, record]，等待writer弹出
-    * read_task：读任务队列，预分配batch并压入[batch, pos, chunk_id, offset, physical_length]，等待reader弹出
-        * batch: 具体读取请求下创建的与indices等长的二元组
-        * pos：在batch中的位置，保证等序返回
-    * gc_task：垃圾回收任务队列，压入batch，等待cleaner弹出
-
-2. 工作线程：执行具体任务的工作线程
-    * writer：线程池执行写任务，但每个线程固定对应一个chunk
-        * writer只会向自己的chunk写入（写满时新建chunk提交给执行器完成）
-        * num_writer推荐：1-4
-    * reader：线程池执行读任务
-        * num_reader推荐：8-32
-    * cleaner：线程池执行垃圾回收任务
-        * num_cleaner推荐：2-4
-
-3. handle：
-    1. mmap：维护全局offset、chunk文件的handle
-        * 初始化阶段注册,关闭前释放
-        * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
-        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
-            * 扩容offset表，一次增长 1M项（16MiB）
-            * 扩容chunk文件，一次增长 1GiB
-    2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
-        * 工作线程完成后注册
-        * 必须调用函数来手动释放，释放时batch压入GC队列
-
-#### 在线任务
-
-1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
-        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
-        * 写入越界：捕捉错误，并对chunk扩容
-        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
-    2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
-
-2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
-    1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy
-
-    读取流程
-    ```
-    [idx]
-    ↓
-    [batch, pos, idx]
-    ↓field_offset
-    [batch, pos, chunk_id, offset, physical_length]
-    ↓
-    [ptr, logical_length]
-    ```
-
-3. 垃圾回收任务：cleaner从GC队列中释放对应batch
-    1. 对于无压缩的数据，不释放内存，只释放 batch 本身
-    2. 对于有压缩的数据，释放内存
-    3. GC后访问ptr属于UB行为
-
-#### 离线任务
-
-1. 重平衡（Rebalance）：zrecord经过大量写操作后，会出现chunk访问不平衡,通过重平衡来恢复chunk的可访问性
-    * 实现：根据offset进行重写入，并在完成后替换
-    * Rebalance只允许手动运行，确保用户知情，提供一个利用率（sum(length)/sum(chunk_size*num_chunks)）
-
-### 分布式扩展
-
-**该部分暂时不会实现，只是提前架构**
-
-1. 分布式下，我们会得到一个更高的层次-cluster,单机将变为cluster内的node
-2. node持有一个shard,包含若干个chunk
-3. 添加一个indirection表，将全局路径映射到具体node的表，indirection也可以使用hash实现，从而无锁，索引路径变为
-```
-outside idx (global)
-↓
-↓indirection[idx]
-↓
-node, idx
-↓
-↓offset[idx]
-↓
-chunk_id, offset, physical_length
-```
-
-## Convert a NumPy tensor to Array_record
-
-*This will create a directory containing file shards, which helps improve I/O performance.*
-
-```
-import numpy as np
-from loaderx import converter
-
-train_data = np.load('train_data.npy',mmap_mode='r')
-converter(train_data, 'train_data')
-```
-
-## Current Limitations
-Currently, loaderx only supports single-host environments and does not yet support multi-host training.
-
-## Quick Start
-```
-import numpy as np
-from loaderx import NPDataset, ARDataset, DataLoader
-
-dataset = ARDataset('train_data')
-labelset = NPDataset('xsub/train_label.npy')
-
-loader = DataLoader(dataset, labelset)
-
-for i, batch in enumerate(loader):
-    if i >= 256:
-        break
-
-print(batch['data'].shape)
-print(batch['label'].shape)
-```
-
-### Integrating with JAX/Flax
-
-For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.3.0/README.md

@@ -1,216 +0,0 @@
-# loaderx
-A record-based data runtime, focused on delivering extreme throughput and low latency
-
-**Only Python3.13_Linux_amd64**
-
-## Sampler
-a high-performance sampler implemented in Zig
-### Index Generator
-1. Sequential generation: indices are produced by traversing the index space in order.
-    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
-2. Random generation: indices are sampled randomly from the index space.
-    * Global random: a set of samples is drawn randomly from the entire index space.
-
-## zrecord
-基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
-
-1. ZRecord基于 record 思想设计，我们不假定record间存在顺序关系，对于顺序语义应在record内部表达或建立timestamp
-2. record是相互独立的逻辑记录，仅在 Python 侧提供与 NumPy 兼容的 array 接口投影
-3. zrecord是无类型的，数据只是Bytes，类型解析由上层完成
-4. chunk只是存储管理单元，提高并发性能，不具备语义
-
-### 存储形式
-1. ZRecord 仅包含一个多流数据集，长度为 N（N 个 record）
-    * 索引空间为 [0, N)，所有索引与切块语义等价于 gather 操作
-    * 多流，类似 `set['A'], set['B']`, 但必须等长
-    ```
-    Dataset:
-    record_id = 0..N-1
-
-    Field A: bytes / tensor / audio / text
-    Field B: bytes / label / meta
-    ```
-2. 为降低存储体积，支持粒度为record级的透明压缩，以下是可选的压缩方法
-```
-| 方法  | 算法    |
-| ----- | ------- |
-| raw   | 无压缩  | 
-| flate | Deflate | 
-```
-3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u32，表示具体chunk
-        * offset为u64, 表示具体chunk内的偏移
-        * physical_length为u32，表示record数据大小
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
-        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
-        * 不同位宽表示的数值范围
-        ```
-        u16 → 2^16 - 1 （65535）
-        u32 → 2^32 - 1 （4.29e9）
-        u64 → 2^64 - 1 （1.84e19）
-        ```
-        * 不同位宽表示的数据大小
-        ```
-        u16 → 64 KiB
-        u32 → 4 GiB
-        u64 → 16 EiB
-        ```
-    ```
-    outside idx (global)
-    ↓
-    ↓offset[idx]
-    ↓
-    chunk_id, offset, physical_length
-    ```
-4. ZRecord的存储是元数据+分块的格式
-    1. 元数据
-        * meta.json：描述全局元数据，包括全局参数与Field参数
-        ```
-        {
-        "length": 65536,
-        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
-        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
-        }
-        ```
-        * Field_offset.zr：索引表
-        ```
-        0 0 255
-        0 256 511
-        ...
-        ```
-    2. 分块数据
-        * chunk/x.zr: 分块的chunk数据,chunk不区分数据所属
-
-存储格式如下
-```
-dataset
-  ├── meta.json
-  ├── A.offset
-  ├── B.offset
-  ├── chunk
-  │    ├── 0.zr
-  └──└── 1.zr
-```
-
-### 执行器
-
-负责维护任务队列，工作线程，handle
-
-1. 任务队列：
-    * write_task：写任务队列，压入[ops, record]，等待writer弹出
-    * read_task：读任务队列，预分配batch并压入[batch, pos, chunk_id, offset, physical_length]，等待reader弹出
-        * batch: 具体读取请求下创建的与indices等长的二元组
-        * pos：在batch中的位置，保证等序返回
-    * gc_task：垃圾回收任务队列，压入batch，等待cleaner弹出
-
-2. 工作线程：执行具体任务的工作线程
-    * writer：线程池执行写任务，但每个线程固定对应一个chunk
-        * writer只会向自己的chunk写入（写满时新建chunk提交给执行器完成）
-        * num_writer推荐：1-4
-    * reader：线程池执行读任务
-        * num_reader推荐：8-32
-    * cleaner：线程池执行垃圾回收任务
-        * num_cleaner推荐：2-4
-
-3. handle：
-    1. mmap：维护全局offset、chunk文件的handle
-        * 初始化阶段注册,关闭前释放
-        * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
-        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
-            * 扩容offset表，一次增长 1M项（16MiB）
-            * 扩容chunk文件，一次增长 1GiB
-    2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
-        * 工作线程完成后注册
-        * 必须调用函数来手动释放，释放时batch压入GC队列
-
-#### 在线任务
-
-1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
-        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
-        * 写入越界：捕捉错误，并对chunk扩容
-        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
-    2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
-
-2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
-    1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy
-
-    读取流程
-    ```
-    [idx]
-    ↓
-    [batch, pos, idx]
-    ↓field_offset
-    [batch, pos, chunk_id, offset, physical_length]
-    ↓
-    [ptr, logical_length]
-    ```
-
-3. 垃圾回收任务：cleaner从GC队列中释放对应batch
-    1. 对于无压缩的数据，不释放内存，只释放 batch 本身
-    2. 对于有压缩的数据，释放内存
-    3. GC后访问ptr属于UB行为
-
-#### 离线任务
-
-1. 重平衡（Rebalance）：zrecord经过大量写操作后，会出现chunk访问不平衡,通过重平衡来恢复chunk的可访问性
-    * 实现：根据offset进行重写入，并在完成后替换
-    * Rebalance只允许手动运行，确保用户知情，提供一个利用率（sum(length)/sum(chunk_size*num_chunks)）
-
-### 分布式扩展
-
-**该部分暂时不会实现，只是提前架构**
-
-1. 分布式下，我们会得到一个更高的层次-cluster,单机将变为cluster内的node
-2. node持有一个shard,包含若干个chunk
-3. 添加一个indirection表，将全局路径映射到具体node的表，indirection也可以使用hash实现，从而无锁，索引路径变为
-```
-outside idx (global)
-↓
-↓indirection[idx]
-↓
-node, idx
-↓
-↓offset[idx]
-↓
-chunk_id, offset, physical_length
-```
-
-## Convert a NumPy tensor to Array_record
-
-*This will create a directory containing file shards, which helps improve I/O performance.*
-
-```
-import numpy as np
-from loaderx import converter
-
-train_data = np.load('train_data.npy',mmap_mode='r')
-converter(train_data, 'train_data')
-```
-
-## Current Limitations
-Currently, loaderx only supports single-host environments and does not yet support multi-host training.
-
-## Quick Start
-```
-import numpy as np
-from loaderx import NPDataset, ARDataset, DataLoader
-
-dataset = ARDataset('train_data')
-labelset = NPDataset('xsub/train_label.npy')
-
-loader = DataLoader(dataset, labelset)
-
-for i, batch in enumerate(loader):
-    if i >= 256:
-        break
-
-print(batch['data'].shape)
-print(batch['label'].shape)
-```
-
-### Integrating with JAX/Flax
-
-For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.3.0/loaderx/__init__.py

@@ -1,5 +0,0 @@
-from .dataset import BaseDataset, NPDataset, ARDataset
-from .dataloader import DataLoader
-from .utils import converter
-
-__version__ = "0.3.0"

loaderx-0.3.0/loaderx/dataset.py

@@ -1,93 +0,0 @@
-import json
-import numpy as np
-from pathlib import Path
-from array_record.python.array_record_data_source import ArrayRecordDataSource
-
-class BaseDataset:
-    def __init__(self, path):
-        """
-        Initialize a dataset.
-        """
-        raise NotImplementedError
-    def __getitem__(self, idx):
-        """
-        Get the item at index idx from the dataset.
-
-        Args:
-            idx (int): Index of the item to retrieve.
-
-        Returns:
-            np.ndarray: The item at index idx from the dataset.
-        """
-        raise NotImplementedError
-    def __len__(self):
-        """
-        Return the number of samples in the dataset.
-
-        Returns:
-            int: The number of samples in the dataset.
-        """
-        raise NotImplementedError
-    def __getitems__(self, idxs):
-        """
-        Get the items at index idxs from the dataset.
-
-        Args:
-            idxs (list of int): indices of the items to retrieve.
-
-        Returns:
-            np.ndarray: The items at index idxs from the dataset.
-        """
-        raise NotImplementedError
-    def close(self):
-        """
-        Close the dataset.
-
-        If the dataset is stored in a memory-mapped file,
-        close the memory map to free up system resources.
-
-        If the dataset is stored in an ArrayRecord file,
-        close the file to free up system resources.
-        """
-        pass
-
-class NPDataset(BaseDataset):
-    def __init__(self, path):
-        self._array = np.load(path, mmap_mode='r')
-
-    def __getitem__(self, idx):
-        return self._array[idx]
-
-    def __getitems__(self, idxs):
-        return self._array[idxs]
-
-    def __len__(self):
-        return len(self._array)
-
-    def close(self):
-        mmap_obj = getattr(self._array, "_mmap", None)
-        if mmap_obj is not None:
-            mmap_obj.close()
-
-class ARDataset(BaseDataset):
-    def __init__(self, path):
-        path = Path(path)
-        with open(path / "meta.json", "r", encoding="utf-8") as f:
-            meta = json.load(f)
-        
-        self._length = meta["length"]
-        self._shape = meta["shape"]
-        self._dtype = meta['dtype']
-        self._array = ArrayRecordDataSource([str(path / f) for f in meta["blocks"]])
-
-    def __getitem__(self, idx):
-        return np.frombuffer(self._array[idx], dtype=self._dtype).reshape(self._shape)
-
-    def __getitems__(self, idxs):
-        return np.frombuffer(b"".join(self._array.__getitems__(idxs)), dtype=self._dtype).reshape(len(idxs), *self._shape)
-
-    def __len__(self):
-        return self._length
-
-    def close(self):
-        self._array.__exit__(None, None, None)

loaderx-0.3.0/loaderx/utils.py

@@ -1,51 +0,0 @@
-import json
-import shutil
-from tqdm import tqdm
-from pathlib import Path
-from array_record.python.array_record_module import ArrayRecordWriter
-
-def converter(data, output_dir, replace=False, blocksize=4096, ar_options= "group_size:1,zstd"):
-    """
-    Convert a NumPy array to an ArrayRecord format.
-
-    Args:
-        data : numpy.ndarray
-            The input data to be converted.
-        output_dir : str
-            The output directory where the ArrayRecord files will be saved.
-        replace : bool, optional
-            If True, the output directory will be deleted if it already exists.
-            Defaults to False.
-        blocksize : int, optional
-            The number of records to write to each ArrayRecord file.
-            Defaults to 4096.
-        ar_options : str, optional
-            The options to be passed to the ArrayRecordWriter.
-            Defaults to "group_size:1,zstd".
-    """
-    output_dir = Path(output_dir)
-    if output_dir.exists():
-        if replace:
-            shutil.rmtree(output_dir)
-            output_dir.mkdir(parents=True, exist_ok=True)
-        elif any(output_dir.iterdir()):
-            raise ValueError(f"output_dir is not empty: {output_dir}")
-    else:
-        output_dir.mkdir(parents=True, exist_ok=True)
-    
-    meta = {"length": data.shape[0], "shape": data.shape[1:], "dtype": str(data.dtype), "blocks":[]}
-
-    bidx = 0
-    writer = None
-    for i,tmp in enumerate(tqdm(data)):
-        if i % blocksize == 0:
-            if writer is not None:
-                writer.close()
-            writer = ArrayRecordWriter(str(output_dir / f"{bidx}.ar"), options=ar_options)
-            meta["blocks"].append(f"{bidx}.ar")
-            bidx += 1
-        writer.write(tmp.tobytes())
-    writer.close()
-
-    with open(output_dir / "meta.json", 'w', encoding='utf-8') as f:
-        json.dump(meta, f)

loaderx-0.3.0/loaderx.egg-info/PKG-INFO

@@ -1,260 +0,0 @@
-Metadata-Version: 2.4
-Name: loaderx
-Version: 0.3.0
-Summary: A record-based data runtime, focused on delivering extreme throughput and low latency
-Author-email: Ben0i0d <ben0i0d@foxmail.com>
-License: MIT License
-        
-        Copyright (c) 2025 EOELAB AI Research
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-        
-Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
-Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
-Project-URL: Source, https://codeberg.org/eoelab/loaderx
-Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
-Keywords: flax,python,dataloader
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Requires-Python: ==3.13
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy
-Requires-Dist: array-record
-Requires-Dist: tqdm
-Dynamic: license-file
-
-# loaderx
-A record-based data runtime, focused on delivering extreme throughput and low latency
-
-**Only Python3.13_Linux_amd64**
-
-## Sampler
-a high-performance sampler implemented in Zig
-### Index Generator
-1. Sequential generation: indices are produced by traversing the index space in order.
-    * Sliding traversal: indices are obtained using a fixed-size sliding window. Note that in this case, the index space is treated as a circular queue to avoid truncation at the tail.
-2. Random generation: indices are sampled randomly from the index space.
-    * Global random: a set of samples is drawn randomly from the entire index space.
-
-## zrecord
-基于Zig实现一个并发友好、实现更简单、单机性能更优的 record 存储系统
-
-1. ZRecord基于 record 思想设计，我们不假定record间存在顺序关系，对于顺序语义应在record内部表达或建立timestamp
-2. record是相互独立的逻辑记录，仅在 Python 侧提供与 NumPy 兼容的 array 接口投影
-3. zrecord是无类型的，数据只是Bytes，类型解析由上层完成
-4. chunk只是存储管理单元，提高并发性能，不具备语义
-
-### 存储形式
-1. ZRecord 仅包含一个多流数据集，长度为 N（N 个 record）
-    * 索引空间为 [0, N)，所有索引与切块语义等价于 gather 操作
-    * 多流，类似 `set['A'], set['B']`, 但必须等长
-    ```
-    Dataset:
-    record_id = 0..N-1
-
-    Field A: bytes / tensor / audio / text
-    Field B: bytes / label / meta
-    ```
-2. 为降低存储体积，支持粒度为record级的透明压缩，以下是可选的压缩方法
-```
-| 方法  | 算法    |
-| ----- | ------- |
-| raw   | 无压缩  | 
-| flate | Deflate | 
-```
-3. 通过offset支持高效随机访问
-    * offset表项是一个三元组 [[chunk_id : u32, offset : u64, physical_length : u32], ...]，长度为N。将外部索引空间（outside[0, N]）指向一个实际存储地址
-        * chunk_id为u32，表示具体chunk
-        * offset为u64, 表示具体chunk内的偏移
-        * physical_length为u32，表示record数据大小
-        * offset表每项的字节大小最好为 8 的整数倍，提升对齐和缓存效率
-        * offset表是mmap访问模式，由于等长，直接将 idx 转换为 ptr + 16*idx
-        * 不同位宽表示的数值范围
-        ```
-        u16 → 2^16 - 1 （65535）
-        u32 → 2^32 - 1 （4.29e9）
-        u64 → 2^64 - 1 （1.84e19）
-        ```
-        * 不同位宽表示的数据大小
-        ```
-        u16 → 64 KiB
-        u32 → 4 GiB
-        u64 → 16 EiB
-        ```
-    ```
-    outside idx (global)
-    ↓
-    ↓offset[idx]
-    ↓
-    chunk_id, offset, physical_length
-    ```
-4. ZRecord的存储是元数据+分块的格式
-    1. 元数据
-        * meta.json：描述全局元数据，包括全局参数与Field参数
-        ```
-        {
-        "length": 65536,
-        "chunks": { "num": 12, "size": 8192, "unfull": {[8, write_pos, record_count], ... } },
-        "fields": [ {"name": "A", "dtype": "f32", "compress": "flate"}, {"name": "B", "dtype": "i32", "compress": "raw"} ]
-        }
-        ```
-        * Field_offset.zr：索引表
-        ```
-        0 0 255
-        0 256 511
-        ...
-        ```
-    2. 分块数据
-        * chunk/x.zr: 分块的chunk数据,chunk不区分数据所属
-
-存储格式如下
-```
-dataset
-  ├── meta.json
-  ├── A.offset
-  ├── B.offset
-  ├── chunk
-  │    ├── 0.zr
-  └──└── 1.zr
-```
-
-### 执行器
-
-负责维护任务队列，工作线程，handle
-
-1. 任务队列：
-    * write_task：写任务队列，压入[ops, record]，等待writer弹出
-    * read_task：读任务队列，预分配batch并压入[batch, pos, chunk_id, offset, physical_length]，等待reader弹出
-        * batch: 具体读取请求下创建的与indices等长的二元组
-        * pos：在batch中的位置，保证等序返回
-    * gc_task：垃圾回收任务队列，压入batch，等待cleaner弹出
-
-2. 工作线程：执行具体任务的工作线程
-    * writer：线程池执行写任务，但每个线程固定对应一个chunk
-        * writer只会向自己的chunk写入（写满时新建chunk提交给执行器完成）
-        * num_writer推荐：1-4
-    * reader：线程池执行读任务
-        * num_reader推荐：8-32
-    * cleaner：线程池执行垃圾回收任务
-        * num_cleaner推荐：2-4
-
-3. handle：
-    1. mmap：维护全局offset、chunk文件的handle
-        * 初始化阶段注册,关闭前释放
-        * 对于full的chunk,使用mmap+close（只读），对于unfull的chunk使用mmap（读写）。一旦写满（writer提交），新建chunk并返回给writer,原mmap close。（避免过多占用fd）
-        * 对于chunk/offset文件，动态完成分块增长，基于ftruncate+mremap实现
-            * 扩容offset表，一次增长 1M项（16MiB）
-            * 扩容chunk文件，一次增长 1GiB
-    2. batch：维护每次返回batch的handle,也就是[ptr, logical_length]中ptr的合规性
-        * 工作线程完成后注册
-        * 必须调用函数来手动释放，释放时batch压入GC队列
-
-#### 在线任务
-
-1. 写任务：写入只允许chunk-level append-only,其余操作基于offset重定向，缺失的field构造为全为0的项（physical_length为0代表无）
-    1. 追加：在chunk当前写入位置追加一个record，offset表更新length+1条目，全局长度增加
-        * 写满：写入完成时检查，如果record_count达到chunk_size, 新建chunk
-        * 写入越界：捕捉错误，并对chunk扩容
-        * offset[ length+1 ]不存在：捕捉错误，并对offset扩容
-    2. 修改：追加一个record，并修改offset项
-    3. 删除：删除offset项(将目标项替换为最后一项)，length减一（越界访问属于UB行为）
-
-2. 读任务：用一个indices（索引数组），从原数组里访问指定位置的元素(gather)
-    1. 对于无压缩的数据，将直接返回文件内地址实现zero-copy
-
-    读取流程
-    ```
-    [idx]
-    ↓
-    [batch, pos, idx]
-    ↓field_offset
-    [batch, pos, chunk_id, offset, physical_length]
-    ↓
-    [ptr, logical_length]
-    ```
-
-3. 垃圾回收任务：cleaner从GC队列中释放对应batch
-    1. 对于无压缩的数据，不释放内存，只释放 batch 本身
-    2. 对于有压缩的数据，释放内存
-    3. GC后访问ptr属于UB行为
-
-#### 离线任务
-
-1. 重平衡（Rebalance）：zrecord经过大量写操作后，会出现chunk访问不平衡,通过重平衡来恢复chunk的可访问性
-    * 实现：根据offset进行重写入，并在完成后替换
-    * Rebalance只允许手动运行，确保用户知情，提供一个利用率（sum(length)/sum(chunk_size*num_chunks)）
-
-### 分布式扩展
-
-**该部分暂时不会实现，只是提前架构**
-
-1. 分布式下，我们会得到一个更高的层次-cluster,单机将变为cluster内的node
-2. node持有一个shard,包含若干个chunk
-3. 添加一个indirection表，将全局路径映射到具体node的表，indirection也可以使用hash实现，从而无锁，索引路径变为
-```
-outside idx (global)
-↓
-↓indirection[idx]
-↓
-node, idx
-↓
-↓offset[idx]
-↓
-chunk_id, offset, physical_length
-```
-
-## Convert a NumPy tensor to Array_record
-
-*This will create a directory containing file shards, which helps improve I/O performance.*
-
-```
-import numpy as np
-from loaderx import converter
-
-train_data = np.load('train_data.npy',mmap_mode='r')
-converter(train_data, 'train_data')
-```
-
-## Current Limitations
-Currently, loaderx only supports single-host environments and does not yet support multi-host training.
-
-## Quick Start
-```
-import numpy as np
-from loaderx import NPDataset, ARDataset, DataLoader
-
-dataset = ARDataset('train_data')
-labelset = NPDataset('xsub/train_label.npy')
-
-loader = DataLoader(dataset, labelset)
-
-for i, batch in enumerate(loader):
-    if i >= 256:
-        break
-
-print(batch['data'].shape)
-print(batch['label'].shape)
-```
-
-### Integrating with JAX/Flax
-
-For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

loaderx-0.3.0/loaderx.egg-info/requires.txt

@@ -1,3 +0,0 @@
-numpy
-array-record
-tqdm