datago 2026.1.2__cp312-cp312-manylinux_2_31_x86_64.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
README.md ADDED
@@ -0,0 +1,319 @@
1
+ # datago
2
+
3
+ [![Rust](https://github.com/Photoroom/datago/actions/workflows/rust.yml/badge.svg)](https://github.com/Photoroom/datago/actions/workflows/rust.yml)
4
+ [![Rust-py](https://github.com/Photoroom/datago/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/Photoroom/datago/actions/workflows/ci-cd.yml)
5
+
6
+ A Rust-written data loader which can be used as a python module. Handles several data sources, from local files to webdataset or a VectorDB focused http stack [soon-to-be open sourced](https://github.com/Photoroom/dataroom). Focused on image data at the moment, could also easily be more generic.
7
+
8
+ Datago handles, outside of the Python GIL
9
+
10
+ - per sample IO
11
+ - deserialization (jpg and png decompression)
12
+ - some optional vision processing (aligning different image payloads)
13
+ - optional serialization
14
+
15
+ Samples are exposed in the Python scope as python native objects, using PIL and Numpy base types. Speed will be network dependent, but GB/s is typical. Depending on the front ends, datago can be rank and world-size aware, in which case the samples are dispatched depending on the samples hash.
16
+
17
+ ![Datago organization](assets/447175851-2277afcb-8abf-4d17-b2db-dae27c6056d0.png)
18
+
19
+ <details> <summary><strong>Use it</strong></summary>
20
+
21
+ You can simply install datago with `[uv] pip install datago`
22
+
23
+ ## Use the package from Python
24
+ Please note that in all the of the following cases, you can directly get an IterableDataset (torch compatible) with the following code snippet
25
+
26
+ ```python
27
+ from dataset import DatagoIterDataset
28
+ client_config = {} # See below for examples
29
+ datago_dataset = DatagoIterDataset(client_config, return_python_types=True)
30
+ ```
31
+
32
+ `return_python_types` enforces that images will be of the PIL.Image sort for instance, being an external binary module should be transparent.
33
+
34
+ <details> <summary><strong>Dataroom</strong></summary>
35
+
36
+ ```python
37
+ from datago import DatagoClient, initialize_logging
38
+ import os
39
+ import json
40
+
41
+ # Respects RUST_LOG=INFO env var for setting log level
42
+ # If omitted the logger will be initialized when the client starts.
43
+ initialize_logging()
44
+
45
+ config = {
46
+ "source_config": {
47
+ "sources": os.environ.get("DATAROOM_TEST_SOURCE", ""),
48
+ "page_size": 500,
49
+ "rank": 0,
50
+ "world_size": 1,
51
+ },
52
+ "limit": 200,
53
+ "samples_buffer_size": 32,
54
+ }
55
+
56
+ client = DatagoClient(json.dumps(config))
57
+
58
+ for _ in range(10):
59
+ sample = client.get_sample()
60
+ ```
61
+
62
+ Please note that the image buffers will be passed around as raw pointers, see below (we provide python utils to convert to PIL types).
63
+
64
+ </details><details> <summary><strong>Local files</strong></summary>
65
+
66
+ To test datago while serving local files (jpg, png, ..), code would look like the following.
67
+ **Note that datago serving files with a lot of concurrent threads means that, even if random_sampling is not set,
68
+ there will be some randomness in the sample ordering.**
69
+
70
+ ```python
71
+ from datago import DatagoClient, initialize_logging
72
+ import os
73
+ import json
74
+
75
+ # Can also set the log level directly instead of using RUST_LOG env var
76
+ initialize_logging(log_level="warn")
77
+
78
+ config = {
79
+ "source_type": "file",
80
+ "source_config": {
81
+ "root_path": "myPath",
82
+ "random_sampling": False, # True if used directly for training
83
+ "rank": 0, # Optional, distributed workloads are possible
84
+ "world_size": 1,
85
+ },
86
+ "limit": 200,
87
+ "samples_buffer_size": 32,
88
+ }
89
+
90
+ client = DatagoClient(json.dumps(config))
91
+
92
+ for _ in range(10):
93
+ sample = client.get_sample()
94
+ ```
95
+
96
+ </details><details> <summary><strong>[experimental] Webdataset</strong></summary>
97
+
98
+ Please note that this implementation is very new, and probably has significant limitations still. It has not yet been tested at scale.
99
+ Please also note that you can find a better example in /python/benchmark_webdataset.py, which will show how to convert everything to more pythonic types (PIL images).
100
+
101
+ ```python
102
+ from datago import DatagoClient, initialize_logging
103
+ import os
104
+ import json
105
+
106
+ # Can also set the log level directly instead of using RUST_LOG env var
107
+ initialize_logging(log_level="warn")
108
+
109
+ # URL of the test bucket
110
+ bucket = "https://storage.googleapis.com/webdataset/fake-imagenet"
111
+ dataset = "/imagenet-train-{000000..001281}.tar"
112
+ url = bucket + dataset
113
+
114
+ client_config = {
115
+ "source_type": "webdataset",
116
+ "source_config": {
117
+ "url": url,
118
+ "random_sampling": False,
119
+ "concurrent_downloads": 8, # The number of TarballSamples which should be handled concurrently
120
+ "rank": 0,
121
+ "world_size": 1,
122
+ },
123
+ "prefetch_buffer_size": 128,
124
+ "samples_buffer_size": 64,
125
+ "limit": 1_000_000, # Dummy example, max number of samples you would like to serve
126
+ }
127
+
128
+ client = DatagoClient(json.dumps(client_config))
129
+
130
+ for _ in range(10):
131
+ sample = client.get_sample()
132
+ ```
133
+
134
+ </details>
135
+
136
+ ## Process images on the fly
137
+
138
+ Datago can also process images on the fly, for instance to align different image payloads. This is done by adding an `image_config` to the configuration. The following example shows how to align different image payloads.
139
+
140
+ Processing can be very CPU heavy, but it will be distributed over all CPU cores wihout requiring multiple python processes. I.e., you can keep a single python process using `get_sample()` on the client and still saturate all CPU cores.
141
+
142
+ There are three main processing topics that you can choose from:
143
+
144
+ - crop the images to within an aspect ratio bucket (which is very handy for all Transformer / patch based architectures)
145
+ - resize the images (setting here will be related to the square aspect ratio bucket, other buckets will differ of course)
146
+ - pre-encode the images to a specific format (jpg, png, ...)
147
+
148
+ ```python
149
+ config = {
150
+ "source_type": "file",
151
+ "source_config": {
152
+ "root_path": "myPath",
153
+ "random_sampling": False, # True if used directly for training
154
+ },
155
+ # Optional pre-processing of the images, placing them in an aspect ratio bucket to preserve as much as possible of the original content
156
+ "image_config": {
157
+ "crop_and_resize": True, # False to turn it off, or just omit this part of the config
158
+ "default_image_size": 1024,
159
+ "downsampling_ratio": 32,
160
+ "min_aspect_ratio": 0.5,
161
+ "max_aspect_ratio": 2.0,
162
+ "pre_encode_images": False,
163
+ },
164
+ "limit": 200,
165
+ "samples_buffer_size": 32,
166
+ }
167
+ ```
168
+
169
+ ## Match the raw exported buffers with typical python types
170
+
171
+ See helper functions provided in `raw_types.py`, should be self explanatory. Check python benchmarks for examples. As mentioned above, we also provide a wrapper so that you get a `dataset` directly.
172
+
173
+ ## Logging
174
+
175
+ We are using the [log](https://docs.rs/log/latest/log/) crate with [env_logger](https://docs.rs/env_logger/latest/env_logger/).
176
+ You can set the log level using the RUST_LOG environment variable. E.g. `RUST_LOG=INFO`.
177
+
178
+ When using the library from Python, `env_logger` will be initialized automatically when creating a `DatagoClient`. There is also a `initialize_logging` function in the `datago` module, which if called before using a client, allows to customize the log level. This only works if RUST_LOG is not set.
179
+
180
+ ## Env variables
181
+
182
+ There are a couple of env variables which will change the behavior of the library, for settings which felt too low level to be exposed in the config.
183
+
184
+ - `DATAGO_MAX_TASKS`: refers to the number of threads which will be used to load the samples. Defaults to a multiple of the CPU cores.
185
+ - `RUST_LOG`: see above, will change the level of logging for the whole library, could be useful for debugging or to report an issue here.
186
+ - `DATAGO_MAX_RETRIES`: number of retries for a failed sample load, defaults to 3.
187
+
188
+ </details><details> <summary><strong>Build it</strong></summary>
189
+
190
+ ## Preamble
191
+
192
+ Just install the rust toolchain via rustup
193
+
194
+ ## [Apple Silicon MacOS only]
195
+
196
+ If you are using an Apple Silicon Mac OS machine, create a `.cargo/config` file and paste the following:
197
+
198
+ ``` cfg
199
+ [target.x86_64-apple-darwin]
200
+ rustflags = [
201
+ "-C", "link-arg=-undefined",
202
+ "-C", "link-arg=dynamic_lookup",
203
+ ]
204
+
205
+ [target.aarch64-apple-darwin]
206
+ rustflags = [
207
+ "-C", "link-arg=-undefined",
208
+ "-C", "link-arg=dynamic_lookup",
209
+ ]
210
+ ```
211
+
212
+ ## Build a benchmark CLI
213
+
214
+ `Cargo run --release -- -h` to get all the information, should be fairly straightforward
215
+
216
+ ## Run the rust test suite
217
+
218
+ From the datago folder
219
+
220
+ ```bash
221
+ cargo test
222
+ ```
223
+
224
+ ## Generate the python package binaries manually
225
+
226
+ Build a wheel useable locally
227
+
228
+ ```bash
229
+ maturin build -i python3.11 --release --target "x86_64-unknown-linux-gnu"
230
+ ```
231
+
232
+ Build a wheel which can be uploaded to pypi or related
233
+
234
+ - either use a manylinux docker image
235
+
236
+ - or cross compile using zip
237
+
238
+ ```bash
239
+ maturin build -i python3.11 --release --target "x86_64-unknown-linux-gnu" --manylinux 2014 --zig
240
+ ```
241
+
242
+ then you can `pip install` from `target/wheels`
243
+
244
+ ## Update the pypi release (maintainers)
245
+
246
+ Create a new tag and a new release in this repo, a new package will be pushed automatically.
247
+
248
+ </details>
249
+
250
+ <details> <summary><strong>Benchmarks</strong></summary>
251
+ As usual, benchmarks are a tricky game, and you shouldn't read too much into the following plots but do your own tests. Some python benchmark examples are provided in the [python](./python/) folder.
252
+
253
+ In general, Datago will be impactful if you want to load a lot of images very fast, but if you consume them as you go at a more leisury pace then it's not really needed. The more CPU work there is with the images and the higher quality they are, the more Datago will shine.
254
+
255
+ ## From disk: ImageNet
256
+
257
+ The following benchmarks are using ImageNet 1k, which is very low resolution and thus kind of a worst case scenario. Data is served from cache (i.e. the OS cache) and the images are not pre-processed. In this case the receiving python process is typically the bottleneck, and caps at around 3000 images per second.
258
+
259
+ ### AMD Zen3 laptop - IN1k - disk - no processing
260
+ ![AMD Zen3 laptop & M2 SSD](assets/zen3_ssd.png)
261
+
262
+ ### AMD EPYC 9454 - IN1k - disk - no processing
263
+ ![AMD EPYC 9454](assets/epyc_vast.png)
264
+
265
+ ## Webdataset: FakeIN
266
+
267
+ This benchmark is using low resolution images. It's accessed through the webdataset front end, datago is compared with the popular python webdataset library. Note that datago will start streaming the images faster here (almost instantly !), which emphasizes throughput differences depending on how long you test it for.
268
+
269
+ Of note is also that this can be bottlenecked by your external bandwidth to the remote storage where WDS is hosted, in which case both solution would yield comparable numbers.
270
+
271
+ ### AMD Zen3 laptop - webdataset - no processing
272
+ ![AMD EPYC 9454](assets/zen3_wds_fakein.png)
273
+
274
+
275
+ ## Webdataset: PD12M
276
+
277
+ This benchmark is using high resolution images. It's accessed through the webdataset front end, datago is compared with the popular python webdataset library. Note that datago will start streaming the images faster here (almost instantly !), which emphasizes throughput differences depending on how long you test it for.
278
+
279
+ Of note is also that this can be bottlenecked by your external bandwidth to the remote storage where WDS is hosted, in which case both solution would yield comparable numbers.
280
+
281
+ ### AMD Zen3 laptop - webdataset - no processing
282
+ ![AMD Zen3 laptop](assets/zen3_wds_pd12m.png)
283
+
284
+
285
+ ### AMD EPYC 9454 - pd12m - webdataset - no processing
286
+ ![AMD EPYC 9454](assets/epyc_wds_pd12m.png)
287
+
288
+
289
+ ### AMD Zen3 laptop - webdataset - processing
290
+ Adding image processing (crop and resize to Transformer compatible size buckets) to the equation changes the picture, as the work spread becomes more important. If you're training a diffusion model or an image encoder from a diverse set of images, this is likely to be the most realistic micro-benchmark.
291
+
292
+ ![AMD Zen3 laptop](assets/zen3_wds_pd12m_processing.png)
293
+
294
+ </details>
295
+
296
+
297
+ ## License
298
+
299
+ MIT License
300
+
301
+ Copyright (c) 2025 Photoroom
302
+
303
+ Permission is hereby granted, free of charge, to any person obtaining a copy
304
+ of this software and associated documentation files (the "Software"), to deal
305
+ in the Software without restriction, including without limitation the rights
306
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
307
+ copies of the Software, and to permit persons to whom the Software is
308
+ furnished to do so, subject to the following conditions:
309
+
310
+ The above copyright notice and this permission notice shall be included in all
311
+ copies or substantial portions of the Software.
312
+
313
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
314
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
315
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
316
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
317
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
318
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
319
+ SOFTWARE.
datago/__init__.py ADDED
@@ -0,0 +1,5 @@
1
+ from .datago import *
2
+
3
+ __doc__ = datago.__doc__
4
+ if hasattr(datago, "__all__"):
5
+ __all__ = datago.__all__
@@ -0,0 +1,337 @@
1
+ Metadata-Version: 2.4
2
+ Name: datago
3
+ Version: 2026.1.2
4
+ Classifier: Programming Language :: Rust
5
+ Classifier: Programming Language :: Python :: Implementation :: CPython
6
+ Classifier: Programming Language :: Python :: Implementation :: PyPy
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ License-File: LICENSE
10
+ Summary: A high performance dataloader for Python, written in Rust
11
+ Author: Benjamin Lefaudeux, Roman Frigg
12
+ Author-email: Photoroom <team@photoroom.com>
13
+ Requires-Python: >=3.8
14
+ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
15
+ Project-URL: Homepage, https://github.com/photoroom/datago
16
+ Project-URL: Issues, https://github.com/photoroom/datago/issues
17
+
18
+ # datago
19
+
20
+ [![Rust](https://github.com/Photoroom/datago/actions/workflows/rust.yml/badge.svg)](https://github.com/Photoroom/datago/actions/workflows/rust.yml)
21
+ [![Rust-py](https://github.com/Photoroom/datago/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/Photoroom/datago/actions/workflows/ci-cd.yml)
22
+
23
+ A Rust-written data loader which can be used as a python module. Handles several data sources, from local files to webdataset or a VectorDB focused http stack [soon-to-be open sourced](https://github.com/Photoroom/dataroom). Focused on image data at the moment, could also easily be more generic.
24
+
25
+ Datago handles, outside of the Python GIL
26
+
27
+ - per sample IO
28
+ - deserialization (jpg and png decompression)
29
+ - some optional vision processing (aligning different image payloads)
30
+ - optional serialization
31
+
32
+ Samples are exposed in the Python scope as python native objects, using PIL and Numpy base types. Speed will be network dependent, but GB/s is typical. Depending on the front ends, datago can be rank and world-size aware, in which case the samples are dispatched depending on the samples hash.
33
+
34
+ ![Datago organization](assets/447175851-2277afcb-8abf-4d17-b2db-dae27c6056d0.png)
35
+
36
+ <details> <summary><strong>Use it</strong></summary>
37
+
38
+ You can simply install datago with `[uv] pip install datago`
39
+
40
+ ## Use the package from Python
41
+ Please note that in all the of the following cases, you can directly get an IterableDataset (torch compatible) with the following code snippet
42
+
43
+ ```python
44
+ from dataset import DatagoIterDataset
45
+ client_config = {} # See below for examples
46
+ datago_dataset = DatagoIterDataset(client_config, return_python_types=True)
47
+ ```
48
+
49
+ `return_python_types` enforces that images will be of the PIL.Image sort for instance, being an external binary module should be transparent.
50
+
51
+ <details> <summary><strong>Dataroom</strong></summary>
52
+
53
+ ```python
54
+ from datago import DatagoClient, initialize_logging
55
+ import os
56
+ import json
57
+
58
+ # Respects RUST_LOG=INFO env var for setting log level
59
+ # If omitted the logger will be initialized when the client starts.
60
+ initialize_logging()
61
+
62
+ config = {
63
+ "source_config": {
64
+ "sources": os.environ.get("DATAROOM_TEST_SOURCE", ""),
65
+ "page_size": 500,
66
+ "rank": 0,
67
+ "world_size": 1,
68
+ },
69
+ "limit": 200,
70
+ "samples_buffer_size": 32,
71
+ }
72
+
73
+ client = DatagoClient(json.dumps(config))
74
+
75
+ for _ in range(10):
76
+ sample = client.get_sample()
77
+ ```
78
+
79
+ Please note that the image buffers will be passed around as raw pointers, see below (we provide python utils to convert to PIL types).
80
+
81
+ </details><details> <summary><strong>Local files</strong></summary>
82
+
83
+ To test datago while serving local files (jpg, png, ..), code would look like the following.
84
+ **Note that datago serving files with a lot of concurrent threads means that, even if random_sampling is not set,
85
+ there will be some randomness in the sample ordering.**
86
+
87
+ ```python
88
+ from datago import DatagoClient, initialize_logging
89
+ import os
90
+ import json
91
+
92
+ # Can also set the log level directly instead of using RUST_LOG env var
93
+ initialize_logging(log_level="warn")
94
+
95
+ config = {
96
+ "source_type": "file",
97
+ "source_config": {
98
+ "root_path": "myPath",
99
+ "random_sampling": False, # True if used directly for training
100
+ "rank": 0, # Optional, distributed workloads are possible
101
+ "world_size": 1,
102
+ },
103
+ "limit": 200,
104
+ "samples_buffer_size": 32,
105
+ }
106
+
107
+ client = DatagoClient(json.dumps(config))
108
+
109
+ for _ in range(10):
110
+ sample = client.get_sample()
111
+ ```
112
+
113
+ </details><details> <summary><strong>[experimental] Webdataset</strong></summary>
114
+
115
+ Please note that this implementation is very new, and probably has significant limitations still. It has not yet been tested at scale.
116
+ Please also note that you can find a better example in /python/benchmark_webdataset.py, which will show how to convert everything to more pythonic types (PIL images).
117
+
118
+ ```python
119
+ from datago import DatagoClient, initialize_logging
120
+ import os
121
+ import json
122
+
123
+ # Can also set the log level directly instead of using RUST_LOG env var
124
+ initialize_logging(log_level="warn")
125
+
126
+ # URL of the test bucket
127
+ bucket = "https://storage.googleapis.com/webdataset/fake-imagenet"
128
+ dataset = "/imagenet-train-{000000..001281}.tar"
129
+ url = bucket + dataset
130
+
131
+ client_config = {
132
+ "source_type": "webdataset",
133
+ "source_config": {
134
+ "url": url,
135
+ "random_sampling": False,
136
+ "concurrent_downloads": 8, # The number of TarballSamples which should be handled concurrently
137
+ "rank": 0,
138
+ "world_size": 1,
139
+ },
140
+ "prefetch_buffer_size": 128,
141
+ "samples_buffer_size": 64,
142
+ "limit": 1_000_000, # Dummy example, max number of samples you would like to serve
143
+ }
144
+
145
+ client = DatagoClient(json.dumps(client_config))
146
+
147
+ for _ in range(10):
148
+ sample = client.get_sample()
149
+ ```
150
+
151
+ </details>
152
+
153
+ ## Process images on the fly
154
+
155
+ Datago can also process images on the fly, for instance to align different image payloads. This is done by adding an `image_config` to the configuration. The following example shows how to align different image payloads.
156
+
157
+ Processing can be very CPU heavy, but it will be distributed over all CPU cores wihout requiring multiple python processes. I.e., you can keep a single python process using `get_sample()` on the client and still saturate all CPU cores.
158
+
159
+ There are three main processing topics that you can choose from:
160
+
161
+ - crop the images to within an aspect ratio bucket (which is very handy for all Transformer / patch based architectures)
162
+ - resize the images (setting here will be related to the square aspect ratio bucket, other buckets will differ of course)
163
+ - pre-encode the images to a specific format (jpg, png, ...)
164
+
165
+ ```python
166
+ config = {
167
+ "source_type": "file",
168
+ "source_config": {
169
+ "root_path": "myPath",
170
+ "random_sampling": False, # True if used directly for training
171
+ },
172
+ # Optional pre-processing of the images, placing them in an aspect ratio bucket to preserve as much as possible of the original content
173
+ "image_config": {
174
+ "crop_and_resize": True, # False to turn it off, or just omit this part of the config
175
+ "default_image_size": 1024,
176
+ "downsampling_ratio": 32,
177
+ "min_aspect_ratio": 0.5,
178
+ "max_aspect_ratio": 2.0,
179
+ "pre_encode_images": False,
180
+ },
181
+ "limit": 200,
182
+ "samples_buffer_size": 32,
183
+ }
184
+ ```
185
+
186
+ ## Match the raw exported buffers with typical python types
187
+
188
+ See helper functions provided in `raw_types.py`, should be self explanatory. Check python benchmarks for examples. As mentioned above, we also provide a wrapper so that you get a `dataset` directly.
189
+
190
+ ## Logging
191
+
192
+ We are using the [log](https://docs.rs/log/latest/log/) crate with [env_logger](https://docs.rs/env_logger/latest/env_logger/).
193
+ You can set the log level using the RUST_LOG environment variable. E.g. `RUST_LOG=INFO`.
194
+
195
+ When using the library from Python, `env_logger` will be initialized automatically when creating a `DatagoClient`. There is also a `initialize_logging` function in the `datago` module, which if called before using a client, allows to customize the log level. This only works if RUST_LOG is not set.
196
+
197
+ ## Env variables
198
+
199
+ There are a couple of env variables which will change the behavior of the library, for settings which felt too low level to be exposed in the config.
200
+
201
+ - `DATAGO_MAX_TASKS`: refers to the number of threads which will be used to load the samples. Defaults to a multiple of the CPU cores.
202
+ - `RUST_LOG`: see above, will change the level of logging for the whole library, could be useful for debugging or to report an issue here.
203
+ - `DATAGO_MAX_RETRIES`: number of retries for a failed sample load, defaults to 3.
204
+
205
+ </details><details> <summary><strong>Build it</strong></summary>
206
+
207
+ ## Preamble
208
+
209
+ Just install the rust toolchain via rustup
210
+
211
+ ## [Apple Silicon MacOS only]
212
+
213
+ If you are using an Apple Silicon Mac OS machine, create a `.cargo/config` file and paste the following:
214
+
215
+ ``` cfg
216
+ [target.x86_64-apple-darwin]
217
+ rustflags = [
218
+ "-C", "link-arg=-undefined",
219
+ "-C", "link-arg=dynamic_lookup",
220
+ ]
221
+
222
+ [target.aarch64-apple-darwin]
223
+ rustflags = [
224
+ "-C", "link-arg=-undefined",
225
+ "-C", "link-arg=dynamic_lookup",
226
+ ]
227
+ ```
228
+
229
+ ## Build a benchmark CLI
230
+
231
+ `Cargo run --release -- -h` to get all the information, should be fairly straightforward
232
+
233
+ ## Run the rust test suite
234
+
235
+ From the datago folder
236
+
237
+ ```bash
238
+ cargo test
239
+ ```
240
+
241
+ ## Generate the python package binaries manually
242
+
243
+ Build a wheel useable locally
244
+
245
+ ```bash
246
+ maturin build -i python3.11 --release --target "x86_64-unknown-linux-gnu"
247
+ ```
248
+
249
+ Build a wheel which can be uploaded to pypi or related
250
+
251
+ - either use a manylinux docker image
252
+
253
+ - or cross compile using zip
254
+
255
+ ```bash
256
+ maturin build -i python3.11 --release --target "x86_64-unknown-linux-gnu" --manylinux 2014 --zig
257
+ ```
258
+
259
+ then you can `pip install` from `target/wheels`
260
+
261
+ ## Update the pypi release (maintainers)
262
+
263
+ Create a new tag and a new release in this repo, a new package will be pushed automatically.
264
+
265
+ </details>
266
+
267
+ <details> <summary><strong>Benchmarks</strong></summary>
268
+ As usual, benchmarks are a tricky game, and you shouldn't read too much into the following plots but do your own tests. Some python benchmark examples are provided in the [python](./python/) folder.
269
+
270
+ In general, Datago will be impactful if you want to load a lot of images very fast, but if you consume them as you go at a more leisury pace then it's not really needed. The more CPU work there is with the images and the higher quality they are, the more Datago will shine.
271
+
272
+ ## From disk: ImageNet
273
+
274
+ The following benchmarks are using ImageNet 1k, which is very low resolution and thus kind of a worst case scenario. Data is served from cache (i.e. the OS cache) and the images are not pre-processed. In this case the receiving python process is typically the bottleneck, and caps at around 3000 images per second.
275
+
276
+ ### AMD Zen3 laptop - IN1k - disk - no processing
277
+ ![AMD Zen3 laptop & M2 SSD](assets/zen3_ssd.png)
278
+
279
+ ### AMD EPYC 9454 - IN1k - disk - no processing
280
+ ![AMD EPYC 9454](assets/epyc_vast.png)
281
+
282
+ ## Webdataset: FakeIN
283
+
284
+ This benchmark is using low resolution images. It's accessed through the webdataset front end, datago is compared with the popular python webdataset library. Note that datago will start streaming the images faster here (almost instantly !), which emphasizes throughput differences depending on how long you test it for.
285
+
286
+ Of note is also that this can be bottlenecked by your external bandwidth to the remote storage where WDS is hosted, in which case both solution would yield comparable numbers.
287
+
288
+ ### AMD Zen3 laptop - webdataset - no processing
289
+ ![AMD EPYC 9454](assets/zen3_wds_fakein.png)
290
+
291
+
292
+ ## Webdataset: PD12M
293
+
294
+ This benchmark is using high resolution images. It's accessed through the webdataset front end, datago is compared with the popular python webdataset library. Note that datago will start streaming the images faster here (almost instantly !), which emphasizes throughput differences depending on how long you test it for.
295
+
296
+ Of note is also that this can be bottlenecked by your external bandwidth to the remote storage where WDS is hosted, in which case both solution would yield comparable numbers.
297
+
298
+ ### AMD Zen3 laptop - webdataset - no processing
299
+ ![AMD Zen3 laptop](assets/zen3_wds_pd12m.png)
300
+
301
+
302
+ ### AMD EPYC 9454 - pd12m - webdataset - no processing
303
+ ![AMD EPYC 9454](assets/epyc_wds_pd12m.png)
304
+
305
+
306
+ ### AMD Zen3 laptop - webdataset - processing
307
+ Adding image processing (crop and resize to Transformer compatible size buckets) to the equation changes the picture, as the work spread becomes more important. If you're training a diffusion model or an image encoder from a diverse set of images, this is likely to be the most realistic micro-benchmark.
308
+
309
+ ![AMD Zen3 laptop](assets/zen3_wds_pd12m_processing.png)
310
+
311
+ </details>
312
+
313
+
314
+ ## License
315
+
316
+ MIT License
317
+
318
+ Copyright (c) 2025 Photoroom
319
+
320
+ Permission is hereby granted, free of charge, to any person obtaining a copy
321
+ of this software and associated documentation files (the "Software"), to deal
322
+ in the Software without restriction, including without limitation the rights
323
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
324
+ copies of the Software, and to permit persons to whom the Software is
325
+ furnished to do so, subject to the following conditions:
326
+
327
+ The above copyright notice and this permission notice shall be included in all
328
+ copies or substantial portions of the Software.
329
+
330
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
331
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
332
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
333
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
334
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
335
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
336
+ SOFTWARE.
337
+
@@ -0,0 +1,7 @@
1
+ README.md,sha256=XwJ68dy_mOJNETAX0ZT4BRY0At9f2feXXlYrysdoBZo,12828
2
+ datago/__init__.py,sha256=kNnGM4GPRHPbxtWqYjr-57Td4rOTtl3kLaKExPLuE7g,107
3
+ datago/datago.cpython-312-x86_64-linux-gnu.so,sha256=sp3KumG2zHx78hQ09cJFBSuymMOPBe_R2-VBzfDksyw,13580424
4
+ datago-2026.1.2.dist-info/METADATA,sha256=7sgPUJ-oKGjIajO_YERXdCGBaZFMDjoJ183-zbF6YvY,13552
5
+ datago-2026.1.2.dist-info/WHEEL,sha256=ULB5_LOdT1XIeV6m04GmfiOw1ifWnpWoDWBprrVKHrs,109
6
+ datago-2026.1.2.dist-info/licenses/LICENSE,sha256=5BkY8hpbSq1ldBOzHL1XHW4h-ognKi_lZa2zeWailV8,1066
7
+ datago-2026.1.2.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: maturin (1.11.2)
3
+ Root-Is-Purelib: false
4
+ Tag: cp312-cp312-manylinux_2_31_x86_64
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2024 Photoroom
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.