litdata 0.2.68__tar.gz → 0.2.69__tar.gz

{litdata-0.2.68/src/litdata.egg-info → litdata-0.2.69}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: litdata
-Version: 0.2.68
+Version: 0.2.69
 Summary: The Deep Learning framework to train, deploy, and ship AI products Lightning fast.
 Home-page: https://github.com/Lightning-AI/litdata
 Download-URL: https://github.com/Lightning-AI/litdata
@@ -989,6 +989,12 @@ for batch_idx, batch in enumerate(dataloader):
         torch.save(dataloader.state_dict(), "dataloader_state.pt")
 ```
 
+Same `seed` and `shuffle` are required. For a **`StreamingDataset`**, **`num_workers` and `world_size` may change**: LitData drops a global `sample_in_epoch` prefix and restripes the rest (never duplicates remaining IDs). For a matching loss curve keep **global batch size** (`world_size * batch_size`) constant and DDP ranks in lockstep. `num_canonical_nodes` (default: first-run `world_size`) is frozen in the checkpoint. POSIX `WindowShuffle` resumes whole remaining chunks. **`CombinedStreamingDataset` and `ParallelStreamingDataset` resume only with the same `world_size`, `num_workers`, and `batch_size`.**
+
+```python
+dataset = StreamingDataset("s3://my-bucket/my-data", shuffle=True, num_canonical_nodes=8)
+```
+
 </details>
 
 
@@ -996,11 +1002,9 @@ for batch_idx, batch in enumerate(dataloader):
   <summary> ✅ Use shared queue for Optimizing <a id="shared-queue" href="#shared-queue">🔗</a> </summary>
 &nbsp;
 
-If you are using multiple workers to optimize your dataset, you can use a shared queue to speed up the process.
+`optimize` / `map` default to a **shared per-node queue** (`keep_data_ordered=False`). Work is packed per node, then every worker on that node pulls the next item, so a slow worker does not leave others idle. Set `keep_data_ordered=True` to keep a static per-worker slice (required for `use_checkpoint` and `align_chunking`).
 
-This is especially useful when optimizing large datasets in parallel, where some workers may be slower than others.
-
-It can also improve fault tolerance when workers fail due to out-of-memory (OOM) errors.
+Local `output_dir` writes chunks in place. Remote inputs and outputs use the streaming downloader (`adownload_file` / `aupload_file`, obstore when available).
 
 ```python
 import numpy as np
@@ -1023,28 +1027,36 @@ if __name__ == "__main__":
         output_dir="fast_data",             # optimized data is stored here
         num_workers=4,                      # The number of workers on the same machine
         chunk_bytes="64MB" ,                 # size of each chunk
-        keep_data_ordered=False,             # Use a shared queue to speed up the process
+        keep_data_ordered=False,             # default: shared queue (set True to keep input order)
     )
 ```
 
 
-### Performance Difference between using a shared queue and not using it:
+### Shared queue vs ordered (skewed local files)
 
-**Note**: The following benchmarks were collected using the ImageNet dataset on an A10G machine with 16 workers.
+`scripts/bench/bench_node_queue.py --files 4000 --workers 8` (first 500 files are 1 MiB). On `main`, unordered optimize sat on a 200s empty-queue timeout after work finished.
 
-| Configuration    | Optimize Time (sec) | Stream 1 (img/sec) | Stream 2 (img/sec) |
-|------------------|---------------------|---------------------|---------------------|
-| shared_queue (`keep_data_ordered=False`)     | 1281                | 5392                | 5732                |
-| no shared_queue (`keep_data_ordered=True (default)`)  | 1187                | 5257                | 5746                |
+| Tree | Mode | Time | Throughput |
+|------|------|-----:|-----------:|
+| `main` (old default) | `keep_data_ordered=True` | 23.7s | 169 files/s |
+| `main` | `keep_data_ordered=False` | 223.6s | 18 files/s |
+| this tree | `keep_data_ordered=True` | 22.9s | 175 files/s |
+| this tree (**new default**) | `keep_data_ordered=False` | **18.8s** | 213 files/s |
 
-📌 Note: The **shared_queue** option impacts optimization time, not streaming speed.
-> While the streaming numbers may appear slightly different, this variation is incidental and not caused by shared_queue.
->
-> Streaming happens after optimization and does not involve inter-process communication where shared_queue plays a role.
+Shared-queue **before → after: ~12×**. New default vs old ordered default: **1.22×**.
+
+### Local / remote input × output
+
+`python scripts/bench/bench_node_queue.py --files 200 --workers 4 --io-matrix` (first 50 files are 1 MiB).
 
-- 📄 Using a shared queue helps balance the load across workers, though it may slightly increase optimization time due to the overhead of pickling items sent between processes.
+| Topology | Ordered | Shared | Speedup |
+|----------|--------:|-------:|--------:|
+| local → local | 6.55s | **2.96s** | 2.21× |
+| remote → local | 7.44s | **3.48s** | 2.14× |
+| local → remote | 10.98s | **6.51s** | 1.69× |
+| remote → remote | 11.48s | **8.55s** | 1.34× |
 
-- ⚡ However, it can significantly improve optimizing performance — especially when some workers are slower than others.
+Shared queue balances uneven workers. It does not change later `StreamingDataset` throughput.
 
 </details>
 
@@ -2344,7 +2356,7 @@ if __name__ == "__main__":
 | `start_method` | spawn† | Multiprocessing start method (†spawn unless IPython) |
 | `optimize_dns` | `None` | Optimized DNS (Studio / cloud) |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared work queue (better for uneven/slow workers) |
+| `keep_data_ordered` | `False` | Shared work queue (faster for uneven workers). `True` keeps a static per-worker slice. Forced `True` with `use_checkpoint` / `align_chunking`. |
 
 </details>
 
@@ -2380,7 +2392,7 @@ Full knob list for `litdata.optimize` (see Quick start for the minimal recipe).
 | `start_method` | spawn† | Multiprocessing start method |
 | `optimize_dns` | `None` | Optimized DNS |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared queue among workers |
+| `keep_data_ordered` | `False` | Shared queue among workers. `True` keeps input order. Forced `True` with `use_checkpoint` / `align_chunking`. |
 | `verbose` | `True` | Progress logging |
 
 Related features: [shared queue](#shared-queue), [queue input](#queue-input), [append/overwrite](#modify-datasets), [compression](#compression), [TokensLoader / LLM](#llm-training), [filter](#filter-data).

{litdata-0.2.68 → litdata-0.2.69}/README.md

@@ -925,6 +925,12 @@ for batch_idx, batch in enumerate(dataloader):
         torch.save(dataloader.state_dict(), "dataloader_state.pt")
 ```
 
+Same `seed` and `shuffle` are required. For a **`StreamingDataset`**, **`num_workers` and `world_size` may change**: LitData drops a global `sample_in_epoch` prefix and restripes the rest (never duplicates remaining IDs). For a matching loss curve keep **global batch size** (`world_size * batch_size`) constant and DDP ranks in lockstep. `num_canonical_nodes` (default: first-run `world_size`) is frozen in the checkpoint. POSIX `WindowShuffle` resumes whole remaining chunks. **`CombinedStreamingDataset` and `ParallelStreamingDataset` resume only with the same `world_size`, `num_workers`, and `batch_size`.**
+
+```python
+dataset = StreamingDataset("s3://my-bucket/my-data", shuffle=True, num_canonical_nodes=8)
+```
+
 </details>
 
 
@@ -932,11 +938,9 @@ for batch_idx, batch in enumerate(dataloader):
   <summary> ✅ Use shared queue for Optimizing <a id="shared-queue" href="#shared-queue">🔗</a> </summary>
 &nbsp;
 
-If you are using multiple workers to optimize your dataset, you can use a shared queue to speed up the process.
+`optimize` / `map` default to a **shared per-node queue** (`keep_data_ordered=False`). Work is packed per node, then every worker on that node pulls the next item, so a slow worker does not leave others idle. Set `keep_data_ordered=True` to keep a static per-worker slice (required for `use_checkpoint` and `align_chunking`).
 
-This is especially useful when optimizing large datasets in parallel, where some workers may be slower than others.
-
-It can also improve fault tolerance when workers fail due to out-of-memory (OOM) errors.
+Local `output_dir` writes chunks in place. Remote inputs and outputs use the streaming downloader (`adownload_file` / `aupload_file`, obstore when available).
 
 ```python
 import numpy as np
@@ -959,28 +963,36 @@ if __name__ == "__main__":
         output_dir="fast_data",             # optimized data is stored here
         num_workers=4,                      # The number of workers on the same machine
         chunk_bytes="64MB" ,                 # size of each chunk
-        keep_data_ordered=False,             # Use a shared queue to speed up the process
+        keep_data_ordered=False,             # default: shared queue (set True to keep input order)
     )
 ```
 
 
-### Performance Difference between using a shared queue and not using it:
+### Shared queue vs ordered (skewed local files)
 
-**Note**: The following benchmarks were collected using the ImageNet dataset on an A10G machine with 16 workers.
+`scripts/bench/bench_node_queue.py --files 4000 --workers 8` (first 500 files are 1 MiB). On `main`, unordered optimize sat on a 200s empty-queue timeout after work finished.
 
-| Configuration    | Optimize Time (sec) | Stream 1 (img/sec) | Stream 2 (img/sec) |
-|------------------|---------------------|---------------------|---------------------|
-| shared_queue (`keep_data_ordered=False`)     | 1281                | 5392                | 5732                |
-| no shared_queue (`keep_data_ordered=True (default)`)  | 1187                | 5257                | 5746                |
+| Tree | Mode | Time | Throughput |
+|------|------|-----:|-----------:|
+| `main` (old default) | `keep_data_ordered=True` | 23.7s | 169 files/s |
+| `main` | `keep_data_ordered=False` | 223.6s | 18 files/s |
+| this tree | `keep_data_ordered=True` | 22.9s | 175 files/s |
+| this tree (**new default**) | `keep_data_ordered=False` | **18.8s** | 213 files/s |
 
-📌 Note: The **shared_queue** option impacts optimization time, not streaming speed.
-> While the streaming numbers may appear slightly different, this variation is incidental and not caused by shared_queue.
->
-> Streaming happens after optimization and does not involve inter-process communication where shared_queue plays a role.
+Shared-queue **before → after: ~12×**. New default vs old ordered default: **1.22×**.
+
+### Local / remote input × output
+
+`python scripts/bench/bench_node_queue.py --files 200 --workers 4 --io-matrix` (first 50 files are 1 MiB).
 
-- 📄 Using a shared queue helps balance the load across workers, though it may slightly increase optimization time due to the overhead of pickling items sent between processes.
+| Topology | Ordered | Shared | Speedup |
+|----------|--------:|-------:|--------:|
+| local → local | 6.55s | **2.96s** | 2.21× |
+| remote → local | 7.44s | **3.48s** | 2.14× |
+| local → remote | 10.98s | **6.51s** | 1.69× |
+| remote → remote | 11.48s | **8.55s** | 1.34× |
 
-- ⚡ However, it can significantly improve optimizing performance — especially when some workers are slower than others.
+Shared queue balances uneven workers. It does not change later `StreamingDataset` throughput.
 
 </details>
 
@@ -2280,7 +2292,7 @@ if __name__ == "__main__":
 | `start_method` | spawn† | Multiprocessing start method (†spawn unless IPython) |
 | `optimize_dns` | `None` | Optimized DNS (Studio / cloud) |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared work queue (better for uneven/slow workers) |
+| `keep_data_ordered` | `False` | Shared work queue (faster for uneven workers). `True` keeps a static per-worker slice. Forced `True` with `use_checkpoint` / `align_chunking`. |
 
 </details>
 
@@ -2316,7 +2328,7 @@ Full knob list for `litdata.optimize` (see Quick start for the minimal recipe).
 | `start_method` | spawn† | Multiprocessing start method |
 | `optimize_dns` | `None` | Optimized DNS |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared queue among workers |
+| `keep_data_ordered` | `False` | Shared queue among workers. `True` keeps input order. Forced `True` with `use_checkpoint` / `align_chunking`. |
 | `verbose` | `True` | Progress logging |
 
 Related features: [shared queue](#shared-queue), [queue input](#queue-input), [append/overwrite](#modify-datasets), [compression](#compression), [TokensLoader / LLM](#llm-training), [filter](#filter-data).

{litdata-0.2.68 → litdata-0.2.69}/src/litdata/__about__.py

@@ -14,7 +14,7 @@
 
 import time
 
-__version__ = "0.2.68"
+__version__ = "0.2.69"
 __author__ = "Lightning AI et al."
 __author_email__ = "pytorch@lightning.ai"
 __license__ = "Apache-2.0"