litdata 0.2.67__tar.gz → 0.2.69__tar.gz

{litdata-0.2.67 → litdata-0.2.69}/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices and integrations that the amazing PyTorch team and other research organization rolls out!
 
-If you are new to open source, check out [this blog to get started with your first Open Source contribution](https://devblog.pytorchlightning.ai/quick-contribution-guide-86d977171b3a).
+If you are new to open source, check out [GitHub's guide to making your first contribution](https://docs.github.com/en/get-started/quickstart/contributing-to-projects).
 
 ## Main Core Value: One less thing to remember
 

{litdata-0.2.67/src/litdata.egg-info → litdata-0.2.69}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: litdata
-Version: 0.2.67
+Version: 0.2.69
 Summary: The Deep Learning framework to train, deploy, and ship AI products Lightning fast.
 Home-page: https://github.com/Lightning-AI/litdata
 Download-URL: https://github.com/Lightning-AI/litdata
@@ -279,6 +279,22 @@ for sample in dataloader:
     img, cls = sample["image"], sample["class"]
 ```
 
+**Keyed lookup and in-place patches** (needs `polars` and `optimize(..., key_fn=...)` or `build_keys_index`):
+
+```python
+ld.optimize(fn=fn, inputs=inputs, output_dir="fast_data", chunk_bytes="64MB", key_fn=lambda s: s["id"])
+
+ds = ld.StreamingDataset("fast_data")
+sample = ds["entity-id"]          # str keys
+sample = ds.get_by_key(42)        # int entity keys; ds[42] is still positional
+
+with ld.dataset_update("fast_data") as update:  # local directory only
+    update["entity-id"] = {"id": "entity-id", "x": 1}
+    update.commit()
+```
+
+`mode="append"` continues chunk numbering. `use_checkpoint=True` tries to resume an interrupted optimize. They are not the same.
+
 **Key benefits:**
 
 ✅ **Accelerate training:**       Optimized datasets load 20x faster.      
@@ -785,6 +801,10 @@ Shuffling is **deterministic** and designed for distributed training:
 
 The permutation depends on `seed`, the epoch, and chunk metadata — the same settings always yield the same order (required for resumable `state_dict`).
 
+**Object storage (`s3://`, `gs://`, …)** globally permutes chunks (`FullShuffle`). Random chunk order is cheap once files are already copied into the local cache.
+
+**POSIX-fast** (automatic for any local path) mmaps chunks in place. **Vast / NFS / Lustre / GPFS** (and `LITDATA_POSIX_FAST=1`) use `WindowShuffle`: each worker gets **whole chunks** in a sequential stripe, then shuffles only inside a sliding window (default **16**, `LITDATA_POSIX_SHUFFLE_WINDOW`) for both chunk order and in-chunk items. Local disks (ext4/xfs) keep global `FullShuffle`. Object URLs stay on `FullShuffle`. `LITDATA_POSIX_FAST=0` disables in-place mmap.
+
 ```python
 from litdata import StreamingDataset, StreamingDataLoader
 
@@ -853,7 +873,7 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 | `seed` | `42` | Shuffle / subsample RNG |
 | `serializers` | built-ins | Custom serialize/deserialize map |
 | `max_cache_size` | `"100GB"` | Evict consumed chunks beyond this size |
-| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk) |
+| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk / RAM) |
 | `subsample` | `1.0` | Fraction of data (`0.01`) or upsample (`2.5`) |
 | `encryption` | `None` | `FernetEncryption` / `RSAEncryption` / custom |
 | `storage_options` | `{}` | Cloud client options |
@@ -864,6 +884,8 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 
 Peak disk ≈ `num_workers × max_pre_download × mean_chunk_size`.
 
+On **Vast / NFS / local disk**, POSIX-fast is on by default (`LITDATA_POSIX_FAST=0` to disable). `WILLNEED` prefetch and `num_workers` are capped when they would exceed about half of `MemAvailable`. Idle **hugepages** (common on GPU nodes) do not count as available RAM — drop unused `nr_hugepages` if `MemAvailable` looks tiny next to `MemTotal`.
+
 **`StreamingDataLoader`**
 
 | Argument | Description |
@@ -967,6 +989,12 @@ for batch_idx, batch in enumerate(dataloader):
         torch.save(dataloader.state_dict(), "dataloader_state.pt")
 ```
 
+Same `seed` and `shuffle` are required. For a **`StreamingDataset`**, **`num_workers` and `world_size` may change**: LitData drops a global `sample_in_epoch` prefix and restripes the rest (never duplicates remaining IDs). For a matching loss curve keep **global batch size** (`world_size * batch_size`) constant and DDP ranks in lockstep. `num_canonical_nodes` (default: first-run `world_size`) is frozen in the checkpoint. POSIX `WindowShuffle` resumes whole remaining chunks. **`CombinedStreamingDataset` and `ParallelStreamingDataset` resume only with the same `world_size`, `num_workers`, and `batch_size`.**
+
+```python
+dataset = StreamingDataset("s3://my-bucket/my-data", shuffle=True, num_canonical_nodes=8)
+```
+
 </details>
 
 
@@ -974,11 +1002,9 @@ for batch_idx, batch in enumerate(dataloader):
   <summary> ✅ Use shared queue for Optimizing <a id="shared-queue" href="#shared-queue">🔗</a> </summary>
 &nbsp;
 
-If you are using multiple workers to optimize your dataset, you can use a shared queue to speed up the process.
+`optimize` / `map` default to a **shared per-node queue** (`keep_data_ordered=False`). Work is packed per node, then every worker on that node pulls the next item, so a slow worker does not leave others idle. Set `keep_data_ordered=True` to keep a static per-worker slice (required for `use_checkpoint` and `align_chunking`).
 
-This is especially useful when optimizing large datasets in parallel, where some workers may be slower than others.
-
-It can also improve fault tolerance when workers fail due to out-of-memory (OOM) errors.
+Local `output_dir` writes chunks in place. Remote inputs and outputs use the streaming downloader (`adownload_file` / `aupload_file`, obstore when available).
 
 ```python
 import numpy as np
@@ -1001,28 +1027,36 @@ if __name__ == "__main__":
         output_dir="fast_data",             # optimized data is stored here
         num_workers=4,                      # The number of workers on the same machine
         chunk_bytes="64MB" ,                 # size of each chunk
-        keep_data_ordered=False,             # Use a shared queue to speed up the process
+        keep_data_ordered=False,             # default: shared queue (set True to keep input order)
     )
 ```
 
 
-### Performance Difference between using a shared queue and not using it:
+### Shared queue vs ordered (skewed local files)
 
-**Note**: The following benchmarks were collected using the ImageNet dataset on an A10G machine with 16 workers.
+`scripts/bench/bench_node_queue.py --files 4000 --workers 8` (first 500 files are 1 MiB). On `main`, unordered optimize sat on a 200s empty-queue timeout after work finished.
 
-| Configuration    | Optimize Time (sec) | Stream 1 (img/sec) | Stream 2 (img/sec) |
-|------------------|---------------------|---------------------|---------------------|
-| shared_queue (`keep_data_ordered=False`)     | 1281                | 5392                | 5732                |
-| no shared_queue (`keep_data_ordered=True (default)`)  | 1187                | 5257                | 5746                |
+| Tree | Mode | Time | Throughput |
+|------|------|-----:|-----------:|
+| `main` (old default) | `keep_data_ordered=True` | 23.7s | 169 files/s |
+| `main` | `keep_data_ordered=False` | 223.6s | 18 files/s |
+| this tree | `keep_data_ordered=True` | 22.9s | 175 files/s |
+| this tree (**new default**) | `keep_data_ordered=False` | **18.8s** | 213 files/s |
 
-📌 Note: The **shared_queue** option impacts optimization time, not streaming speed.
-> While the streaming numbers may appear slightly different, this variation is incidental and not caused by shared_queue.
->
-> Streaming happens after optimization and does not involve inter-process communication where shared_queue plays a role.
+Shared-queue **before → after: ~12×**. New default vs old ordered default: **1.22×**.
 
-- 📄 Using a shared queue helps balance the load across workers, though it may slightly increase optimization time due to the overhead of pickling items sent between processes.
+### Local / remote input × output
 
-- ⚡ However, it can significantly improve optimizing performance — especially when some workers are slower than others.
+`python scripts/bench/bench_node_queue.py --files 200 --workers 4 --io-matrix` (first 50 files are 1 MiB).
+
+| Topology | Ordered | Shared | Speedup |
+|----------|--------:|-------:|--------:|
+| local → local | 6.55s | **2.96s** | 2.21× |
+| remote → local | 7.44s | **3.48s** | 2.14× |
+| local → remote | 10.98s | **6.51s** | 1.69× |
+| remote → remote | 11.48s | **8.55s** | 1.34× |
+
+Shared queue balances uneven workers. It does not change later `StreamingDataset` throughput.
 
 </details>
 
@@ -1806,7 +1840,7 @@ Only **worker 0** is instrumented. When an `int` is used, the tracer wraps `fetc
 
 - Delete or change `profile_dir` between runs — LitData removes an existing `result.json` before starting.
 - Pair with a wiped chunk cache if you care about **cold** epoch behavior (`litdata cache clear`).
-- For deeper LitData internals (download / lock / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/deependujha/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
+- For deeper LitData internals (download / read / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/Lightning-AI/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
 
 </details>
 
@@ -1919,6 +1953,9 @@ export LITDATA_ASYNC_MIN_PRE_DOWNLOAD=0
 | `LITDATA_DISABLE_VERSION_CHECK` | `0` | `1` skips the upgrade tip |
 | `HF_TOKEN` | — | Gated Hugging Face datasets |
 | `DEBUG_LITDATA` / `PRINT_DEBUG_LOGS` | `0` | Internal debug / stdout logs |
+| `LITDATA_LOG_FILE` | `litdata_debug.log` | `enable_tracer()` output path |
+| `LITDATA_TRACE_LEVEL` | unset | `batch` / `chunk` / `sample` / `debug` / `off` (see [Debug & Profile](#debug-profile)) |
+| `LITDATA_TRACE_CATEGORIES` | from level | Comma-separated cats, e.g. `download,read,delete` |
 
 Multi-node `optimize`/`map` on Studios also uses `DATA_OPTIMIZER_*` (set by the platform). Full catalog (debug logs, Studio injects, torchrun): see the LitData skill `reference/env-vars.md` when using agent skills, or the source modules `constants.py` / `async_prefetch.py`.
 
@@ -2084,66 +2121,68 @@ ds = StreamingDataset(input_dir=data_dir, encryption=rsa)
 
 &nbsp;
 
-LitData comes with built-in logging and profiling capabilities to help you debug and profile your data streaming workloads.
+`enable_tracer()` records the streaming pipeline (download vs read vs delete vs batch) as one-line events. [Litracer](https://github.com/Lightning-AI/litracer) converts that log into a Chrome / [Perfetto](https://ui.perfetto.dev) trace.
 
-<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
+This is complementary to [`profile_batches`](#profile-loading) (viztracer = DataLoader worker **CPU**; Litracer = LitData **pipeline** events).
 
-- e.g., with LitData Streaming
+<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
 
 ```python
 import litdata as ld
 from litdata.debugger import enable_tracer
 
-# WARNING: Remove existing trace `litdata_debug.log` file if it exists before re-tracing
-enable_tracer()
+# Call once per process, before the DataLoader. Delete an existing log before re-tracing (append).
+enable_tracer(level="chunk", log_file="litdata_debug.log")
+# level="batch" | "chunk" (default) | "sample" | "debug" | "off"
+# enable_tracer(categories=["download", "read", "delete"])
 
 if __name__ == "__main__":
     dataset = ld.StreamingDataset("s3://my-bucket/my-data", shuffle=True)
-    dataloader = ld.StreamingDataLoader(dataset, batch_size=64)
-
-    for batch in dataloader:
-        print(batch)  # Replace with your data processing logic
+    for batch in ld.StreamingDataLoader(dataset, batch_size=64, num_workers=8):
+        ...
 ```
 
-1. Generate Debug Log:
+| Level | Events |
+| ----- | ------ |
+| `batch` | Epoch + per-batch spans, plus crashes |
+| `chunk` (default) | + `download`, `read`, `delete`, `decompress`, `prefetch` |
+| `sample` | + per-item `__getitem__` (high volume) |
+| `debug` | + `.cnt` lock refcount spans |
+| `off` | Disable |
 
-    - Run your Python program and it'll create a log file containing detailed debug information.
+Event **names** are stable (`download`, `read`, `delete`, `batch`, `sample`, `crash`). Chunk / sample indexes live in args so Perfetto groups all downloads together. Each line is `key: value;` pairs with Chrome **microsecond** timestamps. Crashes are a one-line instant (`ph: I`, `name: crash`); the Python traceback is printed to **stderr**, not the log file (a multi-line `logger.exception` would break Litracer).
 
-    ```bash
-      python main.py
-    ```
+Env overrides: `LITDATA_LOG_FILE`, `LITDATA_TRACE_LEVEL`, `LITDATA_TRACE_CATEGORIES` (comma-separated). Tracer calls are no-ops when tracing is off.
 
-2. Install [Litracer](https://github.com/deependujha/litracer/):
+1. Generate the log:
 
-    - Option 1: Using Go (recommended)
-        - Install Go on your system.
-        - Run the following command to install Litracer:
+    ```bash
+    python train.py   # writes litdata_debug.log
+    ```
 
-        ```bash
-          go install github.com/deependujha/litracer@latest
-        ```
+2. Install [Litracer](https://github.com/Lightning-AI/litracer) (Go 1.23+):
 
-    - Option 2: Download Binary
-        - Visit the [LitRacer GitHub Releases](https://github.com/deependujha/litracer/releases) page.
-        - Download the appropriate binary for your operating system and follow the installation instructions.
+    ```bash
+    git clone https://github.com/Lightning-AI/litracer.git
+    cd litracer && go build -o litracer .
+    ```
 
-3. Convert Debug Log to trace JSON:
+    Or `go install github.com/deependujha/litracer@latest` (published Go module path). Until `go.mod` is renamed, `go install github.com/Lightning-AI/litracer@latest` does not work. Release binaries: [GitHub Releases](https://github.com/Lightning-AI/litracer/releases).
 
-    - Use litracer to convert the generated log file into a trace JSON file. This command uses 100 workers for conversion:
+3. Convert and open in Perfetto:
 
     ```bash
-      litracer litdata_debug.log -o litdata_trace.json -w 100
+    litracer --quiet --validate -o litdata_trace.json.gz litdata_debug.log
+    litracer --quiet --cat download,read,delete -o io.json.gz litdata_debug.log
+    # open the .json.gz at https://ui.perfetto.dev (preferred) or chrome://tracing
     ```
 
-4. Visualize the trace:
-
-    - Use either `chrome://tracing` in the Chrome browser or `ui.perfetto.dev` to view the `litdata_trace.json` file for in-depth performance insights. You can also use `SQL queries` to analyze the logs.
-    - `Perfetto` is recommended over `chrome://tracing` for visualization & analyzing.
+    `--quiet` prints a one-line summary (per-category durations, unmatched B/E, crashes). `--cat` keeps only those categories. Matched B/E pairs become complete (`ph: X`) spans unless `--no-complete`. Default output is gzip Chrome JSON (`.json.gz`) — both Perfetto and `chrome://tracing` open it; pass `-o file.json` for uncompressed.
 
-- Key Points:
+- For trace files `> 2GB`, see [Perfetto large traces](https://perfetto.dev/docs/visualization/large-traces).
+- If you connect Perfetto to the RPC server, prefer Chrome over Brave (Brave often does not autodetect the RPC server).
 
-    - For very large trace.json files (`> 2GB`), refer to the [Perfetto documentation](https://perfetto.dev/docs/visualization/large-traces) for using native accelerators.
-    - If you are trying to connect Perfetto to the RPC server, it is recommended to use Chrome over Brave, as it has been observed that Perfetto in Brave does not autodetect the RPC server.
+**Multi-worker `s3://` `FileNotFoundError` after ~120s:** `num_workers=0` working while `num_workers>0` fails usually means the DataLoader parent started obstore (tokio) before fork and worker GETs hung. Current LitData fetches `index.json` with boto3 so workers can lazy-init obstore; they fall back to boto3 if the parent already started the runtime. On Studio R2 / `lightning_storage`, the same symptom can be a prefetch-thread crash (`data_connection_id` / `endpoint_url` into `boto3.Session`) — look for `[litdata] PrepareChunksThread CRASHED` on stderr and a `crash` instant in the trace.
 
 </details>
 
@@ -2317,7 +2356,7 @@ if __name__ == "__main__":
 | `start_method` | spawn† | Multiprocessing start method (†spawn unless IPython) |
 | `optimize_dns` | `None` | Optimized DNS (Studio / cloud) |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared work queue (better for uneven/slow workers) |
+| `keep_data_ordered` | `False` | Shared work queue (faster for uneven workers). `True` keeps a static per-worker slice. Forced `True` with `use_checkpoint` / `align_chunking`. |
 
 </details>
 
@@ -2353,7 +2392,7 @@ Full knob list for `litdata.optimize` (see Quick start for the minimal recipe).
 | `start_method` | spawn† | Multiprocessing start method |
 | `optimize_dns` | `None` | Optimized DNS |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared queue among workers |
+| `keep_data_ordered` | `False` | Shared queue among workers. `True` keeps input order. Forced `True` with `use_checkpoint` / `align_chunking`. |
 | `verbose` | `True` | Progress logging |
 
 Related features: [shared queue](#shared-queue), [queue input](#queue-input), [append/overwrite](#modify-datasets), [compression](#compression), [TokensLoader / LLM](#llm-training), [filter](#filter-data).
@@ -2428,6 +2467,17 @@ Speed to stream Imagenet 1.2M from local disk with ffcv vs LitData:
 | ffcv(os_cache=True) | JPEG 90% | 20 GB | 7653 | 8051 |
 | ffcv(os_cache=False) | JPEG 90% | 20 GB | 8149 | 8607 |
 
+Speed to stream a **synthetic ImageNet-scale set from Vast NFS** (NFSv3 `nconnect=32`, 208-CPU host, ~1 TiB RAM). Dataset: **1.08M** JPEG q95 256×256 (~160 GiB, 64 MiB chunks). `StreamingDataLoader`, batch **256**, `shuffle=True`, `drop_last=True`, decode only unless noted. POSIX-fast mmaps chunks **in place** (no copy into `~/.lightning/chunks`).
+
+| Setup | Workers | Images / sec |
+|---|---|---|
+| Copy into local cache (`LITDATA_POSIX_FAST=0`) | 48 | **16.7k** (2-epoch avg) |
+| POSIX-fast (this default on local/Vast paths) | 48 | **18.2k** |
+| POSIX-fast + README ImageNet augs (crop 224, flip, float32) | 48 | **12.9k** |
+| POSIX-fast, all CPU cores | **208** | **35.8k** |
+
+Notes: 208 workers need enough **MemAvailable**. This host had **928×1 GiB hugepages** reserved and idle (~900 GiB locked); after `nr_hugepages=0`, 208 workers stayed healthy. If `num_workers=os.cpu_count()` would crowd RAM, LitData **clamps** workers (`LITDATA_POSIX_MAX_WORKERS=0` disables) and skips `WILLNEED` prefetch. Real ImageNet JPEG 90% is much smaller (~12 GiB) and usually decodes faster than this q95 noise set.
+
 ### Raw Dataset
 
 Speed to stream raw Imagenet 1.2M from different cloud storage providers:

{litdata-0.2.67 → litdata-0.2.69}/README.md

@@ -215,6 +215,22 @@ for sample in dataloader:
     img, cls = sample["image"], sample["class"]
 ```
 
+**Keyed lookup and in-place patches** (needs `polars` and `optimize(..., key_fn=...)` or `build_keys_index`):
+
+```python
+ld.optimize(fn=fn, inputs=inputs, output_dir="fast_data", chunk_bytes="64MB", key_fn=lambda s: s["id"])
+
+ds = ld.StreamingDataset("fast_data")
+sample = ds["entity-id"]          # str keys
+sample = ds.get_by_key(42)        # int entity keys; ds[42] is still positional
+
+with ld.dataset_update("fast_data") as update:  # local directory only
+    update["entity-id"] = {"id": "entity-id", "x": 1}
+    update.commit()
+```
+
+`mode="append"` continues chunk numbering. `use_checkpoint=True` tries to resume an interrupted optimize. They are not the same.
+
 **Key benefits:**
 
 ✅ **Accelerate training:**       Optimized datasets load 20x faster.      
@@ -721,6 +737,10 @@ Shuffling is **deterministic** and designed for distributed training:
 
 The permutation depends on `seed`, the epoch, and chunk metadata — the same settings always yield the same order (required for resumable `state_dict`).
 
+**Object storage (`s3://`, `gs://`, …)** globally permutes chunks (`FullShuffle`). Random chunk order is cheap once files are already copied into the local cache.
+
+**POSIX-fast** (automatic for any local path) mmaps chunks in place. **Vast / NFS / Lustre / GPFS** (and `LITDATA_POSIX_FAST=1`) use `WindowShuffle`: each worker gets **whole chunks** in a sequential stripe, then shuffles only inside a sliding window (default **16**, `LITDATA_POSIX_SHUFFLE_WINDOW`) for both chunk order and in-chunk items. Local disks (ext4/xfs) keep global `FullShuffle`. Object URLs stay on `FullShuffle`. `LITDATA_POSIX_FAST=0` disables in-place mmap.
+
 ```python
 from litdata import StreamingDataset, StreamingDataLoader
 
@@ -789,7 +809,7 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 | `seed` | `42` | Shuffle / subsample RNG |
 | `serializers` | built-ins | Custom serialize/deserialize map |
 | `max_cache_size` | `"100GB"` | Evict consumed chunks beyond this size |
-| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk) |
+| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk / RAM) |
 | `subsample` | `1.0` | Fraction of data (`0.01`) or upsample (`2.5`) |
 | `encryption` | `None` | `FernetEncryption` / `RSAEncryption` / custom |
 | `storage_options` | `{}` | Cloud client options |
@@ -800,6 +820,8 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 
 Peak disk ≈ `num_workers × max_pre_download × mean_chunk_size`.
 
+On **Vast / NFS / local disk**, POSIX-fast is on by default (`LITDATA_POSIX_FAST=0` to disable). `WILLNEED` prefetch and `num_workers` are capped when they would exceed about half of `MemAvailable`. Idle **hugepages** (common on GPU nodes) do not count as available RAM — drop unused `nr_hugepages` if `MemAvailable` looks tiny next to `MemTotal`.
+
 **`StreamingDataLoader`**
 
 | Argument | Description |
@@ -903,6 +925,12 @@ for batch_idx, batch in enumerate(dataloader):
         torch.save(dataloader.state_dict(), "dataloader_state.pt")
 ```
 
+Same `seed` and `shuffle` are required. For a **`StreamingDataset`**, **`num_workers` and `world_size` may change**: LitData drops a global `sample_in_epoch` prefix and restripes the rest (never duplicates remaining IDs). For a matching loss curve keep **global batch size** (`world_size * batch_size`) constant and DDP ranks in lockstep. `num_canonical_nodes` (default: first-run `world_size`) is frozen in the checkpoint. POSIX `WindowShuffle` resumes whole remaining chunks. **`CombinedStreamingDataset` and `ParallelStreamingDataset` resume only with the same `world_size`, `num_workers`, and `batch_size`.**
+
+```python
+dataset = StreamingDataset("s3://my-bucket/my-data", shuffle=True, num_canonical_nodes=8)
+```
+
 </details>
 
 
@@ -910,11 +938,9 @@ for batch_idx, batch in enumerate(dataloader):
   <summary> ✅ Use shared queue for Optimizing <a id="shared-queue" href="#shared-queue">🔗</a> </summary>
 &nbsp;
 
-If you are using multiple workers to optimize your dataset, you can use a shared queue to speed up the process.
+`optimize` / `map` default to a **shared per-node queue** (`keep_data_ordered=False`). Work is packed per node, then every worker on that node pulls the next item, so a slow worker does not leave others idle. Set `keep_data_ordered=True` to keep a static per-worker slice (required for `use_checkpoint` and `align_chunking`).
 
-This is especially useful when optimizing large datasets in parallel, where some workers may be slower than others.
-
-It can also improve fault tolerance when workers fail due to out-of-memory (OOM) errors.
+Local `output_dir` writes chunks in place. Remote inputs and outputs use the streaming downloader (`adownload_file` / `aupload_file`, obstore when available).
 
 ```python
 import numpy as np
@@ -937,28 +963,36 @@ if __name__ == "__main__":
         output_dir="fast_data",             # optimized data is stored here
         num_workers=4,                      # The number of workers on the same machine
         chunk_bytes="64MB" ,                 # size of each chunk
-        keep_data_ordered=False,             # Use a shared queue to speed up the process
+        keep_data_ordered=False,             # default: shared queue (set True to keep input order)
     )
 ```
 
 
-### Performance Difference between using a shared queue and not using it:
+### Shared queue vs ordered (skewed local files)
 
-**Note**: The following benchmarks were collected using the ImageNet dataset on an A10G machine with 16 workers.
+`scripts/bench/bench_node_queue.py --files 4000 --workers 8` (first 500 files are 1 MiB). On `main`, unordered optimize sat on a 200s empty-queue timeout after work finished.
 
-| Configuration    | Optimize Time (sec) | Stream 1 (img/sec) | Stream 2 (img/sec) |
-|------------------|---------------------|---------------------|---------------------|
-| shared_queue (`keep_data_ordered=False`)     | 1281                | 5392                | 5732                |
-| no shared_queue (`keep_data_ordered=True (default)`)  | 1187                | 5257                | 5746                |
+| Tree | Mode | Time | Throughput |
+|------|------|-----:|-----------:|
+| `main` (old default) | `keep_data_ordered=True` | 23.7s | 169 files/s |
+| `main` | `keep_data_ordered=False` | 223.6s | 18 files/s |
+| this tree | `keep_data_ordered=True` | 22.9s | 175 files/s |
+| this tree (**new default**) | `keep_data_ordered=False` | **18.8s** | 213 files/s |
 
-📌 Note: The **shared_queue** option impacts optimization time, not streaming speed.
-> While the streaming numbers may appear slightly different, this variation is incidental and not caused by shared_queue.
->
-> Streaming happens after optimization and does not involve inter-process communication where shared_queue plays a role.
+Shared-queue **before → after: ~12×**. New default vs old ordered default: **1.22×**.
 
-- 📄 Using a shared queue helps balance the load across workers, though it may slightly increase optimization time due to the overhead of pickling items sent between processes.
+### Local / remote input × output
 
-- ⚡ However, it can significantly improve optimizing performance — especially when some workers are slower than others.
+`python scripts/bench/bench_node_queue.py --files 200 --workers 4 --io-matrix` (first 50 files are 1 MiB).
+
+| Topology | Ordered | Shared | Speedup |
+|----------|--------:|-------:|--------:|
+| local → local | 6.55s | **2.96s** | 2.21× |
+| remote → local | 7.44s | **3.48s** | 2.14× |
+| local → remote | 10.98s | **6.51s** | 1.69× |
+| remote → remote | 11.48s | **8.55s** | 1.34× |
+
+Shared queue balances uneven workers. It does not change later `StreamingDataset` throughput.
 
 </details>
 
@@ -1742,7 +1776,7 @@ Only **worker 0** is instrumented. When an `int` is used, the tracer wraps `fetc
 
 - Delete or change `profile_dir` between runs — LitData removes an existing `result.json` before starting.
 - Pair with a wiped chunk cache if you care about **cold** epoch behavior (`litdata cache clear`).
-- For deeper LitData internals (download / lock / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/deependujha/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
+- For deeper LitData internals (download / read / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/Lightning-AI/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
 
 </details>
 
@@ -1855,6 +1889,9 @@ export LITDATA_ASYNC_MIN_PRE_DOWNLOAD=0
 | `LITDATA_DISABLE_VERSION_CHECK` | `0` | `1` skips the upgrade tip |
 | `HF_TOKEN` | — | Gated Hugging Face datasets |
 | `DEBUG_LITDATA` / `PRINT_DEBUG_LOGS` | `0` | Internal debug / stdout logs |
+| `LITDATA_LOG_FILE` | `litdata_debug.log` | `enable_tracer()` output path |
+| `LITDATA_TRACE_LEVEL` | unset | `batch` / `chunk` / `sample` / `debug` / `off` (see [Debug & Profile](#debug-profile)) |
+| `LITDATA_TRACE_CATEGORIES` | from level | Comma-separated cats, e.g. `download,read,delete` |
 
 Multi-node `optimize`/`map` on Studios also uses `DATA_OPTIMIZER_*` (set by the platform). Full catalog (debug logs, Studio injects, torchrun): see the LitData skill `reference/env-vars.md` when using agent skills, or the source modules `constants.py` / `async_prefetch.py`.
 
@@ -2020,66 +2057,68 @@ ds = StreamingDataset(input_dir=data_dir, encryption=rsa)
 
 &nbsp;
 
-LitData comes with built-in logging and profiling capabilities to help you debug and profile your data streaming workloads.
+`enable_tracer()` records the streaming pipeline (download vs read vs delete vs batch) as one-line events. [Litracer](https://github.com/Lightning-AI/litracer) converts that log into a Chrome / [Perfetto](https://ui.perfetto.dev) trace.
 
-<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
+This is complementary to [`profile_batches`](#profile-loading) (viztracer = DataLoader worker **CPU**; Litracer = LitData **pipeline** events).
 
-- e.g., with LitData Streaming
+<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
 
 ```python
 import litdata as ld
 from litdata.debugger import enable_tracer
 
-# WARNING: Remove existing trace `litdata_debug.log` file if it exists before re-tracing
-enable_tracer()
+# Call once per process, before the DataLoader. Delete an existing log before re-tracing (append).
+enable_tracer(level="chunk", log_file="litdata_debug.log")
+# level="batch" | "chunk" (default) | "sample" | "debug" | "off"
+# enable_tracer(categories=["download", "read", "delete"])
 
 if __name__ == "__main__":
     dataset = ld.StreamingDataset("s3://my-bucket/my-data", shuffle=True)
-    dataloader = ld.StreamingDataLoader(dataset, batch_size=64)
-
-    for batch in dataloader:
-        print(batch)  # Replace with your data processing logic
+    for batch in ld.StreamingDataLoader(dataset, batch_size=64, num_workers=8):
+        ...
 ```
 
-1. Generate Debug Log:
+| Level | Events |
+| ----- | ------ |
+| `batch` | Epoch + per-batch spans, plus crashes |
+| `chunk` (default) | + `download`, `read`, `delete`, `decompress`, `prefetch` |
+| `sample` | + per-item `__getitem__` (high volume) |
+| `debug` | + `.cnt` lock refcount spans |
+| `off` | Disable |
 
-    - Run your Python program and it'll create a log file containing detailed debug information.
+Event **names** are stable (`download`, `read`, `delete`, `batch`, `sample`, `crash`). Chunk / sample indexes live in args so Perfetto groups all downloads together. Each line is `key: value;` pairs with Chrome **microsecond** timestamps. Crashes are a one-line instant (`ph: I`, `name: crash`); the Python traceback is printed to **stderr**, not the log file (a multi-line `logger.exception` would break Litracer).
 
-    ```bash
-      python main.py
-    ```
+Env overrides: `LITDATA_LOG_FILE`, `LITDATA_TRACE_LEVEL`, `LITDATA_TRACE_CATEGORIES` (comma-separated). Tracer calls are no-ops when tracing is off.
 
-2. Install [Litracer](https://github.com/deependujha/litracer/):
+1. Generate the log:
 
-    - Option 1: Using Go (recommended)
-        - Install Go on your system.
-        - Run the following command to install Litracer:
+    ```bash
+    python train.py   # writes litdata_debug.log
+    ```
 
-        ```bash
-          go install github.com/deependujha/litracer@latest
-        ```
+2. Install [Litracer](https://github.com/Lightning-AI/litracer) (Go 1.23+):
 
-    - Option 2: Download Binary
-        - Visit the [LitRacer GitHub Releases](https://github.com/deependujha/litracer/releases) page.
-        - Download the appropriate binary for your operating system and follow the installation instructions.
+    ```bash
+    git clone https://github.com/Lightning-AI/litracer.git
+    cd litracer && go build -o litracer .
+    ```
 
-3. Convert Debug Log to trace JSON:
+    Or `go install github.com/deependujha/litracer@latest` (published Go module path). Until `go.mod` is renamed, `go install github.com/Lightning-AI/litracer@latest` does not work. Release binaries: [GitHub Releases](https://github.com/Lightning-AI/litracer/releases).
 
-    - Use litracer to convert the generated log file into a trace JSON file. This command uses 100 workers for conversion:
+3. Convert and open in Perfetto:
 
     ```bash
-      litracer litdata_debug.log -o litdata_trace.json -w 100
+    litracer --quiet --validate -o litdata_trace.json.gz litdata_debug.log
+    litracer --quiet --cat download,read,delete -o io.json.gz litdata_debug.log
+    # open the .json.gz at https://ui.perfetto.dev (preferred) or chrome://tracing
     ```
 
-4. Visualize the trace:
-
-    - Use either `chrome://tracing` in the Chrome browser or `ui.perfetto.dev` to view the `litdata_trace.json` file for in-depth performance insights. You can also use `SQL queries` to analyze the logs.
-    - `Perfetto` is recommended over `chrome://tracing` for visualization & analyzing.
+    `--quiet` prints a one-line summary (per-category durations, unmatched B/E, crashes). `--cat` keeps only those categories. Matched B/E pairs become complete (`ph: X`) spans unless `--no-complete`. Default output is gzip Chrome JSON (`.json.gz`) — both Perfetto and `chrome://tracing` open it; pass `-o file.json` for uncompressed.
 
-- Key Points:
+- For trace files `> 2GB`, see [Perfetto large traces](https://perfetto.dev/docs/visualization/large-traces).
+- If you connect Perfetto to the RPC server, prefer Chrome over Brave (Brave often does not autodetect the RPC server).
 
-    - For very large trace.json files (`> 2GB`), refer to the [Perfetto documentation](https://perfetto.dev/docs/visualization/large-traces) for using native accelerators.
-    - If you are trying to connect Perfetto to the RPC server, it is recommended to use Chrome over Brave, as it has been observed that Perfetto in Brave does not autodetect the RPC server.
+**Multi-worker `s3://` `FileNotFoundError` after ~120s:** `num_workers=0` working while `num_workers>0` fails usually means the DataLoader parent started obstore (tokio) before fork and worker GETs hung. Current LitData fetches `index.json` with boto3 so workers can lazy-init obstore; they fall back to boto3 if the parent already started the runtime. On Studio R2 / `lightning_storage`, the same symptom can be a prefetch-thread crash (`data_connection_id` / `endpoint_url` into `boto3.Session`) — look for `[litdata] PrepareChunksThread CRASHED` on stderr and a `crash` instant in the trace.
 
 </details>
 
@@ -2253,7 +2292,7 @@ if __name__ == "__main__":
 | `start_method` | spawn† | Multiprocessing start method (†spawn unless IPython) |
 | `optimize_dns` | `None` | Optimized DNS (Studio / cloud) |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared work queue (better for uneven/slow workers) |
+| `keep_data_ordered` | `False` | Shared work queue (faster for uneven workers). `True` keeps a static per-worker slice. Forced `True` with `use_checkpoint` / `align_chunking`. |
 
 </details>
 
@@ -2289,7 +2328,7 @@ Full knob list for `litdata.optimize` (see Quick start for the minimal recipe).
 | `start_method` | spawn† | Multiprocessing start method |
 | `optimize_dns` | `None` | Optimized DNS |
 | `storage_options` | `{}` | Cloud credentials / endpoints |
-| `keep_data_ordered` | `True` | `False` = shared queue among workers |
+| `keep_data_ordered` | `False` | Shared queue among workers. `True` keeps input order. Forced `True` with `use_checkpoint` / `align_chunking`. |
 | `verbose` | `True` | Progress logging |
 
 Related features: [shared queue](#shared-queue), [queue input](#queue-input), [append/overwrite](#modify-datasets), [compression](#compression), [TokensLoader / LLM](#llm-training), [filter](#filter-data).
@@ -2364,6 +2403,17 @@ Speed to stream Imagenet 1.2M from local disk with ffcv vs LitData:
 | ffcv(os_cache=True) | JPEG 90% | 20 GB | 7653 | 8051 |
 | ffcv(os_cache=False) | JPEG 90% | 20 GB | 8149 | 8607 |
 
+Speed to stream a **synthetic ImageNet-scale set from Vast NFS** (NFSv3 `nconnect=32`, 208-CPU host, ~1 TiB RAM). Dataset: **1.08M** JPEG q95 256×256 (~160 GiB, 64 MiB chunks). `StreamingDataLoader`, batch **256**, `shuffle=True`, `drop_last=True`, decode only unless noted. POSIX-fast mmaps chunks **in place** (no copy into `~/.lightning/chunks`).
+
+| Setup | Workers | Images / sec |
+|---|---|---|
+| Copy into local cache (`LITDATA_POSIX_FAST=0`) | 48 | **16.7k** (2-epoch avg) |
+| POSIX-fast (this default on local/Vast paths) | 48 | **18.2k** |
+| POSIX-fast + README ImageNet augs (crop 224, flip, float32) | 48 | **12.9k** |
+| POSIX-fast, all CPU cores | **208** | **35.8k** |
+
+Notes: 208 workers need enough **MemAvailable**. This host had **928×1 GiB hugepages** reserved and idle (~900 GiB locked); after `nr_hugepages=0`, 208 workers stayed healthy. If `num_workers=os.cpu_count()` would crowd RAM, LitData **clamps** workers (`LITDATA_POSIX_MAX_WORKERS=0` disables) and skips `WILLNEED` prefetch. Real ImageNet JPEG 90% is much smaller (~12 GiB) and usually decodes faster than this q95 noise set.
+
 ### Raw Dataset
 
 Speed to stream raw Imagenet 1.2M from different cloud storage providers:

{litdata-0.2.67 → litdata-0.2.69}/src/litdata/__about__.py

@@ -14,7 +14,7 @@
 
 import time
 
-__version__ = "0.2.67"
+__version__ = "0.2.69"
 __author__ = "Lightning AI et al."
 __author_email__ = "pytorch@lightning.ai"
 __license__ = "Apache-2.0"