litdata 0.2.67__tar.gz → 0.2.68__tar.gz

{litdata-0.2.67 → litdata-0.2.68}/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices and integrations that the amazing PyTorch team and other research organization rolls out!
 
-If you are new to open source, check out [this blog to get started with your first Open Source contribution](https://devblog.pytorchlightning.ai/quick-contribution-guide-86d977171b3a).
+If you are new to open source, check out [GitHub's guide to making your first contribution](https://docs.github.com/en/get-started/quickstart/contributing-to-projects).
 
 ## Main Core Value: One less thing to remember
 

{litdata-0.2.67/src/litdata.egg-info → litdata-0.2.68}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: litdata
-Version: 0.2.67
+Version: 0.2.68
 Summary: The Deep Learning framework to train, deploy, and ship AI products Lightning fast.
 Home-page: https://github.com/Lightning-AI/litdata
 Download-URL: https://github.com/Lightning-AI/litdata
@@ -279,6 +279,22 @@ for sample in dataloader:
     img, cls = sample["image"], sample["class"]
 ```
 
+**Keyed lookup and in-place patches** (needs `polars` and `optimize(..., key_fn=...)` or `build_keys_index`):
+
+```python
+ld.optimize(fn=fn, inputs=inputs, output_dir="fast_data", chunk_bytes="64MB", key_fn=lambda s: s["id"])
+
+ds = ld.StreamingDataset("fast_data")
+sample = ds["entity-id"]          # str keys
+sample = ds.get_by_key(42)        # int entity keys; ds[42] is still positional
+
+with ld.dataset_update("fast_data") as update:  # local directory only
+    update["entity-id"] = {"id": "entity-id", "x": 1}
+    update.commit()
+```
+
+`mode="append"` continues chunk numbering. `use_checkpoint=True` tries to resume an interrupted optimize. They are not the same.
+
 **Key benefits:**
 
 ✅ **Accelerate training:**       Optimized datasets load 20x faster.      
@@ -785,6 +801,10 @@ Shuffling is **deterministic** and designed for distributed training:
 
 The permutation depends on `seed`, the epoch, and chunk metadata — the same settings always yield the same order (required for resumable `state_dict`).
 
+**Object storage (`s3://`, `gs://`, …)** globally permutes chunks (`FullShuffle`). Random chunk order is cheap once files are already copied into the local cache.
+
+**POSIX-fast** (automatic for any local path) mmaps chunks in place. **Vast / NFS / Lustre / GPFS** (and `LITDATA_POSIX_FAST=1`) use `WindowShuffle`: each worker gets **whole chunks** in a sequential stripe, then shuffles only inside a sliding window (default **16**, `LITDATA_POSIX_SHUFFLE_WINDOW`) for both chunk order and in-chunk items. Local disks (ext4/xfs) keep global `FullShuffle`. Object URLs stay on `FullShuffle`. `LITDATA_POSIX_FAST=0` disables in-place mmap.
+
 ```python
 from litdata import StreamingDataset, StreamingDataLoader
 
@@ -853,7 +873,7 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 | `seed` | `42` | Shuffle / subsample RNG |
 | `serializers` | built-ins | Custom serialize/deserialize map |
 | `max_cache_size` | `"100GB"` | Evict consumed chunks beyond this size |
-| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk) |
+| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk / RAM) |
 | `subsample` | `1.0` | Fraction of data (`0.01`) or upsample (`2.5`) |
 | `encryption` | `None` | `FernetEncryption` / `RSAEncryption` / custom |
 | `storage_options` | `{}` | Cloud client options |
@@ -864,6 +884,8 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 
 Peak disk ≈ `num_workers × max_pre_download × mean_chunk_size`.
 
+On **Vast / NFS / local disk**, POSIX-fast is on by default (`LITDATA_POSIX_FAST=0` to disable). `WILLNEED` prefetch and `num_workers` are capped when they would exceed about half of `MemAvailable`. Idle **hugepages** (common on GPU nodes) do not count as available RAM — drop unused `nr_hugepages` if `MemAvailable` looks tiny next to `MemTotal`.
+
 **`StreamingDataLoader`**
 
 | Argument | Description |
@@ -1806,7 +1828,7 @@ Only **worker 0** is instrumented. When an `int` is used, the tracer wraps `fetc
 
 - Delete or change `profile_dir` between runs — LitData removes an existing `result.json` before starting.
 - Pair with a wiped chunk cache if you care about **cold** epoch behavior (`litdata cache clear`).
-- For deeper LitData internals (download / lock / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/deependujha/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
+- For deeper LitData internals (download / read / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/Lightning-AI/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
 
 </details>
 
@@ -1919,6 +1941,9 @@ export LITDATA_ASYNC_MIN_PRE_DOWNLOAD=0
 | `LITDATA_DISABLE_VERSION_CHECK` | `0` | `1` skips the upgrade tip |
 | `HF_TOKEN` | — | Gated Hugging Face datasets |
 | `DEBUG_LITDATA` / `PRINT_DEBUG_LOGS` | `0` | Internal debug / stdout logs |
+| `LITDATA_LOG_FILE` | `litdata_debug.log` | `enable_tracer()` output path |
+| `LITDATA_TRACE_LEVEL` | unset | `batch` / `chunk` / `sample` / `debug` / `off` (see [Debug & Profile](#debug-profile)) |
+| `LITDATA_TRACE_CATEGORIES` | from level | Comma-separated cats, e.g. `download,read,delete` |
 
 Multi-node `optimize`/`map` on Studios also uses `DATA_OPTIMIZER_*` (set by the platform). Full catalog (debug logs, Studio injects, torchrun): see the LitData skill `reference/env-vars.md` when using agent skills, or the source modules `constants.py` / `async_prefetch.py`.
 
@@ -2084,66 +2109,68 @@ ds = StreamingDataset(input_dir=data_dir, encryption=rsa)
 
 &nbsp;
 
-LitData comes with built-in logging and profiling capabilities to help you debug and profile your data streaming workloads.
+`enable_tracer()` records the streaming pipeline (download vs read vs delete vs batch) as one-line events. [Litracer](https://github.com/Lightning-AI/litracer) converts that log into a Chrome / [Perfetto](https://ui.perfetto.dev) trace.
 
-<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
+This is complementary to [`profile_batches`](#profile-loading) (viztracer = DataLoader worker **CPU**; Litracer = LitData **pipeline** events).
 
-- e.g., with LitData Streaming
+<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
 
 ```python
 import litdata as ld
 from litdata.debugger import enable_tracer
 
-# WARNING: Remove existing trace `litdata_debug.log` file if it exists before re-tracing
-enable_tracer()
+# Call once per process, before the DataLoader. Delete an existing log before re-tracing (append).
+enable_tracer(level="chunk", log_file="litdata_debug.log")
+# level="batch" | "chunk" (default) | "sample" | "debug" | "off"
+# enable_tracer(categories=["download", "read", "delete"])
 
 if __name__ == "__main__":
     dataset = ld.StreamingDataset("s3://my-bucket/my-data", shuffle=True)
-    dataloader = ld.StreamingDataLoader(dataset, batch_size=64)
-
-    for batch in dataloader:
-        print(batch)  # Replace with your data processing logic
+    for batch in ld.StreamingDataLoader(dataset, batch_size=64, num_workers=8):
+        ...
 ```
 
-1. Generate Debug Log:
+| Level | Events |
+| ----- | ------ |
+| `batch` | Epoch + per-batch spans, plus crashes |
+| `chunk` (default) | + `download`, `read`, `delete`, `decompress`, `prefetch` |
+| `sample` | + per-item `__getitem__` (high volume) |
+| `debug` | + `.cnt` lock refcount spans |
+| `off` | Disable |
 
-    - Run your Python program and it'll create a log file containing detailed debug information.
+Event **names** are stable (`download`, `read`, `delete`, `batch`, `sample`, `crash`). Chunk / sample indexes live in args so Perfetto groups all downloads together. Each line is `key: value;` pairs with Chrome **microsecond** timestamps. Crashes are a one-line instant (`ph: I`, `name: crash`); the Python traceback is printed to **stderr**, not the log file (a multi-line `logger.exception` would break Litracer).
 
-    ```bash
-      python main.py
-    ```
+Env overrides: `LITDATA_LOG_FILE`, `LITDATA_TRACE_LEVEL`, `LITDATA_TRACE_CATEGORIES` (comma-separated). Tracer calls are no-ops when tracing is off.
 
-2. Install [Litracer](https://github.com/deependujha/litracer/):
+1. Generate the log:
 
-    - Option 1: Using Go (recommended)
-        - Install Go on your system.
-        - Run the following command to install Litracer:
+    ```bash
+    python train.py   # writes litdata_debug.log
+    ```
 
-        ```bash
-          go install github.com/deependujha/litracer@latest
-        ```
+2. Install [Litracer](https://github.com/Lightning-AI/litracer) (Go 1.23+):
 
-    - Option 2: Download Binary
-        - Visit the [LitRacer GitHub Releases](https://github.com/deependujha/litracer/releases) page.
-        - Download the appropriate binary for your operating system and follow the installation instructions.
+    ```bash
+    git clone https://github.com/Lightning-AI/litracer.git
+    cd litracer && go build -o litracer .
+    ```
 
-3. Convert Debug Log to trace JSON:
+    Or `go install github.com/deependujha/litracer@latest` (published Go module path). Until `go.mod` is renamed, `go install github.com/Lightning-AI/litracer@latest` does not work. Release binaries: [GitHub Releases](https://github.com/Lightning-AI/litracer/releases).
 
-    - Use litracer to convert the generated log file into a trace JSON file. This command uses 100 workers for conversion:
+3. Convert and open in Perfetto:
 
     ```bash
-      litracer litdata_debug.log -o litdata_trace.json -w 100
+    litracer --quiet --validate -o litdata_trace.json.gz litdata_debug.log
+    litracer --quiet --cat download,read,delete -o io.json.gz litdata_debug.log
+    # open the .json.gz at https://ui.perfetto.dev (preferred) or chrome://tracing
     ```
 
-4. Visualize the trace:
-
-    - Use either `chrome://tracing` in the Chrome browser or `ui.perfetto.dev` to view the `litdata_trace.json` file for in-depth performance insights. You can also use `SQL queries` to analyze the logs.
-    - `Perfetto` is recommended over `chrome://tracing` for visualization & analyzing.
+    `--quiet` prints a one-line summary (per-category durations, unmatched B/E, crashes). `--cat` keeps only those categories. Matched B/E pairs become complete (`ph: X`) spans unless `--no-complete`. Default output is gzip Chrome JSON (`.json.gz`) — both Perfetto and `chrome://tracing` open it; pass `-o file.json` for uncompressed.
 
-- Key Points:
+- For trace files `> 2GB`, see [Perfetto large traces](https://perfetto.dev/docs/visualization/large-traces).
+- If you connect Perfetto to the RPC server, prefer Chrome over Brave (Brave often does not autodetect the RPC server).
 
-    - For very large trace.json files (`> 2GB`), refer to the [Perfetto documentation](https://perfetto.dev/docs/visualization/large-traces) for using native accelerators.
-    - If you are trying to connect Perfetto to the RPC server, it is recommended to use Chrome over Brave, as it has been observed that Perfetto in Brave does not autodetect the RPC server.
+**Multi-worker `s3://` `FileNotFoundError` after ~120s:** `num_workers=0` working while `num_workers>0` fails usually means the DataLoader parent started obstore (tokio) before fork and worker GETs hung. Current LitData fetches `index.json` with boto3 so workers can lazy-init obstore; they fall back to boto3 if the parent already started the runtime. On Studio R2 / `lightning_storage`, the same symptom can be a prefetch-thread crash (`data_connection_id` / `endpoint_url` into `boto3.Session`) — look for `[litdata] PrepareChunksThread CRASHED` on stderr and a `crash` instant in the trace.
 
 </details>
 
@@ -2428,6 +2455,17 @@ Speed to stream Imagenet 1.2M from local disk with ffcv vs LitData:
 | ffcv(os_cache=True) | JPEG 90% | 20 GB | 7653 | 8051 |
 | ffcv(os_cache=False) | JPEG 90% | 20 GB | 8149 | 8607 |
 
+Speed to stream a **synthetic ImageNet-scale set from Vast NFS** (NFSv3 `nconnect=32`, 208-CPU host, ~1 TiB RAM). Dataset: **1.08M** JPEG q95 256×256 (~160 GiB, 64 MiB chunks). `StreamingDataLoader`, batch **256**, `shuffle=True`, `drop_last=True`, decode only unless noted. POSIX-fast mmaps chunks **in place** (no copy into `~/.lightning/chunks`).
+
+| Setup | Workers | Images / sec |
+|---|---|---|
+| Copy into local cache (`LITDATA_POSIX_FAST=0`) | 48 | **16.7k** (2-epoch avg) |
+| POSIX-fast (this default on local/Vast paths) | 48 | **18.2k** |
+| POSIX-fast + README ImageNet augs (crop 224, flip, float32) | 48 | **12.9k** |
+| POSIX-fast, all CPU cores | **208** | **35.8k** |
+
+Notes: 208 workers need enough **MemAvailable**. This host had **928×1 GiB hugepages** reserved and idle (~900 GiB locked); after `nr_hugepages=0`, 208 workers stayed healthy. If `num_workers=os.cpu_count()` would crowd RAM, LitData **clamps** workers (`LITDATA_POSIX_MAX_WORKERS=0` disables) and skips `WILLNEED` prefetch. Real ImageNet JPEG 90% is much smaller (~12 GiB) and usually decodes faster than this q95 noise set.
+
 ### Raw Dataset
 
 Speed to stream raw Imagenet 1.2M from different cloud storage providers:

{litdata-0.2.67 → litdata-0.2.68}/README.md

@@ -215,6 +215,22 @@ for sample in dataloader:
     img, cls = sample["image"], sample["class"]
 ```
 
+**Keyed lookup and in-place patches** (needs `polars` and `optimize(..., key_fn=...)` or `build_keys_index`):
+
+```python
+ld.optimize(fn=fn, inputs=inputs, output_dir="fast_data", chunk_bytes="64MB", key_fn=lambda s: s["id"])
+
+ds = ld.StreamingDataset("fast_data")
+sample = ds["entity-id"]          # str keys
+sample = ds.get_by_key(42)        # int entity keys; ds[42] is still positional
+
+with ld.dataset_update("fast_data") as update:  # local directory only
+    update["entity-id"] = {"id": "entity-id", "x": 1}
+    update.commit()
+```
+
+`mode="append"` continues chunk numbering. `use_checkpoint=True` tries to resume an interrupted optimize. They are not the same.
+
 **Key benefits:**
 
 ✅ **Accelerate training:**       Optimized datasets load 20x faster.      
@@ -721,6 +737,10 @@ Shuffling is **deterministic** and designed for distributed training:
 
 The permutation depends on `seed`, the epoch, and chunk metadata — the same settings always yield the same order (required for resumable `state_dict`).
 
+**Object storage (`s3://`, `gs://`, …)** globally permutes chunks (`FullShuffle`). Random chunk order is cheap once files are already copied into the local cache.
+
+**POSIX-fast** (automatic for any local path) mmaps chunks in place. **Vast / NFS / Lustre / GPFS** (and `LITDATA_POSIX_FAST=1`) use `WindowShuffle`: each worker gets **whole chunks** in a sequential stripe, then shuffles only inside a sliding window (default **16**, `LITDATA_POSIX_SHUFFLE_WINDOW`) for both chunk order and in-chunk items. Local disks (ext4/xfs) keep global `FullShuffle`. Object URLs stay on `FullShuffle`. `LITDATA_POSIX_FAST=0` disables in-place mmap.
+
 ```python
 from litdata import StreamingDataset, StreamingDataLoader
 
@@ -789,7 +809,7 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 | `seed` | `42` | Shuffle / subsample RNG |
 | `serializers` | built-ins | Custom serialize/deserialize map |
 | `max_cache_size` | `"100GB"` | Evict consumed chunks beyond this size |
-| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk) |
+| `max_pre_download` | `2` | Chunks each worker may prefetch (raise for throughput; watch disk / RAM) |
 | `subsample` | `1.0` | Fraction of data (`0.01`) or upsample (`2.5`) |
 | `encryption` | `None` | `FernetEncryption` / `RSAEncryption` / custom |
 | `storage_options` | `{}` | Cloud client options |
@@ -800,6 +820,8 @@ Rough ImageNet order-of-magnitude on a Studio (not hard guarantees; right tuning
 
 Peak disk ≈ `num_workers × max_pre_download × mean_chunk_size`.
 
+On **Vast / NFS / local disk**, POSIX-fast is on by default (`LITDATA_POSIX_FAST=0` to disable). `WILLNEED` prefetch and `num_workers` are capped when they would exceed about half of `MemAvailable`. Idle **hugepages** (common on GPU nodes) do not count as available RAM — drop unused `nr_hugepages` if `MemAvailable` looks tiny next to `MemTotal`.
+
 **`StreamingDataLoader`**
 
 | Argument | Description |
@@ -1742,7 +1764,7 @@ Only **worker 0** is instrumented. When an `int` is used, the tracer wraps `fetc
 
 - Delete or change `profile_dir` between runs — LitData removes an existing `result.json` before starting.
 - Pair with a wiped chunk cache if you care about **cold** epoch behavior (`litdata cache clear`).
-- For deeper LitData internals (download / lock / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/deependujha/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
+- For deeper LitData internals (download / read / delete timeline), use `enable_tracer()` + [Litracer](https://github.com/Lightning-AI/litracer) instead — see [Debug & Profile LitData](#debug-profile). That path is complementary: viztracer = DataLoader worker CPU timeline; Litracer = LitData pipeline events.
 
 </details>
 
@@ -1855,6 +1877,9 @@ export LITDATA_ASYNC_MIN_PRE_DOWNLOAD=0
 | `LITDATA_DISABLE_VERSION_CHECK` | `0` | `1` skips the upgrade tip |
 | `HF_TOKEN` | — | Gated Hugging Face datasets |
 | `DEBUG_LITDATA` / `PRINT_DEBUG_LOGS` | `0` | Internal debug / stdout logs |
+| `LITDATA_LOG_FILE` | `litdata_debug.log` | `enable_tracer()` output path |
+| `LITDATA_TRACE_LEVEL` | unset | `batch` / `chunk` / `sample` / `debug` / `off` (see [Debug & Profile](#debug-profile)) |
+| `LITDATA_TRACE_CATEGORIES` | from level | Comma-separated cats, e.g. `download,read,delete` |
 
 Multi-node `optimize`/`map` on Studios also uses `DATA_OPTIMIZER_*` (set by the platform). Full catalog (debug logs, Studio injects, torchrun): see the LitData skill `reference/env-vars.md` when using agent skills, or the source modules `constants.py` / `async_prefetch.py`.
 
@@ -2020,66 +2045,68 @@ ds = StreamingDataset(input_dir=data_dir, encryption=rsa)
 
 &nbsp;
 
-LitData comes with built-in logging and profiling capabilities to help you debug and profile your data streaming workloads.
+`enable_tracer()` records the streaming pipeline (download vs read vs delete vs batch) as one-line events. [Litracer](https://github.com/Lightning-AI/litracer) converts that log into a Chrome / [Perfetto](https://ui.perfetto.dev) trace.
 
-<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
+This is complementary to [`profile_batches`](#profile-loading) (viztracer = DataLoader worker **CPU**; Litracer = LitData **pipeline** events).
 
-- e.g., with LitData Streaming
+<img width="1439" alt="431247797-0e955e71-2f9a-4aad-b7c1-a8218fed2e2e" src="https://github.com/user-attachments/assets/4e40676c-ba0b-49af-acac-975977173669" />
 
 ```python
 import litdata as ld
 from litdata.debugger import enable_tracer
 
-# WARNING: Remove existing trace `litdata_debug.log` file if it exists before re-tracing
-enable_tracer()
+# Call once per process, before the DataLoader. Delete an existing log before re-tracing (append).
+enable_tracer(level="chunk", log_file="litdata_debug.log")
+# level="batch" | "chunk" (default) | "sample" | "debug" | "off"
+# enable_tracer(categories=["download", "read", "delete"])
 
 if __name__ == "__main__":
     dataset = ld.StreamingDataset("s3://my-bucket/my-data", shuffle=True)
-    dataloader = ld.StreamingDataLoader(dataset, batch_size=64)
-
-    for batch in dataloader:
-        print(batch)  # Replace with your data processing logic
+    for batch in ld.StreamingDataLoader(dataset, batch_size=64, num_workers=8):
+        ...
 ```
 
-1. Generate Debug Log:
+| Level | Events |
+| ----- | ------ |
+| `batch` | Epoch + per-batch spans, plus crashes |
+| `chunk` (default) | + `download`, `read`, `delete`, `decompress`, `prefetch` |
+| `sample` | + per-item `__getitem__` (high volume) |
+| `debug` | + `.cnt` lock refcount spans |
+| `off` | Disable |
 
-    - Run your Python program and it'll create a log file containing detailed debug information.
+Event **names** are stable (`download`, `read`, `delete`, `batch`, `sample`, `crash`). Chunk / sample indexes live in args so Perfetto groups all downloads together. Each line is `key: value;` pairs with Chrome **microsecond** timestamps. Crashes are a one-line instant (`ph: I`, `name: crash`); the Python traceback is printed to **stderr**, not the log file (a multi-line `logger.exception` would break Litracer).
 
-    ```bash
-      python main.py
-    ```
+Env overrides: `LITDATA_LOG_FILE`, `LITDATA_TRACE_LEVEL`, `LITDATA_TRACE_CATEGORIES` (comma-separated). Tracer calls are no-ops when tracing is off.
 
-2. Install [Litracer](https://github.com/deependujha/litracer/):
+1. Generate the log:
 
-    - Option 1: Using Go (recommended)
-        - Install Go on your system.
-        - Run the following command to install Litracer:
+    ```bash
+    python train.py   # writes litdata_debug.log
+    ```
 
-        ```bash
-          go install github.com/deependujha/litracer@latest
-        ```
+2. Install [Litracer](https://github.com/Lightning-AI/litracer) (Go 1.23+):
 
-    - Option 2: Download Binary
-        - Visit the [LitRacer GitHub Releases](https://github.com/deependujha/litracer/releases) page.
-        - Download the appropriate binary for your operating system and follow the installation instructions.
+    ```bash
+    git clone https://github.com/Lightning-AI/litracer.git
+    cd litracer && go build -o litracer .
+    ```
 
-3. Convert Debug Log to trace JSON:
+    Or `go install github.com/deependujha/litracer@latest` (published Go module path). Until `go.mod` is renamed, `go install github.com/Lightning-AI/litracer@latest` does not work. Release binaries: [GitHub Releases](https://github.com/Lightning-AI/litracer/releases).
 
-    - Use litracer to convert the generated log file into a trace JSON file. This command uses 100 workers for conversion:
+3. Convert and open in Perfetto:
 
     ```bash
-      litracer litdata_debug.log -o litdata_trace.json -w 100
+    litracer --quiet --validate -o litdata_trace.json.gz litdata_debug.log
+    litracer --quiet --cat download,read,delete -o io.json.gz litdata_debug.log
+    # open the .json.gz at https://ui.perfetto.dev (preferred) or chrome://tracing
     ```
 
-4. Visualize the trace:
-
-    - Use either `chrome://tracing` in the Chrome browser or `ui.perfetto.dev` to view the `litdata_trace.json` file for in-depth performance insights. You can also use `SQL queries` to analyze the logs.
-    - `Perfetto` is recommended over `chrome://tracing` for visualization & analyzing.
+    `--quiet` prints a one-line summary (per-category durations, unmatched B/E, crashes). `--cat` keeps only those categories. Matched B/E pairs become complete (`ph: X`) spans unless `--no-complete`. Default output is gzip Chrome JSON (`.json.gz`) — both Perfetto and `chrome://tracing` open it; pass `-o file.json` for uncompressed.
 
-- Key Points:
+- For trace files `> 2GB`, see [Perfetto large traces](https://perfetto.dev/docs/visualization/large-traces).
+- If you connect Perfetto to the RPC server, prefer Chrome over Brave (Brave often does not autodetect the RPC server).
 
-    - For very large trace.json files (`> 2GB`), refer to the [Perfetto documentation](https://perfetto.dev/docs/visualization/large-traces) for using native accelerators.
-    - If you are trying to connect Perfetto to the RPC server, it is recommended to use Chrome over Brave, as it has been observed that Perfetto in Brave does not autodetect the RPC server.
+**Multi-worker `s3://` `FileNotFoundError` after ~120s:** `num_workers=0` working while `num_workers>0` fails usually means the DataLoader parent started obstore (tokio) before fork and worker GETs hung. Current LitData fetches `index.json` with boto3 so workers can lazy-init obstore; they fall back to boto3 if the parent already started the runtime. On Studio R2 / `lightning_storage`, the same symptom can be a prefetch-thread crash (`data_connection_id` / `endpoint_url` into `boto3.Session`) — look for `[litdata] PrepareChunksThread CRASHED` on stderr and a `crash` instant in the trace.
 
 </details>
 
@@ -2364,6 +2391,17 @@ Speed to stream Imagenet 1.2M from local disk with ffcv vs LitData:
 | ffcv(os_cache=True) | JPEG 90% | 20 GB | 7653 | 8051 |
 | ffcv(os_cache=False) | JPEG 90% | 20 GB | 8149 | 8607 |
 
+Speed to stream a **synthetic ImageNet-scale set from Vast NFS** (NFSv3 `nconnect=32`, 208-CPU host, ~1 TiB RAM). Dataset: **1.08M** JPEG q95 256×256 (~160 GiB, 64 MiB chunks). `StreamingDataLoader`, batch **256**, `shuffle=True`, `drop_last=True`, decode only unless noted. POSIX-fast mmaps chunks **in place** (no copy into `~/.lightning/chunks`).
+
+| Setup | Workers | Images / sec |
+|---|---|---|
+| Copy into local cache (`LITDATA_POSIX_FAST=0`) | 48 | **16.7k** (2-epoch avg) |
+| POSIX-fast (this default on local/Vast paths) | 48 | **18.2k** |
+| POSIX-fast + README ImageNet augs (crop 224, flip, float32) | 48 | **12.9k** |
+| POSIX-fast, all CPU cores | **208** | **35.8k** |
+
+Notes: 208 workers need enough **MemAvailable**. This host had **928×1 GiB hugepages** reserved and idle (~900 GiB locked); after `nr_hugepages=0`, 208 workers stayed healthy. If `num_workers=os.cpu_count()` would crowd RAM, LitData **clamps** workers (`LITDATA_POSIX_MAX_WORKERS=0` disables) and skips `WILLNEED` prefetch. Real ImageNet JPEG 90% is much smaller (~12 GiB) and usually decodes faster than this q95 noise set.
+
 ### Raw Dataset
 
 Speed to stream raw Imagenet 1.2M from different cloud storage providers:

{litdata-0.2.67 → litdata-0.2.68}/src/litdata/__about__.py

@@ -14,7 +14,7 @@
 
 import time
 
-__version__ = "0.2.67"
+__version__ = "0.2.68"
 __author__ = "Lightning AI et al."
 __author_email__ = "pytorch@lightning.ai"
 __license__ = "Apache-2.0"

{litdata-0.2.67 → litdata-0.2.68}/src/litdata/__init__.py

@@ -14,16 +14,19 @@ import warnings
 
 from litdata.__about__ import *  # noqa: F403
 from litdata.constants import _LIGHTNING_SDK_AVAILABLE
+from litdata.exceptions import ChunkWaitTimeoutError
 from litdata.processing.functions import map, merge_datasets, optimize, walk
 from litdata.raw.dataset import StreamingRawDataset
 from litdata.streaming.combined import CombinedStreamingDataset
 from litdata.streaming.dataloader import StreamingDataLoader
 from litdata.streaming.dataset import StreamingDataset
+from litdata.streaming.dataset_update import dataset_update
 from litdata.streaming.item_loader import TokensLoader
 from litdata.streaming.parallel import ParallelStreamingDataset
 from litdata.streaming.writer import index_parquet_dataset
 from litdata.utilities.breakpoint import breakpoint
 from litdata.utilities.hf_dataset import index_hf_dataset
+from litdata.utilities.keys_index import build_keys_index
 from litdata.utilities.train_test_split import train_test_split
 
 warnings.filterwarnings(
@@ -41,12 +44,15 @@ __all__ = [
     "ParallelStreamingDataset",
     "map",
     "optimize",
+    "dataset_update",
+    "build_keys_index",
     "walk",
     "train_test_split",
     "merge_datasets",
     "index_parquet_dataset",
     "index_hf_dataset",
     "breakpoint",
+    "ChunkWaitTimeoutError",
 ]
 
 if _LIGHTNING_SDK_AVAILABLE:

{litdata-0.2.67 → litdata-0.2.68}/src/litdata/constants.py

@@ -20,6 +20,12 @@ import torch
 from lightning_utilities.core.imports import RequirementCache
 
 _INDEX_FILENAME = "index.json"
+_KEYS_DIRNAME = "keys"
+_KEYS_SHARD_TEMPLATE = "shard-{:05d}.parquet"
+# Legacy single-file sidecar (still read if present).
+_KEYS_FILENAME = "keys.parquet"
+_RANK_KEYS_SUFFIX = ".keys.parquet"
+_DEFAULT_KEYS_NUM_SHARDS = 1
 _DEFAULT_CHUNK_BYTES = 1 << 26  # 64M B
 _DEFAULT_FAST_DEV_RUN_ITEMS = 10
 _DEFAULT_CACHE_DIR = os.path.join(Path.home(), ".lightning", "chunks")