cdmltrain 0.2.0__tar.gz

cdmltrain-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 prem85642
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cdmltrain-0.2.0/PKG-INFO

@@ -0,0 +1,316 @@
+Metadata-Version: 2.4
+Name: cdmltrain
+Version: 0.2.0
+Summary: Stream ML datasets from ZIP/ZSTD archives into PyTorch without disk extraction.
+Home-page: https://github.com/prem85642/cdmltrain
+Author: prem85642
+Author-email: your.email@domain.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Intended Audience :: Science/Research
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: Pillow>=8.0.0
+Provides-Extra: zstd
+Requires-Dist: zstandard>=0.20.0; extra == "zstd"
+Provides-Extra: gpu
+Requires-Dist: torch>=1.9.0; extra == "gpu"
+Provides-Extra: full
+Requires-Dist: zstandard>=0.20.0; extra == "full"
+Requires-Dist: torch>=1.9.0; extra == "full"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# cdmltrain 🚀
+### Stream ML Datasets Directly from ZIP / ZSTD Archives — No Extraction. No Wasted Storage. Zero OOM.
+
+[![MIT License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Python 3.7+](https://img.shields.io/badge/Python-3.7%2B-blue.svg)](https://www.python.org/)
+[![Platform](https://img.shields.io/badge/Platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey)]()
+
+---
+
+## 🔥 The Problem This Solves
+
+Every Data Scientist / ML Engineer hits this wall:
+
+```
+"Your 100 GB Kaggle dataset is a ZIP file.
+ Extracting it takes 2 hours and needs 300 GB of free disk space.
+ Your Colab/Kaggle notebook crashes with Out-of-Memory errors."
+```
+
+**`cdmltrain` eliminates this problem entirely.**
+
+It lets PyTorch read images, audio, text, CSV, JSON — **any data** — directly from a compressed archive **into RAM**, skipping disk extraction completely.
+
+---
+
+## ✨ Key Features
+
+| Feature | Description |
+|---|---|
+| 🗜️ **Format Agnostic** | Images, audio (.wav), text, CSV, JSON, binary — all supported |
+| ⚡ **O(1) Random Access** | ZIP Central Directory indexing — jumps to any file instantly |
+| 🧠 **Memory-Safe Cache** | Custom LRU cache enforces strict RAM limits — zero OOM crashes |
+| 🔒 **Thread Safe** | Concurrent reads for PyTorch `DataLoader(num_workers=N)` |
+| 🔧 **3-Tier Architecture** | Auto-selects best engine based on your hardware |
+| 🏎️ **ZSTD Support** | `.tar.zst` archives — 25x faster than ZIP deflate |
+| 🎮 **GPU Direct Loader** | Streams data to CUDA VRAM with async prefetch |
+| 🌍 **Cross-Platform** | Windows, Linux, macOS — works everywhere |
+
+---
+
+## 🏗️ Architecture (3 Tiers — Auto-Selected)
+
+```
+Your ZIP/ZSTD File
+        │
+        ▼
+┌─────────────────────────────────────────────────────┐
+│  Tier 1: Python CoreStreamEngine                    │
+│  ✅ Works on ANY machine, no dependencies            │
+│  → O(1) Index + LRU Cache + Thread Safety           │
+├─────────────────────────────────────────────────────┤
+│  Tier 2: C++ FastCoreEngine (pybind11)              │
+│  ✅ Auto-enabled if C++ Build Tools installed        │
+│  → Bypasses Python GIL, faster multi-worker reads   │
+├─────────────────────────────────────────────────────┤
+│  Tier 3: ZSTD Engine + GPU Direct Loader            │
+│  ✅ pip install zstandard   (for .tar.zst files)     │
+│  ✅ NVIDIA GPU (for direct VRAM streaming)           │
+│  → 25x faster decompression + near-zero GPU latency │
+└─────────────────────────────────────────────────────┘
+        │
+        ▼
+  PyTorch DataLoader → Model Training
+```
+
+**The library auto-detects your hardware and picks the best tier. You write the same code regardless.**
+
+---
+
+## 📦 Installation
+
+> **Note:** PyPI release coming soon. Install locally for now.
+
+```bash
+# Step 1: Clone the repo
+git clone https://github.com/prem85642/cdmltrain.git
+cd cdmltrain
+
+# Step 2: Install
+pip install .
+
+# Step 3 (Optional): ZSTD support for .tar.zst archives
+pip install zstandard
+```
+
+> **C++ Acceleration (Tier 2):**
+> - Windows: Install [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+> - Linux: `sudo apt-get install build-essential`
+> - If skipped: pure-Python engine runs automatically — no errors.
+
+---
+
+## 🚀 Quick Start
+
+### Basic Usage (ZIP — Any Data Type)
+```python
+from cdmltrain import CDMLStreamDataset
+from torch.utils.data import DataLoader
+
+# Point directly to your ZIP — no extraction needed!
+dataset = CDMLStreamDataset(
+    zip_path="my_dataset.zip",
+    max_cache_mb=2048    # RAM cache limit (MB)
+)
+
+dataloader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)
+
+for batch in dataloader:
+    # Train your model normally
+    pass
+```
+
+### Image Dataset (with PyTorch Transforms)
+```python
+from cdmltrain import CDMLStreamDataset
+from torchvision import transforms
+
+transform = transforms.Compose([
+    transforms.Resize((224, 224)),
+    transforms.ToTensor(),
+    transforms.Normalize([0.5], [0.5])
+])
+
+dataset = CDMLStreamDataset("images.zip", transform=transform, is_image=True)
+```
+
+### ZSTD Archive (25x Faster Decompression) ⚡
+```python
+# Just change the file extension — everything else is identical!
+dataset = CDMLStreamDataset("my_dataset.tar.zst", max_cache_mb=2048)
+```
+
+### GPU Direct Loader (NVIDIA VRAM Streaming) 🎮
+```python
+from cdmltrain.gpu_loader import GPUDirectLoader
+
+loader = GPUDirectLoader(dataset, batch_size=32)
+# Auto-detects GPU. Falls back to CPU if no GPU found.
+
+for batch in loader:
+    output = model(batch.float())  # batch already on GPU VRAM!
+```
+
+---
+
+## ⚙️ Configuration Parameters
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `zip_path` | `str` | *(required)* | Path to `.zip` or `.tar.zst` file |
+| `transform` | `callable` | `None` | PyTorch/torchvision transform |
+| `is_image` | `bool` | `False` | `True` enables PIL image decoding |
+| `max_cache_mb` | `int` | `2048` | Max RAM for caching (MB) |
+
+**`max_cache_mb` Guide:**
+
+| Your RAM | Recommended |
+|---|---|
+| 4 GB (Colab Free) | `512` |
+| 8 GB (Laptop) | `2048` |
+| 16 GB (PC) | `6000` |
+| 32 GB+ (Server) | `16000` |
+
+---
+
+## 📊 Benchmarks (Real Tests)
+
+### ZSTD vs ZIP Speed (200 files × 10KB)
+| Engine | Speed | Speedup |
+|---|---|---|
+| ZIP (Deflate) — Tier 1/2 | 29,204 files/sec | baseline |
+| ZSTD — Tier 3 | **739,653 files/sec** | **🔥 25x faster** |
+
+### GPU Direct Loader (Google Colab T4)
+| Metric | Value |
+|---|---|
+| GPU | Tesla T4 (15.6 GB VRAM) |
+| Batch device | `cuda:0` — data streamed to VRAM |
+| Throughput | **17,512 items/sec** |
+| Epoch time (100 items) | 0.0045s |
+
+### Memory Safety Test
+| Test | Result |
+|---|---|
+| Cache limit: 1MB, Data: 2MB | ✅ Stayed under 1MB |
+| Thread safety: 8 workers | ✅ Zero race conditions |
+| Corrupted ZIP | ✅ Rejected cleanly |
+| 50MB single file | ✅ Byte-exact in 0.108s |
+| 1000-file archive | ✅ Indexed in 0.04s |
+
+---
+
+## 🐛 Debugging / Inspection
+
+Inspect any specific file without unzipping:
+```python
+dataset.extract_sample_to_disk(idx=42, export_path="./inspection/")
+```
+
+---
+
+## 🛠️ Troubleshooting
+
+### `ModuleNotFoundError: No module named 'cdmltrain'`
+```bash
+git clone https://github.com/prem85642/cdmltrain.git && cd cdmltrain && pip install .
+```
+
+### `ModuleNotFoundError: No module named 'zstandard'`
+```bash
+pip install zstandard
+```
+
+### `Microsoft Visual C++ 14.0 required` (Windows)
+Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/). Or skip — pure Python engine works fine.
+
+### Out of Memory on Colab
+```python
+dataset = CDMLStreamDataset("data.zip", max_cache_mb=512)  # Reduce cache
+```
+
+### `Bad CRC-32` error
+```bash
+python -c "import zipfile; print(zipfile.ZipFile('file.zip').testzip())"
+# None = healthy, anything else = re-download
+```
+
+### `PIL.UnidentifiedImageError`
+```python
+dataset = CDMLStreamDataset("data.zip", is_image=False)  # Not an image dataset
+```
+
+---
+
+## 🖥️ OS Compatibility
+
+| Feature | Windows | Linux | macOS |
+|---|---|---|---|
+| ZIP Engine (Tier 1) | ✅ | ✅ | ✅ |
+| ZSTD Engine (Tier 3) | ✅ | ✅ | ✅ |
+| GPU Direct Loader | ✅ | ✅ | ✅ |
+| C++ Fast Engine (Tier 2) | ✅ pre-built | ✅ compile via `pip install .` | ✅ compile via `pip install .` |
+
+---
+
+## 📁 Project Structure
+
+```
+cdmltrain/
+├── cdmltrain/
+│   ├── __init__.py          # Package entry point
+│   ├── core.py              # Tier 1: Python CoreStreamEngine
+│   ├── dataset.py           # CDMLStreamDataset (auto-tier selection)
+│   ├── zstd_engine.py       # Tier 3: ZSTD streaming engine
+│   ├── gpu_loader.py        # Tier 3: GPU Direct Loader (CUDA pinned memory)
+│   └── src/
+│       └── fast_core.cpp    # Tier 2: C++ FastCoreEngine (pybind11)
+├── demo.py                  # Quickstart demo
+├── quickstart.ipynb         # Jupyter Notebook tutorial
+├── test_enterprise_audit.py # Enterprise QA suite (8 tests)
+├── test_zstd_benchmark.py   # ZSTD vs ZIP benchmark
+├── test_zstd_compat.py      # Cross-format compatibility test
+├── test_gpu_loader.py       # GPU Direct Loader test
+├── setup.py                 # pip install configuration
+├── requirements.txt         # Dependencies
+└── LICENSE                  # MIT License
+```
+
+---
+
+## 🤝 Contributing
+
+Pull requests are welcome! For major changes, please open an issue first.
+
+---
+
+## 📄 License
+
+MIT License — see [LICENSE](LICENSE) for details.
+
+**Made with ❤️ for the ML community — because your model matters more than your storage bill.**

cdmltrain-0.2.0/README.md

@@ -0,0 +1,281 @@
+# cdmltrain 🚀
+### Stream ML Datasets Directly from ZIP / ZSTD Archives — No Extraction. No Wasted Storage. Zero OOM.
+
+[![MIT License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Python 3.7+](https://img.shields.io/badge/Python-3.7%2B-blue.svg)](https://www.python.org/)
+[![Platform](https://img.shields.io/badge/Platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey)]()
+
+---
+
+## 🔥 The Problem This Solves
+
+Every Data Scientist / ML Engineer hits this wall:
+
+```
+"Your 100 GB Kaggle dataset is a ZIP file.
+ Extracting it takes 2 hours and needs 300 GB of free disk space.
+ Your Colab/Kaggle notebook crashes with Out-of-Memory errors."
+```
+
+**`cdmltrain` eliminates this problem entirely.**
+
+It lets PyTorch read images, audio, text, CSV, JSON — **any data** — directly from a compressed archive **into RAM**, skipping disk extraction completely.
+
+---
+
+## ✨ Key Features
+
+| Feature | Description |
+|---|---|
+| 🗜️ **Format Agnostic** | Images, audio (.wav), text, CSV, JSON, binary — all supported |
+| ⚡ **O(1) Random Access** | ZIP Central Directory indexing — jumps to any file instantly |
+| 🧠 **Memory-Safe Cache** | Custom LRU cache enforces strict RAM limits — zero OOM crashes |
+| 🔒 **Thread Safe** | Concurrent reads for PyTorch `DataLoader(num_workers=N)` |
+| 🔧 **3-Tier Architecture** | Auto-selects best engine based on your hardware |
+| 🏎️ **ZSTD Support** | `.tar.zst` archives — 25x faster than ZIP deflate |
+| 🎮 **GPU Direct Loader** | Streams data to CUDA VRAM with async prefetch |
+| 🌍 **Cross-Platform** | Windows, Linux, macOS — works everywhere |
+
+---
+
+## 🏗️ Architecture (3 Tiers — Auto-Selected)
+
+```
+Your ZIP/ZSTD File
+        │
+        ▼
+┌─────────────────────────────────────────────────────┐
+│  Tier 1: Python CoreStreamEngine                    │
+│  ✅ Works on ANY machine, no dependencies            │
+│  → O(1) Index + LRU Cache + Thread Safety           │
+├─────────────────────────────────────────────────────┤
+│  Tier 2: C++ FastCoreEngine (pybind11)              │
+│  ✅ Auto-enabled if C++ Build Tools installed        │
+│  → Bypasses Python GIL, faster multi-worker reads   │
+├─────────────────────────────────────────────────────┤
+│  Tier 3: ZSTD Engine + GPU Direct Loader            │
+│  ✅ pip install zstandard   (for .tar.zst files)     │
+│  ✅ NVIDIA GPU (for direct VRAM streaming)           │
+│  → 25x faster decompression + near-zero GPU latency │
+└─────────────────────────────────────────────────────┘
+        │
+        ▼
+  PyTorch DataLoader → Model Training
+```
+
+**The library auto-detects your hardware and picks the best tier. You write the same code regardless.**
+
+---
+
+## 📦 Installation
+
+> **Note:** PyPI release coming soon. Install locally for now.
+
+```bash
+# Step 1: Clone the repo
+git clone https://github.com/prem85642/cdmltrain.git
+cd cdmltrain
+
+# Step 2: Install
+pip install .
+
+# Step 3 (Optional): ZSTD support for .tar.zst archives
+pip install zstandard
+```
+
+> **C++ Acceleration (Tier 2):**
+> - Windows: Install [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+> - Linux: `sudo apt-get install build-essential`
+> - If skipped: pure-Python engine runs automatically — no errors.
+
+---
+
+## 🚀 Quick Start
+
+### Basic Usage (ZIP — Any Data Type)
+```python
+from cdmltrain import CDMLStreamDataset
+from torch.utils.data import DataLoader
+
+# Point directly to your ZIP — no extraction needed!
+dataset = CDMLStreamDataset(
+    zip_path="my_dataset.zip",
+    max_cache_mb=2048    # RAM cache limit (MB)
+)
+
+dataloader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)
+
+for batch in dataloader:
+    # Train your model normally
+    pass
+```
+
+### Image Dataset (with PyTorch Transforms)
+```python
+from cdmltrain import CDMLStreamDataset
+from torchvision import transforms
+
+transform = transforms.Compose([
+    transforms.Resize((224, 224)),
+    transforms.ToTensor(),
+    transforms.Normalize([0.5], [0.5])
+])
+
+dataset = CDMLStreamDataset("images.zip", transform=transform, is_image=True)
+```
+
+### ZSTD Archive (25x Faster Decompression) ⚡
+```python
+# Just change the file extension — everything else is identical!
+dataset = CDMLStreamDataset("my_dataset.tar.zst", max_cache_mb=2048)
+```
+
+### GPU Direct Loader (NVIDIA VRAM Streaming) 🎮
+```python
+from cdmltrain.gpu_loader import GPUDirectLoader
+
+loader = GPUDirectLoader(dataset, batch_size=32)
+# Auto-detects GPU. Falls back to CPU if no GPU found.
+
+for batch in loader:
+    output = model(batch.float())  # batch already on GPU VRAM!
+```
+
+---
+
+## ⚙️ Configuration Parameters
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `zip_path` | `str` | *(required)* | Path to `.zip` or `.tar.zst` file |
+| `transform` | `callable` | `None` | PyTorch/torchvision transform |
+| `is_image` | `bool` | `False` | `True` enables PIL image decoding |
+| `max_cache_mb` | `int` | `2048` | Max RAM for caching (MB) |
+
+**`max_cache_mb` Guide:**
+
+| Your RAM | Recommended |
+|---|---|
+| 4 GB (Colab Free) | `512` |
+| 8 GB (Laptop) | `2048` |
+| 16 GB (PC) | `6000` |
+| 32 GB+ (Server) | `16000` |
+
+---
+
+## 📊 Benchmarks (Real Tests)
+
+### ZSTD vs ZIP Speed (200 files × 10KB)
+| Engine | Speed | Speedup |
+|---|---|---|
+| ZIP (Deflate) — Tier 1/2 | 29,204 files/sec | baseline |
+| ZSTD — Tier 3 | **739,653 files/sec** | **🔥 25x faster** |
+
+### GPU Direct Loader (Google Colab T4)
+| Metric | Value |
+|---|---|
+| GPU | Tesla T4 (15.6 GB VRAM) |
+| Batch device | `cuda:0` — data streamed to VRAM |
+| Throughput | **17,512 items/sec** |
+| Epoch time (100 items) | 0.0045s |
+
+### Memory Safety Test
+| Test | Result |
+|---|---|
+| Cache limit: 1MB, Data: 2MB | ✅ Stayed under 1MB |
+| Thread safety: 8 workers | ✅ Zero race conditions |
+| Corrupted ZIP | ✅ Rejected cleanly |
+| 50MB single file | ✅ Byte-exact in 0.108s |
+| 1000-file archive | ✅ Indexed in 0.04s |
+
+---
+
+## 🐛 Debugging / Inspection
+
+Inspect any specific file without unzipping:
+```python
+dataset.extract_sample_to_disk(idx=42, export_path="./inspection/")
+```
+
+---
+
+## 🛠️ Troubleshooting
+
+### `ModuleNotFoundError: No module named 'cdmltrain'`
+```bash
+git clone https://github.com/prem85642/cdmltrain.git && cd cdmltrain && pip install .
+```
+
+### `ModuleNotFoundError: No module named 'zstandard'`
+```bash
+pip install zstandard
+```
+
+### `Microsoft Visual C++ 14.0 required` (Windows)
+Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/). Or skip — pure Python engine works fine.
+
+### Out of Memory on Colab
+```python
+dataset = CDMLStreamDataset("data.zip", max_cache_mb=512)  # Reduce cache
+```
+
+### `Bad CRC-32` error
+```bash
+python -c "import zipfile; print(zipfile.ZipFile('file.zip').testzip())"
+# None = healthy, anything else = re-download
+```
+
+### `PIL.UnidentifiedImageError`
+```python
+dataset = CDMLStreamDataset("data.zip", is_image=False)  # Not an image dataset
+```
+
+---
+
+## 🖥️ OS Compatibility
+
+| Feature | Windows | Linux | macOS |
+|---|---|---|---|
+| ZIP Engine (Tier 1) | ✅ | ✅ | ✅ |
+| ZSTD Engine (Tier 3) | ✅ | ✅ | ✅ |
+| GPU Direct Loader | ✅ | ✅ | ✅ |
+| C++ Fast Engine (Tier 2) | ✅ pre-built | ✅ compile via `pip install .` | ✅ compile via `pip install .` |
+
+---
+
+## 📁 Project Structure
+
+```
+cdmltrain/
+├── cdmltrain/
+│   ├── __init__.py          # Package entry point
+│   ├── core.py              # Tier 1: Python CoreStreamEngine
+│   ├── dataset.py           # CDMLStreamDataset (auto-tier selection)
+│   ├── zstd_engine.py       # Tier 3: ZSTD streaming engine
+│   ├── gpu_loader.py        # Tier 3: GPU Direct Loader (CUDA pinned memory)
+│   └── src/
+│       └── fast_core.cpp    # Tier 2: C++ FastCoreEngine (pybind11)
+├── demo.py                  # Quickstart demo
+├── quickstart.ipynb         # Jupyter Notebook tutorial
+├── test_enterprise_audit.py # Enterprise QA suite (8 tests)
+├── test_zstd_benchmark.py   # ZSTD vs ZIP benchmark
+├── test_zstd_compat.py      # Cross-format compatibility test
+├── test_gpu_loader.py       # GPU Direct Loader test
+├── setup.py                 # pip install configuration
+├── requirements.txt         # Dependencies
+└── LICENSE                  # MIT License
+```
+
+---
+
+## 🤝 Contributing
+
+Pull requests are welcome! For major changes, please open an issue first.
+
+---
+
+## 📄 License
+
+MIT License — see [LICENSE](LICENSE) for details.
+
+**Made with ❤️ for the ML community — because your model matters more than your storage bill.**

cdmltrain-0.2.0/cdmltrain/__init__.py

@@ -0,0 +1,4 @@
+from .dataset import CDMLStreamDataset
+from .core import CoreStreamEngine
+
+__all__ = ["CDMLStreamDataset", "CoreStreamEngine"]