tcache 0.2.0__tar.gz

tcache-0.2.0/.gitignore

@@ -0,0 +1,37 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.egg-info/
+dist/
+build/
+*.whl
+.venv/
+.venv-mdv6/
+.pytest_cache/
+*.egg
+
+# Data / large binaries
+*.tgz
+*.tar
+*.zip
+data/
+*_int8.bin
+*_scales.bin
+*_zp.bin
+*_pixels.bin
+*_meta.json
+*_pixel_meta.json
+!compression_benchmark_results.json
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+.DS_Store
+Thumbs.db
+
+# Triton cache
+triton_cache/

tcache-0.2.0/AGENTS.md

@@ -0,0 +1,133 @@
+# AGENTS.md — TensorCache
+
+> Guidance for AI coding agents working in this repo. Keep this file concise, factual, and up-to-date.
+
+## 1. Project Overview
+
+**TensorCache** (`tensorcache`, v0.1.0) — Ultra-fast, high-fidelity block-wise INT8 feature & pixel cache engine for PyTorch.
+
+Two bottlenecks solved:
+1. **Feature Cache Bloat:** Block-wise microscaled INT8 (`G=32`, BF16 scales) — `1.88×` compression vs BF16, `0.54%` rel RMSE (see `compression_benchmark_results.json:190-201`, `README.md:77-81`).
+2. **JPEG/PNG CPU Decode:** Zero-copy `np.memmap` + GPU stream prefetching — `>2000 MB/s` throughput (`README.md:6-7`, `src/tensorcache/pixel_cache.py:95-137`, `src/tensorcache/feature_cache.py:112-210`).
+
+Installation: `pip install tensorcache` or `pip install -e .` from repo root. Python `>=3.9`, `torch>=2.0`, `numpy>=1.20` (`pyproject.toml:16-18`). Optional `fast` extras: `triton`, `zstandard`, `blosc2`, `safetensors` (`pyproject.toml:21-27`) — note `triton` is skipped on Windows (`platform_system != 'Windows'`).
+
+## 2. Repo Structure
+
+```
+/
+├── src/tensorcache/
+│   ├── __init__.py        # public API re-exports
+│   ├── codec.py           # BlockwiseInt8Codec, quantize/dequantize (core)
+│   ├── fused_ops.py       # Triton fused kernels (requires CUDA/ROCm + Triton)
+│   ├── feature_cache.py   # FeatureCacheWriter / FeatureCacheDataset (mmap .bin + .json)
+│   ├── pixel_cache.py     # PixelCacheWriter / PixelCacheDataset (raw uint8 mmap)
+│   ├── prefetcher.py      # AsyncGPUPrefetcher (double-buffered CUDA stream)
+│   └── streamer.py        # ZeroCopyTensorStreamer (pinned + ring-buffered)
+├── tests/
+│   ├── test_codec.py      # codec + feature/pixel disk I/O tests
+│   └── test_fused_ops.py  # GPU-only, skipped if !torch.cuda.is_available()
+├── benchmarks/
+│   ├── bench_memory_opt.py
+│   ├── benchmark_dual_gpu.py
+│   ├── speed_test_rigorous.py
+│   └── tune_dequant.py
+├── analyze_compression_error.py  # full pixel+feature benchmark (tabulate)
+├── experiment_blockwise_int8.py  # G-sweep, symmetric/asymmetric, bitwidth
+├── download_test_dataset.py      # Imagenette-320 fetcher -> data/test_images
+├── pyproject.toml
+└── README.md
+```
+
+Data dirs: `data/` (contains `imagenette2-320.tgz`, `flowers-102/102flowers.tgz` — do not commit large binaries). Cache files are runtime-generated (`*_int8.bin`, `*_scales.bin`, `*_pixels.bin`, `*_meta.json`, `*_pixel_meta.json`) — gitignore them.
+
+## 3. Build / Run / Test Commands
+
+```bash
+# install (dev)
+pip install -e ".[fast,dev]"
+
+# run all tests (fused tests auto-skip on CPU/Windows)
+pytest -v
+pytest tests/test_codec.py -v
+pytest tests/test_fused_ops.py -v  # requires CUDA + Triton
+
+# single-file direct run (no pytest)
+python tests/test_codec.py
+
+# benchmarks / analysis (require data images)
+python analyze_compression_error.py          # needs Plant dataset or data/test_images fallback
+python experiment_blockwise_int8.py          # needs CUDA + timm + Plant data
+python download_test_dataset.py              # fetches Imagenette-320 (~326 MB) to data/test_images
+python benchmarks/speed_test_rigorous.py
+python benchmarks/tune_dequant.py
+
+# Windows note: Triton unavailable -> codec falls back to vectorized PyTorch path (codec.py:167-177)
+```
+
+No `Makefile`, no `opencode.json` yet, no CI config in repo. Use `pytest>=7.0` (`pyproject.toml:29`).
+
+## 4. Core Architecture — Read Before Editing
+
+### 4.1 Codec (`src/tensorcache/codec.py:40-202`)
+- `quantize_int8_g32(x, group_size=32)` — flatten, pad to multiple of `G`, `view(-1,G)`, `amax`, `scales = max/127 -> BF16`, `round(x/scale) clamp [-128,127] -> int8`. Returns `(q_int8 (flat 1D), scales (1D BF16), orig_shape)` (`codec.py:40-75`).
+- `quantize_int8_adaptive` — AdaRound-style parallel candidate search over `linspace(0.90,1.05,31)` per block, picks min L2 error (`codec.py:78-126`). ~10% lower error, heavier.
+- `dequantize_int8_g32` — Triton path if `HAS_TRITON and device.type in (cuda,hip)` using `_triton_dequant_kernel` (`codec.py:22-37`, `150-165`) else padded `view(-1,G) * scales` + copy into `out_buffer` (`codec.py:167-177`). `out_buffer` avoids allocator overhead — preserve this API.
+- `BlockwiseInt8Codec` wrapper (`codec.py:180-202`) — `group_size` + `adaptive` flag. Keep `group_size=32` as default; benchmarks assume it.
+
+**Invariants:**
+- Scales stored as BF16 (`torch.bfloat16`) on disk as `uint16` bitcast: `scales.view(torch.int16).cpu().numpy().view(np.uint16)` (`feature_cache.py:80`) and restored via `view(np.int16).copy()).view(torch.bfloat16)` (`feature_cache.py:153`). Do not change dtype without migrating file format.
+- Padding is trimmed on return (`flatten()[:numel]`). Keep.
+- Storage cost: `1 + 2/32 = 1.0625 B/elem` (`analyze_compression_error.py:515`, `559`).
+
+### 4.2 Feature Cache (`src/tensorcache/feature_cache.py:21-211`)
+- **Writer:** pre-allocates `np.memmap` `w+` with shape `(num_samples, seq_len, dim)` int8 and `(num_samples, scales_per_sample)` uint16 (`feature_cache.py:53-60`). `append` handles both `[seq_len,dim]` and `[B,seq_len,dim]` (`feature_cache.py:63-81`). `close()` flushes, closes `_mmap`, writes `_meta.json` with `num_samples = current_idx` (`feature_cache.py:83-109`). Must call `close()` — Windows holds file lock otherwise.
+- **Dataset:** read-only mmap `mode="r"` (`feature_cache.py:134-142`). `__getitem__` copies via `arr.copy()` before `torch.from_numpy` to avoid memmap lifetime issues, then optional `auto_dequant_device` fused dequant (`feature_cache.py:147-163`). `iter_batches` does C-level batch slice `mmap[batch_idx]` → `torch.from_numpy(...).to(device)` (`feature_cache.py:165-197`).
+- `close()` releases `_mmap` handles (`feature_cache.py:199-210`) — critical on Windows.
+
+### 4.3 Pixel Cache (`src/tensorcache/pixel_cache.py:25-137`)
+- Raw `uint8` mmap `(N,H,W,C)` (`pixel_cache.py:48-51`). `append_image` accepts `np.ndarray | PIL.Image | torch.Tensor | str|Path` and resizes to `(width,height)` via `BILINEAR` (`pixel_cache.py:54-74`). `PixelCacheDataset.__getitem__` returns `torch.uint8 [H,W,C]` with `arr.copy()` (`pixel_cache.py:122-129`).
+
+### 4.4 Fused Ops (`src/tensorcache/fused_ops.py:1-233`)
+- **Requires Triton + CUDA/ROCm** — hard import `import triton` at top (`fused_ops.py:17-18`) will fail on Windows/CPU. Guard imports or make optional if editing.
+- Three kernels: `_fused_quant_kernel` (`fused_ops.py:24-52`), `_fused_dequant_kernel` (`fused_ops.py:82-96`), `_fused_dequant_matmul_kernel` (`fused_ops.py:119-184`). `FusedDequantLinear` (`fused_ops.py:187-233`) fuses `Dequant + GEMM + bias` in registers — zero intermediate VRAM.
+- Tests skip if `!torch.cuda.is_available()` (`tests/test_fused_ops.py:15,32`).
+
+### 4.5 Prefetcher & Streamer
+- `AsyncGPUPrefetcher` (`src/tensorcache/prefetcher.py:11-67`) — wraps any `DataLoader`, uses `torch.cuda.Stream` + `non_blocking=True` double buffer. `device.type in (cuda,hip)` check; `stream=None` on CPU.
+- `ZeroCopyTensorStreamer` (`src/tensorcache/streamer.py:19-163`) — pinnned CPU buffers `pin_memory=True` + GPU ring buffers `out_bf16_0/1` (`streamer.py:62-77`), `np.copyto` into pinned, async `copy_(non_blocking=True)`, `wait_stream`, dequant into ring target (`streamer.py:96-117`). Fixed ~45 MB footprint. `close()` synchronizes and deletes buffers (`streamer.py:129-163`).
+
+## 5. Conventions & Pitfalls
+
+- **Paths:** Windows dev machine (`win32`, `C:\Users\armor\Desktop\AI pipeline\...`). Use `pathlib.Path`, `os.path`, never hardcode `/`. Prefix handling: `str(prefix)+"_int8.bin"` etc. (`feature_cache.py:48-50`).
+- **Dtype discipline:** Features are `bfloat16` throughout; scales are BF16 on wire; raw pixels are `uint8`. Don't silently upcast to FP32 except in metrics (`analyze_compression_error.py:170-205`).
+- **Triton fallback:** Any change to `codec.py` dequant must keep both Triton and PyTorch fallback paths bit-identical. Test on CPU.
+- **Memmap lifecycle:** Always provide `close()` and call it in tests (`tests/test_codec.py:78-80,102-104`). On Windows, open mmap prevents deletion.
+- **No dynamic allocations in hot path:** Streamer/prefetcher are designed for zero allocation — avoid `torch.empty` inside loops without `out_buffer`.
+- **Benchmark thresholds:** Rel RMSE `<1.0%` is the pass criterion (`tests/test_codec.py:36,51`, `tests/test_fused_ops.py:28`). Real Block-32 RMSE ~0.54% vs Naive FP8 ~2.6-5.2% (`compression_benchmark_results.json:126-148`, `190-201`).
+- **Security:** `tar.extractall` in `download_test_dataset.py:28` — keep as is for now but don't expand without validation if hardening.
+- **Formatting:** No enforced formatter; follow existing style (4-space indent, `from __future__ import annotations`, type hints).
+
+## 6. Editing Guidelines for Agents
+
+- Prefer `Read`/`Edit` over `bash` for files; use `bash` only for `pytest`, `python`, `git` ops.
+- Verify with `pytest tests/test_codec.py -v` after codec/cache edits; run `python tests/test_codec.py` as quick sanity.
+- If touching `fused_ops.py`, guard `import triton` with try/except like `codec.py:13-19` or ensure CI skips gracefully.
+- When adding new cache formats, bump `meta.json` fields and handle backward compat in `FeatureCacheDataset.__init__`.
+- Don't commit `data/*.tgz`, `*_int8.bin`, `*_scales.bin`, `*_pixels.bin`, `__pycache__`.
+- Keep `src/tensorcache/__init__.py:1-42` exports in sync when adding public symbols (update `__all__`).
+
+## 7. Useful References
+
+- API docs & benchmarks: `README.md:1-86`
+- Public API: `src/tensorcache/__init__.py:5-42`
+- Error metrics: `analyze_compression_error.py:170-205`, `experiment_blockwise_int8.py:20-40`
+- Compression tables: `compression_benchmark_results.json:2-216` (pixel + feature)
+- Dataset prep: `download_test_dataset.py:7-44`
+
+## 8. Agent Context — Current Environment
+
+- **OS:** Windows (`win32`), PowerShell 7+ (`pwsh`), `workdir` param preferred over `cd`.
+- **Python:** `>=3.9`, `torch>=2.0` installed; Triton unavailable on Windows (fallback path active).
+- **No git repo** at workspace root (`Is directory a git repo: no`) — `git` commands will fail until `git init`.
+- **Today:** 2026-08-29 (use for searches).

tcache-0.2.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 TensorCache Authors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

tcache-0.2.0/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.5
+Name: tcache
+Version: 0.2.0
+Summary: Ultra-fast, high-fidelity block-wise INT8 feature & pixel cache engine for ML pipelines
+Author: TensorCache Authors
+License: Apache-2.0
+License-File: LICENSE
+Keywords: caching,cnn,compression,cuda,gpu,int8,pytorch,quantization,rocm,triton,vit
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: torch>=2.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: rich>=13.0.0; extra == 'dev'
+Requires-Dist: tabulate>=0.9.0; extra == 'dev'
+Provides-Extra: fast
+Requires-Dist: blosc2>=2.0.0; extra == 'fast'
+Requires-Dist: safetensors>=0.4.0; extra == 'fast'
+Requires-Dist: triton>=2.0.0; (platform_system != 'Windows') and extra == 'fast'
+Requires-Dist: zstandard>=0.20.0; extra == 'fast'
+Description-Content-Type: text/markdown
+
+# TensorCache ⚡
+
+**Ultra-fast, high-fidelity block-wise INT8 feature & pixel cache engine for PyTorch.**
+
+`tensorcache` eliminates two bottlenecks:
+1. **Feature Cache Bloat:** `AMO-BQ` asymmetric MSE-optimal `G32` `1.09B` `1.83x` vs BF16 `0.47%` `rel RMSE` (`sym 1.06B 0.54%`) — near `G16` floor `0.39%`.
+2. **JPEG/PNG CPU Decode:** Zero-copy `mmap` + `GPU` stream prefetch `>2,000 MB/s`, ring-buffer `6.8MB` `VRAM` `batch8 128x768`.
+
+---
+
+## 🚀 Key Features
+
+* **AMO-BQ (Asymmetric MSE-Optimal, G32):** `min-max + zp uint8 + 48×` clipping search `[0.95,1.10]` — `0.47%` `hetero 0.54%` vs `sym 0.74%` (`-26%`), `per-token 1.73%` (`cache\dtype_comparison.json`). Presets `fast/balanced/accurate/max`.
+* **Microsecond Dequant:** `Triton` `(q-zp)*scale -> BF16` `0.06ms 5.4M 337GB/s` (`codec.py:40`), `FusedDequantLinear` `0` intermediate `fused_ops.py:119`.
+* **Minimal VRAM:** `q 5.22MB + scales 0.32MB + zp 0.16MB + out 10.45MB` `5.4M`; `Streamer` fixed `~43MB` `double-buffer` `out_buffer` reuse, `G64` halves `scales/zp`.
+* **Cross-Platform:** `CUDA`/`ROCm` `Triton` else `PyTorch` fallback, `Windows` `mmap` safe `close()`.
+* **CLI + Python one-liners:** `tc.compress` / `tc.benchmark_tensor` / `tensorcache benchmark`.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install tensor-cache          # PyPI (import tensorcache)
+pip install -e .          # dev
+pip install -e ".[fast]"  # triton+zstd+blosc2 (Linux)
+```
+
+---
+
+## ⚡ Quick Start
+
+### 1. In-Memory (one-liners)
+```python
+import torch, tensorcache as tc
+
+x = torch.randn(16,446,768, dtype=torch.bfloat16, device="cuda")
+
+# AMO-BQ presets: fast (16,0.95-1.05) 6.9ms 0.49%, balanced (32,0.95-1.05) 13ms 0.478% (default), accurate (48,0.95-1.10) 49ms 0.473%
+q,s,zp,shape = tc.compress(x, mode="balanced")  # or "fast"/"accurate"/"max"/"sym"/"adaptive"
+rec = tc.decompress(q,s,shape,zp)               # <0.1ms BF16
+tc.benchmark_tensor(x)                          # rich table
+tc.estimate_compression(x.shape, group_size=32) # 1.09375 B 1.83x
+tc.auto_select_mode(x, target_rmse=0.5)         # -> "balanced"
+tc.help()                                       # python help
+
+# Codec object
+codec = tc.BlockwiseInt8Codec(group_size=32, amo_bq=True, amo_mode="balanced")
+print(codec) # G=32, amo_bq=balanced 1.0938B 1:1.83x
+```
+
+### 2. Feature Cache to Disk (mmap)
+```python
+import tensorcache as tc
+from torch.utils.data import DataLoader
+
+# Write (amo_bq, G32 default balanced, G64 for minimal VRAM 1.046B 0.55%)
+writer = tc.FeatureCacheWriter("./cache/dinov3", 10000,446,768, group_size=32, amo_bq=True, amo_mode="balanced")
+writer.append(x) # [seq,dim] or [B,seq,dim]
+writer.close()   # writes _int8.bin (uint8) _scales.bin _zp.bin _meta.json
+
+# Load
+ds = tc.FeatureCacheDataset("./cache/dinov3") # -> (q uint8, s BF16, zp uint8)
+ds = tc.FeatureCacheDataset("./cache/dinov3", auto_dequant_device="cuda") # -> BF16 directly
+for q,s,zp in tc.AsyncGPUPrefetcher(DataLoader(ds,batch_size=256,pin_memory=True), device="cuda"):
+    batch = tc.dequantize_int8_amo_bq(q,s,zp, shape, group_size=32)
+
+# Minimal VRAM streamer (fixed ~43MB, zero alloc)
+streamer = tc.ZeroCopyTensorStreamer("./cache/dinov3", batch_size=32, device="cuda")
+for batch in streamer: # BF16 [B,seq,dim] from ring buffer out_bf16_0/1
+    train(batch)
+streamer.close()
+
+# Fused head (no BF16 intermediate)
+from tensorcache import FusedDequantLinear
+head = FusedDequantLinear(768, num_classes, group_size=32).cuda()
+logits = head(q, s) # dequant+GEMM in regs
+```
+
+### 3. CLI
+```bash
+python -m tensorcache info
+python -m tensorcache benchmark --shape 16,446,768 --device cuda
+python -m tensorcache cache-info --prefix ./cache/dinov3
+python -m tensorcache compress-demo --shape 4,197,768 --mode balanced
+tensorcache --help
+```
+
+---
+
+## 📊 Benchmark (DINOv3 ViT-Base, 5.4M randn + RSNA hetero)
+
+| Format | B/elem | vs BF16 | rel RMSE |Outlier 0.1%| Dequant |
+|---|:---:|---|:---:|:---:|---|
+| Raw BF16 |2.00|1.00x|0.167%|0.119%|—|
+| Naive FP8 E5M2 |1.00|2.00x|5.24%|6.81%|—|
+| Naive FP8 E4M3 |1.00|2.00x|2.63%|3.19%|—|
+| MXFP8 G32 |1.09|1.83x|2.38%|—|—|
+| **Sym G32** |**1.06**|**1.88x**|**0.540%**|**0.11%**|**0.036ms 435GB/s**|
+| **AMO-BQ fast G32** |1.093|1.83x|0.490%|0.15%|0.06ms 337GB/s|
+| **AMO-BQ balanced G32** |**1.093**|**1.83x**|**0.478%**|0.15%|13ms quant|
+| **AMO-BQ accurate G32** |1.093|1.83x|0.473%|—|49ms quant|
+| **AMO-BQ G16** |1.187|1.68x|**0.395%**|—|13ms|
+| **Sym G64** |1.046|1.91x|0.55%|—|—|
+
+---
+
+## 📜 License
+Apache 2.0 — see `LICENSE`.

tcache-0.2.0/README.md

@@ -0,0 +1,109 @@
+# TensorCache ⚡
+
+**Ultra-fast, high-fidelity block-wise INT8 feature & pixel cache engine for PyTorch.**
+
+`tensorcache` eliminates two bottlenecks:
+1. **Feature Cache Bloat:** `AMO-BQ` asymmetric MSE-optimal `G32` `1.09B` `1.83x` vs BF16 `0.47%` `rel RMSE` (`sym 1.06B 0.54%`) — near `G16` floor `0.39%`.
+2. **JPEG/PNG CPU Decode:** Zero-copy `mmap` + `GPU` stream prefetch `>2,000 MB/s`, ring-buffer `6.8MB` `VRAM` `batch8 128x768`.
+
+---
+
+## 🚀 Key Features
+
+* **AMO-BQ (Asymmetric MSE-Optimal, G32):** `min-max + zp uint8 + 48×` clipping search `[0.95,1.10]` — `0.47%` `hetero 0.54%` vs `sym 0.74%` (`-26%`), `per-token 1.73%` (`cache\dtype_comparison.json`). Presets `fast/balanced/accurate/max`.
+* **Microsecond Dequant:** `Triton` `(q-zp)*scale -> BF16` `0.06ms 5.4M 337GB/s` (`codec.py:40`), `FusedDequantLinear` `0` intermediate `fused_ops.py:119`.
+* **Minimal VRAM:** `q 5.22MB + scales 0.32MB + zp 0.16MB + out 10.45MB` `5.4M`; `Streamer` fixed `~43MB` `double-buffer` `out_buffer` reuse, `G64` halves `scales/zp`.
+* **Cross-Platform:** `CUDA`/`ROCm` `Triton` else `PyTorch` fallback, `Windows` `mmap` safe `close()`.
+* **CLI + Python one-liners:** `tc.compress` / `tc.benchmark_tensor` / `tensorcache benchmark`.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install tensor-cache          # PyPI (import tensorcache)
+pip install -e .          # dev
+pip install -e ".[fast]"  # triton+zstd+blosc2 (Linux)
+```
+
+---
+
+## ⚡ Quick Start
+
+### 1. In-Memory (one-liners)
+```python
+import torch, tensorcache as tc
+
+x = torch.randn(16,446,768, dtype=torch.bfloat16, device="cuda")
+
+# AMO-BQ presets: fast (16,0.95-1.05) 6.9ms 0.49%, balanced (32,0.95-1.05) 13ms 0.478% (default), accurate (48,0.95-1.10) 49ms 0.473%
+q,s,zp,shape = tc.compress(x, mode="balanced")  # or "fast"/"accurate"/"max"/"sym"/"adaptive"
+rec = tc.decompress(q,s,shape,zp)               # <0.1ms BF16
+tc.benchmark_tensor(x)                          # rich table
+tc.estimate_compression(x.shape, group_size=32) # 1.09375 B 1.83x
+tc.auto_select_mode(x, target_rmse=0.5)         # -> "balanced"
+tc.help()                                       # python help
+
+# Codec object
+codec = tc.BlockwiseInt8Codec(group_size=32, amo_bq=True, amo_mode="balanced")
+print(codec) # G=32, amo_bq=balanced 1.0938B 1:1.83x
+```
+
+### 2. Feature Cache to Disk (mmap)
+```python
+import tensorcache as tc
+from torch.utils.data import DataLoader
+
+# Write (amo_bq, G32 default balanced, G64 for minimal VRAM 1.046B 0.55%)
+writer = tc.FeatureCacheWriter("./cache/dinov3", 10000,446,768, group_size=32, amo_bq=True, amo_mode="balanced")
+writer.append(x) # [seq,dim] or [B,seq,dim]
+writer.close()   # writes _int8.bin (uint8) _scales.bin _zp.bin _meta.json
+
+# Load
+ds = tc.FeatureCacheDataset("./cache/dinov3") # -> (q uint8, s BF16, zp uint8)
+ds = tc.FeatureCacheDataset("./cache/dinov3", auto_dequant_device="cuda") # -> BF16 directly
+for q,s,zp in tc.AsyncGPUPrefetcher(DataLoader(ds,batch_size=256,pin_memory=True), device="cuda"):
+    batch = tc.dequantize_int8_amo_bq(q,s,zp, shape, group_size=32)
+
+# Minimal VRAM streamer (fixed ~43MB, zero alloc)
+streamer = tc.ZeroCopyTensorStreamer("./cache/dinov3", batch_size=32, device="cuda")
+for batch in streamer: # BF16 [B,seq,dim] from ring buffer out_bf16_0/1
+    train(batch)
+streamer.close()
+
+# Fused head (no BF16 intermediate)
+from tensorcache import FusedDequantLinear
+head = FusedDequantLinear(768, num_classes, group_size=32).cuda()
+logits = head(q, s) # dequant+GEMM in regs
+```
+
+### 3. CLI
+```bash
+python -m tensorcache info
+python -m tensorcache benchmark --shape 16,446,768 --device cuda
+python -m tensorcache cache-info --prefix ./cache/dinov3
+python -m tensorcache compress-demo --shape 4,197,768 --mode balanced
+tensorcache --help
+```
+
+---
+
+## 📊 Benchmark (DINOv3 ViT-Base, 5.4M randn + RSNA hetero)
+
+| Format | B/elem | vs BF16 | rel RMSE |Outlier 0.1%| Dequant |
+|---|:---:|---|:---:|:---:|---|
+| Raw BF16 |2.00|1.00x|0.167%|0.119%|—|
+| Naive FP8 E5M2 |1.00|2.00x|5.24%|6.81%|—|
+| Naive FP8 E4M3 |1.00|2.00x|2.63%|3.19%|—|
+| MXFP8 G32 |1.09|1.83x|2.38%|—|—|
+| **Sym G32** |**1.06**|**1.88x**|**0.540%**|**0.11%**|**0.036ms 435GB/s**|
+| **AMO-BQ fast G32** |1.093|1.83x|0.490%|0.15%|0.06ms 337GB/s|
+| **AMO-BQ balanced G32** |**1.093**|**1.83x**|**0.478%**|0.15%|13ms quant|
+| **AMO-BQ accurate G32** |1.093|1.83x|0.473%|—|49ms quant|
+| **AMO-BQ G16** |1.187|1.68x|**0.395%**|—|13ms|
+| **Sym G64** |1.046|1.91x|0.55%|—|—|
+
+---
+
+## 📜 License
+Apache 2.0 — see `LICENSE`.