quantbench-engine 0.1.0__tar.gz

quantbench_engine-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --all-groups
+
+      - name: Run linter
+        run: uv run ruff check .
+
+      - name: Run test suite with coverage
+        run: uv run pytest

quantbench_engine-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,43 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  build:
+    name: Build package
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+      - name: Build sdist and wheel
+        run: uv build
+      - name: Store distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+  publish-pypi:
+    name: Publish to PyPI
+    needs: [build]
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/quantbench
+    permissions:
+      id-token: write
+    steps:
+      - name: Download distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          skip-existing: true

quantbench_engine-0.1.0/.gitignore

@@ -0,0 +1,52 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# IDE & Editor
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Testing & Coverage
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+.ruff_cache/
+
+# Artifacts & Local Weights
+*.safetensors
+*.bin
+*.pt
+*.onnx
+models/
+cache/
+results/
+decisions.md
+flow.md

quantbench_engine-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,146 @@
+# Contributing to quantbench ⚖️
+
+Thank you for your interest in contributing to `quantbench`!
+
+`quantbench` is an open-source, first-principles quantization engine, bit-packing runtime, 2:4 sparsity toolkit, and Quality-vs-Speed Pareto evaluation benchmark written in pure Python and PyTorch. We welcome contributions that improve mathematical transparency, expand sub-byte compression layouts, add new post-training calibration algorithms, and bridge inference serving systems with precision engineering.
+
+---
+
+## 🛠️ Development Setup
+
+We use [uv](https://docs.astral.sh/uv/) for Python environment and dependency management.
+
+1. **Clone the repository**:
+   ```bash
+   git clone https://github.com/Prajwal7-2002/quantbench.git
+   cd quantbench
+   ```
+
+2. **Sync dependencies and virtual environment**:
+   ```bash
+   uv sync
+   ```
+   This automatically installs all runtime, benchmark, and development dependencies into a isolated `.venv`.
+
+---
+
+## 🧪 Testing Protocol
+
+We maintain high test coverage across mathematical primitives, sub-byte bit manipulation, neural network layers, and calibration algorithms.
+
+1. **Run the entire test suite**:
+   ```bash
+   uv run pytest
+   ```
+
+2. **Run with coverage report**:
+   ```bash
+   uv run pytest --cov=quantbench --cov-report=term-missing
+   ```
+
+3. **Run a specific test category**:
+   ```bash
+   # Test bit-packing roundtrip invariants
+   uv run pytest tests/test_packing.py
+
+   # Test calibration algorithms (SmoothQuant, AWQ, GPTQ)
+   uv run pytest tests/test_algorithms.py
+
+   # Test drop-in QuantLinear layers
+   uv run pytest tests/test_quant_linear.py
+   ```
+
+### Invariant Testing Rules
+When contributing new quantization schemes or packing runtimes, you **must** include tests verifying:
+- **Roundtrip Fidelity**: $x = \text{unpack}(\text{pack}(x))$ for all valid integer representations.
+- **Mathematical Invariance**: Continuous exactness proofs (e.g. SmoothQuant $(X s^{-1})(s W) = XW$ within $10^{-5}$ drift).
+- **Error Bounds**: Maximum absolute error bounded by $\frac{\Delta}{2}$ for uniform quantizers.
+- **Extreme Inputs**: Subnormal values, exact zeros, all-positive, and all-negative tensor ranges.
+
+---
+
+## 🎨 Code Quality & Style
+
+We enforce modern Python conventions, strict type hints, and clean code formatting via [ruff](https://docs.astral.sh/ruff/):
+
+1. **Lint and format checks**:
+   ```bash
+   uv run ruff check .
+   ```
+
+2. **Auto-fix formatting**:
+   ```bash
+   uv run ruff format .
+   ```
+
+### Architectural Principles
+
+1. **Zero Compiled C++/CUDA Dependencies**:
+   `quantbench` is designed for radical algorithmic transparency. Core quantization logic and bit-packing must run in pure Python and PyTorch without requiring NVCC or compiler toolchains.
+2. **Vectorized PyTorch Operations**:
+   Avoid Python `for` loops across tensor dimensions. Sub-byte bit-packing must be implemented via vectorized tensor operations (`&`, `|`, `>>`, `<<`) on `torch.uint8` containers.
+3. **Hardware Agnostic**:
+   All algorithms must run seamlessly on consumer CPUs, Apple Silicon, and CUDA GPUs without hardcoded device assumptions.
+4. **Honest Pareto Benchmarks**:
+   Never cherry-pick perplexity numbers. Always evaluate against standard validation corpora (WikiText-2, C4) using standardized sliding-window stride lengths.
+
+---
+
+## 🎯 High-Impact Areas for Contribution
+
+We actively welcome contributions in the following domains:
+
+### 1. New Sub-Byte Formats & Precision Schemes (`src/quantbench/quant/`)
+- [ ] **NVIDIA Blackwell NVFP4 Emulation**: Bit-exact emulation of 4-bit floating point formats (E2M1) with micro-scaling factors.
+- [ ] **2-bit Non-Linear Quantization**: Quantization via Lloyd-Max optimal non-uniform codebooks.
+- [ ] **NF4 (NormalFloat4)**: Quantile-based optimal representation for zero-mean normally distributed weights.
+
+### 2. Quantized KV-Cache Runtimes
+- [ ] **INT4 / FP8 KV-Cache Page Allocator**: Interfacing with `mini-infer`'s paged memory manager to benchmark memory savings during multi-turn generation.
+- [ ] **Dynamic Per-Token Key/Value Quantization**: Asymmetric scaling per token vector along the sequence dimension.
+
+### 3. Advanced Calibration & Optimization (`src/quantbench/quant/algorithms/`)
+- [ ] **SqueezeLLM Clustering**: Non-uniform sensitivity-weighted k-means clustering with sparse outlier matrices.
+- [ ] **QuIP# Vector Quantization**: Randomized Hadamard transform pre-processing to eliminate activation outliers.
+- [ ] **HQQ (Half-Quadratic Quantization)**: Closed-form optimization without calibration data.
+
+### 4. Real-World Model Support (`src/quantbench/nn/`)
+- [ ] **GGUF Dequantization Loader**: Direct parsing and loading of `.gguf` quantized tensors into `QuantLinear` modules.
+- [ ] **SafeTensors Model Converter**: Auto-convert and shard Hugging Face SafeTensors checkpoints directly into disk-mapped quantized weights.
+
+---
+
+## 📬 Pull Request Rules & Submission Process
+
+To keep `quantbench` reliable and maintainable, all contributions must follow these conventions:
+
+### 1. Branch Naming Conventions
+Create a descriptive branch off `main`:
+* `feat/short-feature-name` (e.g. `feat/nvfp4-packing`)
+* `fix/short-bug-description` (e.g. `fix/ternary-base3-rounding`)
+* `perf/short-perf-description` (e.g. `perf/vectorized-hessian-chol`)
+* `docs/short-doc-description` (e.g. `docs/smoothquant-proof-derivation`)
+* `test/short-test-description` (e.g. `test/gptq-corner-cases`)
+
+### 2. Conventional Commit Standards
+Commits must follow the Conventional Commits specification:
+```
+feat(quant): add NVIDIA NVFP4 analytical emulator and bit-packing runtime
+fix(packing): correct uint8 mask overflow in sparse24 index condensation
+docs(guide): add mathematical derivation of optimal brain surgeon Hessian update
+test(eval): add sliding window perplexity regression test on small corpus
+```
+
+### 3. PR Verification Checklist
+Before submitting a PR, ensure:
+- [ ] `uv run pytest` passes 100% of tests.
+- [ ] `uv run ruff check .` reports 0 errors.
+- [ ] Test coverage does not decrease.
+- [ ] Relevant documentation or examples are updated.
+- [ ] Single author attribution is preserved.
+
+---
+
+## 📜 License
+
+By contributing to `quantbench`, you agree that your contributions will be licensed under the [Apache 2.0 License](LICENSE).

quantbench_engine-0.1.0/DEVLOG.md

@@ -0,0 +1,72 @@
+# DEVLOG: quantbench
+
+## 2026-10-26: Project Inception & Phase 1 Mathematical Primitives
+
+- Initialized Project #5 `quantbench` (Unified Quantization Engine & Quality-vs-Speed Pareto Benchmark).
+- Architecture decision: Unified from-scratch quantization algorithms (`quantbench.quant`), drop-in PyTorch modules (`quantbench.nn`), and sliding-window Pareto evaluation (`quantbench.eval`, `quantbench.bench`) in a single zero-dependency-friction repository.
+- Initialized Apache 2.0 license, `.gitignore`, and `pyproject.toml` with `hatchling` build backend and `uv` package manager.
+- Implemented Phase 1 Mathematical Primitives:
+  - Affine asymmetric quantization ($s, z$) and symmetric AbsMax quantization ($s, z=0$) with dynamic per-tensor and per-channel scaling along arbitrary dimensions.
+  - Provable clamp invariants verifying $|x - \hat{x}| \le \frac{s}{2}$.
+  - Group-wise / block-wise quantization for group sizes $G \in \{32, 64, 128\}$ with arbitrary tensor padding and dequantization reconstruction.
+  - Bit-level simulation of OCP FP8 E4M3FN (range $\pm 448$, 3 mantissa bits) and E5M2 (range $\pm 57344$, 2 mantissa bits).
+  - 100% test coverage across mathematical invariants and error bounds.
+
+## 2026-10-26: Phase 2 — Integer Bit-Packing Runtimes (Sub-Byte Storage)
+
+- Implemented vectorized sub-byte bit-packing runtimes in `quantbench.quant.packing`:
+  - `int4_pack.py`: Vectorized bit-shift packing of two INT4 values (unsigned $[0, 15]$ or signed two's complement $[-8, 7]$) into a single `uint8` byte container: `byte = (v0 & 0x0F) | ((v1 & 0x0F) << 4)`. Full support for odd dimension length padding and exact shape restoration.
+  - `ternary_pack.py`: BitNet 1.58b packing supporting both hardware-friendly 2-bit SIMD packing (4 weights/byte using biased offset $u = v + 1 \in \{0, 1, 2\}$) and theoretical maximal density Radix-3 base-3 packing (5 weights/byte via $3^5 = 243 < 256$).
+  - `sparse_pack.py`: NVIDIA Ampere/Ada/Hopper 2:4 structured sparsity layout. Automated row pruning, value condensation to $50\%$ storage ($M, N/2$), and 2-bit metadata column index packing into 4-bit nibbles ($M, N/8$). Decompression scatter preserves exact zeros and non-zero positions.
+- Added CLI command `pack-inspect` to visualize sub-byte bit layouts and theoretical memory compression directly from the terminal.
+- 22/22 unit tests passing with 93% test coverage in 2.18s. All Ruff checks passing.
+
+## 2026-10-26: Phase 3 — Drop-in Neural Layers (`QuantLinear`) & Model Conversion
+
+- Created `QuantLinear` (`quantbench.nn.quant_linear`):
+  - A drop-in replacement for `torch.nn.Linear` storing compressed integer weights inside persistent buffers (`self.register_buffer("packed_weight", ...)`).
+  - Supports INT4 (groupwise and per-channel), INT8, BitNet Ternary, and 2:4 Structured Sparsity.
+  - Implements dynamic vector dequantization in `forward(x)`.
+  - Overrides `_load_from_state_dict` to dynamically bind serialized buffer tensors during checkpoint restoration.
+  - Computes exact layer parameter memory bytes and compression factor relative to FP16.
+- Created `convert_model_to_quant` (`quantbench.nn.model_convert`):
+  - In-place recursive model converter traversing PyTorch module trees.
+  - Swaps targeted linear layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`) while cleanly preserving sensitive layers (`lm_head`).
+  - Reports layer conversion counts, memory reduction statistics, and elapsed conversion duration.
+## 2026-10-26: Phase 4 — Advanced Calibration Algorithms (SmoothQuant, AWQ, GPTQ)
+
+- Created `quantbench.quant.algorithms`:
+  - `smoothquant.py`: Implemented activation outlier channel profiling and per-channel migration scale computation ($Y = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W)$). Mathematical verification confirms exact continuous output invariance before discrete rounding.
+  - `awq.py`: Implemented Activation-aware Weight Quantization with mean activation channel profiling and grid-search over scale candidate $\alpha \in [0, 1]$ to minimize Frobenius reconstruction error $\| W X - \hat{W} X \|_F^2$.
+  - `gptq.py`: Implemented Accurate Post-Training Quantization for Generative Pre-trained Transformers using empirical activation Hessian ($H = 2 X X^T$), diagonal damping regularization ($H_{\text{damp}} = H + \lambda \cdot \text{mean}(\text{diag}(H)) \cdot I$), Cholesky decomposition inversion ($H^{-1} = L^{-T} L^{-1}$), Optimal Brain Surgeon (OBS) coordinate updates, and vectorized block GEMM updates ($W_{:, j+B:} \mathrel{-}= E_{\text{block}} \cdot [H^{-1}]_{j:j+B, j+B:}$).
+- Added comprehensive unit tests in `tests/test_algorithms.py`:
+  - Outlier profiling and continuous mathematical invariance testing.
+  - AWQ reconstruction error reduction testing on synthetic salient channels.
+  - Hessian symmetry, positive semi-definiteness, and GPTQ OBS error reduction testing.
+- 39/39 unit tests passing with 93% test coverage across the repository. All Ruff checks passing.
+
+## 2026-10-26: Phase 5 & 6 — Sliding-Window Perplexity, SQNR & Pareto Benchmarking
+
+- Created `quantbench.eval`:
+  - `sqnr.py`: Implemented Signal-to-Quantization-Noise Ratio ($10 \log_{10}(P_{\text{signal}} / P_{\text{noise}})$) in decibels (dB) for layer-level numerical fidelity tracking without forward inference overhead.
+  - `perplexity.py`: Implemented statistically rigorous sliding-window causal language model evaluation engine. Strided contexts ($S < L$) eliminate context boundary collapse by evaluating loss strictly on trailing $S$ tokens.
+- Created `quantbench.bench.pareto`:
+  - `ParetoBenchmark`: Automated sweep engine mapping (Compression Ratio, Footprint MB, Delta PPL, SQNR dB) across FP16 baseline, INT8, INT4 (G=128, G=64), BitNet Ternary, and 2:4 Structured Sparsity.
+  - Formats GitHub Markdown tables and ASCII terminal scatter plots.
+- Added CLI commands `eval` and `pareto` in `quantbench.cli`.
+- Added unit tests in `tests/test_eval_and_pareto.py`:
+  - 47/47 unit tests passing across entire repo with 95% test coverage in 3.33s. All Ruff checks passing.
+
+## 2026-10-26: Phase 7 — Public Learning Guide & Packaging Complete
+
+- Authoring of `GUIDE.md`:
+  - Comprehensive educational engineering guide detailing arithmetic intensity rooflines, zero-preserving affine invariant derivation, banker's rounding drift, OCP FP8 specifications, vectorized sub-byte packing runtimes, SmoothQuant, AWQ, GPTQ second-order Hessian compensation, persistent buffer architecture, and dual-metric Pareto evaluation.
+- Validated PyPI distribution packaging with `uv build`:
+  - Successfully produced source distribution (`.tar.gz`) and platform-independent wheel (`.whl`).
+- Repository verified across 47 automated test suites with 95% line coverage. All Ruff checks passing.
+- All 7 staged roadmap milestones in `README.md` completed.
+
+
+
+
+

quantbench_engine-0.1.0/GUIDE.md

@@ -0,0 +1,193 @@
+# The Comprehensive Guide to LLM Quantization & Hardware Sparsity ⚡
+**From First-Principles Mathematical Formulations to Vectorized Bit-Packing Runtimes and Pareto Evaluation**
+
+*Author: Prajwal Pujari ([@Prajwal7-2002](https://github.com/Prajwal7-2002))*  
+*Project: [quantbench](https://github.com/Prajwal7-2002/quantbench)*
+
+---
+
+## 1. Introduction: Why Quantization is Mandatory in LLM Serving
+
+When serving modern Generative Large Language Models (LLMs), the primary inference bottleneck during autoregressive token generation (the decoding phase) is **memory bandwidth, not compute capacity**.
+
+### The Arithmetic Intensity Bottleneck
+In the autoregressive decode loop, the model processes tokens with batch size $B=1$ and query length 1:
+* The GEMV (General Matrix-Vector multiplication) arithmetic intensity is:
+  $$\text{Arithmetic Intensity} \approx \frac{2 \cdot P \text{ FLOPs}}{2 \cdot P \text{ Bytes}} = 1.0 \text{ FLOP/Byte} \quad (\text{in FP16})$$
+* On an NVIDIA A100 SXM (312 TFLOPS FP16 Tensor Core, 2,039 GB/s HBM2e bandwidth), the hardware ridge point is:
+  $$\text{Ridge Point} = \frac{312 \times 10^{12} \text{ FLOP/s}}{2,039 \times 10^9 \text{ B/s}} \approx 153 \text{ FLOPs/Byte}$$
+* Since $1.0 \ll 153$, the GPU Tensor Cores sit **99.3% idle**, starved for weights streaming from high-bandwidth memory (HBM).
+
+### Memory Bandwidth Reduction = Linear Decode Acceleration
+By compressing weights from 16-bit to 4-bit (INT4):
+* Weight memory footprint is cut by $\mathbf{4.0\times}$.
+* The byte volume transferred over the memory bus per generated token drops by $\mathbf{4.0\times}$.
+* Autoregressive decoding throughput increases nearly linearly with the compression factor!
+
+---
+
+## 2. Mathematical Foundations of Quantization
+
+### 2.1. The Zero-Preserving Affine Invariant
+Quantization maps continuous real numbers $x \in [\alpha, \beta]$ to a discrete integer grid $q \in [q_{\min}, q_{\max}]$.
+
+$$\text{scale} = s = \frac{\beta - \alpha}{q_{\max} - q_{\min}}$$
+$$\text{zero\_point} = z = \text{round}\left( - \frac{\alpha}{s} \right) + q_{\min}$$
+$$q = \text{clamp}\left( \text{round}\left( \frac{x}{s} \right) + z, \; q_{\min}, \; q_{\max} \right)$$
+$$\hat{x} = (q - z) \cdot s$$
+
+#### Why Real $0.0$ Must Map Exactly to an Integer Zero-Point $z$:
+In neural network operations:
+1. **Padded Tokens**: Sequences of varying lengths use zero-padding ($x = 0.0$).
+2. **ReLU / SwiGLU Zeros**: Non-linear activations introduce exact zero activations.
+3. If $x = 0.0$ did not map to an integer $z$:
+   $$\hat{x} = (\text{round}(0 + z_{\text{float}}) - z_{\text{float}}) \cdot s \ne 0.0$$
+   Padding tokens would leak non-zero bias signals across self-attention projections, corrupting transformer output logits!
+
+### 2.2. Symmetric AbsMax Quantization
+When data distributions are symmetric around zero (e.g. weights centered at mean $0$):
+* Restrict zero point to strictly $z = 0$.
+* For signed $b$-bit integer ($q_{\max} = 2^{b-1} - 1$):
+  $$s = \frac{\max(|x|)}{2^{b-1} - 1}$$
+  $$q = \text{clamp}\left( \text{round}\left( \frac{x}{s} \right), \; -q_{\max}, \; q_{\max} \right)$$
+  $$\hat{x} = q \cdot s$$
+* In symmetric AbsMax, dequantization simplifies from $(q - z) s$ to pure multiplication $q \cdot s$, eliminating subtraction instructions from GPU GEMM inner loops.
+
+### 2.3. Group-wise / Block-wise Quantization
+Per-tensor quantization uses a single scale $s$ for the entire weight matrix. A single outlier value pushes $s$ up, severely degrading the precision of normal values.
+
+Block-wise quantization slices the weight matrix into independent groups of size $G \in \{32, 64, 128\}$:
+* Each block $k$ computes its own independent scale $s_k$ and zero-point $z_k$.
+* **The Granularity Trade-off**:
+  * Smaller $G$ (e.g. 32): Higher precision, but scales consume $16 / 32 = 0.5$ bits/weight ($12.5\%$ metadata overhead).
+  * Larger $G$ (e.g. 128): Lower metadata overhead ($16 / 128 = 0.125$ bits/weight), but slightly reduced ability to isolate local outliers.
+  * In practice, **$G = 128$ or $G = 64$** represents the Pareto-optimal operating point for LLMs.
+
+---
+
+## 3. Modern FP8 Formats (OCP Standard)
+
+The Open Compute Project (OCP) standardizes two distinct 8-bit floating point formats:
+
+| Specification | OCP FP8 E4M3FN | OCP FP8 E5M2 |
+|---|---|---|
+| **Bit Layout** | 1 sign, 4 exponent, 3 mantissa | 1 sign, 5 exponent, 2 mantissa |
+| **Exponent Bias** | 7 | 15 |
+| **Max Absolute Value** | $\pm 448.0$ | $\pm 57344.0$ |
+| **Mantissa Resolution** | High ($2^{-3} = 0.125$) | Low ($2^{-2} = 0.25$) |
+| **Primary Use-Case** | **Forward GEMM** (Weights & Activations) | **KV Cache, Gradients, Optimizer States** |
+
+### Analytical Emulation on Older GPUs / CPUs
+Hardware lacking native FP8 instructions (e.g. RTX 30-series, Apple Silicon, CPUs) cannot natively execute `torch.float8_e4m3fn`. `quantbench` provides a bit-exact software emulator that analytically simulates exponent bias, mantissa truncation, and saturation bounds:
+```python
+# Analytical bit-level reconstruction
+e = clamp(floor(log2(|x|)) + bias, 0, 2^exp_bits - 1)
+m = round((|x| / 2^(e - bias) - 1.0) * 2^mantissa_bits)
+x_hat = sign(x) * (1.0 + m / 2^mantissa_bits) * 2^(e - bias)
+```
+
+---
+
+## 4. Sub-Byte Vectorized Bit-Packing Runtimes
+
+Modern CPU and GPU memory subsystems are byte-addressable (minimum addressable boundary is **8 bits**). Storing 4-bit, 2-bit, or ternary weights requires sub-byte bit packing into `uint8` containers.
+
+### 4.1. Vectorized INT4 Packing
+* Two 4-bit values $v_0$ and $v_1$ pack into 1 byte:
+  $$\text{byte} = (v_0 \& 0x0F) \mid ((v_1 \& 0x0F) \ll 4)$$
+* Unpacking:
+  $$v_0 = \text{byte} \& 0x0F, \quad v_1 = (\text{byte} \gg 4) \& 0x0F$$
+* **Signed Two's Complement Extension**:
+  If bit 3 is set ($v \ge 8$), sign-extend to negative: $v_{\text{signed}} = v - 16$.
+
+### 4.2. BitNet 1.58b Ternary Packing
+Constrains weights strictly to $\{-1, 0, +1\}$:
+1. **2-bit Biased SIMD Packing (Fastest)**:
+   * Maps $u = v + 1 \in \{0, 1, 2\}$.
+   * 4 weights fit into 1 byte:
+     $$\text{byte} = u_0 \mid (u_1 \ll 2) \mid (u_2 \ll 4) \mid (u_3 \ll 6)$$
+   * Decodes via parallel SIMD bit shifts without integer division.
+2. **Radix-3 Base-3 Packing (Maximum Density)**:
+   * 5 weights fit into 1 byte since $3^5 = 243 \le 255$:
+     $$\text{byte} = u_0 + 3 u_1 + 9 u_2 + 27 u_3 + 81 u_4$$
+   * Achieves theoretical $\log_2(3) \approx 1.585 \text{ bits/weight}$.
+
+### 4.3. NVIDIA Ampere/Ada/Hopper 2:4 Structured Sparsity
+NVIDIA Tensor Cores natively support 2:4 sparsity (in every 4 contiguous elements, at least 2 must be zero):
+1. **Condensed Matrix**: Compresses non-zeros into dense matrix $(M, N/2)$ (50% reduction).
+2. **Metadata Nibbles**: Records 2-bit column indices ($0..3$) for each non-zero element.
+3. Two 4-element blocks (8 dense elements, 4 non-zeros) pack into **1 metadata byte**:
+   $$\text{byte} = (idx_{A,0} \& 3) \mid ((idx_{A,1} \& 3) \ll 2) \mid ((idx_{B,0} \& 3) \ll 4) \mid ((idx_{B,1} \& 3) \ll 6)$$
+4. Metadata overhead is strictly **2 bits per non-zero value** (only 12.5% of FP16 weight size).
+
+---
+
+## 5. Advanced Post-Training Calibration Algorithms
+
+### 5.1. SmoothQuant: Activation Outlier Migration
+* **Insight**: LLM activations have systematic outlier channels ($100\times$ normal values), while weights are smooth.
+* **Invariant Transformation**:
+  $$Y = X W = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W)$$
+* **Scale Derivation**:
+  $$s_j = \frac{\max(|X_j|)^\alpha}{\max(|W_j|)^{1 - \alpha}} \quad (\alpha=0.5 \text{ balances activation and weight})$$
+* Scale vector $s$ is folded into preceding LayerNorm/RMSNorm weights, enabling standard INT8 Tensor Core GEMM.
+
+### 5.2. AWQ: Activation-Aware Weight Quantization
+* **Insight**: Only top 1% salient weight channels preserve perplexity.
+* Salience is determined by mean activation magnitude: $s_X = \frac{1}{N} \sum |X|$.
+* Instead of keeping salient weights in FP16 (which ruins uniform SIMD kernels), AWQ applies a per-channel protection scale:
+  $$s = s_X^\alpha$$
+  where $\alpha \in [0, 1]$ is optimized via grid search on calibration tokens to minimize Frobenius reconstruction error:
+  $$\min_\alpha \| W X - \text{dequantize}(\text{quantize}(W \text{diag}(s))) \text{diag}(s)^{-1} X \|_F^2$$
+
+### 5.3. GPTQ: Second-Order Hessian Compensation
+* Uses second-order Taylor expansion of quantization error:
+  $$E \approx \frac{1}{2} (w - \hat{w})^T H (w - \hat{w}), \quad H = 2 X X^T$$
+* **Damping Regularization**:
+  $$H_{\text{damp}} = H + \lambda \cdot \text{mean}(\text{diag}(H)) \cdot I \quad (\lambda = 0.01)$$
+* Inverts via Cholesky decomposition: $H = L L^T \implies H^{-1} = L^{-T} L^{-1}$.
+* **Optimal Brain Surgeon (OBS) Update**:
+  When rounding column $q$, updates remaining columns $F$:
+  $$\Delta W_{:, F} = - \frac{W_{:, q} - \hat{W}_{:, q}}{[H^{-1}]_{qq}} \cdot [H^{-1}]_{q, F}$$
+* **Block Updates**: Accumulates errors in blocks of $B$ columns (e.g. 128) and performs dense matrix multiplications (GEMM) to update the remaining matrix.
+
+---
+
+## 6. Drop-in PyTorch Architecture & In-Place Conversion
+
+### Why Persistent Buffers Beat `nn.Parameter` for Inference
+1. `nn.Parameter` forces float dtypes and triggers Autograd gradient tracking (`requires_grad=True`).
+2. If touched by an optimizer, it allocates momentum and gradient buffers ($2\times\text{--}3\times$ model size).
+3. `QuantLinear` stores packed weights as persistent buffers (`self.register_buffer("packed_weight", ...)`):
+   * Zero gradient tracking.
+   * Zero optimizer memory overhead.
+   * Automatically serializes into `state_dict()` and `safetensors`.
+4. **Dynamic Checkpoint Hook (`_load_from_state_dict`)**:
+   Overrides PyTorch's hook to register serialized buffer shapes dynamically before strict key validation runs.
+
+---
+
+## 7. Quality-vs-Speed Pareto Benchmarking
+
+### Dual-Metric Evaluation
+1. **Layer-Level SQNR (dB)**:
+   $$\text{SQNR (dB)} = 10 \cdot \log_{10} \left( \frac{\sum |W|^2}{\sum |W - \hat{W}|^2 + \epsilon} \right)$$
+   Diagnoses numerical reconstruction fidelity layer-by-layer without inference passes.
+2. **Model-Level Sliding-Window Perplexity**:
+   Uses context window $L$ with stride $S < L$, evaluating loss strictly on trailing $S$ tokens to eliminate chunk boundary context collapse.
+3. **Pareto Table Output**:
+   Maps configurations across (Compression Ratio, Footprint MB, Delta PPL, SQNR dB).
+
+---
+
+## 8. Summary Comparison of Quantization Schemes
+
+| Scheme | Target Bits | Memory Reduction | Compute Speedup | Accuracy Impact | Recommended Use-Case |
+|---|---|---|---|---|---|
+| **FP16 Baseline** | 16-bit | $1.0\times$ | $1.0\times$ | Baseline | Reference quality |
+| **FP8 (E4M3)** | 8-bit | $2.0\times$ | $1.8\times\text{--}2.0\times$ | Near zero ($\Delta \text{PPL} < 0.01$) | Standard serving on Ada/Hopper |
+| **INT8 AbsMax** | 8-bit | $2.0\times$ | $1.7\times\text{--}1.9\times$ | Near zero ($\Delta \text{PPL} < 0.02$) | Universal CPU/GPU acceleration |
+| **INT4 Blockwise** | 4-bit | $3.2\times\text{--}3.6\times$ | $2.5\times\text{--}3.5\times$ | Low ($\Delta \text{PPL} < 0.15$) | VRAM-constrained edge / consumer GPUs |
+| **AWQ / GPTQ INT4** | 4-bit | $3.5\times$ | $3.0\times$ | Minimal ($\Delta \text{PPL} < 0.05$) | Production 4-bit LLM deployment |
+| **Ampere 2:4 Sparse**| 8-bit eff. | $2.0\times$ | $2.0\times$ math | Negligible ($\Delta \text{PPL} < 0.08$) | Dense-throughput acceleration |
+| **BitNet Ternary** | 1.58-bit | $7.5\times\text{--}8.0\times$ | $5.0\times\text{--}7.0\times$ | Requires fine-tuning | Ultra-low power / specialized silicon |