kernel-lens 1.2.0__tar.gz → 1.3.0__tar.gz

{kernel_lens-1.2.0 → kernel_lens-1.3.0}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: kernel-lens
-Version: 1.2.0
-Summary: A production-grade, multi-backend compiler for Triton kernels (ONNX & TensorRT)
-Author: Compiler Engineer
+Version: 1.3.0
+Summary: Compile PyTorch models with custom @triton.jit kernels into ONNX Runtime and TensorRT C++ plugins
+Author-email: Justin Duc <justin.duc@galiad-research.com>
 License: MIT License
         
         Copyright (c) 2026 Duc Justin
@@ -25,11 +25,23 @@ License: MIT License
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
         
+Project-URL: Homepage, https://just1truc.github.io/kernel_lens/
+Project-URL: Documentation, https://just1truc.github.io/kernel_lens/
+Project-URL: Changelog, https://github.com/Just1truc/kernel_lens/blob/main/CHANGELOG.md
 Project-URL: Source Code, https://github.com/Just1truc/kernel_lens
 Project-URL: Bug Tracker, https://github.com/Just1truc/kernel_lens/issues
+Project-URL: Paper, https://doi.org/10.5281/zenodo.23194523
+Keywords: triton,tensorrt,onnxruntime,compiler,gpu,cuda,kernel,inference,deep-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: POSIX :: Linux
+Classifier: Environment :: GPU :: NVIDIA CUDA
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -46,6 +58,8 @@ Provides-Extra: all
 Requires-Dist: onnxruntime-gpu>=1.17.0; extra == "all"
 Requires-Dist: tensorrt<11.0.0,>=8.6.0; extra == "all"
 Requires-Dist: tensorrt-cu12-libs<11.0.0; extra == "all"
+Provides-Extra: bench
+Requires-Dist: transformers>=4.40.0; extra == "bench"
 Dynamic: license-file
 
 # KernelLens
@@ -100,6 +114,12 @@ compiled_model.run(inputs, backend="onnx"|"tensorrt")
 - **Dynamic grids in C++** — launch-grid expressions are transpiled to C++
   integer math over runtime shapes, so one plugin serves dynamic batch and
   sequence lengths.
+- **Shape-generic engines** — `kl.compile(..., dynamic_shapes=True)` exports the
+  leading dimension as a symbolic axis, so a single engine serves any size in a
+  declared range via a TensorRT optimization profile. Verified on Blackwell
+  (`sm_120a`) and Ampere (`sm_80`): one compile at batch 8 serving batches
+  1-16, bit-exact on both engines. Off by default, because a shape-specialised
+  engine is what the published benchmarks measure.
 - **Zero-copy VRAM binding** — PyTorch `data_ptr()` values are registered
   directly with each engine; no host staging copies.
 - **Blackwell PTX ISA clamping** — a Triton-generated `.version 9.3` header is
@@ -113,11 +133,16 @@ compiled_model.run(inputs, backend="onnx"|"tensorrt")
   dynamic scalars, strided and dynamic inputs, loops, conditionals, nested
   helpers, in-place mutation, 3-D grids, autotune, and fp16/bf16.
 
-> **Honest scope.** KernelLens is research-grade. The documentation clearly
-> separates what is **implemented and measured** from what is **planned**. Two
-> TensorRT paths (in-place aliasing, INT8 dequantisation) are known to be
-> incorrect and are reported rather than hidden. See the
-> [verification page](https://just1truc.github.io/kernel_lens/verification/).
+> **Honest scope.** KernelLens is research-grade, and the results are reported
+> per configuration rather than blended. The composition is bit-exact against
+> native Triton (`0.00e+00`) across the research kernels, the fourteen
+> conformance patterns and the quantized GEMMs, on both backends; two
+> multi-operator fusions land just above zero and are reported as such. Keeping
+> the kernel pays when the baseline leaves the graph unfused and breaks even when
+> it already fuses. The TensorRT-LLM backend is design-only (see
+> [`docs/assets/data/a100/trtllm_backend_design.md`](docs/assets/data/a100/trtllm_backend_design.md)),
+> and the quantized GEMMs are validated on ONNX Runtime only. Full methodology
+> and every raw JSON are under [`docs/assets/data/`](docs/assets/data/).
 
 ## Installation
 
@@ -129,7 +154,16 @@ pip install kernel-lens[all]     # both
 ```
 
 **Requirements:** Linux x86-64, Python ≥ 3.10, CUDA ≥ 12.0, PyTorch ≥ 2.0,
-Triton ≥ 2.1, and `nvcc` + `g++` on `PATH`.
+Triton ≥ 2.1, and `nvcc` + `g++` on `PATH`. TensorRT must be `< 11` (the
+generator emits `IPluginV2DynamicExt`).
+
+**Environment:** `KERNEL_LENS_CACHE_DIR` (put it on a persistent volume so the
+`.so` / `.engine` artifacts survive `kl.load`), `KERNEL_LENS_CCBIN` (host
+compiler for `nvcc` — CUDA toolkits before 12.9 reject GCC 13's exception
+specifiers, so set it to `g++-12` there), `KERNEL_LENS_DEBUG` (or `KL_VERBOSE`),
+`TENSORRT_INCLUDE_DIR`, and `KERNEL_LENS_STREAM_READER=1` to deserialize engines
+above 2 GB from disk rather than from a memory buffer (opt-in; see the
+[changelog](CHANGELOG.md)).
 
 ## Quickstart
 
@@ -168,26 +202,73 @@ compiled = kl.compile(RMSNorm(4096).cuda(), (x,), name="llama3_rmsnorm",
                       backends=["onnx", "tensorrt"])
 out_ort = compiled.run((x,), backend="onnx")
 out_trt = compiled.run((x,), backend="tensorrt")
+
+# One engine for any batch: mark the leading dim dynamic. dynamic_max bounds
+# the optimization profile (default 4x the traced size).
+serving = kl.compile(RMSNorm(4096).cuda(), (x,), name="llama3_rmsnorm_dyn",
+                     backends=["onnx", "tensorrt"], dynamic_shapes=True)
+for b in (1, 2, 4, 8, 16):                 # all through the engine built at b=4
+    serving.run((torch.randn(b, 512, 4096, device="cuda"),), backend="tensorrt")
 ```
 
 ## Results at a glance
 
-Measured on an **RTX 5060 Ti (Blackwell, `sm_120a`)**, CUDA 12.8, PyTorch 2.14,
-Triton 3.8, ONNX Runtime 1.30, TensorRT 10.16. Full tables and methodology are on
-the [results page](https://just1truc.github.io/kernel_lens/#results).
-
-- **Fold + keep, and it pays**: on a two-layer block whose attention is a custom
-  squared-ReLU flash kernel, KernelLens runs it in **7.63 ms**, versus **15.15 ms**
-  for the folded-only `torch.compile` formulation and **28.04 ms** for eager
-  folding — **1.99×** and **3.68×** faster — and **1.08×** faster than the
-  unfused Triton-kernel path. The engine folds the standard layers; the kernel
-  keeps the operations folding cannot express.
-- **Exact parity with native Triton** (`MaxDiff = 0.00e+00`) for LLaMA 3 RMSNorm,
-  PaLM SwiGLU, Qwen 2.5 RoPE, Liger fused cross-entropy, in-place mutation and
-  the quantized GEMMs, on both backends.
-- **Lowest VRAM** in a 4-layer LLaMA decoder: **464 MB** peak vs 478 MB (eager)
-  and 473 MB (`torch.compile`), with the fastest time-to-first-token (4.14 ms vs
-  7.85/7.92 ms).
+Two architectures: an **RTX 5060 Ti** (Blackwell, `sm_120a`) for the folding,
+single-operator, end-to-end and quantized results, and an **A100-SXM4-80GB**
+(Ampere, `sm_80`) for the conformance, parity and real-model results. Results are
+reported per configuration and never blended. Full methodology, complete tables
+and every raw JSON are on the
+[results page](https://just1truc.github.io/kernel_lens/#results) and under
+[`docs/assets/data/`](docs/assets/data/).
+
+**The code you don't write.** The deployment is **nine lines of Python**; the C++
+it replaces is **1,137 lines across six files** for one fused RMSNorm kernel on
+both backends, measured with `tests/measure_codegen.py`.
+
+**One engine, many shapes.** With `dynamic_shapes=True` a single compiled engine
+serves any leading-dimension size in range. Measured with
+`tests/test_conformance.py`, which compiles once and then runs every shape through
+that engine — `0.00e+00` on both backends, on Blackwell and on Ampere:
+
+| Case | Compiled at | Served |
+| :--- | :--- | :--- |
+| `dynamic_batch_sweep` | batch 8 | batches 1, 2, 4, 8, 16 |
+| `dynamic_grid_sweep` | 524,288 elements | 65,536 / 131,072 / 524,288 / 1,048,576 |
+
+**Real pretrained models.** Every decoder `LayerNorm` of a real OPT checkpoint
+(125M → 6.7B) replaced by the Triton RMSNorm kernel, graded against
+`torch.compile` — the baseline that *captures the same Triton kernels* rather
+than reimplementing them. The result splits by precision:
+
+- **fp32** — `torch.compile` barely improves on eager (1.02–1.12×) because it
+  leaves the operators around the kernel unfused. KernelLens fuses them:
+  **3.18–5.45× faster** under TensorRT and **2.27–4.62×** under ONNX Runtime,
+  and the margin grows with model size.
+- **fp16** — `torch.compile` optimises well there, so we only match it: TensorRT
+  edges it by **1.06–1.22×**, while ONNX Runtime is consistently **slower
+  (0.67–0.84×)** — per-node dispatch dominates when the graph already fuses.
+- **Accuracy tracks precision, not speed.** fp32 holds the same greedy token on
+  **≥ 98.8%** of positions; fp16 under TensorRT falls to 93.4–97.9%. OPT-350M in
+  fp16 under ONNX Runtime is bit-exact.
+
+**Fold + keep, and it pays.** On a two-layer block whose attention is a custom
+squared-ReLU flash kernel, KernelLens runs it in **7.63 ms**, versus **15.15 ms**
+for the folded-only `torch.compile` formulation and **28.04 ms** for eager
+folding — **1.99×** and **3.68×** faster. Against the unfused kernel path it is
+**1.08×** faster than eager Triton (8.24 ms) and level with `torch.compile`
+(7.55 ms). The engine folds the standard layers; the kernel keeps the operations
+folding cannot express.
+
+**Exact parity with native Triton** (`MaxDiff = 0.00e+00`) for LLaMA 3 RMSNorm,
+PaLM SwiGLU, Qwen 2.5 RoPE, Liger fused cross-entropy, in-place mutation and the
+quantized GEMMs, on both backends — and across all **fourteen** module patterns
+of `tests/test_conformance.py`. Two multi-operator fusions are near-exact rather
+than exact (`≤ 9.5e-07` residual+RMSNorm, `4.8e-07` Mamba's scan) and are
+reported as such.
+
+**Lowest VRAM** in a 4-layer LLaMA decoder: **464 MB** peak vs 478 MB (eager)
+and 473 MB (`torch.compile`), with the fastest time-to-first-token (4.14 ms vs
+7.85/7.92 ms).
 
 ## Documentation
 
@@ -195,9 +276,11 @@ the [results page](https://just1truc.github.io/kernel_lens/#results).
 | :--- | :--- |
 | [Problem](https://just1truc.github.io/kernel_lens/#problem) | The five gaps a naive export hits |
 | [Pipeline](https://just1truc.github.io/kernel_lens/#pipeline) | What you build by hand vs. the compile/run pipeline |
-| [Results](https://just1truc.github.io/kernel_lens/#results) | Folding + kernels, parity, latency, end-to-end |
+| [Results](https://just1truc.github.io/kernel_lens/#results) | Folding + kernels, parity, conformance, single-operator and end-to-end latency, **real pretrained models** |
+| [Why an engine](https://just1truc.github.io/kernel_lens/#why) | `torch.compile` vs TensorRT, both directions |
 | [Install & API](https://just1truc.github.io/kernel_lens/#install) | Installation and API reference |
 | [Status](https://just1truc.github.io/kernel_lens/#status) | Roadmap |
+| [FAQ](https://just1truc.github.io/kernel_lens/#faq) | Fair objections |
 | [Paper](https://just1truc.github.io/kernel_lens/#paper) | Abstract and citation |
 
 ## Reproduce the results
@@ -212,14 +295,30 @@ python tests/test_research_kernels.py          # numerical parity
 python tests/benchmark_research_kernels.py     # single-operator latency
 python tests/benchmark_e2e_llama.py            # end-to-end decoder
 python tests/test_fp8_gemm.py                  # FP8 Tensor Core path
+python tests/measure_codegen.py                # generated C++ volume, per file
+python tests/benchmark_real_model.py --model facebook/opt-1.3b   # real OPT, both precisions
 ```
 
+The real-model matrix (OPT-125M → 6.7B, fp32 and fp16) runs unattended on a fresh
+CUDA host: `scripts/remote_setup.sh` provisions nvcc, TensorRT, `onnxruntime-gpu`
+and `transformers` and installs the package; `scripts/remote_bench_all.sh` then
+sweeps the matrix into `results/` and writes `summary.json`, recording failures
+instead of aborting. See
+[`docs/assets/data/a100/runpod_runbook.md`](docs/assets/data/a100/runpod_runbook.md) for the
+exact invocation and the environment traps (CUDA 12.8's `nvcc` does not build
+against a modern libstdc++; CUDA 13.3's does).
+
 Raw measurement JSON is committed under [`docs/assets/data/`](docs/assets/data/).
 
 ## Technical report
 
-The LaTeX source is [`architecture_report.tex`](architecture_report.tex); the TMLR
-variant is generated from it with `python3 scripts/make_tmlr.py`.
+The paper source is [`kernel_lens_fresh.tex`](kernel_lens_fresh.tex). Two derived
+variants are generated from it and must not be edited by hand:
+`scripts/make_mlsys.py` produces the 2-column MLSys form
+(`kernel_lens_mlsys.tex`) and `scripts/make_tmlr_fresh.py` produces the TMLR
+forms — `kernel_lens_tmlr_fresh.tex` (named preprint) and
+`kernel_lens_tmlr_submission.tex` (anonymized; **do not publish it alongside the
+named preprint**).
 
 ```bibtex
 @article{duc2026kernellens,
@@ -227,7 +326,7 @@ variant is generated from it with `python3 scripts/make_tmlr.py`.
   author      = {Duc, Justin},
   institution = {Galiad Research},
   year        = {2026},
-  version     = {1.1.9},
+  version     = {1.2.1},
   doi         = {10.5281/zenodo.23194523},
   url         = {https://doi.org/10.5281/zenodo.23194523}
 }

{kernel_lens-1.2.0 → kernel_lens-1.3.0}/README.md

@@ -50,6 +50,12 @@ compiled_model.run(inputs, backend="onnx"|"tensorrt")
 - **Dynamic grids in C++** — launch-grid expressions are transpiled to C++
   integer math over runtime shapes, so one plugin serves dynamic batch and
   sequence lengths.
+- **Shape-generic engines** — `kl.compile(..., dynamic_shapes=True)` exports the
+  leading dimension as a symbolic axis, so a single engine serves any size in a
+  declared range via a TensorRT optimization profile. Verified on Blackwell
+  (`sm_120a`) and Ampere (`sm_80`): one compile at batch 8 serving batches
+  1-16, bit-exact on both engines. Off by default, because a shape-specialised
+  engine is what the published benchmarks measure.
 - **Zero-copy VRAM binding** — PyTorch `data_ptr()` values are registered
   directly with each engine; no host staging copies.
 - **Blackwell PTX ISA clamping** — a Triton-generated `.version 9.3` header is
@@ -63,11 +69,16 @@ compiled_model.run(inputs, backend="onnx"|"tensorrt")
   dynamic scalars, strided and dynamic inputs, loops, conditionals, nested
   helpers, in-place mutation, 3-D grids, autotune, and fp16/bf16.
 
-> **Honest scope.** KernelLens is research-grade. The documentation clearly
-> separates what is **implemented and measured** from what is **planned**. Two
-> TensorRT paths (in-place aliasing, INT8 dequantisation) are known to be
-> incorrect and are reported rather than hidden. See the
-> [verification page](https://just1truc.github.io/kernel_lens/verification/).
+> **Honest scope.** KernelLens is research-grade, and the results are reported
+> per configuration rather than blended. The composition is bit-exact against
+> native Triton (`0.00e+00`) across the research kernels, the fourteen
+> conformance patterns and the quantized GEMMs, on both backends; two
+> multi-operator fusions land just above zero and are reported as such. Keeping
+> the kernel pays when the baseline leaves the graph unfused and breaks even when
+> it already fuses. The TensorRT-LLM backend is design-only (see
+> [`docs/assets/data/a100/trtllm_backend_design.md`](docs/assets/data/a100/trtllm_backend_design.md)),
+> and the quantized GEMMs are validated on ONNX Runtime only. Full methodology
+> and every raw JSON are under [`docs/assets/data/`](docs/assets/data/).
 
 ## Installation
 
@@ -79,7 +90,16 @@ pip install kernel-lens[all]     # both
 ```
 
 **Requirements:** Linux x86-64, Python ≥ 3.10, CUDA ≥ 12.0, PyTorch ≥ 2.0,
-Triton ≥ 2.1, and `nvcc` + `g++` on `PATH`.
+Triton ≥ 2.1, and `nvcc` + `g++` on `PATH`. TensorRT must be `< 11` (the
+generator emits `IPluginV2DynamicExt`).
+
+**Environment:** `KERNEL_LENS_CACHE_DIR` (put it on a persistent volume so the
+`.so` / `.engine` artifacts survive `kl.load`), `KERNEL_LENS_CCBIN` (host
+compiler for `nvcc` — CUDA toolkits before 12.9 reject GCC 13's exception
+specifiers, so set it to `g++-12` there), `KERNEL_LENS_DEBUG` (or `KL_VERBOSE`),
+`TENSORRT_INCLUDE_DIR`, and `KERNEL_LENS_STREAM_READER=1` to deserialize engines
+above 2 GB from disk rather than from a memory buffer (opt-in; see the
+[changelog](CHANGELOG.md)).
 
 ## Quickstart
 
@@ -118,26 +138,73 @@ compiled = kl.compile(RMSNorm(4096).cuda(), (x,), name="llama3_rmsnorm",
                       backends=["onnx", "tensorrt"])
 out_ort = compiled.run((x,), backend="onnx")
 out_trt = compiled.run((x,), backend="tensorrt")
+
+# One engine for any batch: mark the leading dim dynamic. dynamic_max bounds
+# the optimization profile (default 4x the traced size).
+serving = kl.compile(RMSNorm(4096).cuda(), (x,), name="llama3_rmsnorm_dyn",
+                     backends=["onnx", "tensorrt"], dynamic_shapes=True)
+for b in (1, 2, 4, 8, 16):                 # all through the engine built at b=4
+    serving.run((torch.randn(b, 512, 4096, device="cuda"),), backend="tensorrt")
 ```
 
 ## Results at a glance
 
-Measured on an **RTX 5060 Ti (Blackwell, `sm_120a`)**, CUDA 12.8, PyTorch 2.14,
-Triton 3.8, ONNX Runtime 1.30, TensorRT 10.16. Full tables and methodology are on
-the [results page](https://just1truc.github.io/kernel_lens/#results).
-
-- **Fold + keep, and it pays**: on a two-layer block whose attention is a custom
-  squared-ReLU flash kernel, KernelLens runs it in **7.63 ms**, versus **15.15 ms**
-  for the folded-only `torch.compile` formulation and **28.04 ms** for eager
-  folding — **1.99×** and **3.68×** faster — and **1.08×** faster than the
-  unfused Triton-kernel path. The engine folds the standard layers; the kernel
-  keeps the operations folding cannot express.
-- **Exact parity with native Triton** (`MaxDiff = 0.00e+00`) for LLaMA 3 RMSNorm,
-  PaLM SwiGLU, Qwen 2.5 RoPE, Liger fused cross-entropy, in-place mutation and
-  the quantized GEMMs, on both backends.
-- **Lowest VRAM** in a 4-layer LLaMA decoder: **464 MB** peak vs 478 MB (eager)
-  and 473 MB (`torch.compile`), with the fastest time-to-first-token (4.14 ms vs
-  7.85/7.92 ms).
+Two architectures: an **RTX 5060 Ti** (Blackwell, `sm_120a`) for the folding,
+single-operator, end-to-end and quantized results, and an **A100-SXM4-80GB**
+(Ampere, `sm_80`) for the conformance, parity and real-model results. Results are
+reported per configuration and never blended. Full methodology, complete tables
+and every raw JSON are on the
+[results page](https://just1truc.github.io/kernel_lens/#results) and under
+[`docs/assets/data/`](docs/assets/data/).
+
+**The code you don't write.** The deployment is **nine lines of Python**; the C++
+it replaces is **1,137 lines across six files** for one fused RMSNorm kernel on
+both backends, measured with `tests/measure_codegen.py`.
+
+**One engine, many shapes.** With `dynamic_shapes=True` a single compiled engine
+serves any leading-dimension size in range. Measured with
+`tests/test_conformance.py`, which compiles once and then runs every shape through
+that engine — `0.00e+00` on both backends, on Blackwell and on Ampere:
+
+| Case | Compiled at | Served |
+| :--- | :--- | :--- |
+| `dynamic_batch_sweep` | batch 8 | batches 1, 2, 4, 8, 16 |
+| `dynamic_grid_sweep` | 524,288 elements | 65,536 / 131,072 / 524,288 / 1,048,576 |
+
+**Real pretrained models.** Every decoder `LayerNorm` of a real OPT checkpoint
+(125M → 6.7B) replaced by the Triton RMSNorm kernel, graded against
+`torch.compile` — the baseline that *captures the same Triton kernels* rather
+than reimplementing them. The result splits by precision:
+
+- **fp32** — `torch.compile` barely improves on eager (1.02–1.12×) because it
+  leaves the operators around the kernel unfused. KernelLens fuses them:
+  **3.18–5.45× faster** under TensorRT and **2.27–4.62×** under ONNX Runtime,
+  and the margin grows with model size.
+- **fp16** — `torch.compile` optimises well there, so we only match it: TensorRT
+  edges it by **1.06–1.22×**, while ONNX Runtime is consistently **slower
+  (0.67–0.84×)** — per-node dispatch dominates when the graph already fuses.
+- **Accuracy tracks precision, not speed.** fp32 holds the same greedy token on
+  **≥ 98.8%** of positions; fp16 under TensorRT falls to 93.4–97.9%. OPT-350M in
+  fp16 under ONNX Runtime is bit-exact.
+
+**Fold + keep, and it pays.** On a two-layer block whose attention is a custom
+squared-ReLU flash kernel, KernelLens runs it in **7.63 ms**, versus **15.15 ms**
+for the folded-only `torch.compile` formulation and **28.04 ms** for eager
+folding — **1.99×** and **3.68×** faster. Against the unfused kernel path it is
+**1.08×** faster than eager Triton (8.24 ms) and level with `torch.compile`
+(7.55 ms). The engine folds the standard layers; the kernel keeps the operations
+folding cannot express.
+
+**Exact parity with native Triton** (`MaxDiff = 0.00e+00`) for LLaMA 3 RMSNorm,
+PaLM SwiGLU, Qwen 2.5 RoPE, Liger fused cross-entropy, in-place mutation and the
+quantized GEMMs, on both backends — and across all **fourteen** module patterns
+of `tests/test_conformance.py`. Two multi-operator fusions are near-exact rather
+than exact (`≤ 9.5e-07` residual+RMSNorm, `4.8e-07` Mamba's scan) and are
+reported as such.
+
+**Lowest VRAM** in a 4-layer LLaMA decoder: **464 MB** peak vs 478 MB (eager)
+and 473 MB (`torch.compile`), with the fastest time-to-first-token (4.14 ms vs
+7.85/7.92 ms).
 
 ## Documentation
 
@@ -145,9 +212,11 @@ the [results page](https://just1truc.github.io/kernel_lens/#results).
 | :--- | :--- |
 | [Problem](https://just1truc.github.io/kernel_lens/#problem) | The five gaps a naive export hits |
 | [Pipeline](https://just1truc.github.io/kernel_lens/#pipeline) | What you build by hand vs. the compile/run pipeline |
-| [Results](https://just1truc.github.io/kernel_lens/#results) | Folding + kernels, parity, latency, end-to-end |
+| [Results](https://just1truc.github.io/kernel_lens/#results) | Folding + kernels, parity, conformance, single-operator and end-to-end latency, **real pretrained models** |
+| [Why an engine](https://just1truc.github.io/kernel_lens/#why) | `torch.compile` vs TensorRT, both directions |
 | [Install & API](https://just1truc.github.io/kernel_lens/#install) | Installation and API reference |
 | [Status](https://just1truc.github.io/kernel_lens/#status) | Roadmap |
+| [FAQ](https://just1truc.github.io/kernel_lens/#faq) | Fair objections |
 | [Paper](https://just1truc.github.io/kernel_lens/#paper) | Abstract and citation |
 
 ## Reproduce the results
@@ -162,14 +231,30 @@ python tests/test_research_kernels.py          # numerical parity
 python tests/benchmark_research_kernels.py     # single-operator latency
 python tests/benchmark_e2e_llama.py            # end-to-end decoder
 python tests/test_fp8_gemm.py                  # FP8 Tensor Core path
+python tests/measure_codegen.py                # generated C++ volume, per file
+python tests/benchmark_real_model.py --model facebook/opt-1.3b   # real OPT, both precisions
 ```
 
+The real-model matrix (OPT-125M → 6.7B, fp32 and fp16) runs unattended on a fresh
+CUDA host: `scripts/remote_setup.sh` provisions nvcc, TensorRT, `onnxruntime-gpu`
+and `transformers` and installs the package; `scripts/remote_bench_all.sh` then
+sweeps the matrix into `results/` and writes `summary.json`, recording failures
+instead of aborting. See
+[`docs/assets/data/a100/runpod_runbook.md`](docs/assets/data/a100/runpod_runbook.md) for the
+exact invocation and the environment traps (CUDA 12.8's `nvcc` does not build
+against a modern libstdc++; CUDA 13.3's does).
+
 Raw measurement JSON is committed under [`docs/assets/data/`](docs/assets/data/).
 
 ## Technical report
 
-The LaTeX source is [`architecture_report.tex`](architecture_report.tex); the TMLR
-variant is generated from it with `python3 scripts/make_tmlr.py`.
+The paper source is [`kernel_lens_fresh.tex`](kernel_lens_fresh.tex). Two derived
+variants are generated from it and must not be edited by hand:
+`scripts/make_mlsys.py` produces the 2-column MLSys form
+(`kernel_lens_mlsys.tex`) and `scripts/make_tmlr_fresh.py` produces the TMLR
+forms — `kernel_lens_tmlr_fresh.tex` (named preprint) and
+`kernel_lens_tmlr_submission.tex` (anonymized; **do not publish it alongside the
+named preprint**).
 
 ```bibtex
 @article{duc2026kernellens,
@@ -177,7 +262,7 @@ variant is generated from it with `python3 scripts/make_tmlr.py`.
   author      = {Duc, Justin},
   institution = {Galiad Research},
   year        = {2026},
-  version     = {1.1.9},
+  version     = {1.2.1},
   doi         = {10.5281/zenodo.23194523},
   url         = {https://doi.org/10.5281/zenodo.23194523}
 }

{kernel_lens-1.2.0 → kernel_lens-1.3.0}/kernel_lens/__init__.py

@@ -7,7 +7,7 @@ from .config import set_verbose, is_verbose
 
 
 
-__version__ = "1.2.0"
+__version__ = "1.3.0"
 
 __all__ = [
     "compile",

{kernel_lens-1.2.0 → kernel_lens-1.3.0}/kernel_lens/backends/builder.py

@@ -6,6 +6,20 @@ import shutil
 
 from ..utils.env_check import find_nvcc
 
+
+def _host_compiler():
+    """Host compiler for nvcc.
+
+    CUDA toolkits before 12.9 declare rsqrt/sinpi/... without noexcept, which
+    GCC 13 rejects ("has a different exception specifier"), so the right choice
+    is environment-specific. Override with KERNEL_LENS_CCBIN.
+    """
+    override = os.environ.get("KERNEL_LENS_CCBIN")
+    if override:
+        return override
+    return "g++-13" if os.path.exists("/usr/bin/g++-13") else "g++"
+
+
 def get_cuda_home() -> str:
     nvcc_path = find_nvcc()
     if nvcc_path:
@@ -152,7 +166,7 @@ def build_ort_plugin(ort_plugins_dir: str, cache_dir: str):
             arch_flag = "-gencode=arch=compute_75,code=sm_75"
 
         user_trt_inc = os.path.expanduser("~/tensorrt_headers")
-        cpp_compiler = "g++-13" if os.path.exists("/usr/bin/g++-13") else "g++"
+        cpp_compiler = _host_compiler()
         ccbin_flag = ["-ccbin", cpp_compiler] if os.path.exists(f"/usr/bin/{cpp_compiler}") else []
 
         cmd = [
@@ -318,7 +332,7 @@ def build_trt_plugin(trt_plugins_dir: str, cache_dir: str):
             and not os.path.exists(os.path.join(d, "crt", "math_functions.h"))
         ]
 
-        cpp_compiler = "g++-13" if os.path.exists("/usr/bin/g++-13") else "g++"
+        cpp_compiler = _host_compiler()
         ccbin_flag = ["-ccbin", cpp_compiler] if os.path.exists(f"/usr/bin/{cpp_compiler}") else []
 
         cmd = [
@@ -383,7 +397,7 @@ def build_trt_plugin(trt_plugins_dir: str, cache_dir: str):
                     nvinfer_link_flag = [os.path.join(d, f)]
                     break
 
-    cpp_compiler = "g++-13" if os.path.exists("/usr/bin/g++-13") else "g++"
+    cpp_compiler = _host_compiler()
     cmd = [
         cpp_compiler, "-shared", "-o", so_path
     ] + obj_files + [

{kernel_lens-1.2.0 → kernel_lens-1.3.0}/kernel_lens/backends/ort_gen.py

@@ -150,31 +150,26 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
         arg_setup_lines.append("auto ref_in = ctx.GetInput(0);")
         arg_setup_lines.append("auto info = ref_in.GetTensorTypeAndShapeInfo();")
         arg_setup_lines.append("std::vector<int64_t> dim_values = info.GetShape();")
+
+        # Dims the caller asked to keep dynamic: read them from the tensor that
+        # is actually being run, not from the traced shape.
+        _dyn_axes = tuple(getattr(manifest, "dynamic_axes", ()) or ())
         
         import re
         fn_match = re.search(r'\.entry\s+([a-zA-Z0-9_]+)', manifest.ptx)
         ptx_entry_name = fn_match.group(1) if fn_match else manifest.kernel_name
         
-        # --- ROBUST GRID EVALUATION WITH GENERIC REGEX SUBSTITUTION ---
-        grid_strs = []
-        if hasattr(manifest, '_sym_grid_asts') and manifest._sym_grid_asts:
-            for g in manifest._sym_grid_asts:
-                expr = str(g.node.expr) if hasattr(g, 'node') else str(g)
-                expr = re.sub(r'([a-zA-Z0-9_]+)\*\*([a-zA-Z0-9_]+)', r'std::pow(\1, \2)', expr)
-                expr = expr.replace("//", "/")
-                expr = re.sub(r'floor\((.*?)\)', r'(\1)', expr)
-                
-                # Replace any s(\d+) symbol dynamically using regex
-                expr = re.sub(r'\bs(\d+)\b', r"(int64_t)(dim_values.size() > \1 ? dim_values[\1] : 1)", expr)
-                grid_strs.append(expr)
-                
+        # --- GRID EVALUATION, from the manifest's own symbol translation ---
+        grid_strs = manifest.grid_exprs(
+            lambda i: f"(int64_t)(dim_values.size() > {i} ? dim_values[{i}] : 1)"
+        )
         while len(grid_strs) < 3:
             grid_strs.append("1")
             
         grid_eval_lines = [
             f"unsigned int grid_x = std::max(1u, (unsigned int)({grid_strs[0]}));",
             f"unsigned int grid_y = std::max(1u, (unsigned int)({grid_strs[1]}));",
-            f"unsigned int grid_z = std::max(1u, (unsigned int)({grid_strs[2]}));"
+            f"unsigned int grid_z = std::max(1u, (unsigned int)({grid_strs[2]}));",
         ]
 
         entry_match = re.search(r'\.entry\s+[a-zA-Z0-9_]+\s*\((.*?)\)\s*(?:\.reqntid|\{)', manifest.ptx, re.DOTALL)
@@ -215,6 +210,23 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
             
             ptx_ordered_slots.append({"type": c_type, "ident": str(param_num), "is_ptr": is_ptr, "decl": p_decl, "param_num": param_num})
 
+        # The node's inputs, in the order ONNX Runtime presents them. This is
+        # the same list GetInputTypeCount() reports, so indexing into it is
+        # authoritative. A running counter over manifest.arguments is not: any
+        # drift between the two silently points a scalar read at the wrong
+        # slot, and the kernel then masks against a garbage element count.
+        inputs_to_node = [
+            a for a in manifest.arguments
+            if (a.kind in ('input', 'inplace')
+                or (a.kind == 'scalar' and not getattr(a, 'is_constexpr', False)))
+        ]
+
+        def _node_input_index(a, fallback):
+            for k, cand in enumerate(inputs_to_node):
+                if cand is a:
+                    return k
+            return fallback
+
         arg_setup_lines.append("// Extract inputs/outputs/scalars from ORT context")
         onnx_input_counter = 0
         ort_output_counter = 0
@@ -227,7 +239,7 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
                     if _esz == 0.5
                     else f"(in_tensor_{slot_idx}.GetTensorTypeAndShapeInfo().GetElementCount() * {int(_esz)})"
                 )
-                arg_setup_lines.append(f"auto in_tensor_{slot_idx} = ctx.GetInput({onnx_input_counter});")
+                arg_setup_lines.append(f"auto in_tensor_{slot_idx} = ctx.GetInput({_node_input_index(arg, onnx_input_counter)});")
                 arg_setup_lines.append(f"static thread_local const void* arg_ptr_{slot_idx};")
                 arg_setup_lines.append(f"const void* raw_ptr_{slot_idx} = (const void*)in_tensor_{slot_idx}.GetTensorData<uint8_t>();")
                 arg_setup_lines.append(f"static thread_local void* align_buf_{slot_idx} = nullptr;")
@@ -242,6 +254,9 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
             elif arg.kind == 'output':
                 out_dim_exprs = []
                 for dim_idx, d in enumerate(arg.shape):
+                    if dim_idx in _dyn_axes:
+                        out_dim_exprs.append(f"(int64_t)(dim_values.size() > {dim_idx} ? dim_values[{dim_idx}] : 1)")
+                        continue
                     try:
                         d_int = int(d)
                         out_dim_exprs.append(str(d_int))
@@ -257,6 +272,9 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
             elif arg.kind == 'inplace':
                 out_dim_exprs = []
                 for dim_idx, d in enumerate(arg.shape):
+                    if dim_idx in _dyn_axes:
+                        out_dim_exprs.append(f"(int64_t)(dim_values.size() > {dim_idx} ? dim_values[{dim_idx}] : 1)")
+                        continue
                     try:
                         d_int = int(d)
                         out_dim_exprs.append(str(d_int))
@@ -264,7 +282,7 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
                         out_dim_exprs.append(f"(int64_t)(dim_values.size() > {dim_idx} ? dim_values[{dim_idx}] : 1)")
                 out_shape_str = "{" + ", ".join(out_dim_exprs) + "}" if out_dim_exprs else "dim_values"
                 
-                arg_setup_lines.append(f"auto in_tensor_{slot_idx} = ctx.GetInput({onnx_input_counter});")
+                arg_setup_lines.append(f"auto in_tensor_{slot_idx} = ctx.GetInput({_node_input_index(arg, onnx_input_counter)});")
                 arg_setup_lines.append(f"std::vector<int64_t> out_dims_{ort_output_counter} = {out_shape_str};")
                 arg_setup_lines.append(f"auto out_tensor_{slot_idx} = ctx.GetOutput({ort_output_counter}, out_dims_{ort_output_counter}.data(), out_dims_{ort_output_counter}.size());")
                 _esz = element_size_bytes(arg.dtype)
@@ -282,16 +300,33 @@ struct {op_name} : Ort::CustomOpBase<{op_name}, {kernel_name}> {{
                 onnx_input_counter += 1
                 ort_output_counter += 1
             elif arg.kind == 'scalar':
-                if not getattr(arg, 'is_constexpr', False):
-                    arg_setup_lines.append(f"auto scalar_tensor_{slot_idx} = ctx.GetInput({onnx_input_counter});")
+                expr = getattr(arg, 'cxx_expr', '') or ''
+                if 'dim_values' in expr:
+                    # Shape-derived (n = x.numel()): compute it from the tensor
+                    # being run. The exported graph folds this scalar into a
+                    # constant -- it is not a graph input -- so reading the slot
+                    # would hand the kernel the element count captured while
+                    # tracing, and it would then process a fraction of the
+                    # tensor and leave the rest untouched.
+                    arg_setup_lines.append(f"double scalar_val_{slot_idx} = (double)({expr});")
+                    onnx_input_counter += 1
+                elif not getattr(arg, 'is_constexpr', False):
+                    arg_setup_lines.append(f"auto scalar_tensor_{slot_idx} = ctx.GetInput({_node_input_index(arg, onnx_input_counter)});")
                     arg_setup_lines.append(f"double scalar_val_{slot_idx} = 0;")
                     arg_setup_lines.append(f"if (scalar_tensor_{slot_idx}.GetTensorTypeAndShapeInfo().GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64) {{ scalar_val_{slot_idx} = (double)(scalar_tensor_{slot_idx}.GetTensorData<int64_t>()[0]); }}")
                     arg_setup_lines.append(f"else if (scalar_tensor_{slot_idx}.GetTensorTypeAndShapeInfo().GetElementType() == ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32) {{ scalar_val_{slot_idx} = (double)(scalar_tensor_{slot_idx}.GetTensorData<int32_t>()[0]); }}")
                     arg_setup_lines.append(f"else {{ scalar_val_{slot_idx} = (double)(scalar_tensor_{slot_idx}.GetTensorData<float>()[0]); }}")
                     onnx_input_counter += 1
                 else:
-                    s_val = arg.value if getattr(arg, 'value', None) is not None else 0.0
-                    arg_setup_lines.append(f"double scalar_val_{slot_idx} = {s_val};")
+                    # Prefer the manifest's translated expression: for a scalar
+                    # derived from shapes (n = x.numel()) that is a runtime read
+                    # of the input dims; otherwise it is the baked constant.
+                    expr = getattr(arg, 'cxx_expr', '') or None
+                    if expr:
+                        arg_setup_lines.append(f"double scalar_val_{slot_idx} = (double)({expr});")
+                    else:
+                        s_val = arg.value if getattr(arg, 'value', None) is not None else 0.0
+                        arg_setup_lines.append(f"double scalar_val_{slot_idx} = {s_val};")
 
         arg_setup_lines.append("static thread_local const void* null_ptr_arg = nullptr;")
         arg_setup_lines.append("static thread_local std::vector<void*> kp;")