inferlast 0.1.0__tar.gz

inferlast-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yuvraj Singh Bhadoria
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inferlast-0.1.0/PKG-INFO

@@ -0,0 +1,211 @@
+Metadata-Version: 2.4
+Name: inferlast
+Version: 0.1.0
+Summary: CPU-first LLM inference optimizer: prove a real win on your laptop CPU before you pay to rent a GPU. Trustcheck catches any benchmark that is really just noise.
+Author-email: Yuvraj Singh Bhadoria <yuvrajsinghbhado2030@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/YuvrajSinghBhadoria2/inferlast
+Project-URL: Repository, https://github.com/YuvrajSinghBhadoria2/inferlast
+Project-URL: Releases, https://github.com/YuvrajSinghBhadoria2/inferlast/releases
+Keywords: llm,inference,optimization,cpu,quantization,benchmark,trustcheck
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch<2.5,>=2.2
+Requires-Dist: transformers<4.47,>=4.40
+Requires-Dist: numpy<2,>=1.26
+Dynamic: license-file
+
+<p align="center">
+  <br/>
+  <img src="https://img.shields.io/badge/status-Phase%201%20(CPU)-informational" alt="Phase 1 CPU"/>
+  <img src="https://img.shields.io/badge/tests-55%20passing-brightgreen" alt="tests"/>
+  <img src="https://img.shields.io/badge/license-MIT-blue" alt="license"/>
+  <img src="https://img.shields.io/badge/python-3.10%2F3.11%2F3.12-blue" alt="python"/>
+  <a href="https://github.com/YuvrajSinghBhadoria2/inferlast/releases"><img src="https://img.shields.io/github/v/release/YuvrajSinghBhadoria2/inferlast" alt="release"/></a>
+</p>
+
+<h1 align="center">inferlast</h1>
+<p align="center"><b>Fine-tune-free, GPU-free inference:</b> prove a real optimization win on a plain laptop CPU <i>before</i> you ever pay to rent one.<br/>
+Most demos tell you a change is <b>3x faster</b> — inferlast catches when that number is just machine noise.</p>
+
+<p align="center">
+  <a href="#why">Why</a> · <a href="#30-second-try">30-second try</a> · <a href="#what-it-caught">What it caught on my machine</a> · <a href="#how-it-works">How it works</a> · <a href="#roadmap">Roadmap</a>
+</p>
+
+---
+
+## Why
+
+Most LLM-inference guides tell you what to do: *quantize to INT8, batch bigger, grab a GPU.* They don't tell you **whether it helps *your* model on *your* hardware** — and they quietly assume you can rent a GPU to find out.
+
+inferlast is the opposite. It is **CPU-first by design**: it profiles and optimizes entirely on the CPU you already have, so **anyone can run it — no GPU, no cloud GPU bill, no CUDA install.** And when you *do* move to a GPU later, inferlast tells you honestly whether it was even worth it.
+
+> **Profile → pick → apply → prove.** And if the measurement says an "obvious" optimization doesn't help, inferlast says so — instead of making you guess wrong.
+
+This is the core of what inference engineers actually do: not "apply the standard thing," but **find where the time really goes and only ship changes that provably pay off** — cheaply enough that you don't need a GPU to do it.
+
+> **Hardware scope — Phase 1 is CPU-only.** Built and measured on a 2019 Intel MacBook Pro 16" (i7-9750H, 6C/12T, 16 GB, no GPU). Findings are CPU-specific and stated as such; on GPU the same model is typically weight-bandwidth-bound, not overhead-bound, so results would differ.
+
+The project is organized around one falsifiable claim — **a decision rule for when GPU spend is actually warranted, estimable from CPU-only measurement.** That thesis, its boundaries, and its frozen success test live in [`docs/RESEARCH-SPEC.md`](docs/RESEARCH-SPEC.md).
+
+## The honest insight it encodes
+
+For tiny models on CPU, inferlast measures that **~98–99% of decode wall time is framework overhead, not model math**. So **blindly quantizing the weights will not speed up an overhead-bound model** — and inferlast *measures* that rather than pretending otherwise. That refusal-to-guess behavior is the whole point.
+
+## Install
+
+```bash
+pip install inferlast            # Python 3.10-3.12; CPU-first, no GPU/CUDA needed
+```
+
+This installs the core (`import trustcheck`, `import gpucheck`, ...) and the
+`inferlast` CLI. Or run straight from the repo:
+
+```bash
+git clone https://github.com/YuvrajSinghBhadoria2/inferlast.git
+pip install -r requirements.txt
+```
+
+## 30-second try
+
+```bash
+pip install -r requirements.txt          # torch CPU, transformers, pytest
+python scripts/run_all.py --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+One command. One report that tells you:
+
+```
+Bottleneck:  mlp 47% · attention 41% · norm 11%   (~99% overhead on CPU)
+Decode:      ~1220 ms/token (0.8 tok/s)
+Quantization: INT8 = 1.5x but shifts the answer → KEEP fp32 (not a win)
+Batching:    B=16 → ~48 tok/s total          Pick B for your goal.
+```
+
+And the part that catches the false wins — `trustcheck` on this repo's own
+recorded evidence (the same config that reported 3.0x faster):
+
+<p align="center">
+  <img src="assets/trustcheck-proof.png" alt="trustcheck catches a false win" width="640"/>
+</p>
+
+Run the individual stages to look closer:
+
+```bash
+python scripts/bench.py --model <m>              # prefill profile
+python scripts/bench.py --model <m> --decode      # decode / per-token latency
+python scripts/bench.py --model <m> --quant       # auto-quantization verdict
+python scripts/bench.py --model <m> --batch       # latency vs throughput sweep
+python scripts/bench.py --model <m> --trustcheck  # is that win real? (see below)
+python scripts/bench.py --model <m> --gpucheck    # do you even need a GPU? (see below)
+```
+
+## What it caught on my machine
+
+A quiet, annoying truth that most optimization tutorials skip: **on an overhead-bound CPU model, INT8 quantization is not a free win.** inferlast measured it three ways and told the truth:
+
+| | fp32 | INT8 | verdict |
+|---|---|---|---|
+| speed (Qwen 0.5B) | 1.31 s/tok | 0.87 s/tok | 1.5x — *but* |
+| quality (logit-cosine) | — | 0.68 | preserved-ish |
+| quality (top-5 overlap) | — | **0.15** | **token ranking shifted** |
+
+INT8 was faster but disturbed what the model would actually say. inferlast's call: **KEEP fp32.** That's not a bug — it's the tool doing its job: *refuse to recommend a change that isn't a real win.*
+
+The full measured records are in `benchmarks/` (JSON + markdown), published as measured — the failures included.
+
+## How it works
+
+Six small modules, one job each:
+
+| Module | Job |
+|---|---|
+| `src/profiler.py` | Per-category (attention / mlp / norm / embed / head) wall-clock profile, and the overhead-vs-weight split. |
+| `src/quantize.py` | Auto-quantization (INT8 dynamic). Measures fp32 vs INT8 averaged over repeats, with a **robust quality metric** (per-token logit cosine + top-5 overlap) — not brittle greedy-token identity. |
+| `src/batcher.py` | Latency-vs-throughput sweep over batch size, with a best-batch picker. |
+| `src/trustcheck.py` | **Is that 'win' worth trusting?** Audits any before/after benchmark for the three ways it lies: single-run noise, a brittle/wrong metric, and a "validated, documented, read by nothing" knob. Returns a REAL / MARGINAL / FALSE verdict. |
+| `src/gpucheck.py` | **Do you even need a GPU?** Estimates, from CPU-only measurements, whether GPU spend would actually beat the best-scheduled CPU config. Returns GPU-warranted / CPU-suffices / insufficient-data — and refuses to guess when it can't tell. |
+| `src/auto_optimizer.py` | Orchestrator: runs all four, emits a combined proof report + JSON. |
+| `scripts/` | `run_all.py` (one command) + `bench.py` (per stage). |
+
+A key design decision: `run_all.py` **always persists** a canonical report, so the evidence on disk always matches the latest run — it can't go stale.
+
+## gpucheck — the part that stops you overspending on hardware
+
+The whole project is built around one falsifiable claim (in
+[`docs/RESEARCH-SPEC.md`](docs/RESEARCH-SPEC.md)): **for overhead-bound small models on CPU, you
+usually don't need a GPU at all.** `gpucheck` puts that to the test from local CPU measurement:
+
+```bash
+python scripts/bench.py --model <m> \
+  --gpucheck --num-params 0.5e9 --overhead-fraction 0.98 --latency-target-ms 2000
+```
+
+It labels the decision — **GPU-warranted / CPU-suffices / insufficient-data** — with the exact inputs
+and reasoning, and it **refuses to guess** (returns `insufficient-data`) whenever it genuinely cannot
+tell from the data you gave it. That refusal is a feature: it never sells you a GPU rental it can't
+defend.
+
+## trustcheck — the part that tells you your benchmark lied
+
+Most tools *produce* a number. `trustcheck` tells you whether to believe it. It caught all three lies live on this repo's own evidence:
+
+- **Single-run noise.** The same INT8-vs-fp32 config, measured twice, gave `3.0x` faster in one session and `0.65x` **slower** in another. `trustcheck` computes the confidence interval and says: *"CI [-1.8x, 5.5x] straddles 1.0x → not a reliable win."* A naive dashboard would have reported `3.0x`.
+- **Brittle metric.** A quality comparison at greedy-token level, with no logits, is flagged as "can lie" and, when logits are available, re-measured with logit cosine + top-5 overlap.
+- **Read-by-nothing knob.** A flag that is documented/validated but never read by any code is a silent bug (the class of bug Soup and vLLM chased); `trustcheck`'s static pass flags it.
+
+```bash
+python scripts/bench.py --trustcheck \
+  --controls 1180 1316 --treatments 390 2031      # audit two real sessions
+python scripts/bench.py --model <m> --trustcheck \
+  --collect-repeats 2 --config-key stream_layers   # measure the noise band live
+```
+
+## Tests
+
+```bash
+pip install pytest
+python -m pytest        # 55 fast tests, no model downloads
+```
+
+The suite guards the things that would sink a tool like this: profiler categorisation & no-double-counting, the robust INT8 quality metric + the honest decision rule, batcher best-batch selection, the `trustcheck` noise/brittle-metric/read-by-nothing logic, the `gpucheck` GPU-necessity decision rule (including its refusal to guess when data is missing), and a **regression test that the report always emits the new metrics — never a stale one.**
+
+## Roadmap
+
+Phase 1 is done and runs on a plain laptop CPU — no GPU needed.
+
+**Shipped:**
+
+- **`trustcheck` — the false-win catcher.** Tells you the benchmark you were
+  about to publish is machine noise, not a win. It caught *this repo's own*
+  3.0x-vs-0.65x as noise.
+- **`gpucheck` — "do you even need a GPU?"** A CPU-only decision rule:
+  GPU-warranted / CPU-suffices / insufficient-data. It refuses to guess when it
+  can't tell.
+- Bottleneck / decode / quant / batch selection.
+
+**Where we want help next** (strongest help first):
+
+- **Validate the `gpucheck` boundary on hardware we can't reach.** Your numbers
+  ship behind an honest "requires <hardware>" gate — a 4 GB card, an Apple
+  Silicon box, a desktop with more RAM. Proof on hardware we lack turns a claim
+  into a finding. *This is the most valuable way to contribute right now.*
+- Quantization beyond INT8 (FP4 / INT4), latency percentiles (p50/p99), memory /
+  KV-cache footprint, and serving-engine integration (vLLM / llama.cpp) as an
+  enrichment layer.
+
+PRs welcome. Nothing here is a live claim — it's the plan.
+
+## License
+
+MIT — free, stays free, built in the open. If inferlast saved you a guessing session, a star helps others find it.
+
+<sub>Not a replacement for vLLM / llama.cpp — a *decision layer* that runs on CPU and tells you which setting is right for your model and hardware, with proof, before you spend on a GPU.</sub>

inferlast-0.1.0/README.md

@@ -0,0 +1,186 @@
+<p align="center">
+  <br/>
+  <img src="https://img.shields.io/badge/status-Phase%201%20(CPU)-informational" alt="Phase 1 CPU"/>
+  <img src="https://img.shields.io/badge/tests-55%20passing-brightgreen" alt="tests"/>
+  <img src="https://img.shields.io/badge/license-MIT-blue" alt="license"/>
+  <img src="https://img.shields.io/badge/python-3.10%2F3.11%2F3.12-blue" alt="python"/>
+  <a href="https://github.com/YuvrajSinghBhadoria2/inferlast/releases"><img src="https://img.shields.io/github/v/release/YuvrajSinghBhadoria2/inferlast" alt="release"/></a>
+</p>
+
+<h1 align="center">inferlast</h1>
+<p align="center"><b>Fine-tune-free, GPU-free inference:</b> prove a real optimization win on a plain laptop CPU <i>before</i> you ever pay to rent one.<br/>
+Most demos tell you a change is <b>3x faster</b> — inferlast catches when that number is just machine noise.</p>
+
+<p align="center">
+  <a href="#why">Why</a> · <a href="#30-second-try">30-second try</a> · <a href="#what-it-caught">What it caught on my machine</a> · <a href="#how-it-works">How it works</a> · <a href="#roadmap">Roadmap</a>
+</p>
+
+---
+
+## Why
+
+Most LLM-inference guides tell you what to do: *quantize to INT8, batch bigger, grab a GPU.* They don't tell you **whether it helps *your* model on *your* hardware** — and they quietly assume you can rent a GPU to find out.
+
+inferlast is the opposite. It is **CPU-first by design**: it profiles and optimizes entirely on the CPU you already have, so **anyone can run it — no GPU, no cloud GPU bill, no CUDA install.** And when you *do* move to a GPU later, inferlast tells you honestly whether it was even worth it.
+
+> **Profile → pick → apply → prove.** And if the measurement says an "obvious" optimization doesn't help, inferlast says so — instead of making you guess wrong.
+
+This is the core of what inference engineers actually do: not "apply the standard thing," but **find where the time really goes and only ship changes that provably pay off** — cheaply enough that you don't need a GPU to do it.
+
+> **Hardware scope — Phase 1 is CPU-only.** Built and measured on a 2019 Intel MacBook Pro 16" (i7-9750H, 6C/12T, 16 GB, no GPU). Findings are CPU-specific and stated as such; on GPU the same model is typically weight-bandwidth-bound, not overhead-bound, so results would differ.
+
+The project is organized around one falsifiable claim — **a decision rule for when GPU spend is actually warranted, estimable from CPU-only measurement.** That thesis, its boundaries, and its frozen success test live in [`docs/RESEARCH-SPEC.md`](docs/RESEARCH-SPEC.md).
+
+## The honest insight it encodes
+
+For tiny models on CPU, inferlast measures that **~98–99% of decode wall time is framework overhead, not model math**. So **blindly quantizing the weights will not speed up an overhead-bound model** — and inferlast *measures* that rather than pretending otherwise. That refusal-to-guess behavior is the whole point.
+
+## Install
+
+```bash
+pip install inferlast            # Python 3.10-3.12; CPU-first, no GPU/CUDA needed
+```
+
+This installs the core (`import trustcheck`, `import gpucheck`, ...) and the
+`inferlast` CLI. Or run straight from the repo:
+
+```bash
+git clone https://github.com/YuvrajSinghBhadoria2/inferlast.git
+pip install -r requirements.txt
+```
+
+## 30-second try
+
+```bash
+pip install -r requirements.txt          # torch CPU, transformers, pytest
+python scripts/run_all.py --model Qwen/Qwen2.5-0.5B-Instruct
+```
+
+One command. One report that tells you:
+
+```
+Bottleneck:  mlp 47% · attention 41% · norm 11%   (~99% overhead on CPU)
+Decode:      ~1220 ms/token (0.8 tok/s)
+Quantization: INT8 = 1.5x but shifts the answer → KEEP fp32 (not a win)
+Batching:    B=16 → ~48 tok/s total          Pick B for your goal.
+```
+
+And the part that catches the false wins — `trustcheck` on this repo's own
+recorded evidence (the same config that reported 3.0x faster):
+
+<p align="center">
+  <img src="assets/trustcheck-proof.png" alt="trustcheck catches a false win" width="640"/>
+</p>
+
+Run the individual stages to look closer:
+
+```bash
+python scripts/bench.py --model <m>              # prefill profile
+python scripts/bench.py --model <m> --decode      # decode / per-token latency
+python scripts/bench.py --model <m> --quant       # auto-quantization verdict
+python scripts/bench.py --model <m> --batch       # latency vs throughput sweep
+python scripts/bench.py --model <m> --trustcheck  # is that win real? (see below)
+python scripts/bench.py --model <m> --gpucheck    # do you even need a GPU? (see below)
+```
+
+## What it caught on my machine
+
+A quiet, annoying truth that most optimization tutorials skip: **on an overhead-bound CPU model, INT8 quantization is not a free win.** inferlast measured it three ways and told the truth:
+
+| | fp32 | INT8 | verdict |
+|---|---|---|---|
+| speed (Qwen 0.5B) | 1.31 s/tok | 0.87 s/tok | 1.5x — *but* |
+| quality (logit-cosine) | — | 0.68 | preserved-ish |
+| quality (top-5 overlap) | — | **0.15** | **token ranking shifted** |
+
+INT8 was faster but disturbed what the model would actually say. inferlast's call: **KEEP fp32.** That's not a bug — it's the tool doing its job: *refuse to recommend a change that isn't a real win.*
+
+The full measured records are in `benchmarks/` (JSON + markdown), published as measured — the failures included.
+
+## How it works
+
+Six small modules, one job each:
+
+| Module | Job |
+|---|---|
+| `src/profiler.py` | Per-category (attention / mlp / norm / embed / head) wall-clock profile, and the overhead-vs-weight split. |
+| `src/quantize.py` | Auto-quantization (INT8 dynamic). Measures fp32 vs INT8 averaged over repeats, with a **robust quality metric** (per-token logit cosine + top-5 overlap) — not brittle greedy-token identity. |
+| `src/batcher.py` | Latency-vs-throughput sweep over batch size, with a best-batch picker. |
+| `src/trustcheck.py` | **Is that 'win' worth trusting?** Audits any before/after benchmark for the three ways it lies: single-run noise, a brittle/wrong metric, and a "validated, documented, read by nothing" knob. Returns a REAL / MARGINAL / FALSE verdict. |
+| `src/gpucheck.py` | **Do you even need a GPU?** Estimates, from CPU-only measurements, whether GPU spend would actually beat the best-scheduled CPU config. Returns GPU-warranted / CPU-suffices / insufficient-data — and refuses to guess when it can't tell. |
+| `src/auto_optimizer.py` | Orchestrator: runs all four, emits a combined proof report + JSON. |
+| `scripts/` | `run_all.py` (one command) + `bench.py` (per stage). |
+
+A key design decision: `run_all.py` **always persists** a canonical report, so the evidence on disk always matches the latest run — it can't go stale.
+
+## gpucheck — the part that stops you overspending on hardware
+
+The whole project is built around one falsifiable claim (in
+[`docs/RESEARCH-SPEC.md`](docs/RESEARCH-SPEC.md)): **for overhead-bound small models on CPU, you
+usually don't need a GPU at all.** `gpucheck` puts that to the test from local CPU measurement:
+
+```bash
+python scripts/bench.py --model <m> \
+  --gpucheck --num-params 0.5e9 --overhead-fraction 0.98 --latency-target-ms 2000
+```
+
+It labels the decision — **GPU-warranted / CPU-suffices / insufficient-data** — with the exact inputs
+and reasoning, and it **refuses to guess** (returns `insufficient-data`) whenever it genuinely cannot
+tell from the data you gave it. That refusal is a feature: it never sells you a GPU rental it can't
+defend.
+
+## trustcheck — the part that tells you your benchmark lied
+
+Most tools *produce* a number. `trustcheck` tells you whether to believe it. It caught all three lies live on this repo's own evidence:
+
+- **Single-run noise.** The same INT8-vs-fp32 config, measured twice, gave `3.0x` faster in one session and `0.65x` **slower** in another. `trustcheck` computes the confidence interval and says: *"CI [-1.8x, 5.5x] straddles 1.0x → not a reliable win."* A naive dashboard would have reported `3.0x`.
+- **Brittle metric.** A quality comparison at greedy-token level, with no logits, is flagged as "can lie" and, when logits are available, re-measured with logit cosine + top-5 overlap.
+- **Read-by-nothing knob.** A flag that is documented/validated but never read by any code is a silent bug (the class of bug Soup and vLLM chased); `trustcheck`'s static pass flags it.
+
+```bash
+python scripts/bench.py --trustcheck \
+  --controls 1180 1316 --treatments 390 2031      # audit two real sessions
+python scripts/bench.py --model <m> --trustcheck \
+  --collect-repeats 2 --config-key stream_layers   # measure the noise band live
+```
+
+## Tests
+
+```bash
+pip install pytest
+python -m pytest        # 55 fast tests, no model downloads
+```
+
+The suite guards the things that would sink a tool like this: profiler categorisation & no-double-counting, the robust INT8 quality metric + the honest decision rule, batcher best-batch selection, the `trustcheck` noise/brittle-metric/read-by-nothing logic, the `gpucheck` GPU-necessity decision rule (including its refusal to guess when data is missing), and a **regression test that the report always emits the new metrics — never a stale one.**
+
+## Roadmap
+
+Phase 1 is done and runs on a plain laptop CPU — no GPU needed.
+
+**Shipped:**
+
+- **`trustcheck` — the false-win catcher.** Tells you the benchmark you were
+  about to publish is machine noise, not a win. It caught *this repo's own*
+  3.0x-vs-0.65x as noise.
+- **`gpucheck` — "do you even need a GPU?"** A CPU-only decision rule:
+  GPU-warranted / CPU-suffices / insufficient-data. It refuses to guess when it
+  can't tell.
+- Bottleneck / decode / quant / batch selection.
+
+**Where we want help next** (strongest help first):
+
+- **Validate the `gpucheck` boundary on hardware we can't reach.** Your numbers
+  ship behind an honest "requires <hardware>" gate — a 4 GB card, an Apple
+  Silicon box, a desktop with more RAM. Proof on hardware we lack turns a claim
+  into a finding. *This is the most valuable way to contribute right now.*
+- Quantization beyond INT8 (FP4 / INT4), latency percentiles (p50/p99), memory /
+  KV-cache footprint, and serving-engine integration (vLLM / llama.cpp) as an
+  enrichment layer.
+
+PRs welcome. Nothing here is a live claim — it's the plan.
+
+## License
+
+MIT — free, stays free, built in the open. If inferlast saved you a guessing session, a star helps others find it.
+
+<sub>Not a replacement for vLLM / llama.cpp — a *decision layer* that runs on CPU and tells you which setting is right for your model and hardware, with proof, before you spend on a GPU.</sub>

inferlast-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "inferlast"
+version = "0.1.0"
+description = "CPU-first LLM inference optimizer: prove a real win on your laptop CPU before you pay to rent a GPU. Trustcheck catches any benchmark that is really just noise."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Yuvraj Singh Bhadoria", email = "yuvrajsinghbhado2030@gmail.com" }]
+keywords = ["llm", "inference", "optimization", "cpu", "quantization", "benchmark", "trustcheck"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "torch>=2.2,<2.5",
+    "transformers>=4.40,<4.47",
+    "numpy>=1.26,<2",
+]
+
+[project.urls]
+Homepage = "https://github.com/YuvrajSinghBhadoria2/inferlast"
+Repository = "https://github.com/YuvrajSinghBhadoria2/inferlast"
+Releases = "https://github.com/YuvrajSinghBhadoria2/inferlast/releases"
+
+[project.scripts]
+inferlast = "cli:main"
+
+[tool.setuptools]
+# The src/ modules are intentionally top-level importables (import trustcheck,
+# import gpucheck, ...) — identical to how the scripts and tests already use them.
+package-dir = { "" = "src" }
+py-modules = [
+    "auto_optimizer",
+    "batcher",
+    "cli",
+    "gpucheck",
+    "profiler",
+    "quantize",
+    "trustcheck",
+]

inferlast-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

inferlast-0.1.0/src/auto_optimizer.py

@@ -0,0 +1,159 @@
+"""Auto-Optimizer orchestrator: one command -> optimized config + proof report.
+
+Runs all four measurements on a given model and emits:
+  * profile (prefill per-category bottleneck + overhead-vs-weight fraction)
+  * decode (ms/token, tok/s)
+  * auto-quantization (fp32 vs int8 BEFORE/AFTER + honest verdict)
+  * auto-batching (latency vs throughput sweep + best batch for a goal)
+
+Returns one dict with every result plus a human-readable proof report string.
+Everything is measured on the caller's hardware (CPU here). The whole point:
+refuse to recommend a change that doesn't provably help.
+"""
+
+from __future__ import annotations
+import time
+
+import torch
+
+from profiler import profile_forward, profile_decode
+from quantize import auto_quantize
+from batcher import bench_batch, best_batch
+
+
+def _overhead_frac(pr) -> float:
+    """fraction of wall time NOT accounted for by leaf matmul spans."""
+    leaf = sum(pr.cat_times.values())
+    return 1.0 - (leaf / max(pr.total_time, 1e-9))
+
+
+def run_auto_optimizer(
+    model,
+    tok,
+    prompt: str = "The capital of France is the city of",
+    n_new: int = 12,
+    batches=(1, 2, 4, 8, 16),
+    batch_goal: str = "throughput",
+    quant_speedup_floor: float = 1.05,
+    quant_quality_floor: float = 0.65,
+    quant_top5_floor: float = 0.60,
+) -> tuple[dict, str]:
+    out: dict = {
+        "prompt": prompt,
+        "hardware": "CPU (2019 MacBook Pro 16in i7-9750H, 16GB)",
+        "model_untouched": True,
+    }
+
+    # 1 & 2: bottleneck profile (prefill) + overhead fraction
+    t0 = time.time()
+    ids = tok([prompt], return_tensors="pt").input_ids
+    pr_pre = profile_forward(model, ids, run_repeats=2, warmup=1)
+    out["prefill"] = {
+        "wall_total_s": pr_pre.total_time,
+        "tokens": pr_pre.tokens,
+        "tok_per_s": pr_pre.tokens / max(pr_pre.total_time, 1e-9),
+        "leaf_time_s": sum(pr_pre.cat_times.values()),
+        "pct_of_leaf": {
+            k: round(v / max(sum(pr_pre.cat_times.values()), 1e-9) * 100, 1)
+            for k, v in pr_pre.cat_times.items()
+        },
+        "overhead_fraction": round(_overhead_frac(pr_pre), 3),
+    }
+
+    # decode
+    pr_dec = profile_decode(model, ids, n_new_tokens=n_new,
+                            run_repeats=1, warmup=1)
+    out["decode"] = {
+        "ms_per_token": pr_dec.total_time / n_new * 1000.0,
+        "tok_per_s": n_new / max(pr_dec.total_time, 1e-9),
+        "overhead_fraction": round(_overhead_frac(pr_dec), 3),
+    }
+
+    # quantization
+    qv = auto_quantize(model, tok, prompt=prompt, n_new=n_new,
+                       speedup_floor=quant_speedup_floor,
+                       quality_floor=quant_quality_floor,
+                       top5_floor=quant_top5_floor)
+    out["quantization"] = {
+        "fp32_ms_per_token": qv.before["ms_per_token"],
+        "int8_ms_per_token": qv.after["ms_per_token"],
+        "speedup_int8_over_fp32": round(qv.speedup, 3),
+        "speedup_worst_repeat": round(qv.speedup_min, 3),
+        "logit_cosine": round(qv.logit_cosine, 3),
+        "top5_overlap": round(qv.top5_overlap, 3),
+        "recommended_int8": qv.recommended,
+        "reason": qv.reason,
+    }
+
+    # batching
+    rows = bench_batch(model, tok, prompt=prompt, batches=batches,
+                       n_steps=1, warmup=1)
+    best = best_batch(rows, goal=batch_goal)
+    out["batching"] = {
+        "goal": batch_goal,
+        "rows": [
+            {"batch": r.batch, "wall_ms": r.wall_ms, "tok_per_s": r.tok_per_s,
+             "ms_per_req": r.ms_per_req, "tok_per_s_per_req": r.eff_tok_per_s_per_req}
+            for r in rows
+        ],
+        "best_batch": best.batch,
+        "best_tok_per_s": best.tok_per_s,
+    }
+    out["elapsed_s"] = round(time.time() - t0, 1)
+
+    report = _report(out)
+    return out, report
+
+
+def _report(o: dict) -> str:
+    p = o["prefill"]
+    d = o["decode"]
+    q = o["quantization"]
+    b = o["batching"]
+    overhead_note = (
+        f"{p['overhead_fraction']*100:.0f}% of prefill wall time and "
+        f"{d['overhead_fraction']*100:.0f}% of decode wall time is framework "
+        "overhead on CPU. NOTE: this is CPU-specific; on a GPU the same model "
+        "(with fused kernels) is typically weight/compute-bound, so the picture "
+        "differs. Here it means: on CPU, tiny models are overhead-bound."
+    )
+    lines = [
+        "# Auto-Optimizer proof report",
+        f"hardware: {o['hardware']}   prompt: {o['prompt']}",
+        "",
+        "## 1. Bottleneck (from profile)",
+        f"prefill throughput ~{p['tok_per_s']:.1f} tok/s; wall {p['wall_total_s']:.2f}s "
+        f"for {p['tokens']} tokens.",
+        "per-category (% of measured leaf work): " + ", ".join(
+            f"{k} {v}%" for k, v in sorted(p['pct_of_leaf'].items(),
+                                           key=lambda x: -x[1])
+        ),
+        f"overhead-vs-weight: {overhead_note}",
+        "",
+        "## 2. Decode",
+        f"{d['ms_per_token']:.0f} ms/token (~{d['tok_per_s']:.2f} tok/s). "
+        "Single-token serial, latency-sensitive.",
+        "",
+        "## 3. Quantization (fp32 vs int8, measured BEFORE/AFTER, averaged)",
+        f"fp32 {q['fp32_ms_per_token']:.0f} ms/tok -> int8 {q['int8_ms_per_token']:.0f} ms/tok "
+        f"= {q['speedup_int8_over_fp32']:.2f}x (worst repeat {q['speedup_worst_repeat']:.2f}x), "
+        f"quality logit-cos {q['logit_cosine']:.2f}, top-5 {q['top5_overlap']:.2f}.",
+        f"suggested: {'USE int8' if q['recommended_int8'] else 'KEEP fp32'} — {q['reason']}",
+        "",
+        "## 4. Batching (latency vs throughput)",
+        "B | wall(ms) | tok/s | ms/req | tok/s/req",
+    ]
+    for r in b["rows"]:
+        lines.append(
+            f"{r['batch']} | {r['wall_ms']:.0f} | {r['tok_per_s']:.1f} | "
+            f"{r['ms_per_req']:.0f} | {r['tok_per_s_per_req']:.2f}"
+        )
+    lines += [
+        f"best batch for goal='{b['goal']}': B={b['best_batch']} "
+        f"-> {b['best_tok_per_s']:.1f} tok/s total.",
+        "",
+        "## Honest takeaway",
+        "The model was never permanently modified; any optimization is advised "
+        "ONLY if it provably helps.",
+    ]
+    return "\n".join(lines)