rigtune 0.2.0__tar.gz

rigtune-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aagam Sachin Bothara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rigtune-0.2.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: rigtune
+Version: 0.2.0
+Summary: Autotuner for LLM serving: measures vLLM, SGLang and llama.cpp settings on your GPU and your traffic, then serves the fastest
+Author-email: Aagam Sachin Bothara <mweeb19@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Aagam-Bothara/rigtune
+Project-URL: Documentation, https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md
+Project-URL: Benchmarks, https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md
+Project-URL: Issues, https://github.com/Aagam-Bothara/rigtune/issues
+Keywords: llm,inference,vllm,llama.cpp,serving,autotuning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.14,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: psutil>=5.9
+Requires-Dist: huggingface_hub>=0.23
+Provides-Extra: nvml
+Requires-Dist: nvidia-ml-py>=12.535.133; extra == "nvml"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: ruff==0.15.0; extra == "dev"
+Dynamic: license-file
+
+# rigtune
+
+[![tests](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml/badge.svg)](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml)
+
+*Formerly PolyServe, renamed because a separate [2025 paper](https://arxiv.org/abs/2507.17769) already has that name. Profiles, results and settings from before the rename keep working ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md#renamed-from-polyserve)).*
+
+**rigtune finds the fastest way to serve an LLM on your GPU and your traffic, then serves it behind an OpenAI-compatible API.** It measures real configurations — engine, weight precision, batch size, KV-cache type, speculative decoding — instead of guessing, because the settings that won changed on every card tried.
+
+**It beats a hand-tuned configuration by 56–97%**, which is the comparison that matters — not the one against untouched defaults.
+
+| Llama 3.1 8B, held-out prompts | rigtune | a fixed rule of thumb † | (stock defaults) |
+|---|---|---|---|
+| RTX 4090 | **1041 tok/s** | 667 — **+56%** | SGLang 448 |
+| A100 | **1155 tok/s** | 674 — **+71%** | vLLM 726 |
+| A40 | **506 tok/s** | 257 — **+97%** | vLLM 262 |
+
+† fp8 weights and KV cache, a short context, batch 256: what an informed engineer sets without measuring. That rule was **slower than changing nothing** on both Ampere cards, because an fp8 KV cache costs throughput there — which is the point: the settings that win move from card to card, so a rule that is right somewhere is wrong elsewhere. The stock column is context rather than a claim; nobody serious ships untouched defaults. Each row was measured on prompts held out of calibration ([full sessions](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#llama-31-8b-on-four-gpus)).
+
+## Results
+
+- **Your traffic decides what pays, not the card alone.** On one A40 with Qwen2.5-3B the winning setting changed with the prompts: suffix decoding on Dolly prompts (+122% over stock), a draft model on news-article extraction (+66.5%), an fp8 KV cache on real chat (+10%). A tuner that ignores your prompts cannot find these.
+- **The pick holds on prompts it never saw.** Every number here was measured on prompts held out of calibration: 300 Dolly-15k prompts to tune on, 300 others to score on. An H100 not in the table gained +24% over stock vLLM with fp8 weights.
+- **Calibration pays for itself within hours.** 22–56 minutes per model and workload, repaid by 0.5–2.6 hours of busy serving. Measuring the busiest load first cuts 29–45% off that without changing a pick.
+- **What the search order is worth: one win, one tie.** Given the same time, random sampling of the same space matched this staged search on all three 8B cards. On a 14B in fp8 on a 24 GB card the staged search won by 4.4% (385 against 369 tok/s); on the same card in the 7× larger space that includes 4-bit checkpoints, the two tied (1412 against 1416) even though random search was given less time ([the evidence](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#qwen25-14b-where-memory-binds)). What staging reliably gives is the same answer every run, every strategy tried at least once, and a profile saying what each setting was worth — the gains come from measuring, not from the order.
+
+Every result, with methods, ablations, quality checks and limits: [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md).
+
+## How this differs from vLLM's auto-tune and AIConfigurator
+
+Not benchmarked against either — the comparison below is one of *kind*, not of measured results, and saying otherwise would be the overclaiming this project tries to avoid.
+
+- **vLLM's `auto_tune`** sweeps one engine's throughput knobs — batch size and batched-token budget — to find the best setting that still meets a latency target. Same spirit, narrower question: it tunes vLLM once you have already chosen vLLM, your weight precision and your decoding strategy. rigtune treats the engine itself as a variable, alongside precision, KV-cache type and speculative decoding — the dimensions that carried almost all the gain here. On an RTX 4090 SGLang led the early trials and vLLM only overtook it once speculative decoding and a quantized cache were tried — tuning the early leader alone would have eliminated the eventual winner, and a single-engine sweep never sees the crossover at all.
+- **AIConfigurator** predicts good deployments analytically from a performance model, aimed at TensorRT-LLM, and answers sizing questions — how many GPUs, which parallelism — very cheaply because it does not launch anything. rigtune measures instead, which costs 20–50 minutes per workload and is why its answers are specific to your prompts. We tried the analytical shortcut and kept it small for a reason: our own fitted predictor reaches a rank correlation of 0.88–0.94, enough to prune quantizations that cannot win, but it ranks the true winner at a median of 15th out of 25 measured configurations — so prediction is a filter here, never the decision ([why](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md)).
+- **What none of the three does** is prove the others wrong. The honest gap is that no one outside this project has run rigtune, and we have not run their tools on our hardware.
+
+## Quickstart
+
+```bash
+pip install rigtune
+pip install "vllm==0.29.0"     # driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
+rigtune serve Qwen/Qwen2.5-3B-Instruct --workload chat
+curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen2.5-3B-Instruct","messages":[{"role":"user","content":"hi"}]}'
+```
+
+The first launch calibrates and caches the result per machine, model, objective and workload; later launches serve at once. To tune for your own traffic, pass a sample of it: `--workload-file prompts.jsonl`, one prompt per line. Optional: `pip install arctic-inference==0.1.1` adds suffix decoding; SGLang can live in its own environment (set `SGLANG_PYTHON` to its `python`); llama.cpp needs `llama-server` built with CUDA, on `PATH` or in `LLAMA_SERVER`.
+
+With Docker (built on the official vLLM image):
+
+```bash
+docker build -t rigtune .
+docker run --gpus all --ipc=host -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v ~/.rigtune:/root/.rigtune rigtune serve Qwen/Qwen2.5-3B-Instruct
+```
+
+### Supported hardware
+
+Linux, Python 3.10–3.13. NVIDIA GPUs of compute capability 7.5 or newer run vLLM, SGLang and llama.cpp (measured on an A40, A100, H100 NVL, L4 and RTX 4090); older NVIDIA GPUs and x86 CPUs run llama.cpp. vLLM-CPU and pre-Turing GPUs have never been benchmarked ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#hardware-and-engines-measured)).
+
+## Learn more
+
+- [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md): every result, at a glance and in full, with methods, ablations, quality, limits and what is still unmeasured.
+- [docs/usage.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md): how calibration works, workloads, objectives, every search option and the CLI.
+- [docs/writeup.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/writeup.md): the design of the memory planner, calibration and the predictor, what the evidence does and does not support, and the roadmap.
+- [docs/decisions.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md): why each choice is the way it is — including why not just random search — with the measurement that settled it.
+- [benchmarks/strategies/SUMMARY.md](https://github.com/Aagam-Bothara/rigtune/blob/main/benchmarks/strategies/SUMMARY.md): every table, regenerated from the raw JSON.
+- [CONTRIBUTING.md](https://github.com/Aagam-Bothara/rigtune/blob/main/CONTRIBUTING.md): development setup, tests and adding a backend.
+
+MIT licensed.

rigtune-0.2.0/README.md

@@ -0,0 +1,69 @@
+# rigtune
+
+[![tests](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml/badge.svg)](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml)
+
+*Formerly PolyServe, renamed because a separate [2025 paper](https://arxiv.org/abs/2507.17769) already has that name. Profiles, results and settings from before the rename keep working ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md#renamed-from-polyserve)).*
+
+**rigtune finds the fastest way to serve an LLM on your GPU and your traffic, then serves it behind an OpenAI-compatible API.** It measures real configurations — engine, weight precision, batch size, KV-cache type, speculative decoding — instead of guessing, because the settings that won changed on every card tried.
+
+**It beats a hand-tuned configuration by 56–97%**, which is the comparison that matters — not the one against untouched defaults.
+
+| Llama 3.1 8B, held-out prompts | rigtune | a fixed rule of thumb † | (stock defaults) |
+|---|---|---|---|
+| RTX 4090 | **1041 tok/s** | 667 — **+56%** | SGLang 448 |
+| A100 | **1155 tok/s** | 674 — **+71%** | vLLM 726 |
+| A40 | **506 tok/s** | 257 — **+97%** | vLLM 262 |
+
+† fp8 weights and KV cache, a short context, batch 256: what an informed engineer sets without measuring. That rule was **slower than changing nothing** on both Ampere cards, because an fp8 KV cache costs throughput there — which is the point: the settings that win move from card to card, so a rule that is right somewhere is wrong elsewhere. The stock column is context rather than a claim; nobody serious ships untouched defaults. Each row was measured on prompts held out of calibration ([full sessions](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#llama-31-8b-on-four-gpus)).
+
+## Results
+
+- **Your traffic decides what pays, not the card alone.** On one A40 with Qwen2.5-3B the winning setting changed with the prompts: suffix decoding on Dolly prompts (+122% over stock), a draft model on news-article extraction (+66.5%), an fp8 KV cache on real chat (+10%). A tuner that ignores your prompts cannot find these.
+- **The pick holds on prompts it never saw.** Every number here was measured on prompts held out of calibration: 300 Dolly-15k prompts to tune on, 300 others to score on. An H100 not in the table gained +24% over stock vLLM with fp8 weights.
+- **Calibration pays for itself within hours.** 22–56 minutes per model and workload, repaid by 0.5–2.6 hours of busy serving. Measuring the busiest load first cuts 29–45% off that without changing a pick.
+- **What the search order is worth: one win, one tie.** Given the same time, random sampling of the same space matched this staged search on all three 8B cards. On a 14B in fp8 on a 24 GB card the staged search won by 4.4% (385 against 369 tok/s); on the same card in the 7× larger space that includes 4-bit checkpoints, the two tied (1412 against 1416) even though random search was given less time ([the evidence](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#qwen25-14b-where-memory-binds)). What staging reliably gives is the same answer every run, every strategy tried at least once, and a profile saying what each setting was worth — the gains come from measuring, not from the order.
+
+Every result, with methods, ablations, quality checks and limits: [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md).
+
+## How this differs from vLLM's auto-tune and AIConfigurator
+
+Not benchmarked against either — the comparison below is one of *kind*, not of measured results, and saying otherwise would be the overclaiming this project tries to avoid.
+
+- **vLLM's `auto_tune`** sweeps one engine's throughput knobs — batch size and batched-token budget — to find the best setting that still meets a latency target. Same spirit, narrower question: it tunes vLLM once you have already chosen vLLM, your weight precision and your decoding strategy. rigtune treats the engine itself as a variable, alongside precision, KV-cache type and speculative decoding — the dimensions that carried almost all the gain here. On an RTX 4090 SGLang led the early trials and vLLM only overtook it once speculative decoding and a quantized cache were tried — tuning the early leader alone would have eliminated the eventual winner, and a single-engine sweep never sees the crossover at all.
+- **AIConfigurator** predicts good deployments analytically from a performance model, aimed at TensorRT-LLM, and answers sizing questions — how many GPUs, which parallelism — very cheaply because it does not launch anything. rigtune measures instead, which costs 20–50 minutes per workload and is why its answers are specific to your prompts. We tried the analytical shortcut and kept it small for a reason: our own fitted predictor reaches a rank correlation of 0.88–0.94, enough to prune quantizations that cannot win, but it ranks the true winner at a median of 15th out of 25 measured configurations — so prediction is a filter here, never the decision ([why](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md)).
+- **What none of the three does** is prove the others wrong. The honest gap is that no one outside this project has run rigtune, and we have not run their tools on our hardware.
+
+## Quickstart
+
+```bash
+pip install rigtune
+pip install "vllm==0.29.0"     # driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
+rigtune serve Qwen/Qwen2.5-3B-Instruct --workload chat
+curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen2.5-3B-Instruct","messages":[{"role":"user","content":"hi"}]}'
+```
+
+The first launch calibrates and caches the result per machine, model, objective and workload; later launches serve at once. To tune for your own traffic, pass a sample of it: `--workload-file prompts.jsonl`, one prompt per line. Optional: `pip install arctic-inference==0.1.1` adds suffix decoding; SGLang can live in its own environment (set `SGLANG_PYTHON` to its `python`); llama.cpp needs `llama-server` built with CUDA, on `PATH` or in `LLAMA_SERVER`.
+
+With Docker (built on the official vLLM image):
+
+```bash
+docker build -t rigtune .
+docker run --gpus all --ipc=host -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v ~/.rigtune:/root/.rigtune rigtune serve Qwen/Qwen2.5-3B-Instruct
+```
+
+### Supported hardware
+
+Linux, Python 3.10–3.13. NVIDIA GPUs of compute capability 7.5 or newer run vLLM, SGLang and llama.cpp (measured on an A40, A100, H100 NVL, L4 and RTX 4090); older NVIDIA GPUs and x86 CPUs run llama.cpp. vLLM-CPU and pre-Turing GPUs have never been benchmarked ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#hardware-and-engines-measured)).
+
+## Learn more
+
+- [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md): every result, at a glance and in full, with methods, ablations, quality, limits and what is still unmeasured.
+- [docs/usage.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md): how calibration works, workloads, objectives, every search option and the CLI.
+- [docs/writeup.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/writeup.md): the design of the memory planner, calibration and the predictor, what the evidence does and does not support, and the roadmap.
+- [docs/decisions.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md): why each choice is the way it is — including why not just random search — with the measurement that settled it.
+- [benchmarks/strategies/SUMMARY.md](https://github.com/Aagam-Bothara/rigtune/blob/main/benchmarks/strategies/SUMMARY.md): every table, regenerated from the raw JSON.
+- [CONTRIBUTING.md](https://github.com/Aagam-Bothara/rigtune/blob/main/CONTRIBUTING.md): development setup, tests and adding a backend.
+
+MIT licensed.

rigtune-0.2.0/pyproject.toml

@@ -0,0 +1,67 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rigtune"
+version = "0.2.0"
+description = "Autotuner for LLM serving: measures vLLM, SGLang and llama.cpp settings on your GPU and your traffic, then serves the fastest"
+readme = "README.md"
+requires-python = ">=3.10,<3.14"
+license = {text = "MIT"}
+authors = [{name = "Aagam Sachin Bothara", email = "mweeb19@gmail.com"}]
+keywords = ["llm", "inference", "vllm", "llama.cpp", "serving", "autotuning"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "pydantic>=2.0",
+    "typer>=0.12",
+    "rich>=13.0",
+    "httpx>=0.27",
+    "fastapi>=0.110",
+    "uvicorn>=0.29",
+    "psutil>=5.9",
+    "huggingface_hub>=0.23",
+]
+
+[project.optional-dependencies]
+nvml = ["nvidia-ml-py>=12.535.133"]
+# llmtrace, the GPU tracer rigtune prefers for telemetry (it falls back to pynvml, then psutil), is not an
+# extra: PyPI refuses packages with git dependencies, and "llmtrace" on PyPI is an unrelated project. Install
+# it from git: pip install "llmtrace @ git+https://github.com/Aagam-Bothara/llmtrace.git"
+# ruff is pinned: its default rules change between releases (0.16 added pyupgrade), so an unpinned
+# linter passed locally and failed in CI on the same code.
+dev = ["pytest>=7.0", "pytest-asyncio>=0.23", "ruff==0.15.0"]
+
+[project.urls]
+Homepage = "https://github.com/Aagam-Bothara/rigtune"
+Documentation = "https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md"
+Benchmarks = "https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md"
+Issues = "https://github.com/Aagam-Bothara/rigtune/issues"
+
+[project.scripts]
+rigtune = "rigtune.cli:app"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["rigtune*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+asyncio_mode = "auto"
+
+[tool.ruff]
+line-length = 110
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F"]  # explicit, so the rule set does not move with ruff's defaults

rigtune-0.2.0/rigtune.egg-info/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: rigtune
+Version: 0.2.0
+Summary: Autotuner for LLM serving: measures vLLM, SGLang and llama.cpp settings on your GPU and your traffic, then serves the fastest
+Author-email: Aagam Sachin Bothara <mweeb19@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Aagam-Bothara/rigtune
+Project-URL: Documentation, https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md
+Project-URL: Benchmarks, https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md
+Project-URL: Issues, https://github.com/Aagam-Bothara/rigtune/issues
+Keywords: llm,inference,vllm,llama.cpp,serving,autotuning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.14,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: psutil>=5.9
+Requires-Dist: huggingface_hub>=0.23
+Provides-Extra: nvml
+Requires-Dist: nvidia-ml-py>=12.535.133; extra == "nvml"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: ruff==0.15.0; extra == "dev"
+Dynamic: license-file
+
+# rigtune
+
+[![tests](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml/badge.svg)](https://github.com/Aagam-Bothara/rigtune/actions/workflows/tests.yml)
+
+*Formerly PolyServe, renamed because a separate [2025 paper](https://arxiv.org/abs/2507.17769) already has that name. Profiles, results and settings from before the rename keep working ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md#renamed-from-polyserve)).*
+
+**rigtune finds the fastest way to serve an LLM on your GPU and your traffic, then serves it behind an OpenAI-compatible API.** It measures real configurations — engine, weight precision, batch size, KV-cache type, speculative decoding — instead of guessing, because the settings that won changed on every card tried.
+
+**It beats a hand-tuned configuration by 56–97%**, which is the comparison that matters — not the one against untouched defaults.
+
+| Llama 3.1 8B, held-out prompts | rigtune | a fixed rule of thumb † | (stock defaults) |
+|---|---|---|---|
+| RTX 4090 | **1041 tok/s** | 667 — **+56%** | SGLang 448 |
+| A100 | **1155 tok/s** | 674 — **+71%** | vLLM 726 |
+| A40 | **506 tok/s** | 257 — **+97%** | vLLM 262 |
+
+† fp8 weights and KV cache, a short context, batch 256: what an informed engineer sets without measuring. That rule was **slower than changing nothing** on both Ampere cards, because an fp8 KV cache costs throughput there — which is the point: the settings that win move from card to card, so a rule that is right somewhere is wrong elsewhere. The stock column is context rather than a claim; nobody serious ships untouched defaults. Each row was measured on prompts held out of calibration ([full sessions](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#llama-31-8b-on-four-gpus)).
+
+## Results
+
+- **Your traffic decides what pays, not the card alone.** On one A40 with Qwen2.5-3B the winning setting changed with the prompts: suffix decoding on Dolly prompts (+122% over stock), a draft model on news-article extraction (+66.5%), an fp8 KV cache on real chat (+10%). A tuner that ignores your prompts cannot find these.
+- **The pick holds on prompts it never saw.** Every number here was measured on prompts held out of calibration: 300 Dolly-15k prompts to tune on, 300 others to score on. An H100 not in the table gained +24% over stock vLLM with fp8 weights.
+- **Calibration pays for itself within hours.** 22–56 minutes per model and workload, repaid by 0.5–2.6 hours of busy serving. Measuring the busiest load first cuts 29–45% off that without changing a pick.
+- **What the search order is worth: one win, one tie.** Given the same time, random sampling of the same space matched this staged search on all three 8B cards. On a 14B in fp8 on a 24 GB card the staged search won by 4.4% (385 against 369 tok/s); on the same card in the 7× larger space that includes 4-bit checkpoints, the two tied (1412 against 1416) even though random search was given less time ([the evidence](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#qwen25-14b-where-memory-binds)). What staging reliably gives is the same answer every run, every strategy tried at least once, and a profile saying what each setting was worth — the gains come from measuring, not from the order.
+
+Every result, with methods, ablations, quality checks and limits: [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md).
+
+## How this differs from vLLM's auto-tune and AIConfigurator
+
+Not benchmarked against either — the comparison below is one of *kind*, not of measured results, and saying otherwise would be the overclaiming this project tries to avoid.
+
+- **vLLM's `auto_tune`** sweeps one engine's throughput knobs — batch size and batched-token budget — to find the best setting that still meets a latency target. Same spirit, narrower question: it tunes vLLM once you have already chosen vLLM, your weight precision and your decoding strategy. rigtune treats the engine itself as a variable, alongside precision, KV-cache type and speculative decoding — the dimensions that carried almost all the gain here. On an RTX 4090 SGLang led the early trials and vLLM only overtook it once speculative decoding and a quantized cache were tried — tuning the early leader alone would have eliminated the eventual winner, and a single-engine sweep never sees the crossover at all.
+- **AIConfigurator** predicts good deployments analytically from a performance model, aimed at TensorRT-LLM, and answers sizing questions — how many GPUs, which parallelism — very cheaply because it does not launch anything. rigtune measures instead, which costs 20–50 minutes per workload and is why its answers are specific to your prompts. We tried the analytical shortcut and kept it small for a reason: our own fitted predictor reaches a rank correlation of 0.88–0.94, enough to prune quantizations that cannot win, but it ranks the true winner at a median of 15th out of 25 measured configurations — so prediction is a filter here, never the decision ([why](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md)).
+- **What none of the three does** is prove the others wrong. The honest gap is that no one outside this project has run rigtune, and we have not run their tools on our hardware.
+
+## Quickstart
+
+```bash
+pip install rigtune
+pip install "vllm==0.29.0"     # driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
+rigtune serve Qwen/Qwen2.5-3B-Instruct --workload chat
+curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen2.5-3B-Instruct","messages":[{"role":"user","content":"hi"}]}'
+```
+
+The first launch calibrates and caches the result per machine, model, objective and workload; later launches serve at once. To tune for your own traffic, pass a sample of it: `--workload-file prompts.jsonl`, one prompt per line. Optional: `pip install arctic-inference==0.1.1` adds suffix decoding; SGLang can live in its own environment (set `SGLANG_PYTHON` to its `python`); llama.cpp needs `llama-server` built with CUDA, on `PATH` or in `LLAMA_SERVER`.
+
+With Docker (built on the official vLLM image):
+
+```bash
+docker build -t rigtune .
+docker run --gpus all --ipc=host -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v ~/.rigtune:/root/.rigtune rigtune serve Qwen/Qwen2.5-3B-Instruct
+```
+
+### Supported hardware
+
+Linux, Python 3.10–3.13. NVIDIA GPUs of compute capability 7.5 or newer run vLLM, SGLang and llama.cpp (measured on an A40, A100, H100 NVL, L4 and RTX 4090); older NVIDIA GPUs and x86 CPUs run llama.cpp. vLLM-CPU and pre-Turing GPUs have never been benchmarked ([details](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md#hardware-and-engines-measured)).
+
+## Learn more
+
+- [docs/benchmarks.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/benchmarks.md): every result, at a glance and in full, with methods, ablations, quality, limits and what is still unmeasured.
+- [docs/usage.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/usage.md): how calibration works, workloads, objectives, every search option and the CLI.
+- [docs/writeup.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/writeup.md): the design of the memory planner, calibration and the predictor, what the evidence does and does not support, and the roadmap.
+- [docs/decisions.md](https://github.com/Aagam-Bothara/rigtune/blob/main/docs/decisions.md): why each choice is the way it is — including why not just random search — with the measurement that settled it.
+- [benchmarks/strategies/SUMMARY.md](https://github.com/Aagam-Bothara/rigtune/blob/main/benchmarks/strategies/SUMMARY.md): every table, regenerated from the raw JSON.
+- [CONTRIBUTING.md](https://github.com/Aagam-Bothara/rigtune/blob/main/CONTRIBUTING.md): development setup, tests and adding a backend.
+
+MIT licensed.

rigtune-0.2.0/rigtune.egg-info/SOURCES.txt

@@ -0,0 +1,69 @@
+LICENSE
+README.md
+pyproject.toml
+rigtune.egg-info/PKG-INFO
+rigtune.egg-info/SOURCES.txt
+rigtune.egg-info/dependency_links.txt
+rigtune.egg-info/entry_points.txt
+rigtune.egg-info/requires.txt
+rigtune.egg-info/top_level.txt
+tests/test_ablation.py
+tests/test_ablation_shrink.py
+tests/test_all_levels.py
+tests/test_attention_check.py
+tests/test_baselines.py
+tests/test_break_even.py
+tests/test_budget.py
+tests/test_cache_and_pipeline.py
+tests/test_cli.py
+tests/test_combinations.py
+tests/test_compare.py
+tests/test_compare_heldout.py
+tests/test_compare_repeats.py
+tests/test_confirm.py
+tests/test_contenders.py
+tests/test_cpu_quota.py
+tests/test_dataset_workloads.py
+tests/test_drift.py
+tests/test_early_stop_check.py
+tests/test_explore.py
+tests/test_fit_per_machine.py
+tests/test_gguf_and_backends.py
+tests/test_hardware_and_config.py
+tests/test_hub_search.py
+tests/test_int8_kv.py
+tests/test_load_pick_order.py
+tests/test_make_dolly_prompts.py
+tests/test_make_oasst_prompts.py
+tests/test_measure_and_proxy.py
+tests/test_measure_levels.py
+tests/test_memcal.py
+tests/test_memory.py
+tests/test_nvlink.py
+tests/test_objectives_and_search.py
+tests/test_oom_retry.py
+tests/test_phases.py
+tests/test_power.py
+tests/test_predict.py
+tests/test_process_group.py
+tests/test_progress_line.py
+tests/test_quality.py
+tests/test_quant_default.py
+tests/test_report.py
+tests/test_rescore_ttft.py
+tests/test_search_budget.py
+tests/test_search_view.py
+tests/test_selector.py
+tests/test_sglang_python.py
+tests/test_spec_first.py
+tests/test_spec_methods.py
+tests/test_strategies.py
+tests/test_strategy_report.py
+tests/test_supervisor.py
+tests/test_tail.py
+tests/test_task_quality.py
+tests/test_ttft_percentile.py
+tests/test_vllm_cli.py
+tests/test_w8a8.py
+tests/test_workload_file.py
+tests/test_workloads.py

rigtune-0.2.0/rigtune.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

rigtune-0.2.0/rigtune.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+rigtune = rigtune.cli:app

rigtune-0.2.0/rigtune.egg-info/requires.txt

@@ -0,0 +1,16 @@
+pydantic>=2.0
+typer>=0.12
+rich>=13.0
+httpx>=0.27
+fastapi>=0.110
+uvicorn>=0.29
+psutil>=5.9
+huggingface_hub>=0.23
+
+[dev]
+pytest>=7.0
+pytest-asyncio>=0.23
+ruff==0.15.0
+
+[nvml]
+nvidia-ml-py>=12.535.133

rigtune-0.2.0/rigtune.egg-info/top_level.txt

@@ -0,0 +1 @@
+

rigtune-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

rigtune-0.2.0/tests/test_ablation.py

@@ -0,0 +1,115 @@
+"""The strategy ablation: variants of the pick, result loading, sweeps and crossover."""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from rigtune.backends import get_backend
+from rigtune.bench.ablation import (
+    crossover, load_pick, markdown, score_row, strategy_variants, sweep_workload,
+)
+from rigtune.bench.compare import results_path
+from rigtune.calibrate.objectives import Constraints
+from rigtune.calibrate.workload import get_workload
+from rigtune.models import Config, ModelSpec, TrialMetrics, TrialResult
+
+
+@pytest.fixture
+def drafts(monkeypatch):
+    import rigtune.backends.vllm as vl
+    import rigtune.speculative as S
+
+    monkeypatch.setattr(S, "vllm_supports_draft", lambda version=None: True)
+    real = vl.importlib.util.find_spec
+    monkeypatch.setattr(vl.importlib.util, "find_spec", lambda n: object() if n == "flashinfer" else real(n))
+
+
+@pytest.mark.usefixtures("drafts")
+def test_variants_flip_each_strategy(hw_a100, prepared_vllm):
+    be = get_backend("vllm")
+    model = prepared_vllm.model_copy(update={"weights_bytes": {**prepared_vllm.weights_bytes, "awq": 2_000_000_000},
+                                             "hf_paths": {"awq": "org/Llama-3.2-3B-Instruct-AWQ"}})
+    pick = Config(backend="vllm", quant="fp8", ctx=4096, batch=64, gpu_memory_utilization=0.9)
+    labels = [v.label for v in strategy_variants(pick, be, hw_a100, model, get_workload("chat-system"))]
+    assert labels == ["+awq", "+kv:fp8_e5m2", "-prefix", "+spec:ngram:4",
+                      "+spec:draft:meta-llama/Llama-3.2-1B-Instruct:4"]
+    no_prefix = [v.label for v in strategy_variants(pick, be, hw_a100, model, get_workload("chat"))]
+    assert "-prefix" not in no_prefix  # nothing shared, nothing for a prefix cache to do
+
+    used = pick.model_copy(update={"quant": "awq", "kv_dtype": "fp8", "spec_decode": "ngram:4"})
+    vs = {v.label: v.config for v in strategy_variants(used, be, hw_a100, model, get_workload("chat"))}
+    assert set(vs) == {"-int4 (fp8)", "-kv", "-spec"}
+    assert vs["-int4 (fp8)"].quant == "fp8" and vs["-kv"].kv_dtype == "auto" and vs["-spec"].spec_decode is None
+
+
+def test_removing_kv_quant_steps_the_batch_down_to_fit(hw_gtx1080, prepared_llamacpp):
+    pick = Config(backend="llamacpp-cuda", quant="Q4_K_M", ctx=4096, batch=16, n_gpu_layers=29, n_batch=512,
+                  kv_dtype="q4_0")
+    (kv,) = [v for v in strategy_variants(pick, get_backend("llamacpp-cuda"), hw_gtx1080, prepared_llamacpp,
+                                          get_workload("chat")) if v.strategy == "kv"]
+    assert kv.label == "-kv" and kv.config.kv_dtype == "auto" and kv.config.batch == 8
+    assert "does not fit at batch 16" in kv.note
+
+
+def test_llamacpp_prefix_flags_are_added_or_removed(hw_gtx1080, prepared_llamacpp):
+    be = get_backend("llamacpp-cuda")
+    pick = Config(backend="llamacpp-cuda", quant="Q4_K_M", ctx=4096, batch=4, n_gpu_layers=29, n_batch=512)
+    shared = get_workload("chat-system")
+    added = [v.label for v in strategy_variants(pick, be, hw_gtx1080, prepared_llamacpp, shared)
+             if v.strategy == "prefix"]
+    assert added == ["+prefix:cache_reuse", "+prefix:kv_unified"]
+    tuned = pick.model_copy(update={"extra": {"kv_unified": True}})
+    (removed,) = [v for v in strategy_variants(tuned, be, hw_gtx1080, prepared_llamacpp, shared)
+                  if v.strategy == "prefix"]
+    assert removed.label == "-prefix" and removed.config.extra == {}
+
+
+def test_load_pick_prefers_this_machine(tmp_path):
+    spec = ModelSpec(hf_id="org/M")
+
+    def write(hw_hash: str, quant: str) -> None:
+        rows = [{"label": "vllm-default", "config": Config(backend="vllm", quant="bf16").model_dump()},
+                {"label": "rigtune", "config": Config(backend="vllm", quant=quant).model_dump()}]
+        results_path(hw_hash, spec, "chat", "balanced", tmp_path).write_text(json.dumps({"rows": rows}))
+
+    write("aaaa", "bf16")
+    write("zzzz", "fp8")
+    assert load_pick(tmp_path, "org/M", "chat", "balanced", hw_hash="zzzz").quant == "fp8"
+    assert load_pick(tmp_path, "org/M", "chat", "balanced").quant == "bf16"
+    assert load_pick(tmp_path, "org/M", "rag", "balanced") is None
+
+
+def test_sweep_workload_and_crossover():
+    base = get_workload("rag-shared")
+    wl = sweep_workload(base, [8, 1, 64, 4])
+    assert wl.concurrencies == (1, 4, 8, 64) and wl.n_prompts == 128 and len(wl.prompts) == 128
+    assert wl.prefix_text == base.prefix_text and all(p.startswith(wl.prefix_text) for p in wl.prompts)
+
+    off = {"1": {"ok": True, "tok_s": 60, "e2e_ms": 2000}, "8": {"ok": True, "tok_s": 400, "e2e_ms": 2500},
+           "32": {"ok": True, "tok_s": 1200, "e2e_ms": 3500}}
+    on = {"1": {"ok": True, "tok_s": 110, "e2e_ms": 1200}, "8": {"ok": True, "tok_s": 390, "e2e_ms": 2400},
+          "32": {"ok": True, "tok_s": 900, "e2e_ms": 4800}}
+    assert crossover(off, on, "tok_s") == 8
+    assert crossover(off, on, "e2e_ms") == 32
+    assert crossover(off, {"1": {"ok": True, "tok_s": 999, "e2e_ms": 1}}) is None
+
+
+def test_score_row_and_markdown():
+    def result(tok_s: float, cfg: Config) -> TrialResult:
+        level = TrialMetrics(tok_s=tok_s, ttft_ms=30, tpot_ms=10, requests=16, output_tokens=2048, concurrency=8)
+        return TrialResult(config=cfg, stage="t", metrics=level.model_copy(update={"by_concurrency": {"8": level}}))
+
+    cons = Constraints(ttft_ceiling_ms=500)
+    base = Config(backend="vllm", quant="fp8")
+    rows = [{"label": label, "config_key": cfg.key(), **score_row(result(tok, cfg), "balanced", cons)}
+            for label, tok, cfg in (("rigtune", 500, base), ("+kv:fp8", 550, base.model_copy(update={"kv_dtype": "fp8"})))]
+    rows.append({"label": "-kv", "config_key": "-", "ok": False, "note": "does not fit in memory"})
+    top = rows[0]
+    assert top["ok"] and top["concurrency"] == 8 and top["meets_slo"]
+    assert top["e2e_ms"] == pytest.approx(30 + 10 * 127) and top["levels"]["8"]["tok_s"] == 500
+    md = markdown(rows)
+    assert "+10.0%" in md and "does not fit in memory" in md
+    failed = score_row(TrialResult(config=base, stage="t", metrics=TrialMetrics(), error="boom"), "balanced", cons)
+    assert failed == {"ok": False, "error": "boom"}

rigtune-0.2.0/tests/test_ablation_shrink.py

@@ -0,0 +1,22 @@
+"""Removing a memory-saving strategy steps the batch down to what would have fitted without it."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import rigtune.bench.ablation as AB
+from rigtune.backends import get_backend
+from rigtune.calibrate.workload import get_workload
+from rigtune.models import Config
+
+
+def test_int4_removal_steps_the_batch_down_to_fit(monkeypatch, hw_a100, prepared_vllm):
+    # Measured on an A40 (chat-system): fp8 weights at the 4-bit pick's batch 256 did not fit.
+    monkeypatch.setattr(AB, "estimate",
+                        lambda hw, model, c, mm: SimpleNamespace(feasible=c.quant == "awq" or c.batch <= 64))
+    model = prepared_vllm.model_copy(update={"weights_bytes": {**prepared_vllm.weights_bytes, "awq": 2_000_000_000}})
+    pick = Config(backend="vllm", quant="awq", ctx=16384, batch=256, gpu_memory_utilization=0.95)
+    (w,) = [v for v in AB.strategy_variants(pick, get_backend("vllm"), hw_a100, model, get_workload("chat"))
+            if v.strategy == "weights"]
+    assert w.label == "-int4 (fp8)" and w.config.quant == "fp8" and w.config.batch == 64
+    assert w.note == "batch 64: fp8 weights does not fit at batch 256"