inference-doctor 0.4.0__tar.gz

inference_doctor-0.4.0/CHANGELOG.md

@@ -0,0 +1,94 @@
+# Changelog
+
+## Unreleased
+
+## 0.4.0 - 2026-10-11
+
+### Changed: the compare-runs decision rule
+
+- `compare-runs` decides with a confidence interval instead of a CV cut followed by a fixed threshold on the difference of means. It computes a Welch t interval on the log ratio of means (90% two-sided by default): `FAIL` when the whole interval for the regression is at or past `--regression-percent`, `PASS` when the whole interval is below it, and `INCONCLUSIVE` when it contains the threshold. Validity checks (environments, failed requests, missing metrics, repetition count) still come first.
+- Why: simulating the old rule showed a healthy benchmark (true CV 4%, no change, 5 repetitions) was `INCONCLUSIVE` about a third of the time, and with 2 repetitions a true 12% regression passed about 13% of the time. The new rule's rates are published in `docs/gate-calibration.md`, generated by `tools/gate_simulation.py` and checked in `tests/test_gate.py`.
+- The CV ceiling is now optional and off by default (`--stability-cv-percent`, Action input `stability-cv-percent`). Pass `--stability-cv-percent 5` to keep the old ceiling in addition to the interval.
+- At least 3 repetitions per side are required (`--min-repetitions`, default 3, was 2). 5 or more is recommended.
+- New options `--confidence` and `--min-repetitions`, and matching Action inputs. New Action outputs `regression-percent-low` and `regression-percent-high`.
+- The JSON report is `schema_version: inference-ci-quality-2`. It adds `confidence_interval`, `lower_is_better`, `notes`, and `thresholds.confidence` and `thresholds.min_repetitions`; `measurement_quality` adds `enough_repetitions`, and `environment_valid` no longer depends on the wording of a reason.
+- A candidate better than the baseline by more than the threshold gets a note asking to check that both sides measured the same workload.
+- The quickstart has five repetitions per side and a `regressed-candidate` directory that demonstrates `FAIL`.
+- The Action passes its inputs to the shell through environment variables instead of interpolating them into the script.
+
+### Fixed: compare and diagnose
+
+- `compare` no longer crashes with `UnboundLocalError` when a baseline value is 0; a rise from 0 in a latency, or a fall to 0 in a throughput, is a failure.
+- `compare` returns `INCONCLUSIVE` (exit 2) instead of `PASS` when no latency or throughput metric is present in both files, and its overall status is `WARN` (still exit 0) when any row is in the warning band.
+- `compare` is deprecated: it prints a notice on stderr pointing to `compare-runs`, and will be removed in a future release.
+- `diagnose`: the queue mean is aggregated across series (`sum(rate(_sum)) / sum(rate(_count))`); with two replicas at 0.2 s it reported 0.4 s. The Prometheus client now refuses a query that returns more than one series instead of adding them up.
+- `diagnose`: `QUEUE_PRESSURE` requires queue evidence (waiting requests, or a mean queue time of at least 20% of TTFT p95). TTFT over its SLO with a healthy TPOT used to be enough, which flagged prefill-heavy workloads as queueing.
+- `diagnose`: a new `PREFILL_PRESSURE` finding covers TTFT over its SLO with no queue evidence and prefill time at least half of TTFT p95. Prefill time and prompt length were collected before but not used.
+- `diagnose` exits 1 when it reports a critical finding; `--fail-on warning|never` changes the threshold. It always exited 0 before.
+
+### Added: gate several metrics at once
+
+- `--metric` can be repeated (Action: a comma-separated `metric` input). Each metric is compared in full; the combined result is `FAIL` if any metric fails, `INCONCLUSIVE` if none fails but one is inconclusive, and `PASS` only if all pass. The report notes that each extra metric adds a chance of a false alarm. With one metric the report is unchanged; with several it has `metrics` and `comparisons`. The Action's job summary has one row per metric, and a new `report` output holds the JSON report path.
+
+### Added: read benchmark tools' own output
+
+- `compare-runs` reads the result files of `vllm bench serve --save-result` (including `--append-result` JSON Lines), NVIDIA AIPerf and GenAI-Perf, GuideLLM `benchmarks.json`, and SGLang `bench_serving`, with no conversion step. `--format auto` (the default) detects the format per file; `--format` forces one. See `docs/input-formats.md`.
+- `--gpu` and `--model` record values the result files do not carry; `--guidellm-benchmark` picks one load point of a GuideLLM sweep. Matching Action inputs `format`, `gpu`, `model` and `guidellm-benchmark`.
+- p99 latencies, inter-token latency (`itl_*_ms`), end-to-end latency (`e2el_*_ms`) and total token throughput, in the normalized schema and as `--metric` choices. Every latency family has mean, median, p95 and p99.
+- The report shows the input format, notes when the GPU or model was not recorded or a tool does not record failed requests, and lists skipped files under `input.ignored_files`.
+- `examples/vllm-bench-serve` holds results in vLLM's own format, where throughput passes and mean TTFT fails.
+- Normalized results under any file name are detected, as long as no file in the directory uses one of the standard names.
+
+### Added
+
+- `inference-doctor --version`, and `python -m inference_doctor`.
+
+### Fixed
+
+- When no repeated-run files are found, the error names the accepted file patterns and lists the JSON files that were ignored. Files such as `run1.json` used to be skipped without a word.
+- `compare-runs` explains `INCONCLUSIVE` accurately. A missing or non-finite metric, or a single repetition, used to be reported as "measurements exceeded the allowed stability threshold"; the stability reason now appears only when a CV was measured, and it states that CV.
+- Identical repetitions, including all-zero values, have a CV of 0 instead of no CV. `--metric requests_failed` could never pass before.
+- An integer `schema_version` (`1` instead of `"1"`) gets a message that says so.
+
+### Changed
+
+- One exit-code scheme for every command: 0 PASS, 1 FAIL, 2 INCONCLUSIVE or UNSTABLE, 3 error with no decision. Before, `compare-runs` exited 2 (the INCONCLUSIVE code) for a missing directory or a mistyped option, `compare` exited 1 for UNSTABLE and 2 for invalid input, and `diagnose` exited 1 when Prometheus could not be queried.
+- An internal error now prints one line and exits 3 instead of printing a traceback and exiting 1, which a CI job read as FAIL. Set `INFERENCE_DOCTOR_DEBUG=1` for the traceback.
+- CI runs ruff, the formatter check and mypy, and tests on Python 3.11 and 3.14 on Linux plus 3.14 on macOS and Windows. Actions are pinned by commit SHA, the token is read-only, and checkouts do not keep credentials. The workflow is now `.github/workflows/ci.yml`.
+- The package moved to `src/` and builds with setuptools. The version is defined once, in `inference_doctor/__init__.py`; it said 0.1.0 while the package metadata said 0.3.0.
+
+## 0.3.0 - 2026-09-14
+
+### Added
+
+- Reusable GitHub Action for repeated-run inference regression gating.
+- No-GPU quickstart fixtures for trying `compare-runs` locally in minutes.
+- GitHub Actions documentation and PR smoke coverage.
+- Measurement-quality gate with first-class `PASS`, `FAIL`, and `INCONCLUSIVE` outcomes.
+- Repeated-run statistics including mean, median, sample standard deviation, CV, min/max, and descriptive outlier flags.
+- Optional environment telemetry helpers for benchmark investigations.
+- Experimental `inference-doctor compare` upgrade guard for normalized benchmark result files, with configurable regression thresholds, terminal and JSON output, and CI exit codes.
+- Auditable historical regression and benchmark-reliability experiments that informed the measurement-quality design.
+
+### Changed
+
+- Repositioned the project around trustworthy inference performance CI: measurement quality is checked before a merge decision is made.
+- `compare-runs` is now the primary documented workflow; Prometheus diagnosis remains supported as a secondary feature.
+- Apparent outliers remain included in official decisions rather than being silently removed.
+
+## 0.2.0 - 2026-09-13
+
+### Added
+
+- Queue mean, prefill p95, and prompt-token p95 collection and reporting.
+- Regression coverage for non-finite Prometheus scalar values.
+- Public integration reproduction notes and sanitized validation evidence.
+
+### Changed
+
+- Non-finite Prometheus scalar values now render as unavailable.
+- Queue p95 remains visible but no longer raises queue-pressure confidence.
+
+## 0.1.0
+
+- Initial vertical slice with Prometheus collection, terminal and JSON reports, and queue, decode, and KV-cache diagnostic rules.

inference_doctor-0.4.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing
+
+Thanks for helping improve Inference Doctor.
+
+The primary product goal is narrow: **trustworthy performance regression testing for LLM inference**. Changes should make it easier to compare baseline vs candidate performance without giving false confidence when benchmark data is noisy.
+
+The original read-only Prometheus diagnostic mode remains supported as a secondary feature.
+
+## Issues
+
+For inference-CI issues, include:
+
+- selected metric
+- baseline and candidate repetition counts
+- relevant normalized benchmark metadata
+- configured stability/regression thresholds
+- redacted terminal or JSON output
+
+For Prometheus diagnostic issues, include the vLLM version, relevant metric names, query window, supplied SLOs, and a redacted report.
+
+Never include API tokens, credentials, customer prompts, private model data, or cloud access details.
+
+## Pull requests
+
+- Keep each change focused on one observable behavior or usability improvement.
+- Preserve `PASS`, `FAIL`, and `INCONCLUSIVE` semantics unless the proposal includes evidence and tests.
+- Do not silently remove statistical outliers from official decisions.
+- Add or update regression tests for behavior changes.
+- Update docs when schemas, thresholds, supported metrics, or exit codes change.
+- Keep GPU/cloud provisioning outside the core comparison engine.
+- Do not commit virtual environments, model caches, credentials, `.env` files, or cloud-provider artifacts.
+
+Before opening a pull request:
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest -q
+ruff check . && ruff format --check . && mypy
+```
+
+CI runs the same checks, with tests on Python 3.11 and 3.14 on Linux and on 3.14 on macOS and Windows.
+
+## Releasing
+
+1. Move the `## Unreleased` entries in `CHANGELOG.md` under `## X.Y.Z - date` and set
+   `__version__` in `src/inference_doctor/__init__.py` to `X.Y.Z`. Commit to `main`.
+2. Once CI is green, tag and push: `git tag vX.Y.Z && git push origin vX.Y.Z`. The
+   release workflow checks the tag, builds, and publishes to PyPI through trusted
+   publishing.
+3. Move the Action's major tag to the release when its inputs and outputs are compatible:
+   `git tag -f v1 vX.Y.Z && git push -f origin v1`.
+
+Use clear commit messages and describe the validation performed in the pull request.

inference_doctor-0.4.0/LICENSE

@@ -0,0 +1,201 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

inference_doctor-0.4.0/MANIFEST.in

@@ -0,0 +1,5 @@
+graft tests
+graft examples
+include CHANGELOG.md CONTRIBUTING.md STATUS.md
+recursive-include tools *.py
+global-exclude __pycache__ *.py[co]

inference_doctor-0.4.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: inference-doctor
+Version: 0.4.0
+Summary: Performance regression gate for LLM inference benchmarks that checks measurement quality before it judges a regression.
+Author-email: Dundy Pasupuleti <dundysm@gmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/dundysm/inference-doctor
+Project-URL: Source, https://github.com/dundysm/inference-doctor
+Project-URL: Issues, https://github.com/dundysm/inference-doctor/issues
+Project-URL: Changelog, https://github.com/dundysm/inference-doctor/blob/main/CHANGELOG.md
+Keywords: llm,inference,benchmark,performance-regression,vllm,continuous-integration,github-action,measurement-quality
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.27
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff==0.17.0; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dundysm/inference-doctor/main/docs/assets/banner.svg" alt="inference-doctor: did this change make LLM inference slower, and can your benchmark actually tell?" width="900">
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/inference-doctor/"><img alt="PyPI" src="https://img.shields.io/pypi/v/inference-doctor?color=3776ab"></a>
+  <a href="https://github.com/dundysm/inference-doctor/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/dundysm/inference-doctor/actions/workflows/ci.yml/badge.svg"></a>
+  <img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-3776ab">
+  <img alt="Reads vLLM, AIPerf, GuideLLM and SGLang" src="https://img.shields.io/badge/reads-vLLM%2C%20AIPerf%2C%20GuideLLM%2C%20SGLang-1f6feb">
+  <a href="https://github.com/dundysm/inference-doctor/blob/main/LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-2ea043"></a>
+  <a href="https://github.com/dundysm/inference-doctor/blob/main/STATUS.md"><img alt="Status: early" src="https://img.shields.io/badge/status-early-d29922"></a>
+  <a href="https://github.com/dundysm/inference-doctor/blob/main/CONTRIBUTING.md"><img alt="PRs welcome" src="https://img.shields.io/badge/PRs-welcome-2ea043"></a>
+</p>
+
+A pull request can pass every test and still make inference slower. Putting a benchmark
+in CI is the obvious fix, and it goes wrong in three ways:
+
+* **false alarms**: GPU benchmarks are noisy, so a gate on two numbers fails changes that
+  did nothing, and the team turns it off,
+* **missed regressions**: with two or three runs, a real 12% slowdown can land inside the
+  noise and pass,
+* **apples to oranges**: the two sides ran on a different GPU, model or load, and the
+  difference is the setup, not the change.
+
+inference-doctor compares repeated baseline and candidate runs and returns `PASS` or
+`FAIL` only when the data can support it. Otherwise it returns `INCONCLUSIVE` and says
+why. It reads the files `vllm bench serve`, AIPerf, GuideLLM and SGLang already write.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dundysm/inference-doctor/main/docs/assets/terminal.svg" alt="Terminal output of inference-doctor compare-runs on a real H100 run: six runs per side, throughput 4.19% lower, 90% confidence interval for the regression -3.33% to +11.17%, which contains the 10% threshold, so the result is INCONCLUSIVE." width="860">
+  <br>
+  <sub>A real H100 run (<code>examples/inference-ci-reliability-001</code>), rendered from the CLI by
+  <code>tools/render_readme_media.py</code>. Throughput is 4.19% lower, but one noisy baseline block leaves an interval
+  from -3.33% to +11.17%. Six runs cannot rule out a 10% regression, so there is no merge recommendation.</sub>
+</p>
+
+## How it decides
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dundysm/inference-doctor/main/docs/assets/decision.svg" alt="Three 90% confidence intervals for the regression against a 10% threshold: the quickstart, +1.08% to +2.92%, PASS; a real H100 run, -3.33% to +11.17%, INCONCLUSIVE; a regressed candidate, +13.13% to +14.87%, FAIL." width="900">
+</p>
+
+1. **Is the comparison valid?** Same GPU, model and benchmark parameters on every run, no
+   failed requests, the metric present everywhere, and at least 3 runs per side. If not,
+   the result is `INCONCLUSIVE` and the reason names the check that failed.
+2. **How big is the change, and how sure are we?** A 90% confidence interval for the
+   regression (positive is worse), from a Welch t interval on the log ratio of means.
+3. **Verdict.** `FAIL` when the whole interval is at or past the threshold (default 10%),
+   `PASS` when the whole interval is below it, `INCONCLUSIVE` when it contains the
+   threshold. More runs narrow the interval.
+
+Outliers are flagged but never removed: a bad run widens the interval instead of being
+hidden. Details and references: [measurement quality](docs/measurement-quality.md).
+
+## Quick start
+
+Python 3.11 or newer. No GPU: the example results are checked in, so the clone is only for
+them.
+
+```bash
+pip install inference-doctor
+git clone https://github.com/dundysm/inference-doctor.git && cd inference-doctor
+
+inference-doctor compare-runs \
+  --baseline examples/quickstart/baseline \
+  --candidate examples/quickstart/candidate \
+  --metric texts_per_second
+```
+
+`PASS`: the candidate is 2% slower, and the interval, +1.08% to +2.92%, is well below 10%.
+Point `--candidate` at `examples/quickstart/regressed-candidate` to see a `FAIL`.
+
+## Use your benchmark's own output
+
+No converter needed. Put each side's runs in a directory and point `compare-runs` at it:
+
+```bash
+inference-doctor compare-runs \
+  --baseline results/main \
+  --candidate results/my-branch \
+  --metric output_token_throughput --metric ttft_p99_ms
+```
+
+Repeat `--metric` to gate on several at once: the result is `FAIL` if any metric fails,
+`INCONCLUSIVE` if none fails but one cannot be decided, and `PASS` only when all pass.
+
+| Tool | What it reads |
+| --- | --- |
+| `vllm bench serve --save-result` | each result file, or each line with `--append-result` |
+| NVIDIA AIPerf / GenAI-Perf | each `profile_export_aiperf.json` |
+| GuideLLM | `benchmarks.json`, one load point of the sweep (`--guidellm-benchmark`) |
+| SGLang `bench_serving` | each line of its `.jsonl` output |
+| anything else | the small [normalized format](docs/benchmark-format.md) |
+
+Metrics: output, total and request throughput, and TTFT, TPOT, ITL and end-to-end latency
+at mean, median, p95 and p99. Most tools do not record the GPU; pass `--gpu` to label it,
+or record it in the results (for vLLM, `--metadata gpu=...`). The
+[input formats](docs/input-formats.md) page lists every field read from each tool.
+
+[`examples/vllm-bench-serve`](examples/vllm-bench-serve/README.md) is a small example in
+vLLM's format where throughput passes and TTFT fails, so gating on both catches what a
+throughput-only gate would let through.
+
+## In GitHub Actions
+
+Run your benchmark on your own GPU runner, then gate on the results:
+
+```yaml
+- name: Gate inference performance
+  uses: dundysm/inference-doctor@v1
+  with:
+    baseline: results/main
+    candidate: results/my-branch
+    metric: output_token_throughput, ttft_p99_ms
+    min-repetitions: "5"
+```
+
+The Action writes a job summary with the verdict, the interval and the reason, and exposes
+them as outputs. `FAIL`, `INCONCLUSIVE` and errors fail the step, so noisy evidence cannot
+approve a change. inference-doctor does not provision GPUs or run the benchmark. See
+[GitHub Actions](docs/github-actions.md).
+
+## How much to trust it
+
+Simulated with known truth, at default settings, from
+[gate calibration](docs/gate-calibration.md) (4,000 trials per row):
+
+| Runs per side, noise (CV) | No change | True 5% drop | True 15% drop | True 20% drop |
+| --- | --- | --- | --- | --- |
+| 3 runs, 4% | 78% PASS, 0% FAIL | 34% PASS, 0.1% FAIL | 0% PASS, 37% FAIL | 0% PASS, 85% FAIL |
+| 5 runs, 4% | 98% PASS, 0% FAIL | 60% PASS, 0% FAIL | 0% PASS, 65% FAIL | 0% PASS, 99% FAIL |
+| 10 runs, 4% | 100% PASS, 0% FAIL | 90% PASS, 0% FAIL | 0% PASS, 92% FAIL | 0% PASS, 100% FAIL |
+
+The rest of each row is `INCONCLUSIVE`. Right at the 10% threshold, each wrong verdict
+happens about 5% of the time; further from it, at most 0.5%. Noise costs decisions, not
+correctness. Use 5 or more runs per side. These rates assume independent, roughly normal
+runs; [interleave baseline and candidate runs](docs/measurement-quality.md#assumptions)
+to stay close to that.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `compare-runs` | The gate: repeated baseline and candidate runs, one or more metrics, `PASS`, `FAIL` or `INCONCLUSIVE`. |
+| `compare` | Deprecated. One normalized file against another, every metric. A single run per side cannot measure noise; use `compare-runs`. See [upgrade guard](docs/upgrade-guard.md). |
+| `diagnose` | Experimental. Reads a live vLLM server's Prometheus metrics and reports queue, prefill, decode and KV-cache pressure. Read-only. Exits 1 on a critical finding (`--fail-on` changes that). See [example reports](docs/example-reports.md). |
+
+`compare-runs` options: `--metric` (repeatable), `--regression-percent` (10), `--confidence` (0.90),
+`--min-repetitions` (3), `--stability-cv-percent` (an optional CV ceiling, off),
+`--format` (auto), `--gpu`, `--model`, `--guidellm-benchmark` and `--json`.
+
+**Exit codes**, the same for every command: `0` PASS, `1` FAIL (for `diagnose`, a
+finding at or above `--fail-on`), `2` INCONCLUSIVE (or UNSTABLE for `compare`), `3` error
+with no decision (bad arguments or input, an unreachable Prometheus, or an internal
+error). `inference-doctor --version` prints the
+version.
+
+## What has been measured
+
+* **The decision rule**: the calibration table above, regenerated by
+  `tools/gate_simulation.py` and checked by `tests/test_gate.py` on every CI run.
+* **One real GPU run**: `examples/inference-ci-reliability-001`, six runs per side of
+  BAAI/bge-m3 on one H100 across two vLLM versions. The honest answer was `INCONCLUSIVE`,
+  and it still is.
+* **The format readers**: tested against records built from each tool's source code. They
+  have not yet been run on a large set of real files from each tool, so please
+  [report](https://github.com/dundysm/inference-doctor/issues/new/choose) any file they
+  misread.
+* **Not yet**: a gate running in another project's CI. See [STATUS.md](STATUS.md).
+
+## Help test it
+
+The most useful thing right now is a real change: run `compare-runs` on results from your
+own benchmark and open a **Benchmark feedback** issue saying what it decided, whether that
+was right, and what was confusing. Redact prompts, credentials, private model names and
+customer data.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). The changelog is [CHANGELOG.md](CHANGELOG.md).
+The experiments that informed the design live under `experiments/`.
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest -q
+ruff check . && ruff format --check . && mypy
+```
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE).