tokenwatt-check 0.1.0__tar.gz

tokenwatt_check-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,16 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.9", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q
+      - run: tokenwatt-check --demo throttled-h100 --json /tmp/r.json || test $? -eq 2

tokenwatt_check-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,17 @@
+name: publish
+on:
+  release:
+    types: [published]
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write        # PyPI trusted publishing; no API token stored
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build && python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1

tokenwatt_check-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.egg-info/
+dist/
+.pytest_cache/
+tokenwatt-check-*.json
+tokenwatt-check-*.md

tokenwatt_check-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Teampulse Solutions LLC (TokenWatt)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tokenwatt_check-0.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.5
+Name: tokenwatt-check
+Version: 0.1.0
+Summary: Check whether the GPU you rented performs like it should: power cap, clocks under load, compute, memory bandwidth, PCIe, disk and network.
+Project-URL: Homepage, https://www.tokenwatt.io
+Project-URL: Methodology, https://www.tokenwatt.io/methodology.html
+Author-email: TokenWatt <hello@tokenwatt.io>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmark,cloud,gpu,h100,inference,llm,nvidia,runpod,vast.ai
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Benchmark
+Classifier: Topic :: System :: Hardware
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Provides-Extra: torch
+Requires-Dist: torch>=2.1; extra == 'torch'
+Description-Content-Type: text/markdown
+
+# tokenwatt-check
+
+**Is the GPU you rented performing like it should?**
+
+GPU listings tell you the model — "H100 SXM, verified, 99.5% reliability" — not the card you get. When we
+[benchmarked four rented H100s](https://www.tokenwatt.io/blog/runpod-vs-vast-vs-lambda-h100.html), two had their
+power limit lowered by the host and one ran at **787 MHz instead of ~1,900** under load: every token took 2.9× longer.
+Nothing on the listing showed it.
+
+`tokenwatt-check` finds that in about two minutes:
+
+```
+pip install tokenwatt-check torch    # torch is usually already there on GPU images
+tokenwatt-check
+```
+
+```
+  OVERALL: ❌ FAIL
+
+  ⚠️  power limit            GPU power limit is 500 W of a possible 700 W (71%).
+  ❌ clocks under load      SM clock during the matmul averaged 560 MHz — 41% of a healthy H100 (1360 MHz).
+                            → Far below normal. Possible causes: aggressive power cap, poor cooling or
+                              power delivery, a degraded card. Ask for another machine.
+  ❌ BF16 matmul            280 TFLOPS — 41% of the healthy H100 reference (680 TFLOPS).
+  ❌ memory copy bandwidth  2.2 TB/s — 73% of the healthy H100 reference (3.0 TB/s).
+  ✅ PCIe link              Gen5 x16.
+  ✅ disk                   Sequential write 1,616 MB/s.
+  ✅ download               2,100 Mbit/s from Hugging Face, one stream (~6.3 min per 100 GB of weights).
+```
+
+(Illustrative output modeled on a real under-performing node — see `src/tokenwatt_check/demo.py`. Try `tokenwatt-check --demo throttled-h100`, `capped-h100` or `healthy-h100` without a GPU.)
+
+## What it checks
+
+| Check | How | Why it matters |
+|---|---|---|
+| Power limit | `nvidia-smi` power.limit vs power.max_limit | Hosts lower it to save power; throughput drops under load |
+| Clocks under load | SM clock sampled every second during a 30 s BF16 matmul, compared with a healthy card of the same model in the same test | The clearest sign of a capped, hot or degraded card |
+| Throttle reasons | nvidia-smi clock event reasons during load | Hardware or thermal slowdown means the card is protecting itself |
+| Compute | Sustained BF16 GEMM (8192²) in TFLOPS | Prompt processing and large batches scale with it |
+| Memory bandwidth | Device-to-device copy in TB/s | LLM token generation follows it closely |
+| PCIe link | Generation and width under load vs the card's maximum | Slow links slow model loading |
+| Memory errors | Uncorrected ECC errors, pending retired pages | Unreliable memory |
+| Disk | 2 GB sequential write and read | How long weights take to load from disk |
+| Download | 20 s from the Hugging Face CDN | You pay for the minutes spent downloading weights |
+
+Results: `PASS`, `WARN` or `FAIL` per check with a reason and what to do, plus a JSON file with every reading
+(`--json path`) and an optional Markdown report (`--md path`). Exit code 2 on `FAIL`, for scripts.
+
+## Options
+
+```
+tokenwatt-check --gpu 1            # another GPU
+tokenwatt-check --seconds 60       # longer load test
+tokenwatt-check --no-load          # inventory, disk and network only (no PyTorch needed)
+tokenwatt-check --no-net --no-disk
+```
+
+## Reference values
+
+Compute, bandwidth and clock-under-load are compared with a healthy card of the same model (`src/tokenwatt_check/verdict.py`):
+
+| GPU | BF16 matmul | Clock during matmul | Copy bandwidth | Source |
+|---|---|---|---|---|
+| H100 SXM | 680 TFLOPS | 1,360 MHz | 3.0 TB/s | measured, RunPod Secure Cloud, 700 W |
+| H200 | 660 TFLOPS | 1,460 MHz | 4.3 TB/s | measured, RunPod Secure Cloud, 700 W |
+| others | | | | provisional (vendor specs, public measurements) |
+
+A dense matmul pulls a healthy card to its power limit, so its clock sits well below the maximum — a healthy H100 holds
+about 69% of max clock in this test. That's why the clock check compares against a healthy card rather than the maximum.
+If a healthy card scores low on your machine, please open an issue with your JSON so we can improve the references.
+
+## Privacy
+
+Everything runs locally. Nothing is sent anywhere except the 20-second download test from huggingface.co.
+
+## Supported
+
+NVIDIA GPUs with `nvidia-smi` (H100, H200, B200, A100, L40S, RTX 4090/5090 and others). AMD Instinct support is planned.
+
+## About
+
+Built by [TokenWatt](https://www.tokenwatt.io), which publishes vendor-neutral measurements of LLM inference —
+SLO goodput, tokens per joule and cost per million tokens — on NVIDIA and AMD. MIT licensed.

tokenwatt_check-0.1.0/README.md

@@ -0,0 +1,84 @@
+# tokenwatt-check
+
+**Is the GPU you rented performing like it should?**
+
+GPU listings tell you the model — "H100 SXM, verified, 99.5% reliability" — not the card you get. When we
+[benchmarked four rented H100s](https://www.tokenwatt.io/blog/runpod-vs-vast-vs-lambda-h100.html), two had their
+power limit lowered by the host and one ran at **787 MHz instead of ~1,900** under load: every token took 2.9× longer.
+Nothing on the listing showed it.
+
+`tokenwatt-check` finds that in about two minutes:
+
+```
+pip install tokenwatt-check torch    # torch is usually already there on GPU images
+tokenwatt-check
+```
+
+```
+  OVERALL: ❌ FAIL
+
+  ⚠️  power limit            GPU power limit is 500 W of a possible 700 W (71%).
+  ❌ clocks under load      SM clock during the matmul averaged 560 MHz — 41% of a healthy H100 (1360 MHz).
+                            → Far below normal. Possible causes: aggressive power cap, poor cooling or
+                              power delivery, a degraded card. Ask for another machine.
+  ❌ BF16 matmul            280 TFLOPS — 41% of the healthy H100 reference (680 TFLOPS).
+  ❌ memory copy bandwidth  2.2 TB/s — 73% of the healthy H100 reference (3.0 TB/s).
+  ✅ PCIe link              Gen5 x16.
+  ✅ disk                   Sequential write 1,616 MB/s.
+  ✅ download               2,100 Mbit/s from Hugging Face, one stream (~6.3 min per 100 GB of weights).
+```
+
+(Illustrative output modeled on a real under-performing node — see `src/tokenwatt_check/demo.py`. Try `tokenwatt-check --demo throttled-h100`, `capped-h100` or `healthy-h100` without a GPU.)
+
+## What it checks
+
+| Check | How | Why it matters |
+|---|---|---|
+| Power limit | `nvidia-smi` power.limit vs power.max_limit | Hosts lower it to save power; throughput drops under load |
+| Clocks under load | SM clock sampled every second during a 30 s BF16 matmul, compared with a healthy card of the same model in the same test | The clearest sign of a capped, hot or degraded card |
+| Throttle reasons | nvidia-smi clock event reasons during load | Hardware or thermal slowdown means the card is protecting itself |
+| Compute | Sustained BF16 GEMM (8192²) in TFLOPS | Prompt processing and large batches scale with it |
+| Memory bandwidth | Device-to-device copy in TB/s | LLM token generation follows it closely |
+| PCIe link | Generation and width under load vs the card's maximum | Slow links slow model loading |
+| Memory errors | Uncorrected ECC errors, pending retired pages | Unreliable memory |
+| Disk | 2 GB sequential write and read | How long weights take to load from disk |
+| Download | 20 s from the Hugging Face CDN | You pay for the minutes spent downloading weights |
+
+Results: `PASS`, `WARN` or `FAIL` per check with a reason and what to do, plus a JSON file with every reading
+(`--json path`) and an optional Markdown report (`--md path`). Exit code 2 on `FAIL`, for scripts.
+
+## Options
+
+```
+tokenwatt-check --gpu 1            # another GPU
+tokenwatt-check --seconds 60       # longer load test
+tokenwatt-check --no-load          # inventory, disk and network only (no PyTorch needed)
+tokenwatt-check --no-net --no-disk
+```
+
+## Reference values
+
+Compute, bandwidth and clock-under-load are compared with a healthy card of the same model (`src/tokenwatt_check/verdict.py`):
+
+| GPU | BF16 matmul | Clock during matmul | Copy bandwidth | Source |
+|---|---|---|---|---|
+| H100 SXM | 680 TFLOPS | 1,360 MHz | 3.0 TB/s | measured, RunPod Secure Cloud, 700 W |
+| H200 | 660 TFLOPS | 1,460 MHz | 4.3 TB/s | measured, RunPod Secure Cloud, 700 W |
+| others | | | | provisional (vendor specs, public measurements) |
+
+A dense matmul pulls a healthy card to its power limit, so its clock sits well below the maximum — a healthy H100 holds
+about 69% of max clock in this test. That's why the clock check compares against a healthy card rather than the maximum.
+If a healthy card scores low on your machine, please open an issue with your JSON so we can improve the references.
+
+## Privacy
+
+Everything runs locally. Nothing is sent anywhere except the 20-second download test from huggingface.co.
+
+## Supported
+
+NVIDIA GPUs with `nvidia-smi` (H100, H200, B200, A100, L40S, RTX 4090/5090 and others). AMD Instinct support is planned.
+
+## About
+
+Built by [TokenWatt](https://www.tokenwatt.io), which publishes vendor-neutral measurements of LLM inference —
+SLO goodput, tokens per joule and cost per million tokens — on NVIDIA and AMD. MIT licensed.

tokenwatt_check-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "tokenwatt-check"
+version = "0.1.0"
+description = "Check whether the GPU you rented performs like it should: power cap, clocks under load, compute, memory bandwidth, PCIe, disk and network."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+authors = [{ name = "TokenWatt", email = "hello@tokenwatt.io" }]
+keywords = ["gpu", "benchmark", "h100", "cloud", "llm", "inference", "nvidia", "vast.ai", "runpod"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "Topic :: System :: Benchmark",
+  "Topic :: System :: Hardware",
+]
+dependencies = []          # stdlib only; the load test uses PyTorch if it is installed
+
+[project.optional-dependencies]
+torch = ["torch>=2.1"]
+dev = ["pytest>=7"]
+
+[project.urls]
+Homepage = "https://www.tokenwatt.io"
+Methodology = "https://www.tokenwatt.io/methodology.html"
+
+[project.scripts]
+tokenwatt-check = "tokenwatt_check.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/tokenwatt_check"]

tokenwatt_check-0.1.0/src/tokenwatt_check/__init__.py

@@ -0,0 +1,3 @@
+"""Check whether the GPU you rented performs like it should."""
+
+__version__ = "0.1.0"

tokenwatt_check-0.1.0/src/tokenwatt_check/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+import sys
+sys.exit(main())

tokenwatt_check-0.1.0/src/tokenwatt_check/cli.py

@@ -0,0 +1,115 @@
+"""tokenwatt-check: is the GPU you rented performing like it should?"""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+import sys
+import time
+from pathlib import Path
+
+from . import __version__, gpu, host, load
+from .demo import DEMOS
+from .verdict import evaluate, findings_json
+
+ICON = {"PASS": "✅", "WARN": "⚠️ ", "FAIL": "❌", "INFO": "ℹ️ "}
+
+
+def say(msg: str = "") -> None:
+    print(msg, flush=True)
+
+
+def run_checks(a) -> dict:
+    if not gpu.available():
+        sys.exit("nvidia-smi not found. tokenwatt-check currently supports NVIDIA GPUs (AMD support is planned).")
+    inv_all = gpu.inventory()
+    if a.gpu >= len(inv_all):
+        sys.exit(f"GPU {a.gpu} not found ({len(inv_all)} visible)")
+    inv = inv_all[a.gpu]
+    say(f"GPU {a.gpu}: {inv.get('name')} · driver {inv.get('driver_version')} · {inv.get('memory.total')} MiB")
+    loadres: dict = {}
+    ok, why = load.torch_available()
+    if a.no_load:
+        loadres["skipped"] = "--no-load"
+    elif not ok:
+        loadres["skipped"] = why
+        say(f"load test skipped: {why}")
+    else:
+        sampler = gpu.Sampler(gpu=a.gpu).start()
+        try:
+            say(f"load test: BF16 matmul for {a.seconds:.0f} s ({why}) …")
+            mm = load.matmul_tflops(seconds=a.seconds, device=a.gpu)
+            say(f"load test: memory copy for {a.seconds / 2:.0f} s …")
+            cp = load.copy_bandwidth(seconds=a.seconds / 2, device=a.gpu)
+        finally:
+            samples = sampler.stop()
+        # skip the first few seconds of each phase while clocks settle
+        mm["telemetry"] = gpu.summarize(samples, mm["t_start"] + 3, mm["t_end"])
+        cp["telemetry"] = gpu.summarize(samples, cp["t_start"] + 2, cp["t_end"])
+        loadres = {"matmul": mm, "copy": cp}
+    disk = {"skipped": "--no-disk"} if a.no_disk else (say("disk: 2 GB write/read …") or host.disk(a.disk_path))
+    net = {"skipped": "--no-net"} if a.no_net else (say("download: 20 s from huggingface.co …") or host.download())
+    return {"inventory": inv, "gpus_visible": len(inv_all), "load": loadres, "disk": disk, "download": net}
+
+
+def render(result: dict) -> str:
+    inv, overall, findings, m = result["inventory"], result["overall"], result["findings"], result["metrics"]
+    lines = ["", f"tokenwatt-check {result['version']} · {result['time']}",
+             f"{inv.get('name')} · driver {inv.get('driver_version')} · {result['host'].get('hostname', '')}", "",
+             f"  OVERALL: {ICON[overall]} {overall}", ""]
+    for x in findings:
+        lines.append(f"  {ICON[x['level']]} {x['check']:<22} {x['message']}")
+        if x["advice"]:
+            lines.append(f"     {'':<22} → {x['advice']}")
+    if m:
+        lines += ["", "  " + " · ".join(f"{k}={v}" for k, v in m.items())]
+    if result.get("demo"):
+        lines += ["", f"  (demo: {result['demo']})"]
+    lines += ["", "  Method and reference values: https://www.tokenwatt.io/methodology.html", ""]
+    return "\n".join(lines)
+
+
+def markdown(result: dict) -> str:
+    inv = result["inventory"]
+    rows = "\n".join(f"| {ICON[x['level']].strip()} {x['level']} | {x['check']} | {x['message']} | {x['advice']} |" for x in result["findings"])
+    return (f"# tokenwatt-check report\n\n**{inv.get('name')}** · driver {inv.get('driver_version')} · {result['time']}\n\n"
+            f"**Overall: {result['overall']}**\n\n| Result | Check | Finding | What to do |\n|---|---|---|---|\n{rows}\n\n"
+            f"Generated by [tokenwatt-check](https://www.tokenwatt.io) {result['version']}.\n")
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="tokenwatt-check", description=__doc__)
+    ap.add_argument("--version", action="version", version=f"tokenwatt-check {__version__}")
+    ap.add_argument("--gpu", type=int, default=0, help="GPU index (default 0)")
+    ap.add_argument("--seconds", type=float, default=30, help="matmul load duration; copy test runs half as long")
+    ap.add_argument("--no-load", action="store_true", help="skip the GPU load test")
+    ap.add_argument("--no-disk", action="store_true")
+    ap.add_argument("--no-net", action="store_true")
+    ap.add_argument("--disk-path", default=".", help="where to write the temporary disk-test file")
+    ap.add_argument("--json", help="write the full result to this file")
+    ap.add_argument("--md", help="write a Markdown report to this file")
+    ap.add_argument("--demo", choices=sorted(DEMOS), help="show an example report without a GPU")
+    a = ap.parse_args(argv)
+
+    t0 = time.time()
+    raw = DEMOS[a.demo] if a.demo else run_checks(a)
+    overall, findings, metrics = evaluate(raw["inventory"], raw.get("load", {}), raw.get("disk", {}), raw.get("download", {}))
+    result = {"tool": "tokenwatt-check", "version": __version__,
+              "time": dt.datetime.now(dt.timezone.utc).replace(microsecond=0).isoformat(),
+              "host": raw.get("host") or host.info(), **raw, "overall": overall,
+              "findings": findings_json(findings), "metrics": metrics, "elapsed_s": round(time.time() - t0, 1)}
+    if a.demo:
+        result["demo"] = raw.get("note", a.demo)
+    say(render(result))
+    stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S")
+    jpath = Path(a.json or f"tokenwatt-check-{stamp}.json")
+    jpath.write_text(json.dumps(result, indent=2, default=str))
+    if a.md:
+        Path(a.md).write_text(markdown(result))
+    say(f"  full result: {jpath}")
+    return {"PASS": 0, "WARN": 0, "FAIL": 2}[overall]
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tokenwatt_check-0.1.0/src/tokenwatt_check/demo.py

@@ -0,0 +1,34 @@
+"""Example inputs for --demo, modeled on three H100 SXM nodes TokenWatt rented in October 2026.
+
+The healthy example is a real tokenwatt-check run (RunPod Secure Cloud H100, 2026-10-09). The capped
+and throttled examples use the power limits, inference-load clocks and host speeds we read on two
+Vast.ai hosts; their matmul clocks and throughput are estimated from those readings, because those
+runs predate this tool's load test.
+"""
+
+
+def _h100(name, power_limit, clk, tflops, tb_s, disk_w, mbit, reasons, note):
+    return {
+        "note": note,
+        "host": {"hostname": name},
+        "inventory": {"name": "NVIDIA H100 80GB HBM3", "driver_version": "580.126.09", "memory.total": 81559,
+                      "pcie.link.gen.max": 5, "pcie.link.width.max": 16, "pcie.link.gen.current": 5,
+                      "pcie.link.width.current": 16, "power.limit": power_limit, "power.max_limit": 700,
+                      "clocks.max.sm": 1980, "ecc.errors.uncorrected.volatile.total": 0, "retired_pages.pending": "No"},
+        "load": {"matmul": {"tflops": tflops, "telemetry": {"sm_mhz_mean": clk, "temp_c_max": 58,
+                                                            "pcie_gen_under_load": 5, "pcie_width_under_load": 16,
+                                                            "throttle_reasons": reasons}},
+                 "copy": {"tb_s": tb_s}},
+        "disk": {"write_mb_s": disk_w},
+        "download": {"mbit_s": mbit, "minutes_per_100gb": round(100e3 * 8 / mbit / 60, 1)},
+    }
+
+
+DEMOS = {
+    "healthy-h100": _h100("demo-healthy", 700, 1359, 682.3, 3.05, 1726, 9400, {"sw_power_cap": 1.0},
+                          "healthy H100 SXM (modeled on a RunPod Secure Cloud node)"),
+    "capped-h100": _h100("demo-capped", 525, 1150, 575, 2.9, 4400, 6000, {"sw_power_cap": 1.0},
+                         "power-capped H100 SXM (modeled on a Vast.ai datacenter host capped at 525 W)"),
+    "throttled-h100": _h100("demo-throttled", 500, 560, 280, 2.2, 1616, 2100, {"sw_power_cap": 1.0},
+                            "under-performing H100 SXM (modeled on a Vast.ai community host: 787 MHz at a 500 W cap)"),
+}

tokenwatt_check-0.1.0/src/tokenwatt_check/gpu.py

@@ -0,0 +1,144 @@
+"""GPU inventory and live sampling through nvidia-smi (no Python bindings needed)."""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import threading
+import time
+from dataclasses import dataclass, field
+
+STATIC_FIELDS = [
+    "index", "name", "uuid", "driver_version", "memory.total", "pcie.link.gen.max", "pcie.link.width.max",
+    "pcie.link.gen.current", "pcie.link.width.current", "power.limit", "power.max_limit", "power.default_limit",
+    "clocks.max.sm", "clocks.max.memory", "ecc.mode.current", "ecc.errors.uncorrected.volatile.total",
+    "retired_pages.pending", "temperature.gpu", "persistence_mode", "compute_mode",
+]
+SAMPLE_FIELDS = ["index", "clocks.sm", "clocks.mem", "power.draw", "temperature.gpu", "utilization.gpu",
+                 "pcie.link.gen.current", "pcie.link.width.current", "clocks_event_reasons.active"]
+
+# nvidia-smi clocks_event_reasons bitmask
+REASONS = {
+    0x1: "gpu_idle", 0x2: "applications_clocks_setting", 0x4: "sw_power_cap", 0x8: "hw_slowdown",
+    0x10: "sync_boost", 0x20: "sw_thermal_slowdown", 0x40: "hw_thermal_slowdown", 0x80: "hw_power_brake_slowdown",
+    0x100: "display_clock_setting",
+}
+
+
+def available() -> bool:
+    return shutil.which("nvidia-smi") is not None
+
+
+def _num(v: str):
+    v = v.strip()
+    if v in ("[N/A]", "N/A", "[Not Supported]", "Not Supported", ""):
+        return None
+    for unit in (" MiB", " W", " MHz", " %"):
+        if v.endswith(unit):
+            v = v[: -len(unit)]
+    try:
+        return int(v) if v.lstrip("-").isdigit() else float(v)
+    except ValueError:
+        return v
+
+
+def query(fields: list[str]) -> list[dict]:
+    """Run nvidia-smi --query-gpu for these fields; unsupported fields come back as None."""
+    out = subprocess.run(["nvidia-smi", f"--query-gpu={','.join(fields)}", "--format=csv,noheader"],
+                         capture_output=True, text=True, timeout=20)
+    if out.returncode != 0:
+        # some drivers reject unknown fields as a whole; retry field by field
+        rows: list[dict] = []
+        for f in fields:
+            r = subprocess.run(["nvidia-smi", f"--query-gpu={f}", "--format=csv,noheader"],
+                               capture_output=True, text=True, timeout=20)
+            vals = r.stdout.strip().splitlines() if r.returncode == 0 else []
+            for i, v in enumerate(vals):
+                while len(rows) <= i:
+                    rows.append({})
+                rows[i][f] = _num(v)
+        return rows
+    rows = []
+    for line in out.stdout.strip().splitlines():
+        parts = [p.strip() for p in line.split(",")]
+        rows.append({f: _num(p) for f, p in zip(fields, parts)})
+    return rows
+
+
+def inventory() -> list[dict]:
+    return query(STATIC_FIELDS)
+
+
+def decode_reasons(mask) -> list[str]:
+    if isinstance(mask, str):
+        try:
+            mask = int(mask, 16)
+        except ValueError:
+            return []
+    if not isinstance(mask, int):
+        return []
+    return [name for bit, name in REASONS.items() if mask & bit]
+
+
+@dataclass
+class Sampler:
+    """Samples clocks, power, temperature and throttle reasons once a second in a thread."""
+    gpu: int = 0
+    period: float = 1.0
+    samples: list[dict] = field(default_factory=list)
+    _stop: threading.Event = field(default_factory=threading.Event)
+    _thread: threading.Thread | None = None
+
+    def _loop(self) -> None:
+        while not self._stop.is_set():
+            t = time.time()
+            try:
+                rows = query(SAMPLE_FIELDS)
+                if self.gpu < len(rows):
+                    s = rows[self.gpu]
+                    s["t"] = t
+                    s["reasons"] = decode_reasons(s.get("clocks_event_reasons.active"))
+                    self.samples.append(s)
+            except Exception:  # noqa: BLE001 — keep sampling
+                pass
+            self._stop.wait(max(0.0, self.period - (time.time() - t)))
+
+    def start(self) -> "Sampler":
+        self._thread = threading.Thread(target=self._loop, daemon=True)
+        self._thread.start()
+        return self
+
+    def stop(self) -> list[dict]:
+        self._stop.set()
+        if self._thread:
+            self._thread.join(timeout=10)
+        return self.samples
+
+
+def summarize(samples: list[dict], t0: float, t1: float) -> dict:
+    s = [x for x in samples if t0 <= x["t"] <= t1]
+    if not s:
+        return {}
+
+    def vals(k):
+        return [x[k] for x in s if isinstance(x.get(k), (int, float))]
+    out = {"samples": len(s)}
+    for k, name in (("clocks.sm", "sm_mhz"), ("clocks.mem", "mem_mhz"), ("power.draw", "power_w"),
+                    ("temperature.gpu", "temp_c"), ("utilization.gpu", "util_pct")):
+        v = vals(k)
+        if v:
+            out[f"{name}_mean"] = round(sum(v) / len(v), 1)
+            out[f"{name}_min"] = min(v)
+            out[f"{name}_max"] = max(v)
+    reasons: dict[str, int] = {}
+    for x in s:
+        for r in x.get("reasons", []):
+            reasons[r] = reasons.get(r, 0) + 1
+    out["throttle_reasons"] = {k: round(v / len(s), 2) for k, v in reasons.items()}   # share of samples
+    gens = vals("pcie.link.gen.current")
+    widths = vals("pcie.link.width.current")
+    if gens:
+        out["pcie_gen_under_load"] = max(gens)
+    if widths:
+        out["pcie_width_under_load"] = max(widths)
+    return out

tokenwatt_check-0.1.0/src/tokenwatt_check/host.py

@@ -0,0 +1,78 @@
+"""Host checks that decide how long a model takes to load: disk throughput and download speed."""
+
+from __future__ import annotations
+
+import os
+import platform
+import shutil
+import time
+import urllib.request
+
+# a large public file on the Hugging Face CDN (what model weights come from)
+DOWNLOAD_URL = "https://huggingface.co/Qwen/Qwen3-30B-A3B-FP8/resolve/main/model-00001-of-00007.safetensors"
+
+
+def info() -> dict:
+    out = {"hostname": platform.node(), "os": platform.platform(), "python": platform.python_version(),
+           "cpus": os.cpu_count()}
+    try:
+        with open("/proc/meminfo") as f:
+            kb = int(next(line for line in f if line.startswith("MemTotal")).split()[1])
+        out["ram_gb"] = round(kb / 2 ** 20)
+    except (OSError, StopIteration, ValueError):
+        pass
+    try:
+        with open("/proc/cpuinfo") as f:
+            out["cpu"] = next(line.split(":", 1)[1].strip() for line in f if line.startswith("model name"))
+    except (OSError, StopIteration):
+        pass
+    return out
+
+
+def disk(path: str = ".", gib: float = 2.0) -> dict:
+    """Sequential write (with fsync) and read of a temporary file in `path`."""
+    usage = shutil.disk_usage(path)
+    out = {"path": os.path.abspath(path), "free_gb": round(usage.free / 1e9)}
+    if usage.free < gib * 2 ** 30 * 1.5:
+        out["skipped"] = "not enough free space"
+        return out
+    fn = os.path.join(path, f".tokenwatt-check-{os.getpid()}.tmp")
+    block = os.urandom(16 * 2 ** 20)
+    n = int(gib * 2 ** 30 // len(block))
+    try:
+        t0 = time.perf_counter()
+        with open(fn, "wb", buffering=0) as f:
+            for _ in range(n):
+                f.write(block)
+            os.fsync(f.fileno())
+        out["write_mb_s"] = round(n * len(block) / (time.perf_counter() - t0) / 1e6)
+        t0 = time.perf_counter()
+        with open(fn, "rb", buffering=0) as f:
+            while f.read(16 * 2 ** 20):
+                pass
+        out["read_mb_s"] = round(n * len(block) / (time.perf_counter() - t0) / 1e6)
+        out["read_note"] = "may be served from page cache"
+    finally:
+        try:
+            os.remove(fn)
+        except OSError:
+            pass
+    return out
+
+
+def download(seconds: float = 20.0, url: str = DOWNLOAD_URL) -> dict:
+    req = urllib.request.Request(url, headers={"User-Agent": "tokenwatt-check/0.1"})
+    got, t0 = 0, time.perf_counter()
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            while time.perf_counter() - t0 < seconds:
+                chunk = r.read(4 * 2 ** 20)
+                if not chunk:
+                    break
+                got += len(chunk)
+    except Exception as e:  # noqa: BLE001
+        return {"error": f"{type(e).__name__}: {e}"[:160]}
+    el = time.perf_counter() - t0
+    mbit = got * 8 / el / 1e6 if el else 0
+    return {"mbit_s": round(mbit), "mb": round(got / 1e6), "seconds": round(el, 1), "source": "huggingface.co",
+            "minutes_per_100gb": round(100e3 * 8 / mbit / 60, 1) if mbit else None}

tokenwatt_check-0.1.0/src/tokenwatt_check/load.py

@@ -0,0 +1,73 @@
+"""Short GPU load tests with PyTorch: dense BF16 matmul (compute) and device-to-device copy (memory bandwidth)."""
+
+from __future__ import annotations
+
+import time
+
+
+def torch_available() -> tuple[bool, str]:
+    try:
+        import torch  # noqa: F401
+    except Exception as e:  # noqa: BLE001
+        return False, f"PyTorch not importable ({type(e).__name__}); install it with `pip install torch`"
+    import torch
+    if not torch.cuda.is_available():
+        return False, "PyTorch can't see a CUDA device"
+    return True, f"torch {torch.__version__}, CUDA {torch.version.cuda}"
+
+
+def matmul_tflops(seconds: float = 30.0, n: int = 8192, device: int = 0) -> dict:
+    """Sustained dense BF16 GEMM throughput. Runs long enough for clocks and power to settle."""
+    import torch
+    torch.cuda.set_device(device)
+    a = torch.randn(n, n, device="cuda", dtype=torch.bfloat16)
+    b = torch.randn(n, n, device="cuda", dtype=torch.bfloat16)
+    for _ in range(5):                       # warm up kernels and clocks
+        a @ b
+    torch.cuda.synchronize()
+    flop = 2 * n ** 3
+    iters, t0 = 0, time.perf_counter()
+    t_start = time.time()
+    while True:
+        for _ in range(10):
+            a @ b
+        torch.cuda.synchronize()
+        iters += 10
+        el = time.perf_counter() - t0
+        if el >= seconds:
+            break
+    t_end = time.time()
+    del a, b
+    torch.cuda.empty_cache()
+    return {"tflops": round(flop * iters / el / 1e12, 1), "n": n, "dtype": "bf16", "seconds": round(el, 1),
+            "t_start": t_start, "t_end": t_end}
+
+
+def copy_bandwidth(seconds: float = 15.0, gib: float = 4.0, device: int = 0) -> dict:
+    """Device-to-device copy bandwidth (read + write counted), the bound for LLM decode."""
+    import torch
+    torch.cuda.set_device(device)
+    free, _total = torch.cuda.mem_get_info()
+    gib = min(gib, free / 2 ** 30 / 2.5)      # two buffers plus headroom
+    n = int(gib * 2 ** 30) // 2
+    src = torch.empty(n, device="cuda", dtype=torch.float16)
+    dst = torch.empty_like(src)
+    for _ in range(3):
+        dst.copy_(src)
+    torch.cuda.synchronize()
+    iters, t0 = 0, time.perf_counter()
+    t_start = time.time()
+    while True:
+        for _ in range(5):
+            dst.copy_(src)
+        torch.cuda.synchronize()
+        iters += 5
+        el = time.perf_counter() - t0
+        if el >= seconds:
+            break
+    t_end = time.time()
+    moved = 2 * n * 2 * iters                  # bytes read + written
+    del src, dst
+    torch.cuda.empty_cache()
+    return {"tb_s": round(moved / el / 1e12, 2), "buffer_gib": round(gib, 2), "seconds": round(el, 1),
+            "t_start": t_start, "t_end": t_end}

tokenwatt_check-0.1.0/src/tokenwatt_check/verdict.py

@@ -0,0 +1,157 @@
+"""Turn measurements into findings: PASS / WARN / FAIL with a reason and what to do about it."""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+
+# "Healthy" reference for this tool's own load tests: sustained dense BF16 GEMM (n=8192), the SM
+# clock during that GEMM, and device-to-device copy bandwidth counted as read+write. Entries marked
+# "measured" come from cards TokenWatt ran this tool on; the rest are provisional (vendor specs and
+# public measurements) until we measure one. Matched against nvidia-smi's name, most specific first.
+#
+# A dense GEMM pulls a healthy card to its power limit, so its clock sits well below clocks.max.sm
+# (an H100 SXM at 700 W holds ~1,360 MHz of 1,980). The clock check compares against `gemm_mhz`.
+REFERENCE = [
+    ("H100 NVL", {"tflops": 550, "tb_s": 3.3}),
+    ("H100 PCIe", {"tflops": 450, "tb_s": 1.7}),
+    ("H100", {"tflops": 680, "tb_s": 3.0, "gemm_mhz": 1360, "measured": "RunPod Secure Cloud, 700 W, 2026-10-09"}),  # SXM
+    ("H200", {"tflops": 660, "tb_s": 4.3, "gemm_mhz": 1460, "measured": "RunPod Secure Cloud, 700 W, 2026-10-09"}),
+    ("B200", {"tflops": 1400, "tb_s": 6.5}),
+    ("A100-SXM4-80GB", {"tflops": 270, "tb_s": 1.7}),
+    ("A100 80GB PCIe", {"tflops": 250, "tb_s": 1.6}),
+    ("A100", {"tflops": 250, "tb_s": 1.3}),          # 40 GB
+    ("L40S", {"tflops": 250, "tb_s": 0.75}),
+    ("RTX 6000 Ada", {"tflops": 250, "tb_s": 0.8}),
+    ("RTX 5090", {"tflops": 200, "tb_s": 1.5}),
+    ("RTX 4090", {"tflops": 150, "tb_s": 0.85}),
+    ("A10", {"tflops": 100, "tb_s": 0.5}),
+]
+
+
+def reference_for(name: str | None) -> tuple[str, dict] | None:
+    if not name:
+        return None
+    for key, ref in REFERENCE:
+        if key.lower() in name.lower():
+            return key, ref
+    return None
+
+
+@dataclass
+class Finding:
+    level: str          # PASS | WARN | FAIL | INFO
+    check: str
+    message: str
+    advice: str = ""
+
+
+def evaluate(inv: dict, load: dict, host_disk: dict, host_net: dict) -> tuple[str, list[Finding], dict]:
+    f: list[Finding] = []
+    metrics: dict = {}
+
+    # power cap
+    lim, mx = inv.get("power.limit"), inv.get("power.max_limit")
+    if isinstance(lim, (int, float)) and isinstance(mx, (int, float)) and mx:
+        ratio = lim / mx
+        metrics["power_limit_pct"] = round(ratio * 100)
+        if ratio < 0.95:
+            f.append(Finding("WARN", "power limit",
+                             f"GPU power limit is {lim:.0f} W of a possible {mx:.0f} W ({ratio:.0%}).",
+                             "The host lowered it. Expect lower throughput under heavy load; compare price per token, not per hour."))
+        else:
+            f.append(Finding("PASS", "power limit", f"Full power limit: {lim:.0f} W."))
+
+    gemm = load.get("matmul") or {}
+    tel = gemm.get("telemetry") or {}
+    # clocks under load: against a healthy card's clock in the same GEMM when we have one,
+    # otherwise against clocks.max.sm with looser thresholds
+    ref = reference_for(inv.get("name"))
+    mxclk, clk = inv.get("clocks.max.sm"), tel.get("sm_mhz_mean")
+    base = (ref[1].get("gemm_mhz") if ref else None) or None
+    if isinstance(clk, (int, float)) and (base or (isinstance(mxclk, (int, float)) and mxclk)):
+        r = clk / (base or mxclk)
+        if not base:
+            r = min(1.0, r / 0.7)            # a healthy card at its power limit runs ~70% of max under GEMM
+        metrics["clock_under_load_pct"] = round(r * 100)
+        msg = (f"SM clock during the matmul averaged {clk:.0f} MHz — {r:.0%} of a healthy {ref[0]} ({base} MHz)"
+               if base else f"SM clock during the matmul averaged {clk:.0f} MHz of {mxclk:.0f} MHz max")
+        if r < 0.6:
+            f.append(Finding("FAIL", "clocks under load", msg + ".",
+                             "Far below normal. Possible causes: aggressive power cap, poor cooling or power delivery, a degraded card. Ask for another machine."))
+        elif r < 0.8:
+            f.append(Finding("WARN", "clocks under load", msg + ".", "Lower than a healthy card under this load; usually a power or thermal limit."))
+        else:
+            f.append(Finding("PASS", "clocks under load", msg + "."))
+
+    reasons = tel.get("throttle_reasons") or {}
+    hard = {k: v for k, v in reasons.items() if k in ("hw_slowdown", "hw_thermal_slowdown", "sw_thermal_slowdown", "hw_power_brake_slowdown") and v >= 0.1}
+    if hard:
+        f.append(Finding("FAIL", "throttling", "Hardware or thermal slowdown active during load: " +
+                         ", ".join(f"{k} ({v:.0%} of samples)" for k, v in hard.items()) + ".",
+                         "This is not a setting the host chose — the card is protecting itself. Avoid this machine."))
+    elif reasons.get("sw_power_cap", 0) >= 0.5:
+        f.append(Finding("INFO", "throttling", f"Running at its power limit for {reasons['sw_power_cap']:.0%} of the load test (normal under full load)."))
+
+    # compute and bandwidth vs reference
+    for key, val, unit, label in (("tflops", gemm.get("tflops"), "TFLOPS", "BF16 matmul"),
+                                  ("tb_s", (load.get("copy") or {}).get("tb_s"), "TB/s", "memory copy bandwidth")):
+        if not isinstance(val, (int, float)):
+            continue
+        metrics[key] = val
+        if not ref:
+            f.append(Finding("INFO", label, f"{val} {unit} (no reference yet for this GPU)."))
+            continue
+        pct = val / ref[1][key]
+        metrics[f"{key}_vs_reference_pct"] = round(pct * 100)
+        msg = f"{val} {unit} — {pct:.0%} of the healthy {ref[0]} reference ({ref[1][key]} {unit})"
+        level = "FAIL" if pct < 0.75 else "WARN" if pct < 0.9 else "PASS"
+        why = ("Prompt processing (prefill) and large batches scale with compute." if key == "tflops"
+               else "LLM token generation (decode) speed follows memory bandwidth closely.")
+        f.append(Finding(level, label, msg + ".", "" if level == "PASS" else f"Below what this GPU model normally delivers. {why}"))
+    if ref is None and inv.get("name"):
+        metrics["reference"] = None
+
+    # PCIe link
+    gmax, wmax = inv.get("pcie.link.gen.max"), inv.get("pcie.link.width.max")
+    gcur, wcur = tel.get("pcie_gen_under_load", inv.get("pcie.link.gen.current")), tel.get("pcie_width_under_load", inv.get("pcie.link.width.current"))
+    if all(isinstance(x, (int, float)) for x in (gmax, wmax, gcur, wcur)):
+        if gcur < gmax or wcur < wmax:
+            f.append(Finding("WARN", "PCIe link", f"PCIe link is Gen{gcur} x{wcur}, below the card's Gen{gmax} x{wmax}.",
+                             "Slows model loading and host↔GPU transfers; mostly harmless once the model is loaded."))
+        else:
+            f.append(Finding("PASS", "PCIe link", f"Gen{gcur} x{wcur}."))
+
+    # memory errors
+    ecc, pend = inv.get("ecc.errors.uncorrected.volatile.total"), inv.get("retired_pages.pending")
+    if isinstance(ecc, (int, float)) and ecc > 0:
+        f.append(Finding("FAIL", "memory errors", f"{ecc:.0f} uncorrected ECC errors since the driver loaded.", "Unreliable memory. Do not run production work here."))
+    if isinstance(pend, str) and pend.lower() == "yes":
+        f.append(Finding("WARN", "memory errors", "Retired memory pages are pending a reset.", "The card needs a reset to retire bad memory."))
+
+    tmax = tel.get("temp_c_max")
+    if isinstance(tmax, (int, float)) and tmax >= 87:
+        f.append(Finding("WARN", "temperature", f"Reached {tmax:.0f} °C under load.", "Hot; sustained work may throttle."))
+
+    # host
+    w = host_disk.get("write_mb_s")
+    if isinstance(w, (int, float)):
+        metrics["disk_write_mb_s"] = w
+        f.append(Finding("WARN" if w < 1000 else "PASS", "disk", f"Sequential write {w:,} MB/s.",
+                         "Slow disk: large models will take minutes to load." if w < 1000 else ""))
+    mb = host_net.get("mbit_s")
+    if isinstance(mb, (int, float)):
+        metrics["download_mbit_s"] = mb
+        mins = host_net.get("minutes_per_100gb")
+        # single-stream download; the same host measured 0.9 and 9.5 Gbit/s on different days, so only flag clearly slow links
+        f.append(Finding("WARN" if mb < 500 else "PASS", "download", f"{mb:,} Mbit/s from Hugging Face, one stream" +
+                         (f" (~{mins} min per 100 GB of weights)." if mins else "."),
+                         "You pay for the minutes spent downloading weights." if mb < 500 else ""))
+
+    order = {"FAIL": 3, "WARN": 2, "PASS": 1, "INFO": 0}
+    worst = max((order[x.level] for x in f), default=0)
+    overall = {3: "FAIL", 2: "WARN"}.get(worst, "PASS")
+    return overall, f, metrics
+
+
+def findings_json(findings: list[Finding]) -> list[dict]:
+    return [asdict(x) for x in findings]

tokenwatt_check-0.1.0/tests/test_verdict.py

@@ -0,0 +1,46 @@
+from tokenwatt_check.demo import DEMOS
+from tokenwatt_check.gpu import _num, decode_reasons, summarize
+from tokenwatt_check.verdict import evaluate, reference_for
+
+
+def run(name):
+    d = DEMOS[name]
+    return evaluate(d["inventory"], d["load"], d["disk"], d["download"])
+
+
+def test_demo_verdicts():
+    assert run("healthy-h100")[0] == "PASS"
+    assert run("capped-h100")[0] == "WARN"
+    overall, findings, metrics = run("throttled-h100")
+    assert overall == "FAIL"
+    assert metrics["clock_under_load_pct"] < 60
+    assert any(f.check == "clocks under load" and f.level == "FAIL" for f in findings)
+
+
+def test_reference_matching_prefers_specific_names():
+    assert reference_for("NVIDIA H100 PCIe")[0] == "H100 PCIe"
+    assert reference_for("NVIDIA H100 80GB HBM3")[0] == "H100"
+    assert reference_for("NVIDIA A100-SXM4-80GB")[0] == "A100-SXM4-80GB"
+    assert reference_for("Tesla T4") is None
+
+
+def test_nvidia_smi_parsing():
+    assert _num("700.00 W") == 700.0
+    assert _num("81559 MiB") == 81559
+    assert _num("[N/A]") is None
+    assert decode_reasons("0x0000000000000004") == ["sw_power_cap"]
+    assert decode_reasons("0x0000000000000048") == ["hw_slowdown", "hw_thermal_slowdown"]
+
+
+def test_hardware_slowdown_fails():
+    d = DEMOS["healthy-h100"]
+    load = {"matmul": {**d["load"]["matmul"], "telemetry": {**d["load"]["matmul"]["telemetry"],
+                                                              "throttle_reasons": {"hw_slowdown": 0.4}}}, "copy": d["load"]["copy"]}
+    overall, findings, _ = evaluate(d["inventory"], load, d["disk"], d["download"])
+    assert overall == "FAIL" and any(f.check == "throttling" for f in findings)
+
+
+def test_summarize_windows_and_reasons():
+    s = [{"t": i, "clocks.sm": 1900 + i, "power.draw": 600.0, "reasons": ["sw_power_cap"] if i % 2 else []} for i in range(10)]
+    out = summarize(s, 2, 7)
+    assert out["samples"] == 6 and out["throttle_reasons"]["sw_power_cap"] == 0.5