tuieval 0.2.0.dev56__tar.gz → 0.2.0.dev65__tar.gz

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/CHANGELOG.md

@@ -2,10 +2,12 @@
 
 ## Unreleased
 
+- Nvidia and AMD GPUs on Linux: VRAM from `nvidia-smi`, `amd-smi` or `rocm-smi` sizes models (`gpu_memory_gb` overrides it), runs record peak VRAM, and `tuieval doctor` shows the GPU.
+- Results has four tabs: PTA index, Production readiness, Per question and Failures. The PTA table adds tok/s, TTFT and memory; Per question can show only where models disagree; the settings warning is one line. Scorecard, pairwise, history and test quality stay in `tuieval compare`, `history` and `items`.
 - The run's ETA adds up each pack's answers left × that model's time per answer there (its history, then its own times once it has 5 answers), instead of extrapolating from the first few answers.
 - Faster start on Macs: the GPU core count comes from `ioreg` instead of `system_profiler`, which took seconds on some Macs.
 - A model a server won't serve (HTTP 401-404, e.g. OpenRouter's "No endpoints found") is never judged on it: those answers don't count, and the model stops at once instead of failing every question.
-- PTA index: parsimony (tokens used) and speed (time), each 100 for the best and 15 points less per doubling, and accuracy per model, as bars: the first Results tab (where `r` lands), `tuieval pta` and the report. Server errors aren't answers, and models with few answers are left out unless picked.
+- PTA index: parsimony (tokens), time and accuracy per model as bars, with tokens and time shown as multiples of the best (2.4×): the first Results tab (where `r` lands), `tuieval pta` and the report. Server errors aren't answers, and models with few answers are left out unless picked.
 - Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
 - More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev56
+Version: 0.2.0.dev65
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -51,7 +51,7 @@ Whatever runs your models, the first steps are short:
 | You have | Do this |
 |---|---|
 | **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
-| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`, or a CUDA/ROCm build on Linux), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your Mac's memory or your Nvidia/AMD card's VRAM. |
 | **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
 
 Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
@@ -76,7 +76,7 @@ Afterwards, every answer keeps its reasoning. Enter on a result opens it with th
 - **Verdicts you can act on.** A pack passes only with zero critical failures over enough trials, and an accuracy whose 95% lower bound clears your bar. *INCONCLUSIVE* says what evidence is missing.
 - **Two tiers:** *Screen* a sample of every pack to drop weak models fast, then *Certify* finalists on every test with repeats. Certification reuses the screening answers.
 - **Honest numbers:** every request is a fresh single-turn conversation, with prompt caching off and repeat rounds in different orders. Each answer records what was sent, and the run warns about reused prompts or identical repeats.
-- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
+- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory or your Nvidia/AMD card's VRAM, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
 - **Tests that check themselves.** Every test carries a reference answer and known-wrong answers, and `tuieval selftest` checks the grader accepts the first and rejects the second, and that each gate is reachable at all.
 - **History.** Verdict changes are appended to `results/verdicts.jsonl`, and replaced results are moved to `history/`, never overwritten.
 
@@ -144,7 +144,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
-4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
@@ -176,7 +176,7 @@ tuieval help
 
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
-- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/README.md

@@ -25,7 +25,7 @@ Whatever runs your models, the first steps are short:
 | You have | Do this |
 |---|---|
 | **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
-| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`, or a CUDA/ROCm build on Linux), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your Mac's memory or your Nvidia/AMD card's VRAM. |
 | **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
 
 Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
@@ -50,7 +50,7 @@ Afterwards, every answer keeps its reasoning. Enter on a result opens it with th
 - **Verdicts you can act on.** A pack passes only with zero critical failures over enough trials, and an accuracy whose 95% lower bound clears your bar. *INCONCLUSIVE* says what evidence is missing.
 - **Two tiers:** *Screen* a sample of every pack to drop weak models fast, then *Certify* finalists on every test with repeats. Certification reuses the screening answers.
 - **Honest numbers:** every request is a fresh single-turn conversation, with prompt caching off and repeat rounds in different orders. Each answer records what was sent, and the run warns about reused prompts or identical repeats.
-- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
+- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory or your Nvidia/AMD card's VRAM, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
 - **Tests that check themselves.** Every test carries a reference answer and known-wrong answers, and `tuieval selftest` checks the grader accepts the first and rejects the second, and that each gate is reachable at all.
 - **History.** Verdict changes are appended to `results/verdicts.jsonl`, and replaced results are moved to `history/`, never overwritten.
 
@@ -118,7 +118,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
-4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
@@ -150,7 +150,7 @@ tuieval help
 
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
-- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/docs/models.md

@@ -69,9 +69,21 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 ## Machines
 
-- **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
-- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
-- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4), `gpu_residency_gb` (see the stall guard) and `parallel_models` (see below).
+- **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`, or `rtx4090-128gb` on a Linux machine with a discrete GPU; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
+- **Fit check (llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory (a Mac's unified memory, or a discrete GPU's VRAM), capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
+- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free: default 4 on a Mac, 1.5 in a discrete GPU's VRAM), `gpu_memory_gb` (the GPU memory to size models against, when detection gets it wrong or you want to hold some back), `gpu_residency_gb` (see the stall guard) and `parallel_models` (see below).
+
+### Discrete GPUs (Linux: Nvidia, AMD)
+
+With `nvidia-smi` (Nvidia) or `amd-smi` / `rocm-smi` (AMD) installed, tuieval reads each card's name and VRAM. Several cards add up, since llama.cpp splits a model across them. `tuieval doctor` shows what it found.
+
+- Models are sized to fit **entirely in VRAM** (servers keep `-ngl 99`); splitting a model between GPU and CPU isn't supported. A model that doesn't fit is skipped with what it needs and what's free.
+- **Memory** in results is the server's peak VRAM: its own processes' on Nvidia, the whole machine's on AMD (its tools don't report it per process reliably, so it's only exact when nothing else uses the GPU).
+- `tuieval tune` rejects flags that leave VRAM nearly full. The macOS memory-pressure checks and the stall guard don't apply.
+- A machine that recorded itself before its GPU was detected keeps that id, so its results and tuning still match; `tuieval doctor` names the new id if you want to switch (`EVALS_MACHINE`).
+- Without the vendor tools, models are sized against system RAM, as on any other machine; set `gpu_memory_gb` instead.
+
+Windows isn't supported (WSL2 works like Linux).
 
 ## Several models at a time
 
@@ -86,11 +98,13 @@ By default a run serves one model at a time. On a machine with room for more, se
 
 ## PTA index
 
-The **PTA index** puts parsimony (tokens), speed (time) and accuracy side by side, 0-100 each and higher is better, as a bar per score for each model, with the numbers and a table underneath. It's the first tab in Results, where `r` opens, and it's also `tuieval pta` and a section in `tuieval report`. There's no combined number; verdicts and critical failures stay in Production readiness.
+The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a bar for each per model (longer is better), with a table underneath. It's the first tab in Results, where `r` opens, and it's also `tuieval pta` and a section in `tuieval report`. There's no combined number; verdicts and critical failures stay in Production readiness.
+
+- **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
+- **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
+- **A accuracy**: the share of answers to those questions that passed.
 
-- **P parsimony** (a score out of 100, not a percentage): from the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). The fewest tokens score 100 (★ in the **tokens** column), and every doubling costs 15 points: `100 − 15 × log2(this model's tokens ÷ fewest)`, never below 0. Unlike speed it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
-- **T speed** (a score out of 100, not a percentage): the same from the total time to answer those questions. The fastest model scores 100 (★ on its total time), and every doubling of the total time costs 15 points: `100 − 15 × log2(this model's total ÷ fastest total)`, never below 0 (about 100× slower). Twice as slow is 85, 10× slower is 50. Adding a faster model lowers the others. The table's **total time** column has the times themselves (lower is better).
-- **A accuracy** (a percentage): the share of answers to those questions that passed.
+The P and T bars use a log scale: full for the best, 15 points (of 100) shorter for every doubling, empty at about 100 times the best. So a model at 28× still shows a bar, and adding a better model shortens the others'.
 
 Models are compared on the questions all of them answered, so one that skipped some never looks faster. Server and connection errors aren't answers: a model with only those is listed as having none. A model with fewer than half as many answers as the most-answered one is left out (and listed), or it would shrink the shared questions for everyone. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`: models you pick are always compared.
 

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/docs/writing-packs.md

@@ -127,7 +127,7 @@ def ticket_route(answer, test, meta):
 - `critical=True` makes every test of this grader a critical trial (any of its failures *can* be critical). Otherwise only tests with `critical: true` or `expected: NOT_AVAILABLE` are.
 - `template` is the skeleton `tuieval capture` writes for a new test of this grader.
 
-A grader file can also add **Scorecard columns** (Results → Scorecard, and `tuieval compare`), for numbers your use case cares about that pass/fail doesn't show:
+A grader file can also add **Scorecard columns** (`tuieval compare`), for numbers your use case cares about that pass/fail doesn't show:
 
 ```python
 import json
@@ -175,7 +175,7 @@ Graders mark failures that should disqualify a model whatever its accuracy:
 
 ## Difficulty labels
 
-Every test has a `difficulty` of easy, medium or hard. It's for you, not the model: requests are built only from `input`, `image`, the system prompt and tools, so the label never reaches the model. The TUI shows it in the pack list (e.g. `12E 20M 8H`), next to the current question, in Recent results and Failures, and in Results → By difficulty.
+Every test has a `difficulty` of easy, medium or hard. It's for you, not the model: requests are built only from `input`, `image`, the system prompt and tools, so the label never reaches the model. The TUI shows it in the pack list (e.g. `12E 20M 8H`), next to the current question, in Recent results and Failures, and as a filter in Results → Per question.
 
 A useful convention: **easy** = one rule or a direct read; **medium** = arithmetic, one trap, or two conditions; **hard** = rules in conflict, multi-step reasoning, or misleading data. Once several models have run, `tuieval items` flags labels the results contradict ("easier than labelled": a hard test every model passes; "harder than labelled": an easy test most models fail). `tuieval selftest` warns about tests without a label.
 

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev56'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev56')
+__version__ = version = '0.2.0.dev65'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev65')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/cli.py

@@ -14,7 +14,7 @@ Start here
 
 Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
-  tuieval pta                 the PTA index: parsimony, speed and accuracy per model, as bars
+  tuieval pta                 the PTA index: parsimony (tokens), time and accuracy per model, as bars
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
   tuieval items               tests that don't separate models or look broken

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/compare.py

@@ -35,7 +35,8 @@ def _load_native(data):
     m, pack, run = data["model"], data["pack"], data["run"]
     info = {"label": m["label"], "served": m.get("served_name"), "tags": m.get("tags") or [],
             "settings": data.get("settings", {}), "load_s": run.get("load_s"),
-            "peak_rss_mb": run.get("peak_rss_mb"), "pack": pack["name"], "pack_fp": pack["fingerprint"],
+            "peak_rss_mb": run.get("peak_rss_mb"), "vram_peak_gb": run.get("vram_peak_gb"),
+            "pack": pack["name"], "pack_fp": pack["fingerprint"],
             # hosted (OpenRouter) models: the provider endpoint and its declared quantization
             "endpoint": run.get("endpoint"), "quantization": run.get("quantization"),
             "loaded_alongside": run.get("loaded_alongside") or []}
@@ -121,6 +122,12 @@ def p90(values):
     return statistics.quantiles(values, n=10)[-1] if len(values) > 1 else (values[0] if values else None)
 
 
+def pct(fraction):
+    """A share as a percentage: "100%" and "0%" only when exact, else one decimal ("99.6%"), so a model
+    with a wrong answer never shows as perfect."""
+    return "100%" if fraction >= 1 else "0%" if fraction <= 0 else f"{100 * fraction:.1f}%"
+
+
 def median(values):
     values = [v for v in values if v is not None]
     return statistics.median(values) if values else None
@@ -154,10 +161,19 @@ def settings_notes(infos):
         per_model.setdefault(i["label"], i["settings"])
     keys = sorted({k for s in per_model.values() for k in s})
     differing = [k for k in keys if len({json.dumps(s.get(k)) for s in per_model.values()}) > 1]
-    if differing:
-        detail = "; ".join(f"{k}: " + ", ".join(f"{m}={per_model[m].get(k)}" for m in sorted(per_model))
-                           for k in differing)
-        notes.append(f"Settings differ between models ({detail}). Compare those models knowing that.")
+    parts = []
+    for k in differing:   # only the models that differ from the most common value, named when few
+        groups = collections.defaultdict(list)
+        for m in sorted(per_model):
+            groups[json.dumps(per_model[m].get(k))].append(m)
+        common = max(groups, key=lambda v: len(groups[v]))
+        for value, ms in sorted(groups.items()):
+            if value != common:
+                who = ", ".join(ms) if len(ms) <= 3 else f"{len(ms)} models"
+                shown = "unset" if value == "null" else json.loads(value)
+                parts.append(f"{k} {shown} on {who}")
+    if parts:
+        notes.append("Settings differ: " + "; ".join(parts) + ".")
     fps = collections.defaultdict(set)
     for i in infos:
         if i.get("pack"):
@@ -412,7 +428,7 @@ def by_difficulty(rows):
             cells = []
             for lvl in LEVELS:
                 x = [r["ok"] for r in sub if r.get("difficulty") == lvl]
-                cells.append(f"{sum(x)}/{len(x)} ({100 * sum(x) / len(x):.0f}%)" if x else "-")
+                cells.append(f"{sum(x)}/{len(x)} ({pct(sum(x) / len(x))})" if x else "-")
             table.append([m, s] + cells)
     return header, table
 

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/doctor.py

@@ -10,6 +10,7 @@ import platform
 import sys
 
 from . import engine
+from . import machines
 from . import workspace
 
 OK, WARN, BAD = "✓", "!", "✗"
@@ -44,6 +45,22 @@ def check(models_path=None):
         return r.bad
     r.line(OK, f"workspace {e.root}")
 
+    r.head("Machine")
+    mc = e.machine()
+    r.line(OK, f"{mc.id}: {mc.summary}")
+    if mc.discrete:
+        r.line(OK, f"models are sized to {mc.gpu_limit_gb:.0f} GB of VRAM, keeping {e.headroom_gb(mc):g} GB free "
+                   "(memory_headroom_gb, gpu_memory_gb under [machines.<id>] change it)")
+        new = machines.gpu_id(machines.discrete_gpus(), mc.ram_gb)
+        if mc.id != new and not os.environ.get("EVALS_MACHINE"):
+            r.line(WARN, f"keeping the id {mc.id} this machine had before its GPU was detected, so earlier results "
+                         "and tuning still match", f"to switch: EVALS_MACHINE={new} (results under the old id "
+                                                   "then count as another machine)")
+    elif platform.system() == "Linux" and any(os.path.exists(p) for p in ("/dev/nvidia0", "/dev/kfd")):
+        r.line(WARN, "this machine seems to have an Nvidia or AMD GPU, but nvidia-smi / amd-smi / rocm-smi "
+                     "isn't installed, so models are sized against system RAM",
+               "install the vendor's tools, or set gpu_memory_gb under [machines.<id>] in models.toml")
+
     r.head("Packs")
     for err in e.pack_errors:
         r.line(BAD, err, "fix the pack, then: tuieval selftest")

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/engine.py

@@ -1191,7 +1191,19 @@ class Engine:
 
     # ---- serving on this machine
     def machine(self):
-        return machines.detect()
+        """This machine, with its models.toml [machines.<id>] gpu_memory_gb applied. A Linux machine
+        that recorded itself before its GPU was detected keeps that id, so its results and tuning
+        still match (EVALS_MACHINE renames it)."""
+        mc = machines.detect()
+        if mc.discrete and not os.environ.get("EVALS_MACHINE"):
+            old = machines.machine_id(mc.chip, mc.ram_gb)
+            if os.path.isdir(os.path.join(self.tuning_dir, old)) and \
+                    not os.path.isdir(os.path.join(self.tuning_dir, mc.id)):
+                mc = dataclasses.replace(mc, id=old)
+        override = self.cfg.get("machines", {}).get(mc.id, {}).get("gpu_memory_gb")
+        if override:
+            mc = dataclasses.replace(mc, gpu_limit_gb=float(override))
+        return mc
 
     def machines(self):
         """This machine first, then every other machine recorded in tuning/ (synced)."""
@@ -1203,9 +1215,14 @@ class Engine:
         return [here] + [mc for mc in profiles.known_machines(self.tuning_dir) if mc.id != here.id]
 
     def machine_settings(self, machine_id=None):
-        """models.toml [machines.<id>] (optional): memory_headroom_gb, name."""
+        """models.toml [machines.<id>] (optional): memory_headroom_gb, gpu_memory_gb, name."""
         return self.cfg.get("machines", {}).get(machine_id or self.machine().id, {})
 
+    def headroom_gb(self, mc):
+        """GPU memory kept free: memory_headroom_gb, else 4 GB on a Mac (macOS and other apps share
+        it) and 1.5 GB in a discrete GPU's VRAM."""
+        return float(self.machine_settings(mc.id).get("memory_headroom_gb", 1.5 if mc.discrete else 4.0))
+
     def parallel_models(self):
         """How many models a run serves at a time on this machine: models.toml [machines.<id>]
         parallel_models (default 1)."""
@@ -1224,7 +1241,7 @@ class Engine:
     def memory_available_gb(self):
         """GPU memory models may take here (what the fit check sizes contexts against)."""
         mc = self.machine()
-        return mc.gpu_limit_gb - self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
+        return mc.gpu_limit_gb - self.headroom_gb(mc)
 
     def runs_alone(self, m):
         """A server whose before_start step may stop other servers never runs next to another model."""
@@ -1269,8 +1286,7 @@ class Engine:
                 if fit_check(server):
                     mmproj = expand(m.get("mmproj", ""))
                     extra = os.path.getsize(mmproj) if mmproj and os.path.isfile(mmproj) else 0
-                    headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
-                    f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
+                    f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=self.headroom_gb(mc), extra_bytes=extra,
                                      want_ctx=want)
                     ctx, fits, note, need = f.max_ctx, f.fits, f.note, f.need_gb
             except (OSError, ValueError) as e:
@@ -1484,11 +1500,22 @@ class Engine:
                         time.sleep(0.2)
         threading.Thread(target=run, daemon=True).start()
 
-    def _watch_memory(self, proc, label, stop, peak):
+    VRAM_EVERY_S = 6   # how often to read a discrete GPU's VRAM while a server runs
+
+    def _watch_memory(self, proc, label, stop, peak, info=None):
         """Peak resident memory of the server's process group (MB). On Apple Silicon this includes
-        the model weights mapped for the GPU."""
+        the model weights mapped for the GPU. With discrete GPUs, also the peak VRAM it holds
+        (info["vram_peak_gb"]): its own processes' on Nvidia, the machine's on AMD."""
+        discrete = info is not None and self.machine().discrete
+        last_vram = [0.0]
+
         def run():
             while not stop.is_set():
+                if discrete and time.time() - last_vram[0] >= self.VRAM_EVERY_S:
+                    last_vram[0] = time.time()
+                    used = machines.vram_used_gb(machines.tree_pids(proc.pid))
+                    if used:
+                        info["vram_peak_gb"] = round(max(info.get("vram_peak_gb") or 0, used), 2)
                 try:
                     out = subprocess.run(["ps", "-axo", "pgid=,rss="], capture_output=True, text=True).stdout
                     rss = sum(int(r) for g, r in (l.split() for l in out.splitlines() if l.strip())
@@ -1789,7 +1816,7 @@ class Engine:
                                           f"(see logs/server/{log_name or m['label']}.log)")
                 t0 = time.time()
                 srv = self._start(cmd, log, cwd=cwd)
-                self._watch_memory(srv, m["label"], stop, peak)
+                self._watch_memory(srv, m["label"], stop, peak, info)
             else:
                 key_env = server.get("api_key_env")
                 if key_env and not os.environ.get(key_env):
@@ -1881,6 +1908,7 @@ class Engine:
             def run_info():
                 return {"load_s": info["load_s"], "peak_rss_mb": info["peak_mb"](),
                         "server_facts": info["facts"] or None, "gpu_peak_gb": info.get("gpu_peak_gb"),
+                        "vram_peak_gb": info.get("vram_peak_gb"),
                         "kernel_share_max": info.get("kernel_share_max"),
                         **({"loaded_alongside": loaded_alongside} if loaded_alongside else {}), **serving_info}
             failed_in_row = 0
@@ -1965,7 +1993,8 @@ class Engine:
         return tests
 
     # ---- sittings: a pack run over several sessions (stopped and resumed) keeps its total time
-    SITTING_KEYS = ("load_s", "peak_rss_mb", "gpu_peak_gb", "kernel_share_max", "machine", "loaded_alongside")
+    SITTING_KEYS = ("load_s", "peak_rss_mb", "gpu_peak_gb", "vram_peak_gb", "kernel_share_max", "machine",
+                    "loaded_alongside")
 
     def _sitting(self, job, info):
         now = time.time()
@@ -2037,7 +2066,7 @@ class Engine:
         out = {"started": min(s["started"] for s in sittings),
                "wall_s": round(sum(s.get("wall_s") or 0 for s in sittings), 1),
                "sittings": len(sittings), "sittings_log": sittings}
-        for k in ("peak_rss_mb", "gpu_peak_gb", "kernel_share_max"):
+        for k in ("peak_rss_mb", "gpu_peak_gb", "vram_peak_gb", "kernel_share_max"):
             if peak(k) is not None:
                 out[k] = peak(k)
         along = sorted({l for s in sittings for l in s.get("loaded_alongside") or []})

{tuieval-0.2.0.dev56 → tuieval-0.2.0.dev65}/src/tuieval/machines.py

@@ -1,6 +1,7 @@
 """Which machine this is, what a model needs, and whether it fits.
 
-    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m3max-64gb")
+    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m3max-64gb";
+                             with Nvidia/AMD cards on Linux, their VRAM and an id like "rtx4090-128gb")
     read_gguf(path)       -> GGUF header facts (architecture, layers, KV heads, context length, …)
     fit(model_path, …)    -> Fit (largest context that fits this machine, estimated memory)
 
@@ -9,6 +10,7 @@ the KV cache on its own (that would change answers); a quantized-KV variant is a
 """
 import dataclasses
 import functools
+import json
 import os
 import platform
 import re
@@ -28,15 +30,19 @@ class Machine:
     id: str
     chip: str
     ram_gb: float
-    gpu_limit_gb: float      # what Metal may wire for the GPU
+    gpu_limit_gb: float      # what Metal may wire for the GPU; with discrete GPUs, their VRAM (all cards)
     p_cores: int
     e_cores: int
     gpu_cores: int | None
     os: str
+    gpu_name: str | None = None   # discrete GPUs, e.g. "2× NVIDIA GeForce RTX 4090"
+    discrete: bool = False        # the GPU has its own memory (VRAM), separate from RAM
 
     @property
     def summary(self):
         cores = f"{self.p_cores}P+{self.e_cores}E" if self.e_cores else f"{self.p_cores} cores"
+        if self.discrete:
+            return f"{self.chip} · {self.ram_gb:.0f} GB RAM · {self.gpu_name} · {self.gpu_limit_gb:.0f} GB VRAM · {cores}"
         gpu = f", {self.gpu_cores}-core GPU" if self.gpu_cores else ""
         return f"{self.chip} · {self.ram_gb:.0f} GB (GPU up to {self.gpu_limit_gb:.0f} GB) · {cores}{gpu}"
 
@@ -53,6 +59,129 @@ def machine_id(chip, ram_gb):
     return re.sub(r"[^a-z0-9]+", "", name.lower()) + f"-{round(ram_gb)}gb"
 
 
+def gpu_id(gpus, ram_gb):
+    """A machine id from its discrete GPUs: "rtx4090-128gb", "2xrtx4090-128gb", "rx7900xtx+rtx3090-64gb"."""
+    names = []
+    for name in sorted({n for n, _ in gpus}):
+        short = re.sub(r"\b(nvidia|geforce|amd|radeon|ati|graphics|tm)\b", "", name.lower())
+        short = re.sub(r"[^a-z0-9]+", "", short) or "gpu"
+        count = sum(n == name for n, _ in gpus)
+        names.append(f"{count}x{short}" if count > 1 else short)
+    return "+".join(names) + f"-{round(ram_gb)}gb"
+
+
+# ---------------------------------------------------------------- discrete GPUs (Linux: Nvidia, AMD)
+# Read from the vendors' own tools. Anything unexpected counts as "not detected", never an error.
+def _run(cmd, timeout=10):
+    try:
+        p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
+    except (OSError, subprocess.TimeoutExpired):
+        return None
+    return p.stdout if p.returncode == 0 else None
+
+
+def _nvidia_gpus():
+    out = _run(["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
+    gpus = []
+    for line in (out or "").splitlines():
+        name, _, mib = line.rpartition(",")
+        try:
+            gpus.append((name.strip(), float(mib) / 1024))
+        except ValueError:
+            pass
+    return [g for g in gpus if g[0] and g[1] > 0]
+
+
+def _walk(obj):
+    """Every dict inside a JSON value."""
+    if isinstance(obj, dict):
+        yield obj
+        for v in obj.values():
+            yield from _walk(v)
+    elif isinstance(obj, list):
+        for v in obj:
+            yield from _walk(v)
+
+
+def _bytes(value, unit=None):
+    """VRAM sizes as amd-smi/rocm-smi print them: 25753026560, "24560 MB", {"value": 24560, "unit": "MB"}."""
+    if isinstance(value, dict):
+        return _bytes(value.get("value"), value.get("unit"))
+    m = re.match(r"\s*([\d.]+)\s*([KMGT]i?B|B)?\s*$", str(value), re.I)
+    if not m:
+        return None
+    scale = {"B": 1, "KB": 2**10, "MB": 2**20, "GB": 2**30, "TB": 2**40}
+    u = (m.group(2) or unit or "B").upper().replace("I", "")
+    return float(m.group(1)) * scale.get(u, 1)
+
+
+def _amd_gpus():
+    out = _run(["amd-smi", "static", "--asic", "--vram", "--json"])
+    gpus = []
+    try:
+        for d in _walk(json.loads(out)) if out else []:
+            if "vram" in d and isinstance(d["vram"], dict):
+                size = _bytes(d["vram"].get("size"))
+                name = (d.get("asic") or {}).get("market_name") or "AMD GPU"
+                if size:
+                    gpus.append((str(name), size / GB))
+    except ValueError:
+        gpus = []
+    if gpus:
+        return gpus
+    out = _run(["rocm-smi", "--showmeminfo", "vram", "--showproductname", "--json"])
+    try:
+        cards = json.loads(out) if out else {}
+    except ValueError:
+        return []
+    for key, d in sorted(cards.items()) if isinstance(cards, dict) else []:
+        if not key.startswith("card") or not isinstance(d, dict):
+            continue
+        size = _bytes(d.get("VRAM Total Memory (B)"))
+        name = d.get("Card Series") or d.get("Card series") or d.get("Card SKU") or d.get("Card model") or "AMD GPU"
+        if size:
+            gpus.append((str(name), size / GB))
+    return gpus
+
+
+def discrete_gpus():
+    """[(name, VRAM GB)] for the Nvidia or AMD cards this machine has, or [] (none, or tools missing)."""
+    return _nvidia_gpus() or _amd_gpus()
+
+
+def vram_used_gb(pids=None):
+    """VRAM in use now (GB): by these processes on Nvidia (all of it when pids is None), or on the
+    whole machine on AMD, whose tools don't report it reliably per process. None if unknown."""
+    if pids is not None:
+        out = _run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"])
+        if out is not None:
+            used = 0.0
+            for line in out.splitlines():
+                pid, _, mib = line.partition(",")
+                if pid.strip() in {str(p) for p in pids}:
+                    try:
+                        used += float(mib) / 1024
+                    except ValueError:
+                        pass
+            return used
+    out = _run(["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"])
+    if out is not None:
+        try:
+            return sum(float(x) for x in out.split()) / 1024
+        except ValueError:
+            return None
+    out = _run(["rocm-smi", "--showmeminfo", "vram", "--json"])
+    try:
+        cards = json.loads(out) if out else None
+    except ValueError:
+        return None
+    if not isinstance(cards, dict):
+        return None
+    used = [_bytes(d.get("VRAM Total Used Memory (B)")) for k, d in cards.items()
+            if k.startswith("card") and isinstance(d, dict)]
+    return sum(u for u in used if u) / GB if any(used) else None
+
+
 @functools.lru_cache(maxsize=1)
 def detect():
     """This machine. Override the id with EVALS_MACHINE if two machines would collide."""
@@ -73,11 +202,19 @@ def detect():
         except (OSError, subprocess.TimeoutExpired):
             pass
         os_name = "macOS " + platform.mac_ver()[0]
-    else:  # Linux and others: CPU-side facts only
+    else:  # Linux and others: the CPU side, plus Nvidia/AMD cards if their tools are installed
         chip = platform.processor() or platform.machine()
         ram = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / GB
         p, e, gpu_cores, limit = os.cpu_count() or 1, 0, None, ram * 0.9
         os_name = platform.system()
+        gpus = discrete_gpus()
+        if gpus:
+            names = sorted({n for n, _ in gpus})
+            gpu_name = " + ".join(f"{sum(n == x for n, _ in gpus)}× {x}" if sum(n == x for n, _ in gpus) > 1 else x
+                                  for x in names)
+            mid = os.environ.get("EVALS_MACHINE") or gpu_id(gpus, ram)
+            return Machine(mid, chip, round(ram, 1), round(sum(g for _, g in gpus), 1), p, e, None, os_name,
+                           gpu_name=gpu_name, discrete=True)
     mid = os.environ.get("EVALS_MACHINE") or machine_id(chip, ram)
     return Machine(mid, chip, round(ram, 1), round(limit, 1), p, e, gpu_cores, os_name)
 
@@ -195,14 +332,15 @@ def fit(model_path, machine=None, kv_type="f16", headroom_gb=4.0, extra_bytes=0,
     model_max = info["context_length"] or 131072
     max_ctx = int(min(room / per_tok, model_max, want_ctx or model_max)) if room > 0 else 0
     max_ctx = max_ctx // 1024 * 1024
+    of = " of VRAM" if machine.discrete else ""
     if max_ctx < MIN_CTX:
         need = fixed + MIN_CTX * per_tok / GB
         return Fit(False, max_ctx, round(need, 1), round(available, 1),
                    f"doesn't fit on {machine.id}: needs ~{need:.0f} GB at {MIN_CTX // 1024}k context, "
-                   f"{available:.0f} GB available")
+                   f"{available:.0f} GB{of} available")
     need = fixed + max_ctx * per_tok / GB
     return Fit(True, max_ctx, round(need, 1), round(available, 1),
-               f"fits on {machine.id} up to {max_ctx // 1024}k context (~{need:.0f} of {available:.0f} GB)")
+               f"fits on {machine.id} up to {max_ctx // 1024}k context (~{need:.0f} of {available:.0f} GB{of})")
 
 
 # ---------------------------------------------------------------- memory pressure
@@ -287,6 +425,26 @@ def tree_cpu_seconds(pid):
     return (sum(rows[p][1] for p in tree), sum(rows[p][2] for p in tree))
 
 
+def tree_pids(pid):
+    """A process and all its descendants (pids), or [pid] if ps can't be read."""
+    try:
+        out = subprocess.run(["ps", "-axo", "pid=,ppid="], capture_output=True, text=True, timeout=5).stdout
+    except (OSError, subprocess.TimeoutExpired):
+        return [pid]
+    children = {}
+    for line in out.splitlines():
+        parts = line.split()
+        if len(parts) == 2:
+            children.setdefault(parts[1], []).append(parts[0])
+    tree, frontier = {str(pid)}, [str(pid)]
+    while frontier:
+        for child in children.get(frontier.pop(), []):
+            if child not in tree:
+                tree.add(child)
+                frontier.append(child)
+    return sorted(int(p) for p in tree)
+
+
 def group_cpu_seconds(pgid):
     """(user, kernel) CPU seconds used so far by a process group, or None."""
     try: