tuieval 0.2.0.dev55__tar.gz → 0.2.0.dev62__tar.gz

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/CHANGELOG.md

@@ -2,9 +2,11 @@
 
 ## Unreleased
 
+- Results has four tabs: PTA index, Production readiness, Per question and Failures. The PTA table adds tok/s, TTFT and memory; Per question can show only where models disagree; the settings warning is one line. Scorecard, pairwise, history and test quality stay in `tuieval compare`, `history` and `items`.
+- The run's ETA adds up each pack's answers left × that model's time per answer there (its history, then its own times once it has 5 answers), instead of extrapolating from the first few answers.
 - Faster start on Macs: the GPU core count comes from `ioreg` instead of `system_profiler`, which took seconds on some Macs.
 - A model a server won't serve (HTTP 401-404, e.g. OpenRouter's "No endpoints found") is never judged on it: those answers don't count, and the model stops at once instead of failing every question.
-- PTA index: parsimony (tokens used) and speed (time), each 100 for the best and 15 points less per doubling, and accuracy per model, as bars: the first Results tab (where `r` lands), `tuieval pta` and the report. Server errors aren't answers, and models with few answers are left out unless picked.
+- PTA index: parsimony (tokens), time and accuracy per model as bars, with tokens and time shown as multiples of the best (2.4×): the first Results tab (where `r` lands), `tuieval pta` and the report. Server errors aren't answers, and models with few answers are left out unless picked.
 - Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
 - More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev55
+Version: 0.2.0.dev62
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -144,7 +144,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
-4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
@@ -176,7 +176,7 @@ tuieval help
 
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
-- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/README.md

@@ -118,7 +118,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
-4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
@@ -150,7 +150,7 @@ tuieval help
 
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
-- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/docs/models.md

@@ -86,11 +86,13 @@ By default a run serves one model at a time. On a machine with room for more, se
 
 ## PTA index
 
-The **PTA index** puts parsimony (tokens), speed (time) and accuracy side by side, 0-100 each and higher is better, as a bar per score for each model, with the numbers and a table underneath. It's the first tab in Results, where `r` opens, and it's also `tuieval pta` and a section in `tuieval report`. There's no combined number; verdicts and critical failures stay in Production readiness.
+The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a bar for each per model (longer is better), with a table underneath. It's the first tab in Results, where `r` opens, and it's also `tuieval pta` and a section in `tuieval report`. There's no combined number; verdicts and critical failures stay in Production readiness.
 
-- **P parsimony** (a score out of 100, not a percentage): from the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). The fewest tokens score 100 (★ in the **tokens** column), and every doubling costs 15 points: `100 − 15 × log2(this model's tokens ÷ fewest)`, never below 0. Unlike speed it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
-- **T speed** (a score out of 100, not a percentage): the same from the total time to answer those questions. The fastest model scores 100 (★ on its total time), and every doubling of the total time costs 15 points: `100 − 15 × log2(this model's total ÷ fastest total)`, never below 0 (about 100× slower). Twice as slow is 85, 10× slower is 50. Adding a faster model lowers the others. The table's **total time** column has the times themselves (lower is better).
-- **A accuracy** (a percentage): the share of answers to those questions that passed.
+- **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
+- **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
+- **A accuracy**: the share of answers to those questions that passed.
+
+The P and T bars use a log scale: full for the best, 15 points (of 100) shorter for every doubling, empty at about 100 times the best. So a model at 28× still shows a bar, and adding a better model shortens the others'.
 
 Models are compared on the questions all of them answered, so one that skipped some never looks faster. Server and connection errors aren't answers: a model with only those is listed as having none. A model with fewer than half as many answers as the most-answered one is left out (and listed), or it would shrink the shared questions for everyone. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`: models you pick are always compared.
 

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/docs/writing-packs.md

@@ -127,7 +127,7 @@ def ticket_route(answer, test, meta):
 - `critical=True` makes every test of this grader a critical trial (any of its failures *can* be critical). Otherwise only tests with `critical: true` or `expected: NOT_AVAILABLE` are.
 - `template` is the skeleton `tuieval capture` writes for a new test of this grader.
 
-A grader file can also add **Scorecard columns** (Results → Scorecard, and `tuieval compare`), for numbers your use case cares about that pass/fail doesn't show:
+A grader file can also add **Scorecard columns** (`tuieval compare`), for numbers your use case cares about that pass/fail doesn't show:
 
 ```python
 import json
@@ -175,7 +175,7 @@ Graders mark failures that should disqualify a model whatever its accuracy:
 
 ## Difficulty labels
 
-Every test has a `difficulty` of easy, medium or hard. It's for you, not the model: requests are built only from `input`, `image`, the system prompt and tools, so the label never reaches the model. The TUI shows it in the pack list (e.g. `12E 20M 8H`), next to the current question, in Recent results and Failures, and in Results → By difficulty.
+Every test has a `difficulty` of easy, medium or hard. It's for you, not the model: requests are built only from `input`, `image`, the system prompt and tools, so the label never reaches the model. The TUI shows it in the pack list (e.g. `12E 20M 8H`), next to the current question, in Recent results and Failures, and as a filter in Results → Per question.
 
 A useful convention: **easy** = one rule or a direct read; **medium** = arithmetic, one trap, or two conditions; **hard** = rules in conflict, multi-step reasoning, or misleading data. Once several models have run, `tuieval items` flags labels the results contradict ("easier than labelled": a hard test every model passes; "harder than labelled": an easy test most models fail). `tuieval selftest` warns about tests without a label.
 

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev55'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev55')
+__version__ = version = '0.2.0.dev62'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev62')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/cli.py

@@ -14,7 +14,7 @@ Start here
 
 Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
-  tuieval pta                 the PTA index: parsimony, speed and accuracy per model, as bars
+  tuieval pta                 the PTA index: parsimony (tokens), time and accuracy per model, as bars
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
   tuieval items               tests that don't separate models or look broken

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/compare.py

@@ -121,6 +121,12 @@ def p90(values):
     return statistics.quantiles(values, n=10)[-1] if len(values) > 1 else (values[0] if values else None)
 
 
+def pct(fraction):
+    """A share as a percentage: "100%" and "0%" only when exact, else one decimal ("99.6%"), so a model
+    with a wrong answer never shows as perfect."""
+    return "100%" if fraction >= 1 else "0%" if fraction <= 0 else f"{100 * fraction:.1f}%"
+
+
 def median(values):
     values = [v for v in values if v is not None]
     return statistics.median(values) if values else None
@@ -154,10 +160,19 @@ def settings_notes(infos):
         per_model.setdefault(i["label"], i["settings"])
     keys = sorted({k for s in per_model.values() for k in s})
     differing = [k for k in keys if len({json.dumps(s.get(k)) for s in per_model.values()}) > 1]
-    if differing:
-        detail = "; ".join(f"{k}: " + ", ".join(f"{m}={per_model[m].get(k)}" for m in sorted(per_model))
-                           for k in differing)
-        notes.append(f"Settings differ between models ({detail}). Compare those models knowing that.")
+    parts = []
+    for k in differing:   # only the models that differ from the most common value, named when few
+        groups = collections.defaultdict(list)
+        for m in sorted(per_model):
+            groups[json.dumps(per_model[m].get(k))].append(m)
+        common = max(groups, key=lambda v: len(groups[v]))
+        for value, ms in sorted(groups.items()):
+            if value != common:
+                who = ", ".join(ms) if len(ms) <= 3 else f"{len(ms)} models"
+                shown = "unset" if value == "null" else json.loads(value)
+                parts.append(f"{k} {shown} on {who}")
+    if parts:
+        notes.append("Settings differ: " + "; ".join(parts) + ".")
     fps = collections.defaultdict(set)
     for i in infos:
         if i.get("pack"):
@@ -412,7 +427,7 @@ def by_difficulty(rows):
             cells = []
             for lvl in LEVELS:
                 x = [r["ok"] for r in sub if r.get("difficulty") == lvl]
-                cells.append(f"{sum(x)}/{len(x)} ({100 * sum(x) / len(x):.0f}%)" if x else "-")
+                cells.append(f"{sum(x)}/{len(x)} ({pct(sum(x) / len(x))})" if x else "-")
             table.append([m, s] + cells)
     return header, table
 

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/engine.py

@@ -987,16 +987,16 @@ class Engine:
                     del per_pack[name]
         return {l: p for l, p in out.items() if p}
 
-    def estimate_seconds(self, jobs):
-        """(seconds, notes): expected run time for the waiting jobs, estimated per pack.
+    def seconds_per_answer(self, jobs, waiting_only=True):
+        """({job key: expected seconds per answer}, borrowed, guessed, thin) for the waiting jobs.
         A pack the model has answered uses its own median time there. Otherwise the median of
         other models' times on that pack, scaled by how much slower or faster this model is than
-        them on packs both have answered. notes lists what made the estimate rough ([] = solid)."""
+        them on packs both have answered. The sets say which estimates are rough (see estimate_seconds)."""
         times = self.answer_times()
         med = {l: {p: statistics.median(ts) for p, ts in pp.items()} for l, pp in times.items()}
-        total, borrowed, guessed, thin = 0.0, set(), set(), set()
+        spr_of, borrowed, guessed, thin = {}, set(), set(), set()
         for j in jobs:
-            if j.status != "waiting":
+            if j.status in ("done", "skipped", "failed") or (waiting_only and j.status != "waiting"):
                 continue
             own = med.get(j.label, {})
             ref = {p: statistics.median(v[p] for l, v in med.items() if p in v and l != j.label)
@@ -1013,8 +1013,18 @@ class Engine:
             else:
                 spr = statistics.median(own.values()) if own else DEFAULT_SECONDS_PER_REQUEST
                 guessed.add((j.key, bool(own)))
-            done = len(self._done_keys(j)) if j.merge else 0
-            total += max(0, j.total - done) * spr
+            spr_of[j.key] = spr
+        return spr_of, borrowed, guessed, thin
+
+    def estimate_seconds(self, jobs):
+        """(seconds, notes): expected run time for the waiting jobs, estimated per pack
+        (seconds_per_answer). notes lists what made the estimate rough ([] = solid)."""
+        spr_of, borrowed, guessed, thin = self.seconds_per_answer(jobs)
+        total = 0.0
+        for j in jobs:
+            if j.key in spr_of:
+                done = len(self._done_keys(j)) if j.merge else 0
+                total += max(0, j.total - done) * spr_of[j.key]
         notes = []
         if borrowed:
             factors = sorted({f for _, f, _ in borrowed if f is not None})

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/pta.py

@@ -1,13 +1,14 @@
-"""PTA index: parsimony, speed (time) and accuracy per model, 0-100 each and higher is better, shown
-as a bar per score for each model.
+"""PTA index: parsimony (tokens), time and accuracy per model, a bar each (longer = better).
 
-    P  parsimony  a score out of 100, from the tokens (reasoning and answer) used on every question
-                  compared (each question at its median over repeats): 100 for the model that used the
-                  fewest, POINTS_PER_DOUBLING points less for every doubling (0 at about 100 times more).
-                  Unlike T it doesn't depend on the machine: it's how much a model says to get there.
-    T  speed      a score out of 100, the same from the total time to answer those questions: 100 for
-                  the fastest model. Log scales, so models that are all far behind the best still differ.
-    A  accuracy   a percentage: the share of answers to those questions that passed.
+    P  parsimony (tokens)  the tokens (reasoning and answer) used on every question compared (each
+                           question at its median over repeats), shown as how many times the leanest
+                           model's ("2.4×"). Unlike T it doesn't depend on the machine.
+    T  time                the total time to answer those questions, as how many times the fastest
+                           model's ("28×").
+    A  accuracy            the share of answers to those questions that passed (%).
+
+Behind the P and T bars is a 0-100 score: 100 for the best, POINTS_PER_DOUBLING points less for every
+doubling (0 at about 100 times more), so models far behind the best still get a visible bar.
 
 Models are compared on the questions all of them answered, so one that skipped some never looks
 faster. Server and connection errors aren't answers. By default a model with fewer than half as many
@@ -32,12 +33,23 @@ def score(total, best):
     return max(0.0, 100 - POINTS_PER_DOUBLING * math.log2(total / best))
 
 
-def index(rows, models=None):
+def memory_gb(infos):
+    """{model: the most memory its server held in any run (GB)}, for models tuieval served (hosted: none).
+    On a discrete GPU that's its peak VRAM (vram_peak_gb), where the model lives; otherwise its resident memory."""
+    out = {}
+    for i in infos:
+        gb = i.get("vram_peak_gb") or (i["peak_rss_mb"] / 1024 if i.get("peak_rss_mb") else None)
+        if gb:
+            out[i["label"]] = max(out.get(i["label"], 0), gb)
+    return out
+
+
+def index(rows, models=None, memory=None):
     """{"questions": n shared, "models": [{model, P, T, A, tokens, total_s, passed, answers, alongside,
-    fewest, fastest}],
+    fewest, fastest, tps, ttft, memory_gb}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
     most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
-    with enough answers; models named here are compared whatever their count)."""
+    with enough answers; models named here are compared whatever their count). memory: memory_gb()."""
     no_answers = sorted({r["model"] for r in rows} - {r["model"] for r in rows if not r.get("error")})
     rows = [r for r in rows if not r.get("error")]     # a server or connection error isn't an answer
     tests = compare.per_question(rows)
@@ -61,12 +73,19 @@ def index(rows, models=None):
                     "total_s": sum(c["secs"] or 0 for c in cells),
                     "passed": sum(c["passed"] for c in cells), "answers": answers,
                     "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    # generation speed and time to first token over the same answers (medians)
+                    "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
+                    "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
+                    "memory_gb": (memory or {}).get(m),
                     # answers that ran while other models were served shared the machine's speed
                     "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
     fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
     fewest = min((x["tokens"] for x in out if x["tokens"] > 0), default=None)
     for x in out:
         x["P"], x["T"] = score(x["tokens"], fewest), score(x["total_s"], fastest)
+        # shown on the bars: how many times the leanest / fastest model's total (1.0 = the best)
+        x["tokens_x"] = x["tokens"] / fewest if fewest and x["tokens"] else None
+        x["time_x"] = x["total_s"] / fastest if fastest and x["total_s"] else None
         x["fewest"] = bool(fewest) and x["tokens"] == fewest
         x["fastest"] = bool(fastest) and x["total_s"] == fastest
     out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
@@ -74,29 +93,40 @@ def index(rows, models=None):
             "most": most}
 
 
-LETTERS = (("P", "parsimony", "/100", "cyan"), ("T", "speed", "/100", "magenta"), ("A", "accuracy", "%", "green"))
+# (letter, name, what the number beside the bar shows, colour); the bar itself is the 0-100 score
+LETTERS = (("P", "parsimony (tokens)", "tokens_x", "cyan"), ("T", "time", "time_x", "magenta"),
+           ("A", "accuracy", "A", "green"))
+
+
+def times(v):
+    """A multiple of the best: "1.0×", "2.4×", "28×"."""
+    return "-" if v is None else f"{v:.1f}×" if v < 9.95 else f"{v:.0f}×"
 
 
 def bars(result, width=20, markup=True):
-    """Lines with a row per model and a bar per score (P, T, A), longer = better, each with its number:
-    the at-a-glance view. markup=False gives plain text (the report)."""
+    """Lines with a row per model and a bar per score (P, T, A), longer = better: the at-a-glance view.
+    Beside P and T, how many times the leanest / fastest model's tokens / time; beside A, the percentage.
+    markup=False gives plain text (the report)."""
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
-    cell_w = width + 6                          # the bar, a space and "100%" or " 100"
+    # each column: the bar, a space, "100%" or "9.8×" (5), then a gap; at least as wide as its heading
+    cell_w = max(width + 9, *(len(f"{k} {name}") + 2 for k, name, _, _ in LETTERS))
     head = " " * (name_w + 2) + "".join(f"{f'{k} {name}':<{cell_w}}" for k, name, _, _ in LETTERS)
     lines = [f"[b]{head.rstrip()}[/b]" if markup else head.rstrip()]
     for x in result["models"]:
         cells = []
-        for k, _, unit, color in LETTERS:
+        for k, _, shown, color in LETTERS:
             v = x[k]
             full = 0 if v is None else round(width * max(0, min(100, v)) / 100)
+            if v is not None and v < 100:   # full only when perfect (accuracy) or the best (P, T)
+                full = min(full, width - 1)
             bar, rest = "█" * full, "░" * (width - full)
-            num = "-" if v is None else f"{v:.0f}{'%' if unit == '%' else ''}"
+            num = (("-" if v is None else compare.pct(v / 100)) if shown == "A" else times(x.get(shown)))
             bar = f"[{color}]{bar}[/][grey37]{rest}[/]" if markup else bar + rest
-            cells.append(f"{bar} {num:>4}")
+            cells.append(f"{bar} {num:>6}" + " " * (cell_w - width - 7))
         name = f"{x['model']:<{name_w}}"
-        lines.append(f"{rich_escape(name) if markup else name}  " + "  ".join(cells))
+        lines.append((f"{rich_escape(name) if markup else name}  " + "".join(cells)).rstrip())
     return lines
 
 
@@ -116,22 +146,28 @@ def left_out_lines(result):
     return out
 
 
-def _score(v, unit=""):
-    """A is a percentage (%), P and T scores out of 100 (/100: on a log scale, not percentages)."""
-    return "-" if v is None else f"{v:.0f}{unit}"
-
-
 def table(result):
     """(header, rows) for a plain table of the index."""
-    header = ["model", "P parsimony /100", "T speed /100", "A accuracy %", "tokens", "total time",
-              "answers right"]
-    rows = [[x["model"], _score(x["P"]), _score(x["T"]), _score(x["A"], "%"),
-             f"{round(x['tokens']):,}" + (" ★" if x.get("fewest") else ""),
-             fmt_total(x["total_s"]) + (" ★" if x.get("fastest") else ""),
-             f"{x['passed']}/{x['answers']}"] for x in result["models"]]
+    n = result["questions"] or 1
+    header = ["model", "tokens/answer", "time/answer", "answers right", "tok/s", "TTFT s", "memory GB"]
+    rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
+             fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
+             f"{x['passed']}/{x['answers']}", _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
+             _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
     return header, rows
 
 
+def fmt_secs(secs):
+    """Time per answer: "2.5s", "69s", "4m10s"."""
+    if not secs:
+        return "-"
+    return f"{secs:.1f}s" if secs < 10 else f"{secs:.0f}s" if secs < 60 else fmt_total(secs)
+
+
+def _num(v, fmt):
+    return "-" if v is None else format(v, fmt)
+
+
 def fmt_total(secs):
     if not secs:
         return "-"
@@ -147,19 +183,18 @@ def scope_note(result, models):
     if not result["questions"]:
         return "These models have no question in common yet, so there's nothing to compare."
     return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
-            f"A is a percentage. P parsimony and T speed are scores out of 100, not percentages: 100 = the fewest "
-            f"tokens / the fastest total time (★), {POINTS_PER_DOUBLING} points less for each doubling; a better "
-            "model added lowers the others.")
+            "Longer bars are better. P parsimony (tokens) and T time: how many times the leanest / fastest "
+            "model's (1.0×, ★ in the table). A accuracy: answers right.")
 
 
 def for_engine(e, labels=None, packs=None, results_dir=None):
     """(result, models) from a workspace's finished results (results on their way out excluded)."""
     paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
-    rows, _ = compare.load(paths) if paths else ([], [])
+    rows, infos = compare.load(paths) if paths else ([], [])
     if packs:
         rows = [r for r in rows if r["suite"] in packs]
     models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
-    return index(rows, models if labels else None), models
+    return index(rows, models if labels else None, memory_gb(infos)), models
 
 
 def markdown(result, models):

{tuieval-0.2.0.dev55 → tuieval-0.2.0.dev62}/src/tuieval/run_evals.py

@@ -373,11 +373,12 @@ def report_markdown(e, labels=None):
                              + f"; truncated {100 * ev['truncation']:.1f}%"
                              + (f"; consistency {100 * ev['consistency']:.0f}% of {ev['groups']} groups" if ev.get("consistency") is not None else ""))
                 recs = e.records(label, name)
+                from .compare import pct
                 levels = []
                 for lvl in ("easy", "medium", "hard"):
                     x = [r["pass"] for r in recs if r.get("difficulty") == lvl]
                     if x:
-                        levels.append(f"{lvl} {sum(x)}/{len(x)} ({100 * sum(x) / len(x):.0f}%)")
+                        levels.append(f"{lvl} {sum(x)}/{len(x)} ({pct(sum(x) / len(x))})")
                 if levels:
                     lines.append("  - by difficulty: " + ", ".join(levels))
                 for ex in ev.get("critical_examples", []):
@@ -389,7 +390,7 @@ def report_markdown(e, labels=None):
 
 def cmd_pta(argv):
     p = argparse.ArgumentParser(prog="tuieval pta",
-                                description="The PTA index: parsimony (tokens), speed and accuracy per model (0-100, higher is better), as a "
+                                description="The PTA index: parsimony (tokens), time and accuracy per model, as a "
                                             "bar per score for each model and a table, over the questions all the models answered.")
     p.add_argument("--only", help="comma-separated model labels (default: every model with results)")
     p.add_argument("--packs", help="comma-separated packs (default: all)")
@@ -413,7 +414,7 @@ def cmd_pta(argv):
         return
     out.print()
     for line in pta.bars(result, a.width):
-        out.print(line)
+        out.print(line, soft_wrap=True)   # a bar row stays one line (the terminal wraps it if it must)
     header, rows = pta.table(result)
     widths = [max(len(h), *(len(r[i]) for r in rows)) for i, h in enumerate(header)]
     out.print()