tuieval 0.2.0.dev42__tar.gz → 0.2.0.dev50__tar.gz

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## Unreleased
 
-- PTA index: privacy, time (total, relative to the fastest) and accuracy per model, as a triangle in Results, `tuieval pta` and the report.
+- A model a server won't serve (HTTP 401-404, e.g. OpenRouter's "No endpoints found") is never judged on it: those answers don't count, and the model stops at once instead of failing every question.
+- PTA index: parsimony (tokens used) and speed (time), each 100 for the best and 15 points less per doubling, and accuracy per model, as bars: the first Results tab (where `r` lands), `tuieval pta` and the report. Server errors aren't answers, and models with few answers are left out unless picked.
 - Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
 - More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev42
+Version: 0.2.0.dev50
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/docs/models.md

@@ -86,13 +86,13 @@ By default a run serves one model at a time. On a machine with room for more, se
 
 ## PTA index
 
-The **PTA index** puts privacy, time and accuracy side by side, 0-100 each, as a radar triangle (Results → PTA index, `tuieval pta`, and a section in `tuieval report`). Each model is a triangle reaching each corner as far as its score: bigger means better all round. There's no combined number; verdicts and critical failures stay in Production readiness.
+The **PTA index** puts parsimony (tokens), speed (time) and accuracy side by side, 0-100 each and higher is better, as a bar per score for each model, with the numbers and a table underneath. It's the first tab in Results, where `r` opens, and it's also `tuieval pta` and a section in `tuieval report`. There's no combined number; verdicts and critical failures stay in Production readiness.
 
-- **P privacy**: 100 when prompts stay on machines you control (tuieval starts the server, its `url` is this machine, or the server has `private = true`, e.g. your own box on the network); 0 for a hosted API.
-- **T time**: the total time to answer every question compared (each question at its median over repeats), relative to the fastest model: `100 × fastest total ÷ this model's total`. Adding a faster model lowers the others.
-- **A accuracy**: the share of answers to those questions that passed.
+- **P parsimony** (a score out of 100, not a percentage): from the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). The fewest tokens score 100 (★ in the **tokens** column), and every doubling costs 15 points: `100 − 15 × log2(this model's tokens ÷ fewest)`, never below 0. Unlike speed it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
+- **T speed** (a score out of 100, not a percentage): the same from the total time to answer those questions. The fastest model scores 100 (★ on its total time), and every doubling of the total time costs 15 points: `100 − 15 × log2(this model's total ÷ fastest total)`, never below 0 (about 100× slower). Twice as slow is 85, 10× slower is 50. Adding a faster model lowers the others. The table's **total time** column has the times themselves (lower is better).
+- **A accuracy** (a percentage): the share of answers to those questions that passed.
 
-Models are compared on the questions all of them answered, so one that skipped some never looks faster. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`.
+Models are compared on the questions all of them answered, so one that skipped some never looks faster. Server and connection errors aren't answers: a model with only those is listed as having none. A model with fewer than half as many answers as the most-answered one is left out (and listed), or it would shrink the shared questions for everyone. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`: models you pick are always compared.
 
 ## Tuning
 
@@ -140,7 +140,6 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 |---|---|
 | `cmd` | command that starts the server (no `cmd` = an already-running server at `url`). Placeholders: `{model}`, `{served_name}`, `{port}`, `{mmproj}`, `{ctx}`, `{kv_type}`, `{root}` (the workspace), and the tune placeholders. An argument `env:NAME=value` sets an environment variable instead. |
 | `url` | an already-running server's base URL |
-| `private` | `true` for a server on a machine you control (e.g. your own box on the network): its models score 100 for privacy in the PTA index |
 | `port` | the port `cmd` serves on |
 | `cwd` | folder to start `cmd` in |
 | `model_is_path` | `model` is a file or folder: check it exists before starting |

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev42'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev42')
+__version__ = version = '0.2.0.dev50'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev50')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/cli.py

@@ -14,7 +14,7 @@ Start here
 
 Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
-  tuieval pta                 the PTA index: privacy, time and accuracy per model, as a triangle
+  tuieval pta                 the PTA index: parsimony, speed and accuracy per model, as bars
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
   tuieval items               tests that don't separate models or look broken

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/client.py

@@ -17,6 +17,10 @@ import time
 import urllib.parse
 
 SERVER_ERROR_PREFIXES = ("server returned HTTP 5", "server returned HTTP 429", "server error", "connection error")
+# the model can't be reached there at all (a bad key, no credits, or a hosted API with no provider serving it, e.g.
+# OpenRouter's 404 "No endpoints found"): not the model's answer either, and waiting won't fix it
+UNAVAILABLE_PREFIXES = ("server returned HTTP 401", "server returned HTTP 402", "server returned HTTP 403",
+                        "server returned HTTP 404")
 
 
 def is_server_error(error):
@@ -27,9 +31,15 @@ def is_server_error(error):
         and "timeout" not in error.lower()
 
 
+def is_unavailable(error):
+    """True for an error saying the model isn't available on that server at all: HTTP 401, 402, 403 or
+    404. Never the model's answer, and not worth retrying."""
+    return bool(error) and error.startswith(UNAVAILABLE_PREFIXES)
+
+
 def server_error_row(rec):
     """A stored answer that was really a server failure (never a judgement of the model)."""
-    return not rec.get("pass") and is_server_error(rec.get("reason"))
+    return not rec.get("pass") and (is_server_error(rec.get("reason")) or is_unavailable(rec.get("reason")))
 
 
 class Cancelled(Exception):

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/engine.py

@@ -2105,6 +2105,9 @@ class Engine:
                     self._check()
                     if res["error"] and not alive():
                         raise ModelFailed(f"the server stopped: {res['error']}")
+                    if client.is_unavailable(res["error"]):   # never an answer; retrying won't help
+                        raise ModelFailed(f"the model isn't available there (not counted against it): "
+                                          f"{res['error'][:300]}")
                     if not client.is_server_error(res["error"]):
                         break
                     # the server failed, not the model: never recorded as an answer

tuieval-0.2.0.dev50/src/tuieval/pta.py

@@ -0,0 +1,176 @@
+"""PTA index: parsimony, speed (time) and accuracy per model, 0-100 each and higher is better, shown
+as a bar per score for each model.
+
+    P  parsimony  a score out of 100, from the tokens (reasoning and answer) used on every question
+                  compared (each question at its median over repeats): 100 for the model that used the
+                  fewest, POINTS_PER_DOUBLING points less for every doubling (0 at about 100 times more).
+                  Unlike T it doesn't depend on the machine: it's how much a model says to get there.
+    T  speed      a score out of 100, the same from the total time to answer those questions: 100 for
+                  the fastest model. Log scales, so models that are all far behind the best still differ.
+    A  accuracy   a percentage: the share of answers to those questions that passed.
+
+Models are compared on the questions all of them answered, so one that skipped some never looks
+faster. Server and connection errors aren't answers. By default a model with fewer than half as many
+answers as the most-answered model is left out (listed, so it can be picked), or it would shrink the
+shared questions for everyone. There's no combined number: the three stay side by side, and verdicts
+and critical failures stay where they are (verdict.py).
+"""
+import math
+
+from rich.markup import escape as rich_escape
+
+from . import compare
+
+MIN_SHARE = 0.5   # fewer answers than this share of the most-answered model's: left out by default
+POINTS_PER_DOUBLING = 15
+
+
+def score(total, best):
+    """P and T: 100 for the best (lowest) total, POINTS_PER_DOUBLING less for every doubling, never below 0."""
+    if not total or not best:
+        return None
+    return max(0.0, 100 - POINTS_PER_DOUBLING * math.log2(total / best))
+
+
+def index(rows, models=None):
+    """{"questions": n shared, "models": [{model, P, T, A, tokens, total_s, passed, answers, alongside,
+    fewest, fastest}],
+    "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
+    most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
+    with enough answers; models named here are compared whatever their count)."""
+    no_answers = sorted({r["model"] for r in rows} - {r["model"] for r in rows if not r.get("error")})
+    rows = [r for r in rows if not r.get("error")]     # a server or connection error isn't an answer
+    tests = compare.per_question(rows)
+    counts = {}
+    for t in tests:
+        for m in t["cells"]:
+            counts[m] = counts.get(m, 0) + 1
+    most = max(counts.values(), default=0)
+    if models:
+        no_answers = [m for m in no_answers if m in models]
+        models, left_out = sorted(m for m in models if m in counts), []
+    else:
+        models = sorted(m for m in counts if counts[m] >= MIN_SHARE * most)
+        left_out = sorted(((m, n) for m, n in counts.items() if m not in models), key=lambda x: (-x[1], x[0]))
+    shared = [t for t in tests if models and all(m in t["cells"] for m in models)]
+    out = []
+    for m in models:
+        cells = [t["cells"][m] for t in shared]
+        answers = sum(c["n"] for c in cells)
+        out.append({"model": m, "tokens": sum(c["tokens"] or 0 for c in cells),
+                    "total_s": sum(c["secs"] or 0 for c in cells),
+                    "passed": sum(c["passed"] for c in cells), "answers": answers,
+                    "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    # answers that ran while other models were served shared the machine's speed
+                    "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
+    fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
+    fewest = min((x["tokens"] for x in out if x["tokens"] > 0), default=None)
+    for x in out:
+        x["P"], x["T"] = score(x["tokens"], fewest), score(x["total_s"], fastest)
+        x["fewest"] = bool(fewest) and x["tokens"] == fewest
+        x["fastest"] = bool(fastest) and x["total_s"] == fastest
+    out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
+    return {"questions": len(shared), "models": out, "left_out": left_out, "no_answers": no_answers,
+            "most": most}
+
+
+LETTERS = (("P", "parsimony", "/100", "cyan"), ("T", "speed", "/100", "magenta"), ("A", "accuracy", "%", "green"))
+
+
+def bars(result, width=20, markup=True):
+    """Lines with a row per model and a bar per score (P, T, A), longer = better, each with its number:
+    the at-a-glance view. markup=False gives plain text (the report)."""
+    if not result["models"]:
+        return []
+    name_w = max(len(x["model"]) for x in result["models"])
+    cell_w = width + 6                          # the bar, a space and "100%" or " 100"
+    head = " " * (name_w + 2) + "".join(f"{f'{k} {name}':<{cell_w}}" for k, name, _, _ in LETTERS)
+    lines = [f"[b]{head.rstrip()}[/b]" if markup else head.rstrip()]
+    for x in result["models"]:
+        cells = []
+        for k, _, unit, color in LETTERS:
+            v = x[k]
+            full = 0 if v is None else round(width * max(0, min(100, v)) / 100)
+            bar, rest = "█" * full, "░" * (width - full)
+            num = "-" if v is None else f"{v:.0f}{'%' if unit == '%' else ''}"
+            bar = f"[{color}]{bar}[/][grey37]{rest}[/]" if markup else bar + rest
+            cells.append(f"{bar} {num:>4}")
+        name = f"{x['model']:<{name_w}}"
+        lines.append(f"{rich_escape(name) if markup else name}  " + "  ".join(cells))
+    return lines
+
+
+def left_out_lines(result):
+    """Plain sentences on the models not compared (too few answers, or only server errors) and on
+    answers that shared the machine with other models."""
+    out = []
+    if result["left_out"]:
+        out.append("Left out (fewer than half as many answers as the most-answered model; pick them under "
+                   "Models or --only to include them): "
+                   + ", ".join(f"{m} ({n} of {result['most']})" for m, n in result["left_out"]))
+    if result["no_answers"]:
+        out.append("No answers, only server errors: " + ", ".join(result["no_answers"]))
+    shared = [x["model"] for x in result["models"] if x["alongside"]]
+    if shared:
+        out.append("Some answers ran alongside other models, which shared the machine's speed: " + ", ".join(shared))
+    return out
+
+
+def _score(v, unit=""):
+    """A is a percentage (%), P and T scores out of 100 (/100: on a log scale, not percentages)."""
+    return "-" if v is None else f"{v:.0f}{unit}"
+
+
+def table(result):
+    """(header, rows) for a plain table of the index."""
+    header = ["model", "P parsimony /100", "T speed /100", "A accuracy %", "tokens", "total time",
+              "answers right"]
+    rows = [[x["model"], _score(x["P"]), _score(x["T"]), _score(x["A"], "%"),
+             f"{round(x['tokens']):,}" + (" ★" if x.get("fewest") else ""),
+             fmt_total(x["total_s"]) + (" ★" if x.get("fastest") else ""),
+             f"{x['passed']}/{x['answers']}"] for x in result["models"]]
+    return header, rows
+
+
+def fmt_total(secs):
+    if not secs:
+        return "-"
+    if secs < 60:
+        return f"{secs:.1f}s"
+    secs = round(secs)
+    h, rem = divmod(secs, 3600)
+    return f"{h}h{rem // 60:02d}m" if h else f"{rem // 60}m{rem % 60:02d}s" if rem >= 60 else f"{rem}s"
+
+
+def scope_note(result, models):
+    """What the scores compare, in a sentence."""
+    if not result["questions"]:
+        return "These models have no question in common yet, so there's nothing to compare."
+    return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
+            f"A is a percentage. P parsimony and T speed are scores out of 100, not percentages: 100 = the fewest "
+            f"tokens / the fastest total time (★), {POINTS_PER_DOUBLING} points less for each doubling; a better "
+            "model added lowers the others.")
+
+
+def for_engine(e, labels=None, packs=None, results_dir=None):
+    """(result, models) from a workspace's finished results (results on their way out excluded)."""
+    paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
+    rows, _ = compare.load(paths) if paths else ([], [])
+    if packs:
+        rows = [r for r in rows if r["suite"] in packs]
+    models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
+    return index(rows, models if labels else None), models
+
+
+def markdown(result, models):
+    """The index as a markdown section (tuieval report)."""
+    lines = ["## PTA index", "", scope_note(result, models), ""]
+    if not result["questions"]:
+        return lines
+    lines += ["```"] + bars(result, markup=False) + ["```", ""]
+    header, rows = table(result)
+    lines += ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]
+    lines += ["| " + " | ".join(r) + " |" for r in rows]
+    for line in left_out_lines(result):
+        lines += ["", line]
+    return lines

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/run_evals.py

@@ -389,11 +389,11 @@ def report_markdown(e, labels=None):
 
 def cmd_pta(argv):
     p = argparse.ArgumentParser(prog="tuieval pta",
-                                description="The PTA index: privacy, time and accuracy per model (0-100 each), as a "
-                                            "triangle and a table, over the questions all the models answered.")
+                                description="The PTA index: parsimony (tokens), speed and accuracy per model (0-100, higher is better), as a "
+                                            "bar per score for each model and a table, over the questions all the models answered.")
     p.add_argument("--only", help="comma-separated model labels (default: every model with results)")
     p.add_argument("--packs", help="comma-separated packs (default: all)")
-    p.add_argument("--width", type=int, default=48, help="triangle width in characters (default 48)")
+    p.add_argument("--width", type=int, default=20, help="bar width in characters (default 20)")
     p.add_argument("--models")
     p.add_argument("--results-dir")
     a = p.parse_args(argv)
@@ -408,15 +408,20 @@ def cmd_pta(argv):
         sys.exit("no finished results to compare" + (" for those models or packs" if labels or packs else ""))
     out.print(pta.scope_note(result, models))
     if not result["questions"]:
+        for line in pta.left_out_lines(result):
+            out.print(line, markup=False)
         return
     out.print()
-    for line in pta.triangle(result, a.width) + [""] + pta.legend(result):
+    for line in pta.bars(result, a.width):
         out.print(line)
     header, rows = pta.table(result)
     widths = [max(len(h), *(len(r[i]) for r in rows)) for i, h in enumerate(header)]
     out.print()
     for line in [header] + rows:
         out.print("  ".join(c.ljust(w) for c, w in zip(line, widths)).rstrip(), markup=False)
+    for line in pta.left_out_lines(result):
+        out.print()
+        out.print(line, markup=False)
 
 
 def cmd_report(argv):

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/src/tuieval/tui.py

@@ -10,10 +10,10 @@ Run screen:    live reasoning and answer, per-test ✓/✗, TTFT and tokens/s, p
                k skips the current model, c cancels everything, f pauses auto-scroll.
                r shows results at any time (also mid-run); n starts a new run when it ends.
                Enter on a Recent results row shows that answer in full.
-Results:       opens on Per question, which compares each model's tokens and seconds on the same
-               question. ctrl+r reloads (it also reloads when a run's pack finishes). Enter on a
-               Failures row or a Per question cell shows the answer: grading and checks, reasoning,
-               answer, question.
+Results:       opens on the PTA index (parsimony, speed and accuracy per model, as bars). Per
+               question compares each model's tokens and seconds on the same question. ctrl+r
+               reloads (it also reloads when a run's pack finishes). Enter on a Failures row or a
+               Per question cell shows the answer: grading and checks, reasoning, answer, question.
 Answer detail: [ and ] step to the previous/next answer; esc/q close.
 Anywhere:      esc/q go back (q quits only on Setup), ctrl+q quits, w lists this session's runs and
                tunes. Leaving a live run or tune keeps it going; the header shows its progress.
@@ -331,11 +331,11 @@ with the model's reasoning and answer as they stream, then recent results with t
   v  with several models at a time: stream the next one (k skips the one streaming)""",
     "ResultsScreen": """[b]Results[/b]
 
-Opens on [b]Per question[/b]. [b]Production readiness[/b] gives one verdict per model and use case,
-and what's missing when it's INCONCLUSIVE. The other tabs are the evidence:
+Opens on the [b]PTA index[/b]: parsimony (tokens), speed and accuracy per model, a bar each (0-100,
+longer = better). [b]Production readiness[/b] gives one verdict per model and use case, and what's
+missing when it's INCONCLUSIVE. The other tabs are the evidence:
   Scorecard              accuracy, critical failures and truncation per model and pack
   Speed & tokens         time and tokens per answer (★ = nothing beats it on both accuracy and time)
-  PTA index              privacy, total time and accuracy per model (0-100 each), as a triangle
   Per question           every question, model by model
   Is the difference real? whether one model is really better than another, or it's noise
   Tests that separate    the questions that tell models apart
@@ -2341,7 +2341,18 @@ class ResultsScreen(Screen):
             yield Static(id="results-info")
             if self.smoke_dir:
                 yield Button("Show this Smoke run's results", id="toggle-smoke")
-        with TabbedContent(initial="tab-per-question"):
+        with TabbedContent(initial="tab-pta"):
+            with TabPane("PTA index", id="tab-pta"):
+                with Horizontal(id="pta-controls"):
+                    yield Select([("All packs", "")], id="pta-pack", allow_blank=False, value="")
+                    yield Button("Models: all", id="pta-models")
+                yield Static("[dim]P parsimony: tokens used (reasoning and answer). T speed: total time. A accuracy: "
+                             "answers right. Longer bars are better for all three. Verdicts and critical failures "
+                             "are in Production readiness.[/dim]")
+                yield Static(id="pta-note")
+                yield Static(id="pta-bars")
+                yield DataTable(id="pta", zebra_stripes=True, cursor_type="none")
+                yield Static(id="pta-left-out")
             with TabPane("Production readiness"):
                 yield Static(id="readiness-short")
                 yield Static(self.READINESS_HELP, id="readiness-help")
@@ -2366,19 +2377,6 @@ class ResultsScreen(Screen):
                              "On another machine, times are measured there if the model ran there, otherwise "
                              "projected from each answer's token counts and that machine's tuned speeds.[/dim]")
                 yield DataTable(id="speed", zebra_stripes=True)
-            with TabPane("PTA index", id="tab-pta"):
-                with Horizontal(id="pta-controls"):
-                    yield Select([("All packs", "")], id="pta-pack", allow_blank=False, value="")
-                    yield Button("Models: all", id="pta-models")
-                yield Static("[dim]P privacy: 100 when prompts stay on machines you control (local, or a server "
-                             "marked private = true), 0 for a hosted API. T time: total time for every question "
-                             "compared, relative to the fastest model (100). A accuracy: answers right. Bigger "
-                             "triangle = better all round; verdicts and critical failures are in Production "
-                             "readiness.[/dim]")
-                yield Static(id="pta-note")
-                with Horizontal(id="pta-body"):
-                    yield Static(id="pta-triangle")
-                    yield DataTable(id="pta", zebra_stripes=True, cursor_type="none")
             with TabPane("Per question", id="tab-per-question"):
                 with Horizontal(id="pq-controls"):
                     yield Select([("All packs", "")], id="pq-pack", allow_blank=False, value="")
@@ -2586,21 +2584,22 @@ class ResultsScreen(Screen):
         present = {r["model"] for r in rows}
         chosen = {m for m in self.app.pq_models if m in present}
         self.query_one("#pta-models", Button).label = f"Models: {len(chosen)} picked" if chosen else "Models: all"
-        models = sorted(chosen or present)
         t = self.query_one("#pta", DataTable)
         t.clear(columns=True)
-        if not models:
+        if not present:
             self.query_one("#pta-note", Static).update("No finished results to compare yet.")
-            self.query_one("#pta-triangle", Static).update("")
+            self.query_one("#pta-bars", Static).update("")
+            self.query_one("#pta-left-out", Static).update("")
             return
-        result = pta.index(rows, lambda m: pta.privacy(e.cfg, m), models)
-        extra = (f"  [yellow]{len(pta.COLORS)} models are drawn at most; pick fewer with Models.[/yellow]"
-                 if len(models) > len(pta.COLORS) else "")
-        self.query_one("#pta-note", Static).update(pta.scope_note(result, models) + extra)
-        self.query_one("#pta-triangle", Static).update(
-            "\n".join(pta.triangle(result, 48) + [""] + pta.legend(result)) if result["questions"] else "")
+        # none picked: every model with enough answers (pta.index leaves out the rest, and says so)
+        result = pta.index(rows, sorted(chosen) or None)
+        self.query_one("#pta-note", Static).update(pta.scope_note(result, sorted(present)))
+        self.query_one("#pta-bars", Static).update("\n".join(pta.bars(result)) if result["questions"] else "")
+        self.query_one("#pta-left-out", Static).update(
+            "\n".join(f"[dim]{rich_escape(line)}[/dim]" for line in pta.left_out_lines(result)))
         header, table = pta.table(result)
-        t.add_columns(*header)
+        for i, h in enumerate(header):   # sized to the contents: added before the rows, a column keeps its header's width
+            t.add_column(h, width=max(len(h), *(len(r[i]) for r in table)) if table else None)
         t.add_rows(table)
 
     def fill_per_question(self):
@@ -2908,9 +2907,9 @@ class EvalsApp(App):
     #pq-controls { height: 3; }
     #pta-controls { height: 3; }
     #pta-controls Select { width: 28; margin-right: 1; }
-    #pta-body { height: auto; }
-    #pta-triangle { width: auto; padding: 1 2 0 0; }
+    #pta-bars { height: auto; padding: 1 0 0 0; }
     #pta { width: 1fr; height: auto; margin-top: 1; }
+    #pta-left-out { height: auto; padding: 1 0 0 0; }
     #pq-controls Checkbox { width: auto; }
     #pq-controls Select { width: 28; margin-right: 1; }
     #pq-models { margin-right: 1; min-width: 20; }

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/tests/mock_server.py

@@ -6,6 +6,7 @@ oracle  answers each question with its test's `reference` (or the expected tool
         correct grader passes it
 wrong   answers with the test's first `wrong` entry (or "ANSWER: 0")
 fixed   always "ANSWER: 42"
+unavailable  HTTP 404 "No endpoints found" for every chat request
 
 Streams like llama.cpp: reasoning_content then content deltas, usage and timings at the end.
 """
@@ -77,6 +78,8 @@ class Handler(BaseHTTPRequestHandler):
         if isinstance(content, list):
             content = " ".join(p.get("text", "") for p in content if isinstance(p, dict))
         test = self.answers.get(" ".join(str(content).split()))
+        if self.mode == "unavailable":   # like OpenRouter with no provider serving the model
+            return self._json({"error": {"message": f"No endpoints found for {self.model}.", "code": 404}}, 404)
         text, calls = reply_for(test, self.mode)
         self.send_response(200)
         self.send_header("Content-Type", "text/event-stream")
@@ -105,7 +108,7 @@ def main(argv=None):
     p = argparse.ArgumentParser()
     p.add_argument("--port", type=int, default=18090)
     p.add_argument("--packs", required=True)
-    p.add_argument("--mode", choices=("oracle", "wrong", "fixed"), default="oracle")
+    p.add_argument("--mode", choices=("oracle", "wrong", "fixed", "unavailable"), default="oracle")
     p.add_argument("--model", default="mock")
     p.add_argument("--delay", type=float, default=0.0, help="seconds each answer takes")
     a, _ = p.parse_known_args(argv)   # other flags (e.g. speed knobs under test) are accepted and ignored

{tuieval-0.2.0.dev42 → tuieval-0.2.0.dev50}/tests/test_tuieval.py

@@ -3,6 +3,7 @@
 Uses a temporary workspace and tests/mock_server.py on a free local port; no model or GPU needed.
 """
 import json
+import math
 import os
 import re
 import socket
@@ -198,6 +199,16 @@ class WorkspaceGraders(unittest.TestCase):
 
 
 class Units(unittest.TestCase):
+    def test_server_errors_never_judge_the_model(self):
+        from tuieval import client
+        no_endpoints = 'server returned HTTP 404: {"error":{"message":"No endpoints found for some/model."}}'
+        for reason, retry, unavailable in ((no_endpoints, False, True), ("server returned HTTP 402: no credits", False, True),
+                                           ("server returned HTTP 503: busy", True, False),
+                                           ("server returned HTTP 400: bad request", False, False),
+                                           ("connection error: timed out", False, False)):   # the model too slow
+            self.assertEqual((client.is_server_error(reason), client.is_unavailable(reason)), (retry, unavailable), reason)
+            self.assertEqual(client.server_error_row({"pass": False, "reason": reason}), retry or unavailable, reason)
+
     def test_tune_workload_needs_no_packs(self):
         from tuieval import tune
         work = tune.workload(None, 65536)
@@ -692,40 +703,59 @@ class AddToRun(unittest.TestCase):
 
 
 class PTAIndex(unittest.TestCase):
-    """The PTA index: privacy from where a model runs, time relative to the fastest total, accuracy,
-    all over the questions every compared model answered."""
-
-    def test_privacy(self):
-        from tuieval import pta
-        cfg = {"servers": {"llama": {"cmd": ["llama-server"]}, "lm": {"url": "http://localhost:1234/v1"},
-                           "box": {"url": "http://192.168.1.20:8000/v1"}, "mine": {"url": "http://192.168.1.20:8000/v1",
-                                                                                   "private": True},
-                           "or": {"url": "https://openrouter.ai/api/v1"}},
-               "models": [{"label": l, "server": l} for l in ("llama", "lm", "box", "mine", "or")]}
-        self.assertEqual([pta.privacy(cfg, l) for l in ("llama", "lm", "box", "mine", "or", "gone")],
-                         [100, 100, 0, 100, 0, None])
+    """The PTA index: parsimony from tokens and speed from time, each relative to the best on a log
+    scale, and accuracy, all over the questions every compared model answered."""
 
     def test_index_compares_shared_questions(self):
         from tuieval import pta
 
         def row(model, test, ok, secs, repeat=0):
             return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": secs * 1000,
-                    "tokens": 10, "truncated": False, "gen_tps": None, "repeat": repeat}
+                    "tokens": 10 if model == "slow" else 40, "truncated": False, "gen_tps": None, "repeat": repeat}
         rows = [row("fast", "q1", True, 2), row("fast", "q2", False, 2),
                 row("slow", "q1", True, 8), row("slow", "q2", True, 4), row("slow", "q2", True, 6, repeat=1),
                 row("slow", "q3", True, 100)]               # only slow answered q3: not compared
-        res = pta.index(rows, lambda m: 100 if m == "slow" else 0)
+        res = pta.index(rows)
         self.assertEqual(res["questions"], 2)
         by = {x["model"]: x for x in res["models"]}
         self.assertEqual(by["fast"]["total_s"], 4)
         self.assertEqual(by["slow"]["total_s"], 13)      # q1 8 + q2 median of 4 and 6
         self.assertEqual(by["fast"]["T"], 100)
-        self.assertAlmostEqual(by["slow"]["T"], 100 * 4 / 13)
+        self.assertAlmostEqual(by["slow"]["T"], 100 - 15 * math.log2(13 / 4))   # 15 points per doubling
+        self.assertEqual((by["slow"]["tokens"], by["fast"]["tokens"]), (20, 80))   # medians over repeats
+        self.assertEqual((by["slow"]["P"], by["fast"]["P"]), (100, 70))          # 4x the tokens: two doublings
+        self.assertEqual((pta.score(2, 1), pta.score(1000, 1)), (85, 0))
+        header, table = pta.table(res)
+        self.assertEqual(header[1:4], ["P parsimony /100", "T speed /100", "A accuracy %"])
+        cells = {r[0]: r for r in table}
+        self.assertEqual(cells["fast"][1:4], ["70", "100", "50%"])
+        self.assertEqual((cells["slow"][4], cells["fast"][5][-1]), ("20 ★", "★"))   # fewest tokens, fastest
+        bars = pta.bars(res, 10, markup=False)
+        self.assertEqual(bars[0].split(), ["P", "parsimony", "T", "speed", "A", "accuracy"])
+        self.assertEqual(bars[2], "fast  " + "███████░░░   70  " + "██████████  100  " + "█████░░░░░  50%")
         self.assertEqual((by["fast"]["A"], by["slow"]["A"]), (50, 100))
         self.assertEqual([x["model"] for x in res["models"]], ["slow", "fast"])   # best accuracy first
-        lines = pta.triangle(res, 40)
-        self.assertIn("P[/b] privacy", lines[0])
-        self.assertTrue(any("[cyan]" in l for l in lines) and any("[magenta]" in l for l in lines))
+
+    def test_server_errors_and_models_with_few_answers(self):
+        from tuieval import pta
+
+        def row(model, test, ok=True, error=False):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "error": error,
+                    "latency": 1000, "tokens": 10, "truncated": False, "gen_tps": None, "repeat": 0}
+        rows = ([row(m, f"q{i}") for m in ("a", "b") for i in range(10)] + [row("few", "q0", ok=False)]
+                + [row("down", f"q{i}", ok=False, error=True) for i in range(10)]
+                + [row("b", "q10", ok=False, error=True)])      # an error among real answers: not counted
+        res = pta.index(rows)
+        self.assertEqual([x["model"] for x in res["models"]], ["a", "b"])
+        self.assertEqual(res["questions"], 10)                  # not shrunk to the one "few" answered
+        self.assertEqual(res["left_out"], [("few", 1)])
+        self.assertEqual(res["no_answers"], ["down"])
+        self.assertEqual([x["answers"] for x in res["models"]], [10, 10])
+        legend = "\n".join(pta.left_out_lines(res))
+        self.assertIn("few (1 of 10)", legend)
+        self.assertIn("only server errors: down", legend)
+        picked = pta.index(rows, ["a", "few"])      # picked by hand: compared anyway
+        self.assertEqual((picked["questions"], picked["left_out"]), (1, []))
 
     def test_cli_report_and_tui(self):
         with tempfile.TemporaryDirectory() as tmp:
@@ -749,16 +779,14 @@ class PTAIndex(unittest.TestCase):
             finally:
                 right.stop()
                 wrong.stop()
-            path = os.path.join(ws, "models.toml")    # as if "there" were a hosted API
-            write(path, read(path).replace(f"http://127.0.0.1:{wrong.port}", "https://api.example.com/v1"))
             out = tuieval(ws, "pta").stdout
             self.assertIn("Compared on the 4 questions all 2 model(s) answered", out)
             table = {l.split()[0]: l.split()[1:4] for l in out.splitlines() if l.startswith(("good ", "hosted "))}
-            self.assertEqual(table["good"], ["100", "100", "100"])
-            self.assertEqual(table["hosted"][0], "0")
+            self.assertEqual(table["good"], ["100", "100", "100%"])
+            self.assertEqual(table["hosted"][0], "100")         # the mock says as much either way
             self.assertLess(int(table["hosted"][1]), 100)   # slower in total
-            self.assertEqual(table["hosted"][2], "0")
-            self.assertIn("P privacy", out)
+            self.assertEqual(table["hosted"][2], "0%")
+            self.assertIn("P parsimony", out)
             self.assertIn("4/4", tuieval(ws, "pta", "--only", "good").stdout)
             report = os.path.join(tmp, "r.md")
             tuieval(ws, "report", "-o", report)
@@ -766,7 +794,7 @@ class PTAIndex(unittest.TestCase):
             code = textwrap.dedent("""
                 import asyncio, json
                 from tuieval.tui import EvalsApp, ResultsScreen
-                from textual.widgets import DataTable, Static
+                from textual.widgets import DataTable, Static, TabbedContent
                 async def go():
                     app = EvalsApp({})
                     async with app.run_test(size=(180, 60)) as pilot:
@@ -775,16 +803,53 @@ class PTAIndex(unittest.TestCase):
                         await pilot.pause(0.5)
                         t = app.screen.query_one("#pta", DataTable)
                         rows = [[str(c) for c in t.get_row_at(i)] for i in range(t.row_count)]
-                        tri = str(app.screen.query_one("#pta-triangle", Static).render())
-                        print(json.dumps({"rows": rows, "triangle": "P privacy" in tri}))
+                        tri = str(app.screen.query_one("#pta-bars", Static).render())
+                        print(json.dumps({"rows": rows, "bars": "P parsimony" in tri,
+                                          "tab": app.screen.query_one(TabbedContent).active}))
                 asyncio.run(go())
             """)
             p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=300,
                                env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""))
             got = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
             self.assertIsInstance(got, dict, got)
-            self.assertTrue(got["triangle"])
-            self.assertEqual([r[:4] for r in got["rows"]][0], ["good", "100", "100", "100"])
+            self.assertTrue(got["bars"])
+            self.assertEqual(got["tab"], "tab-pta")                  # Results opens on the PTA index
+            self.assertEqual([r[:4] for r in got["rows"]][0], ["good", "100", "100", "100%"])
+
+
+class Unavailable(unittest.TestCase):
+    """A model no server will serve (OpenRouter's 404 "No endpoints found", a bad key, no credits) stops at
+    once and is never judged on it; the other models run as usual."""
+
+    def test_unavailable_model_stops_and_isnt_judged(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            d = os.path.join(ws, "packs", "apps")
+            os.makedirs(d)
+            write(os.path.join(d, "pack.toml"), 'label = "apps"\n[certify]\nrepeat = 1\n[gate]\nmin_accuracy = 0.1\n')
+            write(os.path.join(d, "tests.yaml"), "".join(
+                f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', difficulty: easy}}\n"
+                for t in "abc"))
+            up, down = Mock(os.path.join(ws, "packs"), "oracle"), Mock(os.path.join(ws, "packs"), "unavailable")
+            try:
+                with open(os.path.join(ws, "models.toml"), "a") as f:
+                    f.write(f'\n[servers.up]\nurl = "http://127.0.0.1:{up.port}"\n'
+                            f'\n[servers.down]\nurl = "http://127.0.0.1:{down.port}"\n')
+                tuieval(ws, "add", "mock", "--server", "up", "--label", "works")
+                tuieval(ws, "add", "mock", "--server", "down", "--label", "gone")
+                t = time.time()
+                out = tuieval(ws, "run", "--tier", "certify", check=False).stdout
+            finally:
+                up.stop()
+                down.stop()
+            self.assertLess(time.time() - t, 60)                      # no retrying for minutes
+            self.assertIn("isn't available there", out)
+            verdict = tuieval(ws, "verdict", check=False).stdout
+            self.assertNotIn("0% right", verdict)                       # never judged on the 404s
+            rows = [l for l in verdict.splitlines() if "gone" in l]
+            self.assertFalse(any("FAIL" in l for l in rows), rows)
+            self.assertIn("works", tuieval(ws, "pta").stdout)
 
 
 class FirstRun(unittest.TestCase):
@@ -844,7 +909,7 @@ class FirstRun(unittest.TestCase):
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60).stdout.strip()
         self.assertEqual(out, "['first'] ['my-model'] smoke False")   # tuning shows once something has run
 
-    def test_results_open_on_per_question(self):
+    def test_results_open_on_pta_index(self):
         tuieval(self.ws, "add", "my-model", "--server", "local")
         code = textwrap.dedent("""
             import asyncio
@@ -861,7 +926,7 @@ class FirstRun(unittest.TestCase):
         """)
         out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60).stdout.strip()
-        self.assertEqual(out, "ResultsScreen tab-per-question")
+        self.assertEqual(out, "ResultsScreen tab-pta")
 
     def test_stream_follows_only_at_the_bottom(self):
         code = textwrap.dedent("""

tuieval-0.2.0.dev42/src/tuieval/pta.py

@@ -1,185 +0,0 @@
-"""PTA index: privacy, time and accuracy per model, 0-100 each, drawn as a radar triangle.
-
-    P  privacy   100 when prompts stay on machines you control: tuieval starts the server, its URL
-                 is this machine, or the server is marked private = true in models.toml. 0 otherwise
-                 (a hosted API: there's no partial privacy once prompts leave).
-    T  time      the total time to answer every question compared (each question at its median over
-                 repeats), relative to the fastest model: 100 x fastest total / this model's total.
-    A  accuracy  the share of answers to those questions that passed.
-
-Models are compared on the questions all of them answered, so one that skipped some never looks
-faster. There's no combined number: the three stay side by side, and verdicts and critical
-failures stay where they are (verdict.py).
-"""
-import urllib.parse
-
-from . import compare
-
-LOCAL_HOSTS = {"localhost", "127.0.0.1", "::1", "0.0.0.0"}
-# one colour per model on the triangle; more models than colours are listed but not drawn
-COLORS = ["cyan", "magenta", "yellow", "green", "red", "blue", "bright_white", "bright_cyan"]
-
-
-def privacy(cfg, label):
-    """100 if a model's prompts stay on machines you control, else 0 (see the module docstring);
-    None for a model that's no longer in models.toml (where it ran isn't known)."""
-    m = next((m for m in cfg["models"] if m["label"] == label), None)
-    server = cfg["servers"].get(m["server"], {}) if m else {}
-    if not server:
-        return None
-    if server.get("cmd") or server.get("private"):
-        return 100
-    host = urllib.parse.urlparse(server.get("url", "")).hostname or ""
-    return 100 if host.lower() in LOCAL_HOSTS else 0
-
-
-def index(rows, privacy_of, models=None):
-    """{"questions": n shared, "models": [{model, P, T, A, total_s, passed, answers, alongside}]},
-    best accuracy first. models: the ones to compare (default: every model in rows)."""
-    tests = compare.per_question(rows)
-    models = sorted(models or {m for t in tests for m in t["cells"]})
-    shared = [t for t in tests if models and all(m in t["cells"] for m in models)]
-    out = []
-    for m in models:
-        cells = [t["cells"][m] for t in shared]
-        answers = sum(c["n"] for c in cells)
-        out.append({"model": m, "P": privacy_of(m), "total_s": sum(c["secs"] or 0 for c in cells),
-                    "passed": sum(c["passed"] for c in cells), "answers": answers,
-                    "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
-                    # answers that ran while other models were served shared the machine's speed
-                    "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
-    fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
-    for x in out:
-        x["T"] = 100 * fastest / x["total_s"] if fastest and x["total_s"] > 0 else None
-    out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
-    return {"questions": len(shared), "models": out}
-
-
-# ---------------------------------------------------------------- the triangle (braille, no dependencies)
-_DOTS = ((0x01, 0x08), (0x02, 0x10), (0x04, 0x20), (0x40, 0x80))   # braille dot bits by (row, column)
-
-
-class _Canvas:
-    """Braille cells, 2 dots wide and 4 tall each; every cell takes the colour of the last line through it."""
-    def __init__(self, cols, rows):
-        self.cols, self.rows = cols, rows
-        self.bits = [[0] * cols for _ in range(rows)]
-        self.color = [[None] * cols for _ in range(rows)]
-
-    def dot(self, x, y, color):
-        cx, cy = int(x) // 2, int(y) // 4
-        if 0 <= cx < self.cols and 0 <= cy < self.rows:
-            self.bits[cy][cx] |= _DOTS[int(y) % 4][int(x) % 2]
-            self.color[cy][cx] = color
-
-    def line(self, a, b, color, dotted=False):
-        steps = int(max(abs(b[0] - a[0]), abs(b[1] - a[1]))) or 1
-        for i in range(steps + 1):
-            if dotted and i % 3:
-                continue
-            self.dot(a[0] + (b[0] - a[0]) * i / steps, a[1] + (b[1] - a[1]) * i / steps, color)
-
-    def text(self):
-        lines = []
-        for bits, colors in zip(self.bits, self.color):
-            out = []
-            for b, c in zip(bits, colors):
-                ch = chr(0x2800 + b) if b else " "
-                out.append(f"[{c}]{ch}[/]" if b and c else ch)
-            lines.append("".join(out).rstrip())
-        return lines
-
-
-def triangle(result, width=60):
-    """The radar triangle as Rich markup lines: P at the top, T bottom left, A bottom right; a
-    model's triangle reaches each corner as far as its score (bigger = better all round)."""
-    cols = max(20, width)
-    w = cols * 2 - 2                            # dots across
-    h = int(w * 0.866)                          # an equilateral triangle (braille dots are about square)
-    rows = h // 4 + 1
-    top, left, right = (w / 2, 0), (0, h), (w, h)
-    center = (w / 2, h * 2 / 3)
-    c = _Canvas(cols, rows)
-
-    def at(corner, score):
-        return (center[0] + (corner[0] - center[0]) * score / 100, center[1] + (corner[1] - center[1]) * score / 100)
-
-    for corner in (top, left, right):           # axes, the 50% ring, then the frame
-        c.line(center, corner, "grey37", dotted=True)
-    half = [at(k, 50) for k in (top, left, right)]
-    for a, b in zip(half, half[1:] + half[:1]):
-        c.line(a, b, "grey37", dotted=True)
-    for a, b in ((top, left), (left, right), (right, top)):
-        c.line(a, b, "grey70")
-    for i, x in enumerate(result["models"][:len(COLORS)][::-1]):   # the best ends on top
-        color = COLORS[len(result["models"][:len(COLORS)]) - 1 - i]
-        pts = [at(k, s or 0) for k, s in ((top, x["P"]), (left, x["T"]), (right, x["A"]))]
-        for a, b in zip(pts, pts[1:] + pts[:1]):
-            c.line(a, b, color)
-    lines = c.text()
-    pad = " " * max(0, cols // 2 - 5)
-    return ([f"{pad}[b]P[/b] privacy"] + lines
-            + [f"[b]T[/b] time{' ' * max(1, cols - 15)}[b]A[/b] accuracy"])
-
-
-def legend(result):
-    """One Rich markup line per model: its colour on the triangle and its three scores."""
-    out = []
-    for i, x in enumerate(result["models"]):
-        mark = f"[{COLORS[i]}]■[/]" if i < len(COLORS) else " "
-        out.append(f"{mark} {x['model']}  P {_score(x['P'])} · T {_score(x['T'])} · A {_score(x['A'])}"
-                   + ("  [yellow](some answers ran alongside other models)[/yellow]" if x["alongside"] else ""))
-    return out
-
-
-def _score(v):
-    return "-" if v is None else f"{v:.0f}"
-
-
-def table(result):
-    """(header, rows) for a plain table of the index."""
-    header = ["model", "P privacy", "T time", "A accuracy", "total time", "answers right"]
-    rows = [[x["model"], _score(x["P"]), _score(x["T"]), _score(x["A"]), fmt_total(x["total_s"]),
-             f"{x['passed']}/{x['answers']}"] for x in result["models"]]
-    return header, rows
-
-
-def fmt_total(secs):
-    if not secs:
-        return "-"
-    if secs < 60:
-        return f"{secs:.1f}s"
-    secs = round(secs)
-    h, rem = divmod(secs, 3600)
-    return f"{h}h{rem // 60:02d}m" if h else f"{rem // 60}m{rem % 60:02d}s" if rem >= 60 else f"{rem}s"
-
-
-def scope_note(result, models):
-    """What the scores compare, in a sentence."""
-    if not result["questions"]:
-        return "These models have no question in common yet, so there's nothing to compare."
-    return (f"Compared on the {result['questions']} questions all {len(models)} model(s) answered. "
-            "T is relative to the fastest total, so adding a faster model lowers the others.")
-
-
-def for_engine(e, labels=None, packs=None, results_dir=None):
-    """(result, models) from a workspace's finished results (results on their way out excluded)."""
-    paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
-    rows, _ = compare.load(paths) if paths else ([], [])
-    if packs:
-        rows = [r for r in rows if r["suite"] in packs]
-    models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
-    return index(rows, lambda m: privacy(e.cfg, m), models), models
-
-
-def markdown(result, models):
-    """The index as a markdown section (tuieval report)."""
-    lines = ["## PTA index", "", scope_note(result, models), ""]
-    if not result["questions"]:
-        return lines
-    header, rows = table(result)
-    lines += ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]
-    lines += ["| " + " | ".join(r) + " |" for r in rows]
-    if any(x["alongside"] for x in result["models"]):
-        lines += ["", "Some answers ran alongside other models, which shared the machine's speed."]
-    return lines