tuieval 0.2.0.dev129__tar.gz → 0.2.0.dev133__tar.gz

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
+- Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).
 - Agent packs (`kind = "agent"`): each test is a small project with a task, worked on by a command-line agent you define in `[agents]`, using the model tuieval serves, and graded by the task's visible and hidden tests. Results show the agent's stats, diff and transcript; `tuieval compare` adds an Agent tasks table; `tuieval selftest` checks each task's reference and wrong fixes. See docs/agent-packs.md.

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev129
+Version: 0.2.0.dev133
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/docs/models.md

@@ -105,6 +105,8 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
+- **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
@@ -118,7 +120,7 @@ Results → **Insights** (also `tuieval insights`) turns the results into charts
 1. **Who's worth keeping**: every model against a reference model, paired on the questions both answered: the difference in accuracy points with its 95% CI, and better, tie or worse. Pick the reference in the tab or with `--reference`; set a default with `reference = "<label>"` under `[defaults]` in `models.toml`. Without one, the most accurate model is the reference.
 2. **Best model per use case**: a heatmap of accuracy per model and use case (pack group); ★ marks the best among models that answered most of it.
 3. **Fine-tunes against their base model**: for each model with `base = "<label>"` in its `[[models]]` block, the difference per pack and the question categories (a test's `category`) it gained or lost most.
-4. **Why models fail**: each model's answers split into right, wrong, cut off at max_tokens, no answer or bad format, code that crashed or hung, and critical failures.
+4. **Why models fail**: each model's answers split into right, near miss (a score of 80% or more, e.g. 5 of 6 checks), partly right, wrong (a score of 0), cut off at max_tokens or the time limit, no answer or bad format, code that crashed or hung, and critical failures. The Failures tab shows each failure's kind (with its checks passed) and filters by it.
 5. **Accuracy against cost**: accuracy against median tokens (or seconds) per question, over the questions every model answered; green models are the ones no other model beats on both. With hosted and local models together, the best local trade-offs get their own line.
 6. **Question health**: questions every model fails (check them), every model passes (they don't separate models), and the ones that split models most.
 7. **Next runs**: the Certify runs that would settle the open verdicts of the strongest models, with their time and command.

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev129'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev129')
+__version__ = version = '0.2.0.dev133'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev133')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/compare.py

@@ -248,12 +248,14 @@ def scorecard(rows, rng=None):
     suites = sorted({r["suite"] for r in rows})
     extra = dict(graders.COLUMNS)   # columns your graders add (graders.scorecard_column)
     over = any(r.get("over_budget") for r in rows)   # a column only when some answer ran past max_tokens
-    header = ["model"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
+    header = ["model", "repeats"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
                                    *(["over budget"] if over else []), "rep"]
     table = []
     for m in models:
         mine = [r for r in rows if r["model"] == m]
-        line = [m]
+        per_test = collections.Counter(r["test"] for r in mine)   # answers per question: the repeats
+        few, most = min(per_test.values()), max(per_test.values())
+        line = [m, f"×{few}" if few == most else f"×{few}-{most}"]
         for s in suites:
             sub = [r for r in mine if r["suite"] == s]
             line.append(f"{sum(r['ok'] for r in sub)}/{len(sub)}" if sub else "-")
@@ -351,10 +353,22 @@ def speed(rows, infos, machine=None, speeds=None):
     return header, table, frontier
 
 
+NEAR_MISS = 0.8   # a failed answer scoring at least this (e.g. 5 of 6 checks) is a near miss
+
+
+def checks_passed(r):
+    """(passed, total) checks of a failed code answer (from its stored checks, or its reason), else None."""
+    checks = (r.get("raw") or {}).get("checks")
+    if checks:
+        return sum(v is None for v in checks.values()), len(checks)
+    m = re.match(r"Failed (\d+)/(\d+) checks", r.get("reason") or "")
+    return (int(m[2]) - int(m[1]), int(m[2])) if m else None
+
+
 def per_question(rows):
     """[{test, name, suite, difficulty, cells: {model: cell}, spread}] with one cell per model that
-    answered: n answers, passes, median tokens / seconds / tok/s over repeats, truncated count,
-    partial (some answers from a pack still running), rows (the answers, for the detail view).
+    answered: n answers, passes, mean score (partial credit), median tokens / seconds / tok/s over repeats,
+    truncated count, partial (some answers from a pack still running), rows (the answers, for the detail view).
     spread = {"tokens": .., "secs": ..}: largest / smallest model median (1 if one model)."""
     tests = {}
     for r in rows:
@@ -369,6 +383,7 @@ def per_question(rows):
             c.update(rows=v, n=len(v), passed=sum(r["ok"] for r in v), truncated=sum(r["truncated"] for r in v),
                      over=sum(bool(r.get("over_budget")) for r in v),
                      partial=any(r.get("partial") for r in v),
+                     score=statistics.fmean(r.get("score", r["ok"]) for r in v),
                      tokens=median(r["tokens"] or None for r in v), secs=median(r["latency"] / 1000 or None for r in v),
                      tps=median(r["gen_tps"] for r in v))
         t["spread"] = _spread(t["cells"])
@@ -404,7 +419,7 @@ def complete_models(rows, suite, test_ids=None):
 def compare_models(tests, models):
     """Per model over per_question entries that all `models` answered: passes, token and time
     totals/medians (totals add each question's median over repeats, so repeats don't inflate them), questions where it used the fewest tokens / was fastest (among passing
-    answers for tokens; ties count for each), answers cut off."""
+    answers for tokens; ties count for each), answers cut off, mean score (partial credit)."""
     out = []
     for m in models:
         cs = [t["cells"][m] for t in tests]
@@ -421,6 +436,7 @@ def compare_models(tests, models):
                 fastest += 1
         rs = [r for c in cs for r in c["rows"]]
         out.append({"model": m, "n": len(cs), "passed": passed, "total_tokens": total,
+                    "score": statistics.fmean(c["score"] for c in cs) if cs else None,
                     "avg_pp_tps": _rate([(r.get("prompt_tokens"), r["prompt_tps"]) for r in rs]),
                     "avg_gen_tps": _rate([(r["tokens"], r["gen_tps"]) for r in rs]),
                     "med_tokens": median(c["tokens"] for c in cs), "med_secs": median(c["secs"] for c in cs),
@@ -664,7 +680,8 @@ def main(argv):
     for note in settings_notes(infos):
         print("NOTE:", note)
     print_table(*scorecard(rows, rng))
-    print("\nmean score = average score, with partial credit; all repeats = tests passed on every repeat.")
+    print("\nrepeats = answers per question (×1: one answer each, the roughest accuracy).")
+    print("mean score = average score, with partial credit; all repeats = tests passed on every repeat.")
     print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern.")
     header, table = by_difficulty(rows)
     if any(r.get("difficulty") in LEVELS for r in rows):

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/engine.py

@@ -989,6 +989,14 @@ class Engine:
         need = {(t["id"], r) for t in pack.tests for r in range(self.certify_repeat(pack))}
         return "certified" if need <= keys else "screened"
 
+    def result_repeats(self, label, pack_name, tier="certify"):
+        """How many times each answered question of a stored result was asked: (fewest, most), or None."""
+        data = self._result(self.result_path(label, pack_name, tier))
+        if not data or not data["results"]:
+            return None
+        per_test = collections.Counter(r["test"] for r in data["results"])
+        return min(per_test.values()), max(per_test.values())
+
     def _partial_lines(self, path):
         """(header, answers) of an unfinished run's .partial.jsonl, cached like _result;
         (None, []) if there is none."""

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/insights.py

@@ -6,8 +6,9 @@
   2 use cases       accuracy per model and use case (pack group) as a heatmap; the best marked
   3 vs base         a fine-tune against its base model (models.toml `base`): per pack, and the question
                     categories it gained or lost most
-  4 why they fail   each model's answers split into right, wrong, cut off, no answer / bad format, code
-                    crashed or hung, and critical failures
+  4 why they fail   each model's answers split into right, near miss (most checks passed), partly right,
+                    wrong, cut off (or out of time), no answer / bad format, code crashed or hung, and
+                    critical failures
   5 accuracy / cost accuracy against median tokens (or seconds) per question; the models no other beats
                     on both
   6 questions       questions every model fails (check them), every model passes (no signal), and the
@@ -267,7 +268,8 @@ def categories(rows, m, base, min_q=2):
 
 
 # ---------------------------------------------------------------- 4 why they fail
-KINDS = (("right", "green", "█"), ("wrong", "red", "█"), ("cut off", "yellow", "█"),
+KINDS = (("right", "green", "█"), ("near miss", "#87af5f", "█"), ("partly right", "#af875f", "█"),
+         ("wrong", "red", "█"), ("cut off", "yellow", "█"),
          ("no answer / format", "magenta", "█"), ("code crashed or hung", "cyan", "█"), ("critical", "#ff5faf", "▓"))
 FORMAT = ("No 'ANSWER:' line", "Empty answer", "No code", "no tool call")
 CRASHED = ("Timed out after", "Harness crashed", "grader error")
@@ -278,13 +280,15 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET")):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"
     if r["reason"].startswith(CRASHED):
         return "code crashed or hung"
-    return "wrong"
+    # partial credit (e.g. checks passed): close isn't the same as all wrong
+    score = r.get("score") or 0
+    return "near miss" if score >= compare.NEAR_MISS else "partly right" if score > 0 else "wrong"
 
 
 def _split(counts, total, width):
@@ -322,7 +326,9 @@ def failure_mix(rows, models, width=36):
             top, n = max(((k, c[k]) for k, *_ in KINDS[1:-1]), key=lambda x: x[1])
             if n / fails >= 0.4 and top != "wrong":
                 actions.append(f"{m}: {n} of its {fails} failures are {top}: " + {
-                    "cut off": "it thinks too long for max_tokens, rather than reasoning wrong.",
+                    "cut off": "it thinks too long for max_tokens (or the time limit), rather than reasoning wrong.",
+                    "near miss": "they pass most of their checks: it's closer than its accuracy shows.",
+                    "partly right": "they pass some of their checks, not none.",
                     "no answer / format": "it doesn't follow the answer format (check its chat template and the "
                                           "pack's instructions).",
                     "code crashed or hung": "its code crashes or never finishes."}[top])

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/pta.py

@@ -7,11 +7,17 @@
                            model's ("28×").
     A  accuracy            the share of answers to those questions that passed (%).
 
+Beside A, the score: partial credit averaged over the same answers (e.g. a code answer passing 5 of 6
+checks scores 83%, where A counts it wrong). Shown when it differs from A; models can be ordered by it.
+
 The P and T bars are as long as the tokens and time used (the model that used the most fills its bar,
 the others in proportion, at least one block), so shorter is better; the A bar: longer is better.
 P and T also carry a 0-100 score (100 for the best, POINTS_PER_DOUBLING points less for every
 doubling), used to order models with the same accuracy.
 
+Beside each name, the repeats: how many times it answered each of those questions ("×3", or "×1-3" when
+it varies). A ×1 model's accuracy rests on one answer per question, so it is rougher than a ×3 one's.
+
 Models are compared on the questions all of them answered, so one that skipped some never looks
 faster. Server and connection errors aren't answers. By default a model with fewer than half as many
 answers as the most-answered model is left out (listed, so it can be picked), or it would shrink the
@@ -46,11 +52,15 @@ def memory_gb(infos):
     return out
 
 
-def index(rows, models=None, memory=None):
-    """{"questions": n shared, "models": [{model, P, T, A, cut, cut_pct, tokens, total_s, passed, answers, alongside,
-    fewest, fastest, tps, ttft, memory_gb}],
+SORTS = ("accuracy", "score")
+
+
+def index(rows, models=None, memory=None, sort="accuracy"):
+    """{"questions": n shared, "models": [{model, P, T, A, S, cut, cut_pct, tokens, total_s, passed, answers, alongside,
+    fewest, fastest, tps, ttft, memory_gb, repeats: (fewest, most) answers per question}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
-    most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
+    most-answered model}, best accuracy (sort="accuracy") or score (sort="score") first. S: the mean
+score (partial credit) in %. models: the ones to compare (default: every model in rows
     with enough answers; models named here are compared whatever their count). memory: memory_gb()."""
     no_answers = sorted({r["model"] for r in rows} - {r["model"] for r in rows if not r.get("error")})
     rows = [r for r in rows if not r.get("error")]     # a server or connection error isn't an answer
@@ -75,6 +85,7 @@ def index(rows, models=None, memory=None):
                     "total_s": sum(c["secs"] or 0 for c in cells),
                     "passed": sum(c["passed"] for c in cells), "answers": answers,
                     "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    "S": 100 * sum(c["score"] * c["n"] for c in cells) / answers if answers else None,
                     # failed only because max_tokens ran out (overthinking, not wrong yet)
                     "cut": sum(c["truncated"] for c in cells),
                     "cut_pct": 100 * sum(c["truncated"] for c in cells) / answers if answers else None,
@@ -84,6 +95,7 @@ def index(rows, models=None, memory=None):
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
                     "memory_gb": (memory or {}).get(m),
+                    "repeats": (min(c["n"] for c in cells), max(c["n"] for c in cells)) if cells else None,
                     # answers that ran while other models were served shared the machine's speed
                     "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
     fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
@@ -95,7 +107,8 @@ def index(rows, models=None, memory=None):
         x["time_x"] = x["total_s"] / fastest if fastest and x["total_s"] else None
         x["fewest"] = bool(fewest) and x["tokens"] == fewest
         x["fastest"] = bool(fastest) and x["total_s"] == fastest
-    out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
+    first, then = ("S", "A") if sort == "score" else ("A", "S")
+    out.sort(key=lambda x: (-(x[first] or 0), -(x[then] or 0), -(x["T"] or 0), x["model"]))
     return {"questions": len(shared), "models": out, "left_out": left_out, "no_answers": no_answers,
             "most": most}
 
@@ -107,6 +120,14 @@ LETTERS = (("P", "parsimony (tokens)", "▼ shorter is better", "tokens", "token
            ("A", "accuracy", "▲ longer is better", "A", "A", "green"))
 
 
+def repeats(x):
+    """How many times a model answered each question compared: "×3", "×1-3", or "" when unknown."""
+    r = x.get("repeats")
+    if not r:
+        return ""
+    return f"×{r[0]}" if r[0] == r[1] else f"×{r[0]}-{r[1]}"
+
+
 def times(v):
     """A multiple of the best: "1.0×", "2.4×", "28×"."""
     return "-" if v is None else f"{v:.1f}×" if v < 9.95 else f"{v:.0f}×"
@@ -129,6 +150,8 @@ def bars(result, width=20, markup=True, click=None):
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
+    rep_w = max(len(repeats(x)) for x in result["models"])
+    name_w += rep_w + 1 if rep_w else 0   # the repeats sit after the name, in its column
     # each column: the bar, a space, "100%" or "9.8×" (6), then a gap; at least as wide as its heading
     cell_w = max(width + 9, *(len(t) + 2 for k, name, better, *_ in LETTERS for t in (f"{k} {name}", better)))
     pad = " " * (name_w + 2)
@@ -145,6 +168,10 @@ def bars(result, width=20, markup=True, click=None):
                 if v is not None and v < 100:   # full only when every answer passed
                     full = min(full, width - 1)
                 num = "-" if v is None else compare.pct(v / 100)
+                score = x.get("S")
+                if score is not None and v is not None and compare.pct(score / 100) != num:
+                    note = f"score {compare.pct(score / 100)}"
+                    num = f"{num:>6}  " + (f"[dim]{note}[/dim]" if markup else note)
                 cut = x.get("cut_pct") or 0
                 part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
                 if cut:   # after the accuracy, which stays aligned with the other rows'
@@ -160,11 +187,14 @@ def bars(result, width=20, markup=True, click=None):
             bar = (f"[{color}]{bar}[/]" + (f"[#ff8c00]{'█' * part}[/]" if part else "") + f"[grey37]{rest}[/]"
                    if markup else bar + cut_bar + rest)
             cells.append(f"{bar} {num:>6}" + " " * max(0, cell_w - width - 1 - max(6, len(num))))
-        name = f"{x['model']:<{name_w}}"
+        tag = repeats(x)
+        name = f"{x['model'] + (' ' + tag if tag else ''):<{name_w}}"
         if markup:
-            pad_w = name_w - len(x["model"])
-            name = (f"[@click={click}({row})]{rich_escape(x['model'])}[/]" + " " * pad_w if click
-                    else rich_escape(name))
+            pad_w = name_w - len(x["model"]) - (len(tag) + 1 if tag else 0)
+            # a single answer per question in yellow: its accuracy is the roughest
+            tag = f" [{'yellow' if x['repeats'][1] == 1 else 'dim'}]{tag}[/]" if tag else ""
+            name = ((f"[@click={click}({row})]{rich_escape(x['model'])}[/]" if click else rich_escape(x["model"]))
+                    + tag + " " * pad_w)
         lines.append((f"{name}  " + "".join(cells)).rstrip())
     return lines
 
@@ -189,11 +219,12 @@ def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
     over = any(x.get("over") for x in result["models"])   # a column only when some answer ran past max_tokens
-    header = ["model", "tokens/answer", "time/answer", "answers right", "truncated", *(["over budget"] if over else []),
+    header = ["model", "repeats", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
               "tok/s", "TTFT s", "memory GB"]
-    rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
+    rows = [[x["model"], repeats(x) or "-", f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
-             f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
+             f"{x['passed']}/{x['answers']}", "-" if x.get("S") is None else compare.pct(x["S"] / 100),
+             f"{x.get('cut') or 0}/{x['answers']}",
              *([f"{x.get('over') or 0}/{x['answers']}"] if over else []),
              _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
              _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
@@ -228,7 +259,11 @@ def scope_note(result, models):
     return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
-            "(longer is better)"
+            "(longer is better); score: partial credit, e.g. 5 of 6 checks passed is 83% (shown when it differs)"
+            + (f"; ×n beside a name: answers per question. ×1 models answered each question once, so one "
+               f"answer more or less right moves their accuracy by {100 / result['questions']:.0f} points; "
+               "repeats steady it" if len({x.get("repeats") for x in result["models"]}) > 1
+               or any(x.get("repeats") and x["repeats"][1] == 1 for x in result["models"]) else "")
             + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
                "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else ".")
             + (" Over budget: answers that used more tokens than the max_tokens sent, because the provider didn't "
@@ -236,14 +271,14 @@ def scope_note(result, models):
                if any(x.get("over") for x in result["models"]) else ""))
 
 
-def for_engine(e, labels=None, packs=None, results_dir=None):
+def for_engine(e, labels=None, packs=None, results_dir=None, sort="accuracy"):
     """(result, models) from a workspace's finished results (results on their way out excluded)."""
     paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
     rows, infos = compare.load(paths) if paths else ([], [])
     if packs:
         rows = [r for r in rows if r["suite"] in packs]
     models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
-    return index(rows, models if labels else None, memory_gb(infos)), models
+    return index(rows, models if labels else None, memory_gb(infos), sort), models
 
 
 def markdown(result, models):

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/run_evals.py

@@ -402,6 +402,8 @@ def cmd_pta(argv):
     p.add_argument("--only", help="comma-separated model labels (default: every model with results)")
     p.add_argument("--packs", help="comma-separated packs (default: all)")
     p.add_argument("--width", type=int, default=20, help="bar width in characters (default 20)")
+    p.add_argument("--sort", choices=("accuracy", "score"), default="accuracy",
+                   help="order models by answers right (default) or by score (partial credit)")
     p.add_argument("--models")
     p.add_argument("--results-dir")
     a = p.parse_args(argv)
@@ -410,7 +412,7 @@ def cmd_pta(argv):
     e = engine.Engine(models_path=a.models, results_dir=a.results_dir)
     labels = [x.strip() for x in a.only.split(",")] if a.only else None
     packs = [x.strip() for x in a.packs.split(",")] if a.packs else None
-    result, models = pta.for_engine(e, labels, packs)
+    result, models = pta.for_engine(e, labels, packs, sort=a.sort)
     out = Console(highlight=False)
     if not models:
         sys.exit("no finished results to compare" + (" for those models or packs" if labels or packs else ""))

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/src/tuieval/tui.py

@@ -937,7 +937,10 @@ class SetupScreen(Screen):
         for n in e.packs:
             st = e.result_status(label, n)
             if st:
-                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}")
+                reps = e.result_repeats(label, n) if st in ("certified", "screened") else None
+                times = ("" if not reps else f"[dim]×{reps[0]}[/dim]" if reps[0] == reps[1]
+                         else f"[dim]×{reps[0]}-{reps[1]}[/dim]")
+                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}{times}")
                 if st.startswith("outdated"):
                     why.setdefault(st.removeprefix("outdated: "), []).append(n)
         if why:  # each reason once, not once per pack
@@ -951,7 +954,7 @@ class SetupScreen(Screen):
         hidden = (f"  [cyan]{' '.join(tags)}[/cyan]" if tags else "") + hidden
         detail.update(f"[b]{label}[/b]  [dim]{m['server']} · {m.get('model') or ''}[/dim]"
                       f"{serving_badge(e, m, not e.has_results())}{hidden}\n{verdicts}\n"
-                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial):[/dim] " + "  ".join(packs)
+                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial; ×n answers per question):[/dim] " + "  ".join(packs)
                          if packs else "[dim]no results yet[/dim]"))
 
     @on(Checkbox.Changed, "#show-hidden")
@@ -2892,6 +2895,10 @@ class ResultsScreen(Screen):
     def action_runs(self):
         self.app.open_sessions()
 
+    FAIL_KINDS = [("Every kind", ""), ("Near misses", "near miss"), ("Partly right", "partly right"),
+                  ("Wrong", "wrong"), ("Cut off", "cut off"), ("No answer / format", "no answer / format"),
+                  ("Code crashed or hung", "code crashed or hung"), ("Critical", "critical")]
+
     READINESS_HELP = ("[dim]PASS needs a Certify run with zero critical failures over enough trials and an "
                       "accuracy lower bound that clears the pack's gate (pack.toml). Screening can only FAIL "
                       "or look promising.[/dim]")
@@ -2913,6 +2920,8 @@ class ResultsScreen(Screen):
             with TabPane("PTA index", id="tab-pta"):
                 with Horizontal(id="pta-controls"):
                     yield Select([("All packs", "")], id="pta-pack", allow_blank=False, value="")
+                    yield Select([("By accuracy", "accuracy"), ("By score", "score")], id="pta-sort",
+                                 allow_blank=False, value="accuracy")
                     yield Button("Models: all", id="pta-models")
                 yield Static("[dim]P parsimony (tokens): tokens used, reasoning and answer. T time: total time. "
                              "A accuracy: answers right. Verdicts and critical failures are in Production "
@@ -2954,6 +2963,7 @@ class ResultsScreen(Screen):
             with TabPane("Failures", id="tab-failures"):
                 with Horizontal(id="fail-controls"):
                     yield Button("Models: all", id="fail-models")
+                    yield Select(self.FAIL_KINDS, id="fail-kind", allow_blank=False, value="")
                     yield Static(id="fail-count")
                 yield Static("[dim]Enter on a row shows the whole answer, its reasoning and what was checked.[/dim]")
                 yield DataTable(id="failures", zebra_stripes=True, cursor_type="row")
@@ -3045,24 +3055,40 @@ class ResultsScreen(Screen):
         self.sort_tables()
 
     def fill_failures(self):
-        """Every wrong answer, of the models picked (Models: shared with the other tabs)."""
+        """Every wrong answer, of the models picked (Models: shared with the other tabs), of the kind picked
+        (insights.kind: near miss, partly right, wrong, cut off, …)."""
+        from . import insights
         f = self.query_one("#failures", DataTable)
         f.clear(columns=True)
-        f.add_columns("Model", "Level", "Test", "Answered", "Reason")
+        f.add_columns("Model", "Level", "Test", "Answered", "Kind", "Reason")
         every = {r["model"] for r in self.rows}
         chosen = {m for m in self.app.pq_models if m in every}
-        rows = [r for r in self.rows if not chosen or r["model"] in chosen]
+        wanted = self.query_one("#fail-kind", Select).value or ""
+        rows = [r for r in self.rows if (not chosen or r["model"] in chosen)
+                and (r["ok"] or not wanted or insights.kind(r) == wanted)]
         # compare.failures lists the failed rows in order, so they line up with fail_rows
         self.fail_rows = [r for r in rows if not r["ok"]]
+        colors = {k: c for k, c, _ in insights.KINDS}
         for i, (model, test, reason, level) in enumerate(compare.failures(rows)):
-            when = dates.day(dates.answered_at(self.fail_rows[i].get("raw") or {}))
-            f.add_row(model, diff_badge(level), rich_escape(test), when, rich_escape(reason[:130]), key=str(i))
+            r = self.fail_rows[i]
+            when = dates.day(dates.answered_at(r.get("raw") or {}))
+            k = insights.kind(r)
+            checks = compare.checks_passed(r)
+            label = f"{k} {checks[0]}/{checks[1]}" if checks and k in ("near miss", "partly right", "wrong") else k
+            f.add_row(model, diff_badge(level), rich_escape(test), when, f"[{colors[k]}]{label}[/]",
+                      rich_escape(reason[:130]), key=str(i))
         self.query_one("#fail-models", Button).label = self.models_label(chosen)
         n = len(self.fail_rows)
         self.query_one("#fail-count", Static).update(
-            f"{n} failure{'s' * (n != 1)}" + (f" of {len(chosen)} model{'s' * (len(chosen) != 1)}" if chosen else ""))
+            f"{n} {'' if not wanted else wanted + ' '}failure{'s' * (n != 1)}"
+            + (f" of {len(chosen)} model{'s' * (len(chosen) != 1)}" if chosen else ""))
         self.sort_table(f)
 
+    @on(Select.Changed, "#fail-kind")
+    def fail_kind_changed(self):
+        if getattr(self, "rows", None) is not None:
+            self.fill_failures()
+
     @on(DataTable.RowSelected, "#failures")
     def failure_selected(self, event):
         # in the order shown, so [ and ] step through them as sorted
@@ -3223,6 +3249,13 @@ class ResultsScreen(Screen):
         out = insights.panels(rows, models, e.packs, ref, bases, by, engine=e if main else None)
         body.update("\n\n".join(insights.render(p, i) for i, p in enumerate(out, 1)))
 
+    @on(Select.Changed, "#pta-sort")
+    def pta_sort_changed(self):
+        """The Sort dropdown orders the bars and the table, taking over from a column sorted by a click."""
+        self.app.result_sorts.pop("pta", None)
+        if getattr(self, "rows", None) is not None:
+            self.fill_pta()
+
     @on(Select.Changed, "#pta-pack")
     @on(Select.Changed, "#pq-pack")
     def pack_changed(self, event):
@@ -3293,7 +3326,8 @@ class ResultsScreen(Screen):
             self.query_one("#pta-left-out", Static).update("")
             return
         # none picked: every model with enough answers (pta.index leaves out the rest, and says so)
-        result = pta.index(rows, sorted(chosen) or None, pta.memory_gb(self.infos))
+        result = pta.index(rows, sorted(chosen) or None, pta.memory_gb(self.infos),
+                           self.query_one("#pta-sort", Select).value or "accuracy")
         self.query_one("#pta-note", Static).update(pta.scope_note(result, sorted(present)))
         self.pta_result = result
         removed = [m for m in self.pta_removed_now() if m in present]
@@ -3439,7 +3473,8 @@ class ResultsScreen(Screen):
             tt = r["total_secs"]
             cells.append([r["model"],
                           hi(tt, ("-" if not tt else f"{tt:.1f}s" if tt < 60 else fmt_secs(tt)), "total_secs", "min"),
-                          hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max"),
+                          hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max")
+                          + (f" [dim]{r['score']:.0%}[/dim]" if r["passed"] < r["n"] and r["score"] else ""),
                           hi(r["med_tokens"], f"{r['med_tokens']:,.0f}" if r["med_tokens"] else "-", "med_tokens", "min"),
                           hi(r["total_tokens"], f"{r['total_tokens']:,.0f}", "total_tokens", "min"),
                           hi(r["tokens_per_pass"], f"{r['tokens_per_pass']:,.0f}" if r["tokens_per_pass"] else "-",
@@ -3469,7 +3504,7 @@ class ResultsScreen(Screen):
                      "shared the machine's speed: compare their times with care.[/yellow]")
         title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
                      "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
-                     "they took (cached prompts left out). Fewest / Fastest: questions won; ties count for both. "
+                     "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Fewest / Fastest: questions won; ties count for both. "
                      "Cut off: answers that hit max_tokens.[/dim]")
 
     @staticmethod
@@ -3478,6 +3513,9 @@ class ResultsScreen(Screen):
                 else f"[yellow]{c['passed']}/{c['n']}[/yellow]")
         if c["n"] > 1 and c["passed"] in (0, c["n"]):
             mark += f"[dim]{c['n']}[/dim]"
+        if c["passed"] < c["n"] and c.get("score"):   # partial credit: how close it came
+            checks = compare.checks_passed(c["rows"][0]) if c["n"] == 1 else None
+            mark += f" [yellow]{checks[0]}/{checks[1]}[/yellow]" if checks else f" [yellow]{c['score']:.0%}[/yellow]"
         tok = c["tokens"]
         tok_s = "-" if not tok else (f"{tok / 1000:.1f}k" if tok >= 1000 else f"{tok:.0f}")
         if tok and best and tok == best and c["passed"]:
@@ -3624,6 +3662,7 @@ class EvalsApp(App):
     #ins-controls #ins-cost { width: 20; }
     #ins-body { height: 1fr; }
     #pta-controls Select { width: 28; margin-right: 1; }
+    #pta-controls #pta-sort { width: 18; }
     #pta-bars { height: auto; padding: 1 0 0 0; }
     #pta { width: 1fr; height: auto; margin-top: 1; }
     #pta-left-out { height: auto; padding: 1 0 0 0; }
@@ -3632,6 +3671,7 @@ class EvalsApp(App):
     #pq-models { margin-right: 1; min-width: 20; }
     #fail-controls { height: 3; }
     #fail-models { margin-right: 1; min-width: 20; }
+    #fail-kind { width: 28; margin-right: 1; }
     #fail-count { width: auto; padding: 1 1; }
     #pq-compare-title { padding: 0 1; height: auto; }
     #pq-summary { height: auto; }   /* every picked model, no scrolling: the tab scrolls instead */

{tuieval-0.2.0.dev129 → tuieval-0.2.0.dev133}/tests/test_tuieval.py

@@ -502,6 +502,9 @@ class PickTests(unittest.TestCase):
         self.assertIn("done earlier", out)                                       # nothing missing: not re-asked
         tuieval(self.ws, "run", "--tests", "apps:d", "--tier", "certify")
         self.assertEqual(self.status(), "certified")
+        code = "from tuieval import engine; print(engine.Engine().result_repeats('m', 'apps'))"
+        self.assertEqual(subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,   # ✓×1 on Setup
+                                        env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip(), "(1, 1)")
 
         self.server("wrong")
         tuieval(self.ws, "run", "--tests", "apps:b", "--force")
@@ -1555,7 +1558,7 @@ class PTAIndex(unittest.TestCase):
     scale, and accuracy, all over the questions every compared model answered."""
 
     def test_index_compares_shared_questions(self):
-        from tuieval import pta
+        from tuieval import compare, pta
 
         def row(model, test, ok, secs, repeat=0):
             return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": secs * 1000,
@@ -1574,9 +1577,9 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((by["slow"]["P"], by["fast"]["P"]), (100, 70))          # 4x the tokens: two doublings
         self.assertEqual((pta.score(2, 1), pta.score(1000, 1)), (85, 0))
         header, table = pta.table(res)
-        self.assertEqual(header[1:4], ["tokens/answer", "time/answer", "answers right"])
+        self.assertEqual(header[1:5], ["repeats", "tokens/answer", "time/answer", "answers right"])
         cells = {r[0]: r for r in table}
-        self.assertEqual((cells["slow"][1], cells["fast"][1:3]), ("10 ★", ["40", "2.0s ★"]))   # per answer
+        self.assertEqual((cells["slow"][2], cells["fast"][2:4]), ("10 ★", ["40", "2.0s ★"]))   # per answer
         bars = pta.bars(res, 10, markup=False)
         self.assertEqual(bars[0].split(), ["P", "parsimony", "(tokens)", "T", "time", "A", "accuracy"])
         self.assertEqual(bars[1].split(), ["▼", "shorter", "is", "better"] * 2 + ["▲", "longer", "is", "better"])
@@ -1585,8 +1588,13 @@ class PTAIndex(unittest.TestCase):
                          (fast.index("██████████"), fast.index("███░░░░░░░")))       # its heading
         # P and T bars as long as tokens and time (shorter is better): fast used 4x the tokens of slow but
         # took 4 s against 13 s, so it has the longest token bar and the shortest time bar
-        self.assertEqual(fast.split(), ["fast", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
-        self.assertEqual(slow.split()[1:5], ["██░░░░░░░░", "1.0×", "██████████", "3.2×"])
+        self.assertEqual(fast.split(), ["fast", "×1", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
+        self.assertEqual(slow.split()[1:6], ["×1-2", "██░░░░░░░░", "1.0×", "██████████", "3.2×"])   # ×: answers per question
+        self.assertEqual((cells["slow"][1], cells["fast"][1]), ("×1-2", "×1"))
+        self.assertIn("×1 models answered each question once", pta.scope_note(res, ["slow", "fast"]))
+        full = [dict(r, score=float(r["ok"]), error=False, output="") for r in rows]
+        header, table = compare.scorecard(full)
+        self.assertEqual({r[0]: r[header.index("repeats")] for r in table}, {"slow": "×1-2", "fast": "×1"})
         self.assertEqual((pta.times(1), pta.times(9.84), pta.times(27.9)), ("1.0×", "9.8×", "28×"))
         self.assertEqual((by["fast"]["A"], by["slow"]["A"]), (50, 100))
         self.assertEqual([x["model"] for x in res["models"]], ["slow", "fast"])   # best accuracy first
@@ -1618,8 +1626,8 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # none truncated: nothing beside it
         self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
         header, table = pta.table(res)
-        self.assertEqual(header[4], "truncated")
-        self.assertEqual({r[0]: r[4] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertEqual(header[5:7], ["score", "truncated"])
+        self.assertEqual({r[0]: r[6] for r in table}, {"thinker": "2/10", "plain": "0/10"})
         self.assertIn("truncated", pta.scope_note(res, ["thinker", "plain"]))
         self.assertNotIn("truncated", pta.scope_note(pta.index(rows[10:]), ["plain"]))
 
@@ -1656,6 +1664,82 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(sorted(cells, key=sort_key), ["10/20", "3/4", "4/4", "abc", "-"])
         self.assertEqual(sorted(cells, key=lambda c: sort_key(c, True)), ["4/4", "3/4", "10/20", "abc", "-"])
 
+    def test_partial_credit_score(self):
+        from tuieval import compare, insights, pta
+
+        def row(model, test, ok, score, reason="Failed", **kw):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "score": score, "latency": 1000,
+                    "tokens": 10, "truncated": False, "gen_tps": None, "repeat": 0, "reason": reason, **kw}
+        rows = ([row("close", f"q{i}", False, 5 / 6, "Failed 1/6 checks: x") for i in range(2)]
+                + [row("far", f"q{i}", False, 0.0, "Failed 6/6 checks: x") for i in range(2)]
+                + [row("half", "q0", True, 1.0, "ok"), row("half", "q1", False, 0.0, "Failed 6/6 checks: x")])
+        res = pta.index(rows)
+        self.assertEqual([(x["model"], x["A"], round(x["S"])) for x in res["models"]],
+                         [("half", 50, 50), ("close", 0, 83), ("far", 0, 0)])   # accuracy first, then score
+        self.assertEqual([x["model"] for x in pta.index(rows, sort="score")["models"]], ["close", "half", "far"])
+        bars = {l.split()[0]: l for l in pta.bars(res, 10, markup=False)[2:]}
+        self.assertIn("0%  score 83.3%", bars["close"])
+        self.assertNotIn("score", bars["half"] + bars["far"])               # shown only when it differs
+        header, table = pta.table(res)
+        self.assertEqual({r[0]: r[header.index("score")] for r in table}, {"half": "50.0%", "close": "83.3%", "far": "0%"})
+        self.assertEqual([compare.checks_passed(r) for r in (rows[0], rows[2], rows[4])], [(5, 6), (0, 6), None])
+        self.assertEqual(compare.checks_passed({"reason": "x", "raw": {"checks": {"a": None, "b": "err"}}}), (1, 2))
+        self.assertEqual([insights.kind(r) for r in (rows[0], rows[2], rows[4])], ["near miss", "wrong", "right"])
+        self.assertEqual(insights.kind(row("m", "q", False, 0.5)), "partly right")
+        self.assertEqual(insights.kind(row("m", "q", False, 0.0, "time limit: timed out after 600 s in all")), "cut off")
+        self.assertEqual(insights.kind(row("m", "q", False, 0.9, severity="critical")), "critical")
+        p = insights.failure_mix(rows, ["close", "far", "half"])
+        self.assertIn("0% right (2 answers; 2 near miss)", re.sub(r"\[[^\]]*\]", "", "\n".join(p["lines"])))
+
+    def test_results_show_partial_credit(self):
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, Select, Static, TabbedContent
+            def row(model, test, ok, score, reason):
+                return {"model": model, "suite": "p", "test": f"p: {test}", "name": f"p: {test}", "ok": ok,
+                        "score": score, "reason": reason, "error": False, "truncated": False, "output": "",
+                        "tokens": 10, "latency": 1000, "gen_tps": None, "prompt_tps": None, "ttft": None,
+                        "repeat": 0, "difficulty": "hard", "raw": {}, "path": "x", "alongside": []}
+            rows = ([row("close", f"q{i}", False, 5 / 6, "Failed 1/6 checks: x") for i in range(2)]
+                    + [row("half", "q0", True, 1.0, "ok"), row("half", "q1", False, 0.0, "Failed 6/6 checks: x")])
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(180, 50)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    s = app.screen
+                    s.rows, s.infos = rows, []
+                    s.fill_pta()
+                    await pilot.pause(0.2)
+                    first = s.pta_result["models"][0]["model"]
+                    s.query_one("#pta-sort", Select).value = "score"
+                    await pilot.pause(0.3)
+                    print(first, s.pta_result["models"][0]["model"])
+                    s.fill_failures()
+                    t = s.query_one("#failures", DataTable)
+                    print(sorted(str(t.get_row_at(i)[4]) for i in range(t.row_count)))
+                    s.query_one("#fail-kind", Select).value = "near miss"
+                    await pilot.pause(0.3)
+                    print(t.row_count, str(s.query_one("#fail-count", Static).render()))
+                    s.fill_per_question()
+                    pq = s.query_one("#per-question", DataTable)
+                    cols = [str(c.label) for c in pq.ordered_columns]
+                    print(sorted(str(pq.get_row_at(i)[cols.index("close")]).split()[1] for i in range(pq.row_count)))
+            asyncio.run(go())
+        """)
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                                 env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        self.assertEqual(out.stdout.strip().splitlines(), [
+            "half close",                                         # by accuracy, then by score
+            "['[#87af5f]near miss 5/6[/]', '[#87af5f]near miss 5/6[/]', '[red]wrong 0/6[/]']",
+            "2 2 near miss failures",
+            "['5/6', '5/6']"], out.stderr[-2000:])
+
     def test_complete_models(self):
         from tuieval import compare
         rows = [{"model": m, "suite": s, "test": f"{s}: {t}", **({"partial": True} if m == "running" else {})}
@@ -1873,7 +1957,7 @@ class PTAIndex(unittest.TestCase):
             self.assertGreaterEqual(len(got["opened"]), 2)
             self.assertEqual(got["opened"], {m: m for m in got["opened"]})   # a cell opens its column's model
             self.assertEqual(got["tab"], "tab-pta")                  # Results opens on the PTA index
-            self.assertEqual([(r[0], r[3]) for r in got["rows"]][0], ("good", "4/4"))
+            self.assertEqual([(r[0], r[4]) for r in got["rows"]][0], ("good", "4/4"))
             s = got["sorts"]   # a click on a column name sorts by it, a second click reverses
             self.assertEqual((s["up"], s["down"], s["kept"]), (["hosted", "good"], ["good", "hosted"], ["good", "hosted"]))
             self.assertIn("answers right ▼", s["labels"])