tuieval 0.2.0.dev127__tar.gz → 0.2.0.dev131__tar.gz

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
+- Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).
 - Agent packs (`kind = "agent"`): each test is a small project with a task, worked on by a command-line agent you define in `[agents]`, using the model tuieval serves, and graded by the task's visible and hidden tests. Results show the agent's stats, diff and transcript; `tuieval compare` adds an Agent tasks table; `tuieval selftest` checks each task's reference and wrong fixes. See docs/agent-packs.md.
 - Answers that used more tokens than the `max_tokens` sent (a provider that didn't stop them) are marked over budget: `(n over budget)` in the PTA index, an **over budget** column, `over` in Per question and a note in the answer view. `[defaults] over_budget = "fail"` fails them instead. Works on stored results.

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev127
+Version: 0.2.0.dev131
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/docs/models.md

@@ -105,6 +105,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
+- **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
@@ -118,7 +119,7 @@ Results → **Insights** (also `tuieval insights`) turns the results into charts
 1. **Who's worth keeping**: every model against a reference model, paired on the questions both answered: the difference in accuracy points with its 95% CI, and better, tie or worse. Pick the reference in the tab or with `--reference`; set a default with `reference = "<label>"` under `[defaults]` in `models.toml`. Without one, the most accurate model is the reference.
 2. **Best model per use case**: a heatmap of accuracy per model and use case (pack group); ★ marks the best among models that answered most of it.
 3. **Fine-tunes against their base model**: for each model with `base = "<label>"` in its `[[models]]` block, the difference per pack and the question categories (a test's `category`) it gained or lost most.
-4. **Why models fail**: each model's answers split into right, wrong, cut off at max_tokens, no answer or bad format, code that crashed or hung, and critical failures.
+4. **Why models fail**: each model's answers split into right, near miss (a score of 80% or more, e.g. 5 of 6 checks), partly right, wrong (a score of 0), cut off at max_tokens or the time limit, no answer or bad format, code that crashed or hung, and critical failures. The Failures tab shows each failure's kind (with its checks passed) and filters by it.
 5. **Accuracy against cost**: accuracy against median tokens (or seconds) per question, over the questions every model answered; green models are the ones no other model beats on both. With hosted and local models together, the best local trade-offs get their own line.
 6. **Question health**: questions every model fails (check them), every model passes (they don't separate models), and the ones that split models most.
 7. **Next runs**: the Certify runs that would settle the open verdicts of the strongest models, with their time and command.

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev127'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev127')
+__version__ = version = '0.2.0.dev131'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev131')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/compare.py

@@ -351,10 +351,22 @@ def speed(rows, infos, machine=None, speeds=None):
     return header, table, frontier
 
 
+NEAR_MISS = 0.8   # a failed answer scoring at least this (e.g. 5 of 6 checks) is a near miss
+
+
+def checks_passed(r):
+    """(passed, total) checks of a failed code answer (from its stored checks, or its reason), else None."""
+    checks = (r.get("raw") or {}).get("checks")
+    if checks:
+        return sum(v is None for v in checks.values()), len(checks)
+    m = re.match(r"Failed (\d+)/(\d+) checks", r.get("reason") or "")
+    return (int(m[2]) - int(m[1]), int(m[2])) if m else None
+
+
 def per_question(rows):
     """[{test, name, suite, difficulty, cells: {model: cell}, spread}] with one cell per model that
-    answered: n answers, passes, median tokens / seconds / tok/s over repeats, truncated count,
-    partial (some answers from a pack still running), rows (the answers, for the detail view).
+    answered: n answers, passes, mean score (partial credit), median tokens / seconds / tok/s over repeats,
+    truncated count, partial (some answers from a pack still running), rows (the answers, for the detail view).
     spread = {"tokens": .., "secs": ..}: largest / smallest model median (1 if one model)."""
     tests = {}
     for r in rows:
@@ -369,6 +381,7 @@ def per_question(rows):
             c.update(rows=v, n=len(v), passed=sum(r["ok"] for r in v), truncated=sum(r["truncated"] for r in v),
                      over=sum(bool(r.get("over_budget")) for r in v),
                      partial=any(r.get("partial") for r in v),
+                     score=statistics.fmean(r.get("score", r["ok"]) for r in v),
                      tokens=median(r["tokens"] or None for r in v), secs=median(r["latency"] / 1000 or None for r in v),
                      tps=median(r["gen_tps"] for r in v))
         t["spread"] = _spread(t["cells"])
@@ -390,10 +403,21 @@ def with_models(test, models):
     return dict(test, cells=cells, spread=_spread(cells))
 
 
+def complete_models(rows, suite, test_ids=None):
+    """The models whose rows answer every question of a pack: test_ids (the pack's current tests),
+    or every question of it seen in the rows. Partial rows (a pack still running) don't count."""
+    rows = [r for r in rows if r["suite"] == suite and not r.get("partial")]
+    wanted = {f"{suite}: {t}" for t in test_ids} if test_ids is not None else {r["test"] for r in rows}
+    answered = {}
+    for r in rows:
+        answered.setdefault(r["model"], set()).add(r["test"])
+    return sorted(m for m, tests in answered.items() if wanted and wanted <= tests)
+
+
 def compare_models(tests, models):
     """Per model over per_question entries that all `models` answered: passes, token and time
     totals/medians (totals add each question's median over repeats, so repeats don't inflate them), questions where it used the fewest tokens / was fastest (among passing
-    answers for tokens; ties count for each), answers cut off."""
+    answers for tokens; ties count for each), answers cut off, mean score (partial credit)."""
     out = []
     for m in models:
         cs = [t["cells"][m] for t in tests]
@@ -410,6 +434,7 @@ def compare_models(tests, models):
                 fastest += 1
         rs = [r for c in cs for r in c["rows"]]
         out.append({"model": m, "n": len(cs), "passed": passed, "total_tokens": total,
+                    "score": statistics.fmean(c["score"] for c in cs) if cs else None,
                     "avg_pp_tps": _rate([(r.get("prompt_tokens"), r["prompt_tps"]) for r in rs]),
                     "avg_gen_tps": _rate([(r["tokens"], r["gen_tps"]) for r in rs]),
                     "med_tokens": median(c["tokens"] for c in cs), "med_secs": median(c["secs"] for c in cs),

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/insights.py

@@ -6,8 +6,9 @@
   2 use cases       accuracy per model and use case (pack group) as a heatmap; the best marked
   3 vs base         a fine-tune against its base model (models.toml `base`): per pack, and the question
                     categories it gained or lost most
-  4 why they fail   each model's answers split into right, wrong, cut off, no answer / bad format, code
-                    crashed or hung, and critical failures
+  4 why they fail   each model's answers split into right, near miss (most checks passed), partly right,
+                    wrong, cut off (or out of time), no answer / bad format, code crashed or hung, and
+                    critical failures
   5 accuracy / cost accuracy against median tokens (or seconds) per question; the models no other beats
                     on both
   6 questions       questions every model fails (check them), every model passes (no signal), and the
@@ -267,7 +268,8 @@ def categories(rows, m, base, min_q=2):
 
 
 # ---------------------------------------------------------------- 4 why they fail
-KINDS = (("right", "green", "█"), ("wrong", "red", "█"), ("cut off", "yellow", "█"),
+KINDS = (("right", "green", "█"), ("near miss", "#87af5f", "█"), ("partly right", "#af875f", "█"),
+         ("wrong", "red", "█"), ("cut off", "yellow", "█"),
          ("no answer / format", "magenta", "█"), ("code crashed or hung", "cyan", "█"), ("critical", "#ff5faf", "▓"))
 FORMAT = ("No 'ANSWER:' line", "Empty answer", "No code", "no tool call")
 CRASHED = ("Timed out after", "Harness crashed", "grader error")
@@ -278,13 +280,15 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET")):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"
     if r["reason"].startswith(CRASHED):
         return "code crashed or hung"
-    return "wrong"
+    # partial credit (e.g. checks passed): close isn't the same as all wrong
+    score = r.get("score") or 0
+    return "near miss" if score >= compare.NEAR_MISS else "partly right" if score > 0 else "wrong"
 
 
 def _split(counts, total, width):
@@ -322,7 +326,9 @@ def failure_mix(rows, models, width=36):
             top, n = max(((k, c[k]) for k, *_ in KINDS[1:-1]), key=lambda x: x[1])
             if n / fails >= 0.4 and top != "wrong":
                 actions.append(f"{m}: {n} of its {fails} failures are {top}: " + {
-                    "cut off": "it thinks too long for max_tokens, rather than reasoning wrong.",
+                    "cut off": "it thinks too long for max_tokens (or the time limit), rather than reasoning wrong.",
+                    "near miss": "they pass most of their checks: it's closer than its accuracy shows.",
+                    "partly right": "they pass some of their checks, not none.",
                     "no answer / format": "it doesn't follow the answer format (check its chat template and the "
                                           "pack's instructions).",
                     "code crashed or hung": "its code crashes or never finishes."}[top])

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/pta.py

@@ -7,6 +7,9 @@
                            model's ("28×").
     A  accuracy            the share of answers to those questions that passed (%).
 
+Beside A, the score: partial credit averaged over the same answers (e.g. a code answer passing 5 of 6
+checks scores 83%, where A counts it wrong). Shown when it differs from A; models can be ordered by it.
+
 The P and T bars are as long as the tokens and time used (the model that used the most fills its bar,
 the others in proportion, at least one block), so shorter is better; the A bar: longer is better.
 P and T also carry a 0-100 score (100 for the best, POINTS_PER_DOUBLING points less for every
@@ -46,11 +49,15 @@ def memory_gb(infos):
     return out
 
 
-def index(rows, models=None, memory=None):
-    """{"questions": n shared, "models": [{model, P, T, A, cut, cut_pct, tokens, total_s, passed, answers, alongside,
+SORTS = ("accuracy", "score")
+
+
+def index(rows, models=None, memory=None, sort="accuracy"):
+    """{"questions": n shared, "models": [{model, P, T, A, S, cut, cut_pct, tokens, total_s, passed, answers, alongside,
     fewest, fastest, tps, ttft, memory_gb}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
-    most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
+    most-answered model}, best accuracy (sort="accuracy") or score (sort="score") first. S: the mean
+score (partial credit) in %. models: the ones to compare (default: every model in rows
     with enough answers; models named here are compared whatever their count). memory: memory_gb()."""
     no_answers = sorted({r["model"] for r in rows} - {r["model"] for r in rows if not r.get("error")})
     rows = [r for r in rows if not r.get("error")]     # a server or connection error isn't an answer
@@ -75,6 +82,7 @@ def index(rows, models=None, memory=None):
                     "total_s": sum(c["secs"] or 0 for c in cells),
                     "passed": sum(c["passed"] for c in cells), "answers": answers,
                     "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    "S": 100 * sum(c["score"] * c["n"] for c in cells) / answers if answers else None,
                     # failed only because max_tokens ran out (overthinking, not wrong yet)
                     "cut": sum(c["truncated"] for c in cells),
                     "cut_pct": 100 * sum(c["truncated"] for c in cells) / answers if answers else None,
@@ -95,7 +103,8 @@ def index(rows, models=None, memory=None):
         x["time_x"] = x["total_s"] / fastest if fastest and x["total_s"] else None
         x["fewest"] = bool(fewest) and x["tokens"] == fewest
         x["fastest"] = bool(fastest) and x["total_s"] == fastest
-    out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
+    first, then = ("S", "A") if sort == "score" else ("A", "S")
+    out.sort(key=lambda x: (-(x[first] or 0), -(x[then] or 0), -(x["T"] or 0), x["model"]))
     return {"questions": len(shared), "models": out, "left_out": left_out, "no_answers": no_answers,
             "most": most}
 
@@ -145,6 +154,10 @@ def bars(result, width=20, markup=True, click=None):
                 if v is not None and v < 100:   # full only when every answer passed
                     full = min(full, width - 1)
                 num = "-" if v is None else compare.pct(v / 100)
+                score = x.get("S")
+                if score is not None and v is not None and compare.pct(score / 100) != num:
+                    note = f"score {compare.pct(score / 100)}"
+                    num = f"{num:>6}  " + (f"[dim]{note}[/dim]" if markup else note)
                 cut = x.get("cut_pct") or 0
                 part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
                 if cut:   # after the accuracy, which stays aligned with the other rows'
@@ -189,11 +202,12 @@ def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
     over = any(x.get("over") for x in result["models"])   # a column only when some answer ran past max_tokens
-    header = ["model", "tokens/answer", "time/answer", "answers right", "truncated", *(["over budget"] if over else []),
+    header = ["model", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
               "tok/s", "TTFT s", "memory GB"]
     rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
-             f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
+             f"{x['passed']}/{x['answers']}", "-" if x.get("S") is None else compare.pct(x["S"] / 100),
+             f"{x.get('cut') or 0}/{x['answers']}",
              *([f"{x.get('over') or 0}/{x['answers']}"] if over else []),
              _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
              _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
@@ -228,7 +242,7 @@ def scope_note(result, models):
     return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
-            "(longer is better)"
+            "(longer is better); score: partial credit, e.g. 5 of 6 checks passed is 83% (shown when it differs)"
             + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
                "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else ".")
             + (" Over budget: answers that used more tokens than the max_tokens sent, because the provider didn't "
@@ -236,14 +250,14 @@ def scope_note(result, models):
                if any(x.get("over") for x in result["models"]) else ""))
 
 
-def for_engine(e, labels=None, packs=None, results_dir=None):
+def for_engine(e, labels=None, packs=None, results_dir=None, sort="accuracy"):
     """(result, models) from a workspace's finished results (results on their way out excluded)."""
     paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
     rows, infos = compare.load(paths) if paths else ([], [])
     if packs:
         rows = [r for r in rows if r["suite"] in packs]
     models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
-    return index(rows, models if labels else None, memory_gb(infos)), models
+    return index(rows, models if labels else None, memory_gb(infos), sort), models
 
 
 def markdown(result, models):

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/run_evals.py

@@ -402,6 +402,8 @@ def cmd_pta(argv):
     p.add_argument("--only", help="comma-separated model labels (default: every model with results)")
     p.add_argument("--packs", help="comma-separated packs (default: all)")
     p.add_argument("--width", type=int, default=20, help="bar width in characters (default 20)")
+    p.add_argument("--sort", choices=("accuracy", "score"), default="accuracy",
+                   help="order models by answers right (default) or by score (partial credit)")
     p.add_argument("--models")
     p.add_argument("--results-dir")
     a = p.parse_args(argv)
@@ -410,7 +412,7 @@ def cmd_pta(argv):
     e = engine.Engine(models_path=a.models, results_dir=a.results_dir)
     labels = [x.strip() for x in a.only.split(",")] if a.only else None
     packs = [x.strip() for x in a.packs.split(",")] if a.packs else None
-    result, models = pta.for_engine(e, labels, packs)
+    result, models = pta.for_engine(e, labels, packs, sort=a.sort)
     out = Console(highlight=False)
     if not models:
         sys.exit("no finished results to compare" + (" for those models or packs" if labels or packs else ""))

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/src/tuieval/tui.py

@@ -2892,6 +2892,10 @@ class ResultsScreen(Screen):
     def action_runs(self):
         self.app.open_sessions()
 
+    FAIL_KINDS = [("Every kind", ""), ("Near misses", "near miss"), ("Partly right", "partly right"),
+                  ("Wrong", "wrong"), ("Cut off", "cut off"), ("No answer / format", "no answer / format"),
+                  ("Code crashed or hung", "code crashed or hung"), ("Critical", "critical")]
+
     READINESS_HELP = ("[dim]PASS needs a Certify run with zero critical failures over enough trials and an "
                       "accuracy lower bound that clears the pack's gate (pack.toml). Screening can only FAIL "
                       "or look promising.[/dim]")
@@ -2913,6 +2917,8 @@ class ResultsScreen(Screen):
             with TabPane("PTA index", id="tab-pta"):
                 with Horizontal(id="pta-controls"):
                     yield Select([("All packs", "")], id="pta-pack", allow_blank=False, value="")
+                    yield Select([("By accuracy", "accuracy"), ("By score", "score")], id="pta-sort",
+                                 allow_blank=False, value="accuracy")
                     yield Button("Models: all", id="pta-models")
                 yield Static("[dim]P parsimony (tokens): tokens used, reasoning and answer. T time: total time. "
                              "A accuracy: answers right. Verdicts and critical failures are in Production "
@@ -2954,6 +2960,7 @@ class ResultsScreen(Screen):
             with TabPane("Failures", id="tab-failures"):
                 with Horizontal(id="fail-controls"):
                     yield Button("Models: all", id="fail-models")
+                    yield Select(self.FAIL_KINDS, id="fail-kind", allow_blank=False, value="")
                     yield Static(id="fail-count")
                 yield Static("[dim]Enter on a row shows the whole answer, its reasoning and what was checked.[/dim]")
                 yield DataTable(id="failures", zebra_stripes=True, cursor_type="row")
@@ -2963,6 +2970,9 @@ class ResultsScreen(Screen):
         self.rows, self.infos, self.partial_rows, self.superseded = [], [], [], set()
         if not hasattr(self.app, "pq_models"):
             self.app.pq_models = set()
+        if getattr(self.app, "pack_pick", None):   # the pack lists open on All packs again
+            self.app.pq_models = self.app.pack_pick["before"]
+        self.app.pack_pick = None
         if not hasattr(self.app, "result_sorts"):
             self.app.result_sorts = {}   # table id -> (column name, descending), kept for this session
         self.fill()
@@ -3042,24 +3052,40 @@ class ResultsScreen(Screen):
         self.sort_tables()
 
     def fill_failures(self):
-        """Every wrong answer, of the models picked (Models: shared with the other tabs)."""
+        """Every wrong answer, of the models picked (Models: shared with the other tabs), of the kind picked
+        (insights.kind: near miss, partly right, wrong, cut off, …)."""
+        from . import insights
         f = self.query_one("#failures", DataTable)
         f.clear(columns=True)
-        f.add_columns("Model", "Level", "Test", "Answered", "Reason")
+        f.add_columns("Model", "Level", "Test", "Answered", "Kind", "Reason")
         every = {r["model"] for r in self.rows}
         chosen = {m for m in self.app.pq_models if m in every}
-        rows = [r for r in self.rows if not chosen or r["model"] in chosen]
+        wanted = self.query_one("#fail-kind", Select).value or ""
+        rows = [r for r in self.rows if (not chosen or r["model"] in chosen)
+                and (r["ok"] or not wanted or insights.kind(r) == wanted)]
         # compare.failures lists the failed rows in order, so they line up with fail_rows
         self.fail_rows = [r for r in rows if not r["ok"]]
+        colors = {k: c for k, c, _ in insights.KINDS}
         for i, (model, test, reason, level) in enumerate(compare.failures(rows)):
-            when = dates.day(dates.answered_at(self.fail_rows[i].get("raw") or {}))
-            f.add_row(model, diff_badge(level), rich_escape(test), when, rich_escape(reason[:130]), key=str(i))
-        self.query_one("#fail-models", Button).label = f"Models: {len(chosen)} picked" if chosen else "Models: all"
+            r = self.fail_rows[i]
+            when = dates.day(dates.answered_at(r.get("raw") or {}))
+            k = insights.kind(r)
+            checks = compare.checks_passed(r)
+            label = f"{k} {checks[0]}/{checks[1]}" if checks and k in ("near miss", "partly right", "wrong") else k
+            f.add_row(model, diff_badge(level), rich_escape(test), when, f"[{colors[k]}]{label}[/]",
+                      rich_escape(reason[:130]), key=str(i))
+        self.query_one("#fail-models", Button).label = self.models_label(chosen)
         n = len(self.fail_rows)
         self.query_one("#fail-count", Static).update(
-            f"{n} failure{'s' * (n != 1)}" + (f" of {len(chosen)} model{'s' * (len(chosen) != 1)}" if chosen else ""))
+            f"{n} {'' if not wanted else wanted + ' '}failure{'s' * (n != 1)}"
+            + (f" of {len(chosen)} model{'s' * (len(chosen) != 1)}" if chosen else ""))
         self.sort_table(f)
 
+    @on(Select.Changed, "#fail-kind")
+    def fail_kind_changed(self):
+        if getattr(self, "rows", None) is not None:
+            self.fill_failures()
+
     @on(DataTable.RowSelected, "#failures")
     def failure_selected(self, event):
         # in the order shown, so [ and ] step through them as sorted
@@ -3072,7 +3098,6 @@ class ResultsScreen(Screen):
         self.app.push_screen(AnswerScreen(keep, keep.index(entries[i]), what="failure"))
 
     # -- per question: tokens and time for every model on the same question
-    @on(Select.Changed, "#pq-pack")
     @on(Select.Changed, "#pq-level")
     @on(Checkbox.Changed, "#pq-partial")
     @on(Checkbox.Changed, "#pq-shared")
@@ -3214,18 +3239,62 @@ class ResultsScreen(Screen):
         finally:
             self.filling_insights = False
         chosen = {m for m in self.app.pq_models if m in every}
-        self.query_one("#ins-models", Button).label = f"Models: {len(chosen)} picked" if chosen else "Models: all"
+        self.query_one("#ins-models", Button).label = self.models_label(chosen)
         main = os.path.abspath(self.results_dir or e.results_dir) == os.path.abspath(e.results_dir)
         bases = {m["label"]: m["base"] for m in e.cfg["models"] if m.get("base")}
         by = self.query_one("#ins-cost", Select).value or "tokens"
         out = insights.panels(rows, models, e.packs, ref, bases, by, engine=e if main else None)
         body.update("\n\n".join(insights.render(p, i) for i, p in enumerate(out, 1)))
 
-    @on(Select.Changed, "#pta-pack")
-    def pta_changed(self):
+    @on(Select.Changed, "#pta-sort")
+    def pta_sort_changed(self):
+        """The Sort dropdown orders the bars and the table, taking over from a column sorted by a click."""
+        self.app.result_sorts.pop("pta", None)
         if getattr(self, "rows", None) is not None:
             self.fill_pta()
 
+    @on(Select.Changed, "#pta-pack")
+    @on(Select.Changed, "#pq-pack")
+    def pack_changed(self, event):
+        """A pack picked (PTA index or Per question, kept the same in both): compare the models that
+        answered every question of it (the Models button can change that); All packs: back to the
+        models picked before."""
+        if getattr(self, "rows", None) is None:
+            return
+        e, pack = self.app.engine, event.value or ""
+        for other, values in (("#pta-pack", "pta_pack_values"), ("#pq-pack", "pq_pack_values")):
+            sel = self.query_one(other, Select)
+            if sel is not event.select and sel.value != pack and pack in getattr(self, values, [""]):
+                with sel.prevent(Select.Changed):
+                    sel.value = pack
+        shared = self.query_one("#pq-shared", Checkbox)
+        p = self.app.pack_pick or {"before": set(self.app.pq_models), "shared": shared.value}
+        before = p["before"]
+        if not pack:
+            self.app.pack_pick = None
+            with shared.prevent(Checkbox.Changed):   # set_models may have ticked it
+                shared.value = p["shared"]
+            self.set_models(before)
+            return
+        rows = [r for r in self.rows if r.get("path") not in self.superseded]
+        complete = set(compare.complete_models(rows, pack, [t["id"] for t in e.packs[pack].tests]
+                                               if pack in e.packs else None))
+        self.app.pack_pick = {"pack": pack, "models": complete, "before": before, "shared": p["shared"]}
+        if not complete:
+            self.notify(f"No model has answered every question of {self.pack_label(pack)} yet: "
+                        "the models picked before are kept.", severity="warning", timeout=8)
+        self.set_models(complete or before)
+
+    def pack_label(self, pack):
+        return self.app.engine.packs[pack].label if pack in self.app.engine.packs else pack
+
+    def models_label(self, chosen):
+        """The Models buttons' text: the models a pack pick chose say so."""
+        p = self.app.pack_pick
+        if chosen and p and chosen == p["models"]:
+            return f"Models: {len(chosen)} complete on {self.pack_label(p['pack'])}"
+        return f"Models: {len(chosen)} picked" if chosen else "Models: all"
+
     def fill_pta(self):
         """The PTA index (pta.py) of the picked models (Models button, shared with Per question) on
         the finished results, over the questions all of them answered."""
@@ -3245,7 +3314,7 @@ class ResultsScreen(Screen):
             rows = [r for r in rows if r["suite"] == sel.value]
         present = {r["model"] for r in rows}
         chosen = {m for m in self.app.pq_models if m in present}
-        self.query_one("#pta-models", Button).label = f"Models: {len(chosen)} picked" if chosen else "Models: all"
+        self.query_one("#pta-models", Button).label = self.models_label(chosen)
         t = self.query_one("#pta", DataTable)
         t.clear(columns=True)
         if not present:
@@ -3254,7 +3323,8 @@ class ResultsScreen(Screen):
             self.query_one("#pta-left-out", Static).update("")
             return
         # none picked: every model with enough answers (pta.index leaves out the rest, and says so)
-        result = pta.index(rows, sorted(chosen) or None, pta.memory_gb(self.infos))
+        result = pta.index(rows, sorted(chosen) or None, pta.memory_gb(self.infos),
+                           self.query_one("#pta-sort", Select).value or "accuracy")
         self.query_one("#pta-note", Static).update(pta.scope_note(result, sorted(present)))
         self.pta_result = result
         removed = [m for m in self.pta_removed_now() if m in present]
@@ -3300,8 +3370,7 @@ class ResultsScreen(Screen):
         all_tests = compare.per_question(rows)
         self.pq_all_tests = all_tests
         chosen = {m for m in self.app.pq_models if any(m in t["cells"] for t in all_tests)}
-        self.query_one("#pq-models", Button).label = (
-            f"Models: {len(chosen)} picked" if chosen else "Models: all")
+        self.query_one("#pq-models", Button).label = self.models_label(chosen)
         tests = []
         for t in all_tests:
             if (pack and t["suite"] != pack) or (level and t["difficulty"] != level):
@@ -3401,7 +3470,8 @@ class ResultsScreen(Screen):
             tt = r["total_secs"]
             cells.append([r["model"],
                           hi(tt, ("-" if not tt else f"{tt:.1f}s" if tt < 60 else fmt_secs(tt)), "total_secs", "min"),
-                          hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max"),
+                          hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max")
+                          + (f" [dim]{r['score']:.0%}[/dim]" if r["passed"] < r["n"] and r["score"] else ""),
                           hi(r["med_tokens"], f"{r['med_tokens']:,.0f}" if r["med_tokens"] else "-", "med_tokens", "min"),
                           hi(r["total_tokens"], f"{r['total_tokens']:,.0f}", "total_tokens", "min"),
                           hi(r["tokens_per_pass"], f"{r['tokens_per_pass']:,.0f}" if r["tokens_per_pass"] else "-",
@@ -3431,7 +3501,7 @@ class ResultsScreen(Screen):
                      "shared the machine's speed: compare their times with care.[/yellow]")
         title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
                      "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
-                     "they took (cached prompts left out). Fewest / Fastest: questions won; ties count for both. "
+                     "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Fewest / Fastest: questions won; ties count for both. "
                      "Cut off: answers that hit max_tokens.[/dim]")
 
     @staticmethod
@@ -3440,6 +3510,9 @@ class ResultsScreen(Screen):
                 else f"[yellow]{c['passed']}/{c['n']}[/yellow]")
         if c["n"] > 1 and c["passed"] in (0, c["n"]):
             mark += f"[dim]{c['n']}[/dim]"
+        if c["passed"] < c["n"] and c.get("score"):   # partial credit: how close it came
+            checks = compare.checks_passed(c["rows"][0]) if c["n"] == 1 else None
+            mark += f" [yellow]{checks[0]}/{checks[1]}[/yellow]" if checks else f" [yellow]{c['score']:.0%}[/yellow]"
         tok = c["tokens"]
         tok_s = "-" if not tok else (f"{tok / 1000:.1f}k" if tok >= 1000 else f"{tok:.0f}")
         if tok and best and tok == best and c["passed"]:
@@ -3586,6 +3659,7 @@ class EvalsApp(App):
     #ins-controls #ins-cost { width: 20; }
     #ins-body { height: 1fr; }
     #pta-controls Select { width: 28; margin-right: 1; }
+    #pta-controls #pta-sort { width: 18; }
     #pta-bars { height: auto; padding: 1 0 0 0; }
     #pta { width: 1fr; height: auto; margin-top: 1; }
     #pta-left-out { height: auto; padding: 1 0 0 0; }
@@ -3594,6 +3668,7 @@ class EvalsApp(App):
     #pq-models { margin-right: 1; min-width: 20; }
     #fail-controls { height: 3; }
     #fail-models { margin-right: 1; min-width: 20; }
+    #fail-kind { width: 28; margin-right: 1; }
     #fail-count { width: auto; padding: 1 1; }
     #pq-compare-title { padding: 0 1; height: auto; }
     #pq-summary { height: auto; }   /* every picked model, no scrolling: the tab scrolls instead */

{tuieval-0.2.0.dev127 → tuieval-0.2.0.dev131}/tests/test_tuieval.py

@@ -1618,8 +1618,8 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # none truncated: nothing beside it
         self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
         header, table = pta.table(res)
-        self.assertEqual(header[4], "truncated")
-        self.assertEqual({r[0]: r[4] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertEqual(header[4:6], ["score", "truncated"])
+        self.assertEqual({r[0]: r[5] for r in table}, {"thinker": "2/10", "plain": "0/10"})
         self.assertIn("truncated", pta.scope_note(res, ["thinker", "plain"]))
         self.assertNotIn("truncated", pta.scope_note(pta.index(rows[10:]), ["plain"]))
 
@@ -1656,6 +1656,159 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(sorted(cells, key=sort_key), ["10/20", "3/4", "4/4", "abc", "-"])
         self.assertEqual(sorted(cells, key=lambda c: sort_key(c, True)), ["4/4", "3/4", "10/20", "abc", "-"])
 
+    def test_partial_credit_score(self):
+        from tuieval import compare, insights, pta
+
+        def row(model, test, ok, score, reason="Failed", **kw):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "score": score, "latency": 1000,
+                    "tokens": 10, "truncated": False, "gen_tps": None, "repeat": 0, "reason": reason, **kw}
+        rows = ([row("close", f"q{i}", False, 5 / 6, "Failed 1/6 checks: x") for i in range(2)]
+                + [row("far", f"q{i}", False, 0.0, "Failed 6/6 checks: x") for i in range(2)]
+                + [row("half", "q0", True, 1.0, "ok"), row("half", "q1", False, 0.0, "Failed 6/6 checks: x")])
+        res = pta.index(rows)
+        self.assertEqual([(x["model"], x["A"], round(x["S"])) for x in res["models"]],
+                         [("half", 50, 50), ("close", 0, 83), ("far", 0, 0)])   # accuracy first, then score
+        self.assertEqual([x["model"] for x in pta.index(rows, sort="score")["models"]], ["close", "half", "far"])
+        bars = {l.split()[0]: l for l in pta.bars(res, 10, markup=False)[2:]}
+        self.assertIn("0%  score 83.3%", bars["close"])
+        self.assertNotIn("score", bars["half"] + bars["far"])               # shown only when it differs
+        header, table = pta.table(res)
+        self.assertEqual({r[0]: r[header.index("score")] for r in table}, {"half": "50.0%", "close": "83.3%", "far": "0%"})
+        self.assertEqual([compare.checks_passed(r) for r in (rows[0], rows[2], rows[4])], [(5, 6), (0, 6), None])
+        self.assertEqual(compare.checks_passed({"reason": "x", "raw": {"checks": {"a": None, "b": "err"}}}), (1, 2))
+        self.assertEqual([insights.kind(r) for r in (rows[0], rows[2], rows[4])], ["near miss", "wrong", "right"])
+        self.assertEqual(insights.kind(row("m", "q", False, 0.5)), "partly right")
+        self.assertEqual(insights.kind(row("m", "q", False, 0.0, "time limit: timed out after 600 s in all")), "cut off")
+        self.assertEqual(insights.kind(row("m", "q", False, 0.9, severity="critical")), "critical")
+        p = insights.failure_mix(rows, ["close", "far", "half"])
+        self.assertIn("0% right (2 answers; 2 near miss)", re.sub(r"\[[^\]]*\]", "", "\n".join(p["lines"])))
+
+    def test_results_show_partial_credit(self):
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, Select, Static, TabbedContent
+            def row(model, test, ok, score, reason):
+                return {"model": model, "suite": "p", "test": f"p: {test}", "name": f"p: {test}", "ok": ok,
+                        "score": score, "reason": reason, "error": False, "truncated": False, "output": "",
+                        "tokens": 10, "latency": 1000, "gen_tps": None, "prompt_tps": None, "ttft": None,
+                        "repeat": 0, "difficulty": "hard", "raw": {}, "path": "x", "alongside": []}
+            rows = ([row("close", f"q{i}", False, 5 / 6, "Failed 1/6 checks: x") for i in range(2)]
+                    + [row("half", "q0", True, 1.0, "ok"), row("half", "q1", False, 0.0, "Failed 6/6 checks: x")])
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(180, 50)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    s = app.screen
+                    s.rows, s.infos = rows, []
+                    s.fill_pta()
+                    await pilot.pause(0.2)
+                    first = s.pta_result["models"][0]["model"]
+                    s.query_one("#pta-sort", Select).value = "score"
+                    await pilot.pause(0.3)
+                    print(first, s.pta_result["models"][0]["model"])
+                    s.fill_failures()
+                    t = s.query_one("#failures", DataTable)
+                    print(sorted(str(t.get_row_at(i)[4]) for i in range(t.row_count)))
+                    s.query_one("#fail-kind", Select).value = "near miss"
+                    await pilot.pause(0.3)
+                    print(t.row_count, str(s.query_one("#fail-count", Static).render()))
+                    s.fill_per_question()
+                    pq = s.query_one("#per-question", DataTable)
+                    cols = [str(c.label) for c in pq.ordered_columns]
+                    print(sorted(str(pq.get_row_at(i)[cols.index("close")]).split()[1] for i in range(pq.row_count)))
+            asyncio.run(go())
+        """)
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                                 env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        self.assertEqual(out.stdout.strip().splitlines(), [
+            "half close",                                         # by accuracy, then by score
+            "['[#87af5f]near miss 5/6[/]', '[#87af5f]near miss 5/6[/]', '[red]wrong 0/6[/]']",
+            "2 2 near miss failures",
+            "['5/6', '5/6']"], out.stderr[-2000:])
+
+    def test_complete_models(self):
+        from tuieval import compare
+        rows = [{"model": m, "suite": s, "test": f"{s}: {t}", **({"partial": True} if m == "running" else {})}
+                for m, s, ts in (("full", "p", "abc"), ("half", "p", "ab"), ("running", "p", "abc"), ("full", "o", "a"))
+                for t in ts]
+        self.assertEqual(compare.complete_models(rows, "p", ["a", "b", "c"]), ["full"])   # half misses c; running is ◐
+        self.assertEqual(compare.complete_models(rows, "p", ["a", "b"]), ["full", "half"])
+        self.assertEqual(compare.complete_models(rows, "p", ["a", "b", "c", "d"]), [])     # a question added since
+        self.assertEqual(compare.complete_models(rows, "p"), ["full"])                     # pack gone: questions seen
+        self.assertEqual(compare.complete_models(rows, "x", []), [])
+
+    def test_results_pack_picks_the_models_that_answered_all_of_it(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            for name, ids in (("apps", "abcd"), ("other", "ab")):
+                d = os.path.join(ws, "packs", name)
+                os.makedirs(d)
+                write(os.path.join(d, "pack.toml"), f'label = "{name.title()}"\n[certify]\nrepeat = 1\n')
+                write(os.path.join(d, "tests.yaml"), "".join(
+                    f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+                    "difficulty: easy}\n" for t in ids))
+            mock = Mock(os.path.join(ws, "packs"), "oracle")
+            try:
+                with open(os.path.join(ws, "models.toml"), "a") as f:
+                    f.write(f'\n[servers.here]\nurl = "http://127.0.0.1:{mock.port}"\n')
+                for label in ("full", "half", "solo"):
+                    tuieval(ws, "add", "mock", "--server", "here", "--label", label)
+                tuieval(ws, "run", "--tier", "certify", "--only", "full")
+                tuieval(ws, "run", "--tier", "certify", "--only", "half", "--packs", "apps", "--tests", "a,b")
+                tuieval(ws, "run", "--tier", "certify", "--only", "solo", "--packs", "other")
+            finally:
+                mock.stop()
+            code = textwrap.dedent("""
+                import asyncio
+                from tuieval.tui import EvalsApp, ResultsScreen
+                from textual.widgets import Button, DataTable, Select
+                async def go():
+                    app = EvalsApp({})
+                    async with app.run_test(size=(180, 50)) as pilot:
+                        await pilot.pause()
+                        app.pq_models = {"half"}                      # picked by hand before
+                        app.push_screen(ResultsScreen())
+                        await pilot.pause(0.5)
+                        s = app.screen
+                        def state():
+                            pq = s.query_one("#per-question", DataTable)
+                            cols = [str(c.label) for c in pq.ordered_columns][len(s.PQ_FIXED):]
+                            return (sorted(app.pq_models), s.query_one("#pta-pack", Select).value,
+                                    str(s.query_one("#pq-models", Button).label), cols, s.query_one("#pq-shared").value)
+                        s.query_one("#pq-pack", Select).value = "apps"
+                        await pilot.pause(0.3)
+                        print(state())
+                        s.query_one("#pta-pack", Select).value = "other"
+                        await pilot.pause(0.3)
+                        print(state())
+                        s.query_one("#pta-pack", Select).value = ""
+                        await pilot.pause(0.3)
+                        print(state())
+                        app.pq_models = set()                         # none picked: All packs gives every model again
+                        s.set_models(set())
+                        await pilot.pause(0.3)
+                        for pack in ("apps", ""):
+                            s.query_one("#pq-pack", Select).value = pack
+                            await pilot.pause(0.3)
+                            print(state())
+                asyncio.run(go())
+            """)
+            out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                                 env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+            self.assertEqual(out.stdout.strip().splitlines(), [
+                "(['full'], 'apps', 'Models: 1 complete on Apps', ['full'], False)",
+                "(['full', 'solo'], 'other', 'Models: 2 complete on Other', ['full', 'solo'], False)",
+                "(['half'], '', 'Models: 1 picked', ['half'], False)",
+                "(['full'], 'apps', 'Models: 1 complete on Apps', ['full'], True)",     # none picked before: ticked
+                "([], '', 'Models: all', ['full', 'half', 'solo'], False)"], out.stderr[-2000:])
+
     def test_cli_report_and_tui(self):
         with tempfile.TemporaryDirectory() as tmp:
             ws = os.path.join(tmp, "ws")