tuieval 0.2.0.dev131__tar.gz → 0.2.0.dev133__tar.gz

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev131
+Version: 0.2.0.dev133
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/docs/models.md

@@ -106,6 +106,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 - **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev131'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev131')
+__version__ = version = '0.2.0.dev133'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev133')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/src/tuieval/compare.py

@@ -248,12 +248,14 @@ def scorecard(rows, rng=None):
     suites = sorted({r["suite"] for r in rows})
     extra = dict(graders.COLUMNS)   # columns your graders add (graders.scorecard_column)
     over = any(r.get("over_budget") for r in rows)   # a column only when some answer ran past max_tokens
-    header = ["model"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
+    header = ["model", "repeats"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
                                    *(["over budget"] if over else []), "rep"]
     table = []
     for m in models:
         mine = [r for r in rows if r["model"] == m]
-        line = [m]
+        per_test = collections.Counter(r["test"] for r in mine)   # answers per question: the repeats
+        few, most = min(per_test.values()), max(per_test.values())
+        line = [m, f"×{few}" if few == most else f"×{few}-{most}"]
         for s in suites:
             sub = [r for r in mine if r["suite"] == s]
             line.append(f"{sum(r['ok'] for r in sub)}/{len(sub)}" if sub else "-")
@@ -678,7 +680,8 @@ def main(argv):
     for note in settings_notes(infos):
         print("NOTE:", note)
     print_table(*scorecard(rows, rng))
-    print("\nmean score = average score, with partial credit; all repeats = tests passed on every repeat.")
+    print("\nrepeats = answers per question (×1: one answer each, the roughest accuracy).")
+    print("mean score = average score, with partial credit; all repeats = tests passed on every repeat.")
     print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern.")
     header, table = by_difficulty(rows)
     if any(r.get("difficulty") in LEVELS for r in rows):

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/src/tuieval/engine.py

@@ -989,6 +989,14 @@ class Engine:
         need = {(t["id"], r) for t in pack.tests for r in range(self.certify_repeat(pack))}
         return "certified" if need <= keys else "screened"
 
+    def result_repeats(self, label, pack_name, tier="certify"):
+        """How many times each answered question of a stored result was asked: (fewest, most), or None."""
+        data = self._result(self.result_path(label, pack_name, tier))
+        if not data or not data["results"]:
+            return None
+        per_test = collections.Counter(r["test"] for r in data["results"])
+        return min(per_test.values()), max(per_test.values())
+
     def _partial_lines(self, path):
         """(header, answers) of an unfinished run's .partial.jsonl, cached like _result;
         (None, []) if there is none."""

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/src/tuieval/pta.py

@@ -15,6 +15,9 @@ the others in proportion, at least one block), so shorter is better; the A bar:
 P and T also carry a 0-100 score (100 for the best, POINTS_PER_DOUBLING points less for every
 doubling), used to order models with the same accuracy.
 
+Beside each name, the repeats: how many times it answered each of those questions ("×3", or "×1-3" when
+it varies). A ×1 model's accuracy rests on one answer per question, so it is rougher than a ×3 one's.
+
 Models are compared on the questions all of them answered, so one that skipped some never looks
 faster. Server and connection errors aren't answers. By default a model with fewer than half as many
 answers as the most-answered model is left out (listed, so it can be picked), or it would shrink the
@@ -54,7 +57,7 @@ SORTS = ("accuracy", "score")
 
 def index(rows, models=None, memory=None, sort="accuracy"):
     """{"questions": n shared, "models": [{model, P, T, A, S, cut, cut_pct, tokens, total_s, passed, answers, alongside,
-    fewest, fastest, tps, ttft, memory_gb}],
+    fewest, fastest, tps, ttft, memory_gb, repeats: (fewest, most) answers per question}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
     most-answered model}, best accuracy (sort="accuracy") or score (sort="score") first. S: the mean
 score (partial credit) in %. models: the ones to compare (default: every model in rows
@@ -92,6 +95,7 @@ score (partial credit) in %. models: the ones to compare (default: every model i
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
                     "memory_gb": (memory or {}).get(m),
+                    "repeats": (min(c["n"] for c in cells), max(c["n"] for c in cells)) if cells else None,
                     # answers that ran while other models were served shared the machine's speed
                     "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
     fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
@@ -116,6 +120,14 @@ LETTERS = (("P", "parsimony (tokens)", "▼ shorter is better", "tokens", "token
            ("A", "accuracy", "▲ longer is better", "A", "A", "green"))
 
 
+def repeats(x):
+    """How many times a model answered each question compared: "×3", "×1-3", or "" when unknown."""
+    r = x.get("repeats")
+    if not r:
+        return ""
+    return f"×{r[0]}" if r[0] == r[1] else f"×{r[0]}-{r[1]}"
+
+
 def times(v):
     """A multiple of the best: "1.0×", "2.4×", "28×"."""
     return "-" if v is None else f"{v:.1f}×" if v < 9.95 else f"{v:.0f}×"
@@ -138,6 +150,8 @@ def bars(result, width=20, markup=True, click=None):
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
+    rep_w = max(len(repeats(x)) for x in result["models"])
+    name_w += rep_w + 1 if rep_w else 0   # the repeats sit after the name, in its column
     # each column: the bar, a space, "100%" or "9.8×" (6), then a gap; at least as wide as its heading
     cell_w = max(width + 9, *(len(t) + 2 for k, name, better, *_ in LETTERS for t in (f"{k} {name}", better)))
     pad = " " * (name_w + 2)
@@ -173,11 +187,14 @@ def bars(result, width=20, markup=True, click=None):
             bar = (f"[{color}]{bar}[/]" + (f"[#ff8c00]{'█' * part}[/]" if part else "") + f"[grey37]{rest}[/]"
                    if markup else bar + cut_bar + rest)
             cells.append(f"{bar} {num:>6}" + " " * max(0, cell_w - width - 1 - max(6, len(num))))
-        name = f"{x['model']:<{name_w}}"
+        tag = repeats(x)
+        name = f"{x['model'] + (' ' + tag if tag else ''):<{name_w}}"
         if markup:
-            pad_w = name_w - len(x["model"])
-            name = (f"[@click={click}({row})]{rich_escape(x['model'])}[/]" + " " * pad_w if click
-                    else rich_escape(name))
+            pad_w = name_w - len(x["model"]) - (len(tag) + 1 if tag else 0)
+            # a single answer per question in yellow: its accuracy is the roughest
+            tag = f" [{'yellow' if x['repeats'][1] == 1 else 'dim'}]{tag}[/]" if tag else ""
+            name = ((f"[@click={click}({row})]{rich_escape(x['model'])}[/]" if click else rich_escape(x["model"]))
+                    + tag + " " * pad_w)
         lines.append((f"{name}  " + "".join(cells)).rstrip())
     return lines
 
@@ -202,9 +219,9 @@ def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
     over = any(x.get("over") for x in result["models"])   # a column only when some answer ran past max_tokens
-    header = ["model", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
+    header = ["model", "repeats", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
               "tok/s", "TTFT s", "memory GB"]
-    rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
+    rows = [[x["model"], repeats(x) or "-", f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
              f"{x['passed']}/{x['answers']}", "-" if x.get("S") is None else compare.pct(x["S"] / 100),
              f"{x.get('cut') or 0}/{x['answers']}",
@@ -243,6 +260,10 @@ def scope_note(result, models):
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
             "(longer is better); score: partial credit, e.g. 5 of 6 checks passed is 83% (shown when it differs)"
+            + (f"; ×n beside a name: answers per question. ×1 models answered each question once, so one "
+               f"answer more or less right moves their accuracy by {100 / result['questions']:.0f} points; "
+               "repeats steady it" if len({x.get("repeats") for x in result["models"]}) > 1
+               or any(x.get("repeats") and x["repeats"][1] == 1 for x in result["models"]) else "")
             + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
                "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else ".")
             + (" Over budget: answers that used more tokens than the max_tokens sent, because the provider didn't "

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/src/tuieval/tui.py

@@ -937,7 +937,10 @@ class SetupScreen(Screen):
         for n in e.packs:
             st = e.result_status(label, n)
             if st:
-                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}")
+                reps = e.result_repeats(label, n) if st in ("certified", "screened") else None
+                times = ("" if not reps else f"[dim]×{reps[0]}[/dim]" if reps[0] == reps[1]
+                         else f"[dim]×{reps[0]}-{reps[1]}[/dim]")
+                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}{times}")
                 if st.startswith("outdated"):
                     why.setdefault(st.removeprefix("outdated: "), []).append(n)
         if why:  # each reason once, not once per pack
@@ -951,7 +954,7 @@ class SetupScreen(Screen):
         hidden = (f"  [cyan]{' '.join(tags)}[/cyan]" if tags else "") + hidden
         detail.update(f"[b]{label}[/b]  [dim]{m['server']} · {m.get('model') or ''}[/dim]"
                       f"{serving_badge(e, m, not e.has_results())}{hidden}\n{verdicts}\n"
-                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial):[/dim] " + "  ".join(packs)
+                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial; ×n answers per question):[/dim] " + "  ".join(packs)
                          if packs else "[dim]no results yet[/dim]"))
 
     @on(Checkbox.Changed, "#show-hidden")

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev133}/tests/test_tuieval.py

@@ -502,6 +502,9 @@ class PickTests(unittest.TestCase):
         self.assertIn("done earlier", out)                                       # nothing missing: not re-asked
         tuieval(self.ws, "run", "--tests", "apps:d", "--tier", "certify")
         self.assertEqual(self.status(), "certified")
+        code = "from tuieval import engine; print(engine.Engine().result_repeats('m', 'apps'))"
+        self.assertEqual(subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,   # ✓×1 on Setup
+                                        env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip(), "(1, 1)")
 
         self.server("wrong")
         tuieval(self.ws, "run", "--tests", "apps:b", "--force")
@@ -1555,7 +1558,7 @@ class PTAIndex(unittest.TestCase):
     scale, and accuracy, all over the questions every compared model answered."""
 
     def test_index_compares_shared_questions(self):
-        from tuieval import pta
+        from tuieval import compare, pta
 
         def row(model, test, ok, secs, repeat=0):
             return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": secs * 1000,
@@ -1574,9 +1577,9 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((by["slow"]["P"], by["fast"]["P"]), (100, 70))          # 4x the tokens: two doublings
         self.assertEqual((pta.score(2, 1), pta.score(1000, 1)), (85, 0))
         header, table = pta.table(res)
-        self.assertEqual(header[1:4], ["tokens/answer", "time/answer", "answers right"])
+        self.assertEqual(header[1:5], ["repeats", "tokens/answer", "time/answer", "answers right"])
         cells = {r[0]: r for r in table}
-        self.assertEqual((cells["slow"][1], cells["fast"][1:3]), ("10 ★", ["40", "2.0s ★"]))   # per answer
+        self.assertEqual((cells["slow"][2], cells["fast"][2:4]), ("10 ★", ["40", "2.0s ★"]))   # per answer
         bars = pta.bars(res, 10, markup=False)
         self.assertEqual(bars[0].split(), ["P", "parsimony", "(tokens)", "T", "time", "A", "accuracy"])
         self.assertEqual(bars[1].split(), ["▼", "shorter", "is", "better"] * 2 + ["▲", "longer", "is", "better"])
@@ -1585,8 +1588,13 @@ class PTAIndex(unittest.TestCase):
                          (fast.index("██████████"), fast.index("███░░░░░░░")))       # its heading
         # P and T bars as long as tokens and time (shorter is better): fast used 4x the tokens of slow but
         # took 4 s against 13 s, so it has the longest token bar and the shortest time bar
-        self.assertEqual(fast.split(), ["fast", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
-        self.assertEqual(slow.split()[1:5], ["██░░░░░░░░", "1.0×", "██████████", "3.2×"])
+        self.assertEqual(fast.split(), ["fast", "×1", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
+        self.assertEqual(slow.split()[1:6], ["×1-2", "██░░░░░░░░", "1.0×", "██████████", "3.2×"])   # ×: answers per question
+        self.assertEqual((cells["slow"][1], cells["fast"][1]), ("×1-2", "×1"))
+        self.assertIn("×1 models answered each question once", pta.scope_note(res, ["slow", "fast"]))
+        full = [dict(r, score=float(r["ok"]), error=False, output="") for r in rows]
+        header, table = compare.scorecard(full)
+        self.assertEqual({r[0]: r[header.index("repeats")] for r in table}, {"slow": "×1-2", "fast": "×1"})
         self.assertEqual((pta.times(1), pta.times(9.84), pta.times(27.9)), ("1.0×", "9.8×", "28×"))
         self.assertEqual((by["fast"]["A"], by["slow"]["A"]), (50, 100))
         self.assertEqual([x["model"] for x in res["models"]], ["slow", "fast"])   # best accuracy first
@@ -1618,8 +1626,8 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # none truncated: nothing beside it
         self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
         header, table = pta.table(res)
-        self.assertEqual(header[4:6], ["score", "truncated"])
-        self.assertEqual({r[0]: r[5] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertEqual(header[5:7], ["score", "truncated"])
+        self.assertEqual({r[0]: r[6] for r in table}, {"thinker": "2/10", "plain": "0/10"})
         self.assertIn("truncated", pta.scope_note(res, ["thinker", "plain"]))
         self.assertNotIn("truncated", pta.scope_note(pta.index(rows[10:]), ["plain"]))
 
@@ -1949,7 +1957,7 @@ class PTAIndex(unittest.TestCase):
             self.assertGreaterEqual(len(got["opened"]), 2)
             self.assertEqual(got["opened"], {m: m for m in got["opened"]})   # a cell opens its column's model
             self.assertEqual(got["tab"], "tab-pta")                  # Results opens on the PTA index
-            self.assertEqual([(r[0], r[3]) for r in got["rows"]][0], ("good", "4/4"))
+            self.assertEqual([(r[0], r[4]) for r in got["rows"]][0], ("good", "4/4"))
             s = got["sorts"]   # a click on a column name sorts by it, a second click reverses
             self.assertEqual((s["up"], s["down"], s["kept"]), (["hosted", "good"], ["good", "hosted"], ["good", "hosted"]))
             self.assertIn("answers right ▼", s["labels"])