tuieval 0.2.0.dev74__tar.gz → 0.2.0.dev77__tar.gz

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- `tuieval compare --reference <model>`: every other model paired against one on the questions both answered, per pack and overall: the difference with a 95% CI, better/tie/worse, critical failures and tokens vs the reference. `--packs` limits any compare to some packs.
+- Fix: a question description or grader reason containing `[` no longer crashes the run screen.
 - Retest asks per pack before it starts: from zero, continue or leave out. It now really starts from zero, and replaced smoke results go to history/.
 - Retest: the setup checkbox "Rerun existing results" is now "Retest (ask again, even if already answered)", and `tuieval run --retest` is the same (`--force` still works).
 - `tuieval logs` follows the server log of the model running now and moves on with the run; `logs live` shows each answer as it finishes, `--list` every log.

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev74
+Version: 0.2.0.dev77
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -160,6 +160,7 @@ tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per m
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time
 tuieval compare --speed --pairwise --failures # scorecards
+tuieval compare --reference <model> --packs X # each model vs one: better, tie or worse
 tuieval selftest                              # check every test's reference and wrong answers
 tuieval items                                 # tests that don't separate models or look broken
 tuieval logs                                  # follow the server log of the model running now
@@ -179,6 +180,7 @@ tuieval help
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
 - **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
+- **Comparing variants of one model (fine-tunes, quantizations)?** `tuieval compare --reference <base>` pairs each against the base on the same questions and says better, tie or worse. Keep the server, quantization and settings the same, or they'll show up as differences too.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/README.md

@@ -134,6 +134,7 @@ tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per m
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time
 tuieval compare --speed --pairwise --failures # scorecards
+tuieval compare --reference <model> --packs X # each model vs one: better, tie or worse
 tuieval selftest                              # check every test's reference and wrong answers
 tuieval items                                 # tests that don't separate models or look broken
 tuieval logs                                  # follow the server log of the model running now
@@ -153,6 +154,7 @@ tuieval help
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
 - **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
+- **Comparing variants of one model (fine-tunes, quantizations)?** `tuieval compare --reference <base>` pairs each against the base on the same questions and says better, tie or worse. Keep the server, quantization and settings the same, or they'll show up as differences too.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev74'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev74')
+__version__ = version = '0.2.0.dev77'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev77')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/src/tuieval/cli.py

@@ -13,7 +13,7 @@ Start here
   tuieval verdict             which model is ready for what, and what to run next
 
 Day to day
-  tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
+  tuieval compare [...]       scorecards (--speed, --pairwise, --reference M, --failures)
   tuieval pta                 the PTA index: parsimony (tokens), time and accuracy per model, as bars
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/src/tuieval/compare.py

@@ -5,6 +5,8 @@ Usage:
     tuieval compare --speed                  # speed & tokens table (TTFT, tok/s, tokens, memory)
     tuieval compare --machine m3max-64gb     # speed on another machine (measured there, or projected)
     tuieval compare --pairwise --failures    # is each difference real? which tests failed?
+    tuieval compare --reference MODEL        # every model paired against one: better, tie or worse
+    tuieval compare --packs coding,rag       # only these packs (works with every option)
     tuieval compare results/some-model/*.json    # specific files
 """
 import collections
@@ -409,6 +411,49 @@ def pairwise(rows, rng=None):
     return out
 
 
+def vs_reference(rows, ref, rng=None):
+    """Every other model against one reference model, paired on the questions both answered:
+    [{model, pack ("all" = every pack together), n, diff, lo, hi, verdict, crit, ref_crit, tokens}].
+
+    diff is the mean over shared questions of (pass rate - the reference's pass rate), with a 95%
+    bootstrap CI; verdict is "better" or "worse" when the CI excludes 0, else "tie". crit/ref_crit
+    count critical failures on those questions; tokens is the median of per-question token ratios
+    (median tokens over repeats, model / reference), so 0.5 means half the tokens."""
+    rng = rng or random.Random(0)
+    by = collections.defaultdict(lambda: collections.defaultdict(list))   # model -> test -> rows
+    suite = {}
+    for r in rows:
+        by[r["model"]][r["test"]].append(r)
+        suite[r["test"]] = r["suite"]
+    if ref not in by:
+        return []
+    out = []
+    for m in sorted(set(by) - {ref}):
+        shared = sorted(set(by[m]) & set(by[ref]))
+        for pack in sorted({suite[t] for t in shared}) + ["all"]:
+            tests = [t for t in shared if pack in ("all", suite[t])]
+            if not tests:
+                continue
+            rate = lambda model, t: statistics.fmean(r["ok"] for r in by[model][t])
+            diffs = [rate(m, t) - rate(ref, t) for t in tests]
+            lo, hi = bootstrap_ci(diffs, rng)
+            crit = lambda model: sum(r.get("severity") == "critical" for t in tests for r in by[model][t])
+            ratios = [a / b for t in tests
+                      if (a := median(r["tokens"] for r in by[m][t])) and (b := median(r["tokens"] for r in by[ref][t]))]
+            out.append({"model": m, "pack": pack, "n": len(tests), "diff": statistics.fmean(diffs), "lo": lo, "hi": hi,
+                        "verdict": "better" if lo > 0 else "worse" if hi < 0 else "tie",
+                        "crit": crit(m), "ref_crit": crit(ref), "tokens": median(ratios)})
+    return out
+
+
+def vs_reference_table(results, ref):
+    header = ["model", "pack", "questions", f"vs {ref}", "95% CI", "verdict", "critical (vs ref)", "tokens vs ref"]
+    table = [[x["model"], x["pack"], x["n"], f"{100 * x['diff']:+.1f} pts", f"{100 * x['lo']:+.1f} to {100 * x['hi']:+.1f}",
+              x["verdict"].upper() if x["pack"] == "all" else x["verdict"], f"{x['crit']} ({x['ref_crit']})",
+              _fmt(x["tokens"], ".2f") + ("×" if x["tokens"] is not None else "")] for x in results]
+    return header, table
+
+
 def failures(rows):
     return [(r["model"], r.get("name", r["test"]), ("TRUNCATED " if r["truncated"] and not r["reason"].startswith("TRUNCATED")
                                                     else "") + r["reason"], r.get("difficulty", "unrated"))
@@ -499,10 +544,22 @@ def main(argv):
     flags = {a for a in argv if a.startswith("--")}
     log_dir = argv[argv.index("--logs") + 1] if "--logs" in argv else None
     machine_arg = argv[argv.index("--machine") + 1] if "--machine" in argv else None
-    paths = [a for a in argv if not a.startswith("--") and a not in (log_dir, machine_arg)] or default_paths()
+    ref = argv[argv.index("--reference") + 1] if "--reference" in argv else None
+    only_packs = argv[argv.index("--packs") + 1].split(",") if "--packs" in argv else None
+    values = (log_dir, machine_arg, ref, ",".join(only_packs) if only_packs else None)
+    paths = [a for a in argv if not a.startswith("--") and a not in values] or default_paths()
+    if only_packs:
+        paths = [p for p in paths if os.path.basename(p)[:-len(".json")] in only_packs]
     if not paths:
         sys.exit(__doc__ + "\n(no results found in results/<model>/<pack>.json: run some evals first)")
     rows, infos = load(paths)
+    if ref:
+        if ref not in {r["model"] for r in rows}:
+            sys.exit(f"no results for reference model {ref!r}" + (f" in {', '.join(only_packs)}" if only_packs else ""))
+        print(f"Paired against {ref} on the questions both answered (pass rate per question, repeats averaged;"
+              "\n95% bootstrap CI over questions; BETTER/WORSE only when the CI excludes 0):")
+        print_table(*vs_reference_table(vs_reference(rows, ref, random.Random(0)), ref))
+        return
     rng = random.Random(0)
     from . import packs as packs_mod
     packs_mod.load_packs(errors=[])   # loads the workspace's graders, and their Scorecard columns

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/src/tuieval/tui.py

@@ -2239,9 +2239,10 @@ class RunScreen(Screen):
             self.req_started, self.req_tokens, self.req_first = time.time(), 0, None
             job = d["job"]
             rep_note = f" · repeat {d['repeat'] + 1}/{job.repeat}" if job.repeat > 1 else ""
-            self.current = (f"[b]{job.label}[/b] · {job.pack.label} · test {job.done + 1}/{job.total}{rep_note}"
+            self.current = (f"[b]{rich_escape(job.label)}[/b] · {rich_escape(job.pack.label)} · test "
+                            f"{job.done + 1}/{job.total}{rep_note}"
                             f"{'  [magenta]🖼 image[/magenta]' if d['has_image'] else ''}\n"
-                            f"{diff_badge(d.get('difficulty', 'unrated'))}  {d['test']}")
+                            f"{diff_badge(d.get('difficulty', 'unrated'))}  {rich_escape(d['test'])}")
         elif kind == "request_done":
             if self.watched(kind, d):
                 self.flush_streams()
@@ -2251,12 +2252,12 @@ class RunScreen(Screen):
             # follow new rows unless you're browsing an older one
             at_end = not recent.has_focus or recent.cursor_row >= recent.row_count - 1
             calls = f"  [cyan]→ {r['tool_calls'][0]['name']}()[/cyan]" if r.get("tool_calls") else ""
-            row_key = recent.add_row(mark, diff_badge(r.get("difficulty", "unrated")), r["description"][:50], job.label,
+            row_key = recent.add_row(mark, diff_badge(r.get("difficulty", "unrated")), rich_escape(r["description"][:50]), job.label,
                            str(r.get("completion_tokens") or ""),
                            (f"{r['ttft_s']:.1f}" + (" ⟲" if r.get("cached_tokens") else ""))
                            if r.get("ttft_s") is not None else "",
                            f"{r['gen_tps']:.0f}" if r.get("gen_tps") else "", f"{r['total_s']:.0f}",
-                           r["reason"][:90] + calls
+                           rich_escape(r["reason"][:90]) + calls
                            + (f"  [red]copy of repeat {r['identical_to_repeat'] + 1}[/red]"
                               if r.get("identical_to_repeat") is not None else ""))
             self.stats.append(r)
@@ -2364,7 +2365,7 @@ class RunScreen(Screen):
             self.event("[bold yellow]REUSE[/bold yellow] " + msg)
             self.notify(msg, severity="warning", timeout=20)
         elif kind == "identical_answer":
-            msg = (f"{d['label']} · {d['pack']} · {d['test']}: repeat {d['repeat'] + 1} is an exact copy of repeat "
+            msg = (f"{d['label']} · {d['pack']} · {rich_escape(d['test'])}: repeat {d['repeat'] + 1} is an exact copy of repeat "
                    f"{d['same'] + 1}, so the server may be returning cached answers rather than fresh ones.")
             self.event("[bold red]COPY[/bold red] " + msg)
             self.notify(msg, severity="error", timeout=20)
@@ -2711,7 +2712,7 @@ class ResultsScreen(Screen):
         # compare.failures lists the failed rows in order, so they line up with fail_rows
         self.fail_rows = [r for r in rows if not r["ok"]]
         for i, (model, test, reason, level) in enumerate(compare.failures(rows)):
-            f.add_row(model, diff_badge(level), test, rich_escape(reason[:130]), key=str(i))
+            f.add_row(model, diff_badge(level), rich_escape(test), rich_escape(reason[:130]), key=str(i))
         self.fill_per_question()
         self.fill_pta()
         self.fill_readiness(results_dir)

{tuieval-0.2.0.dev74 → tuieval-0.2.0.dev77}/tests/test_tuieval.py

@@ -209,6 +209,29 @@ class Units(unittest.TestCase):
         self.assertEqual(compare.settings_notes(infos),
                          ["Settings differ: effort medium on 4 models; penalty 1.5 on j."])
 
+    def test_vs_reference_pairs_on_shared_questions(self):
+        from tuieval import compare
+        row = lambda m, pack, t, ok, tokens=100, sev=None, rep=0: {
+            "model": m, "suite": pack, "test": f"{pack}: {t}", "ok": ok, "tokens": tokens, "severity": sev, "repeat": rep}
+        rows = [row("ref", "a", f"q{i}", i < 10) for i in range(20)]                       # ref: 10/20 on a
+        rows += [row("good", "a", f"q{i}", True, tokens=50) for i in range(20)]            # good: 20/20, half the tokens
+        rows += [row("same", "a", f"q{i}", i < 10) for i in range(20)]
+        rows += [row("bad", "a", f"q{i}", False, sev="critical" if i == 0 else None) for i in range(20)]
+        rows += [row("ref", "b", "x", True), row("good", "b", "x", False), row("good", "b", "x", True, rep=1)]
+        rows += [row("good", "b", "only-good", True)]                                      # not shared: ignored
+        res = {(x["model"], x["pack"]): x for x in compare.vs_reference(rows, "ref")}
+        self.assertEqual(res["good", "a"]["verdict"], "better")
+        self.assertAlmostEqual(res["good", "a"]["diff"], 0.5)
+        self.assertEqual(res["good", "a"]["tokens"], 0.5)
+        self.assertEqual(res["same", "a"]["verdict"], "tie")
+        self.assertEqual((res["bad", "a"]["verdict"], res["bad", "a"]["crit"], res["bad", "a"]["ref_crit"]), ("worse", 1, 0))
+        self.assertEqual(res["good", "b"]["n"], 1)
+        self.assertAlmostEqual(res["good", "b"]["diff"], -0.5)                             # repeats averaged
+        self.assertEqual(res["good", "all"]["n"], 21)
+        self.assertEqual(compare.vs_reference(rows, "nobody"), [])
+        header, table = compare.vs_reference_table([res["good", "all"]], "ref")
+        self.assertEqual(table[0][5], "BETTER")
+
     def test_eta_goes_pack_by_pack(self):
         from tuieval.tui import RunScreen
         job = lambda key, label, total, done=0, status="waiting": types.SimpleNamespace(
@@ -735,6 +758,56 @@ class Parallel(unittest.TestCase):
         self.assertEqual(sum(r["pass"] for r in self.results("b")["results"]), 6)
 
 
+class MarkupInText(unittest.TestCase):
+    """Brackets in a description or a grader's reason are shown as text: an unmatched
+    "[" once crashed the run screen (it swallowed the "[dim]" of the line after it)."""
+
+    def test_run_screen_shows_brackets(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            d = os.path.join(ws, "packs", "br")
+            os.makedirs(d)
+            write(os.path.join(d, "pack.toml"), 'label = "Brackets"\n[certify]\nrepeat = 1\n[gate]\nmin_accuracy = 0.1\n')
+            write(os.path.join(d, "tests.yaml"),
+                  "- {id: a, description: 'notional in [t - 10 s, t), excluding ties', input: q a, expected: 1, "
+                  "reference: 'ANSWER: 1', wrong: ['ANSWER: [2'], difficulty: hard}\n"
+                  "- {id: b, description: 'window [a, b) then [/dim] text', input: q b, expected: 1, "
+                  "reference: 'ANSWER: 1', wrong: ['ANSWER: [2'], difficulty: hard}\n")
+            mock = [sys.executable, os.path.join(HERE, "mock_server.py"), "--port", "{port}", "--packs",
+                    os.path.join(ws, "packs"), "--model", "{served_name}", "--mode", "wrong", "--delay", "0.2"]
+            with open(os.path.join(ws, "models.toml"), "a") as f:
+                f.write(f'\n[servers.mk]\ncmd = {json.dumps(mock)}\nport = {free_port()}\n'
+                        '\n[[models]]\nlabel = "m"\nserver = "mk"\nmodel = "m"\n')
+            code = textwrap.dedent("""
+                import asyncio, json, time
+                from tuieval.tui import EvalsApp, RunScreen
+                from textual.widgets import SelectionList
+                async def go():
+                    app = EvalsApp({})
+                    async with app.run_test(size=(160, 50)) as pilot:
+                        await pilot.pause()
+                        s = app.screen
+                        s.query_one("#suites", SelectionList).select("br")
+                        s.selected_models = {"m"}
+                        s.refresh_models()
+                        s.query_one("#tier-certify").value = True
+                        await pilot.pause()
+                        s.action_start()
+                        deadline = time.time() + 240
+                        while time.time() < deadline:
+                            await pilot.pause(0.2)
+                            if isinstance(app.screen, RunScreen) and not app.screen.running:
+                                break
+                        print(json.dumps([(j.key, j.status, j.done) for j in app.screen.jobs]))
+                asyncio.run(go())
+            """)
+            p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=480,
+                               env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""))
+            self.assertNotIn("MarkupError", p.stderr + p.stdout)
+            self.assertEqual(json.loads(p.stdout.strip().splitlines()[-1]), [["m/br", "done", 2]], p.stderr[-2000:])
+
+
 class AddToRun(unittest.TestCase):
     """More evals while a run is going: added to it (they run after its other jobs) or queued after it."""