tuieval 0.2.0.dev75__tar.gz → 0.2.0.dev77__tar.gz

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- `tuieval compare --reference <model>`: every other model paired against one on the questions both answered, per pack and overall: the difference with a 95% CI, better/tie/worse, critical failures and tokens vs the reference. `--packs` limits any compare to some packs.
 - Fix: a question description or grader reason containing `[` no longer crashes the run screen.
 - Retest asks per pack before it starts: from zero, continue or leave out. It now really starts from zero, and replaced smoke results go to history/.
 - Retest: the setup checkbox "Rerun existing results" is now "Retest (ask again, even if already answered)", and `tuieval run --retest` is the same (`--force` still works).

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev75
+Version: 0.2.0.dev77
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -160,6 +160,7 @@ tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per m
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time
 tuieval compare --speed --pairwise --failures # scorecards
+tuieval compare --reference <model> --packs X # each model vs one: better, tie or worse
 tuieval selftest                              # check every test's reference and wrong answers
 tuieval items                                 # tests that don't separate models or look broken
 tuieval logs                                  # follow the server log of the model running now
@@ -179,6 +180,7 @@ tuieval help
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
 - **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
+- **Comparing variants of one model (fine-tunes, quantizations)?** `tuieval compare --reference <base>` pairs each against the base on the same questions and says better, tie or worse. Keep the server, quantization and settings the same, or they'll show up as differences too.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/README.md

@@ -134,6 +134,7 @@ tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per m
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time
 tuieval compare --speed --pairwise --failures # scorecards
+tuieval compare --reference <model> --packs X # each model vs one: better, tie or worse
 tuieval selftest                              # check every test's reference and wrong answers
 tuieval items                                 # tests that don't separate models or look broken
 tuieval logs                                  # follow the server log of the model running now
@@ -153,6 +154,7 @@ tuieval help
 - **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
 - **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
 - **Trust `tuieval compare --pairwise` over raw percentages.** Differences of one or two tests are usually noise.
+- **Comparing variants of one model (fine-tunes, quantizations)?** `tuieval compare --reference <base>` pairs each against the base on the same questions and says better, tie or worse. Keep the server, quantization and settings the same, or they'll show up as differences too.
 - **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
 - **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
 

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev75'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev75')
+__version__ = version = '0.2.0.dev77'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev77')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/src/tuieval/cli.py

@@ -13,7 +13,7 @@ Start here
   tuieval verdict             which model is ready for what, and what to run next
 
 Day to day
-  tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
+  tuieval compare [...]       scorecards (--speed, --pairwise, --reference M, --failures)
   tuieval pta                 the PTA index: parsimony (tokens), time and accuracy per model, as bars
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/src/tuieval/compare.py

@@ -5,6 +5,8 @@ Usage:
     tuieval compare --speed                  # speed & tokens table (TTFT, tok/s, tokens, memory)
     tuieval compare --machine m3max-64gb     # speed on another machine (measured there, or projected)
     tuieval compare --pairwise --failures    # is each difference real? which tests failed?
+    tuieval compare --reference MODEL        # every model paired against one: better, tie or worse
+    tuieval compare --packs coding,rag       # only these packs (works with every option)
     tuieval compare results/some-model/*.json    # specific files
 """
 import collections
@@ -409,6 +411,49 @@ def pairwise(rows, rng=None):
     return out
 
 
+def vs_reference(rows, ref, rng=None):
+    """Every other model against one reference model, paired on the questions both answered:
+    [{model, pack ("all" = every pack together), n, diff, lo, hi, verdict, crit, ref_crit, tokens}].
+
+    diff is the mean over shared questions of (pass rate - the reference's pass rate), with a 95%
+    bootstrap CI; verdict is "better" or "worse" when the CI excludes 0, else "tie". crit/ref_crit
+    count critical failures on those questions; tokens is the median of per-question token ratios
+    (median tokens over repeats, model / reference), so 0.5 means half the tokens."""
+    rng = rng or random.Random(0)
+    by = collections.defaultdict(lambda: collections.defaultdict(list))   # model -> test -> rows
+    suite = {}
+    for r in rows:
+        by[r["model"]][r["test"]].append(r)
+        suite[r["test"]] = r["suite"]
+    if ref not in by:
+        return []
+    out = []
+    for m in sorted(set(by) - {ref}):
+        shared = sorted(set(by[m]) & set(by[ref]))
+        for pack in sorted({suite[t] for t in shared}) + ["all"]:
+            tests = [t for t in shared if pack in ("all", suite[t])]
+            if not tests:
+                continue
+            rate = lambda model, t: statistics.fmean(r["ok"] for r in by[model][t])
+            diffs = [rate(m, t) - rate(ref, t) for t in tests]
+            lo, hi = bootstrap_ci(diffs, rng)
+            crit = lambda model: sum(r.get("severity") == "critical" for t in tests for r in by[model][t])
+            ratios = [a / b for t in tests
+                      if (a := median(r["tokens"] for r in by[m][t])) and (b := median(r["tokens"] for r in by[ref][t]))]
+            out.append({"model": m, "pack": pack, "n": len(tests), "diff": statistics.fmean(diffs), "lo": lo, "hi": hi,
+                        "verdict": "better" if lo > 0 else "worse" if hi < 0 else "tie",
+                        "crit": crit(m), "ref_crit": crit(ref), "tokens": median(ratios)})
+    return out
+
+
+def vs_reference_table(results, ref):
+    header = ["model", "pack", "questions", f"vs {ref}", "95% CI", "verdict", "critical (vs ref)", "tokens vs ref"]
+    table = [[x["model"], x["pack"], x["n"], f"{100 * x['diff']:+.1f} pts", f"{100 * x['lo']:+.1f} to {100 * x['hi']:+.1f}",
+              x["verdict"].upper() if x["pack"] == "all" else x["verdict"], f"{x['crit']} ({x['ref_crit']})",
+              _fmt(x["tokens"], ".2f") + ("×" if x["tokens"] is not None else "")] for x in results]
+    return header, table
+
+
 def failures(rows):
     return [(r["model"], r.get("name", r["test"]), ("TRUNCATED " if r["truncated"] and not r["reason"].startswith("TRUNCATED")
                                                     else "") + r["reason"], r.get("difficulty", "unrated"))
@@ -499,10 +544,22 @@ def main(argv):
     flags = {a for a in argv if a.startswith("--")}
     log_dir = argv[argv.index("--logs") + 1] if "--logs" in argv else None
     machine_arg = argv[argv.index("--machine") + 1] if "--machine" in argv else None
-    paths = [a for a in argv if not a.startswith("--") and a not in (log_dir, machine_arg)] or default_paths()
+    ref = argv[argv.index("--reference") + 1] if "--reference" in argv else None
+    only_packs = argv[argv.index("--packs") + 1].split(",") if "--packs" in argv else None
+    values = (log_dir, machine_arg, ref, ",".join(only_packs) if only_packs else None)
+    paths = [a for a in argv if not a.startswith("--") and a not in values] or default_paths()
+    if only_packs:
+        paths = [p for p in paths if os.path.basename(p)[:-len(".json")] in only_packs]
     if not paths:
         sys.exit(__doc__ + "\n(no results found in results/<model>/<pack>.json: run some evals first)")
     rows, infos = load(paths)
+    if ref:
+        if ref not in {r["model"] for r in rows}:
+            sys.exit(f"no results for reference model {ref!r}" + (f" in {', '.join(only_packs)}" if only_packs else ""))
+        print(f"Paired against {ref} on the questions both answered (pass rate per question, repeats averaged;"
+              "\n95% bootstrap CI over questions; BETTER/WORSE only when the CI excludes 0):")
+        print_table(*vs_reference_table(vs_reference(rows, ref, random.Random(0)), ref))
+        return
     rng = random.Random(0)
     from . import packs as packs_mod
     packs_mod.load_packs(errors=[])   # loads the workspace's graders, and their Scorecard columns

{tuieval-0.2.0.dev75 → tuieval-0.2.0.dev77}/tests/test_tuieval.py

@@ -209,6 +209,29 @@ class Units(unittest.TestCase):
         self.assertEqual(compare.settings_notes(infos),
                          ["Settings differ: effort medium on 4 models; penalty 1.5 on j."])
 
+    def test_vs_reference_pairs_on_shared_questions(self):
+        from tuieval import compare
+        row = lambda m, pack, t, ok, tokens=100, sev=None, rep=0: {
+            "model": m, "suite": pack, "test": f"{pack}: {t}", "ok": ok, "tokens": tokens, "severity": sev, "repeat": rep}
+        rows = [row("ref", "a", f"q{i}", i < 10) for i in range(20)]                       # ref: 10/20 on a
+        rows += [row("good", "a", f"q{i}", True, tokens=50) for i in range(20)]            # good: 20/20, half the tokens
+        rows += [row("same", "a", f"q{i}", i < 10) for i in range(20)]
+        rows += [row("bad", "a", f"q{i}", False, sev="critical" if i == 0 else None) for i in range(20)]
+        rows += [row("ref", "b", "x", True), row("good", "b", "x", False), row("good", "b", "x", True, rep=1)]
+        rows += [row("good", "b", "only-good", True)]                                      # not shared: ignored
+        res = {(x["model"], x["pack"]): x for x in compare.vs_reference(rows, "ref")}
+        self.assertEqual(res["good", "a"]["verdict"], "better")
+        self.assertAlmostEqual(res["good", "a"]["diff"], 0.5)
+        self.assertEqual(res["good", "a"]["tokens"], 0.5)
+        self.assertEqual(res["same", "a"]["verdict"], "tie")
+        self.assertEqual((res["bad", "a"]["verdict"], res["bad", "a"]["crit"], res["bad", "a"]["ref_crit"]), ("worse", 1, 0))
+        self.assertEqual(res["good", "b"]["n"], 1)
+        self.assertAlmostEqual(res["good", "b"]["diff"], -0.5)                             # repeats averaged
+        self.assertEqual(res["good", "all"]["n"], 21)
+        self.assertEqual(compare.vs_reference(rows, "nobody"), [])
+        header, table = compare.vs_reference_table([res["good", "all"]], "ref")
+        self.assertEqual(table[0][5], "BETTER")
+
     def test_eta_goes_pack_by_pack(self):
         from tuieval.tui import RunScreen
         job = lambda key, label, total, done=0, status="waiting": types.SimpleNamespace(