tuieval 0.2.0.dev137__tar.gz → 0.2.0.dev141__tar.gz

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Per question → Comparison: a **Pass rate** column beside Passed, the share of answers right with repeats included (shown when some model has repeats).
+- Repeats (Setup, `tuieval run --repeat N`) add N rounds to the saved answers: ×1 run with 2 gives ×3, not ×2. Retest, a blank count and presets still mean the total.
 - Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
 - Repeats: the PTA index, Per question, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev137
+Version: 0.2.0.dev141
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/docs/models.md

@@ -106,7 +106,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 - **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
-- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`). To steady it, set **Repeats** in Setup (or `tuieval run --repeat N`): N adds that many rounds to the answers already saved, so a `×1` model run with 2 ends up `×3` (with Retest, N in all; left blank, the tier's default count in all). A preset's repeat count is in all too, until you type another.
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev137'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev137')
+__version__ = version = '0.2.0.dev141'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev141')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/src/tuieval/compare.py

@@ -420,7 +420,8 @@ def complete_models(rows, suite, test_ids=None):
 def compare_models(tests, models):
     """Per model over per_question entries that all `models` answered: passes, token and time
     totals/medians (totals add each question's median over repeats, so repeats don't inflate them), questions where it used the fewest tokens / was fastest (among passing
-    answers for tokens; ties count for each), answers cut off, mean score (partial credit)."""
+    answers for tokens; ties count for each), answers cut off, mean score (partial credit),
+    pass rate (answers right with repeats, each question weighing the same)."""
     out = []
     for m in models:
         cs = [t["cells"][m] for t in tests]
@@ -438,6 +439,7 @@ def compare_models(tests, models):
         rs = [r for c in cs for r in c["rows"]]
         out.append({"model": m, "n": len(cs), "passed": passed, "total_tokens": total,
                     "score": statistics.fmean(c["score"] for c in cs) if cs else None,
+                    "pass_rate": statistics.fmean(c["passed"] / c["n"] for c in cs) if cs else None,
                     "avg_pp_tps": _rate([(r.get("prompt_tokens"), r["prompt_tps"]) for r in rs]),
                     "avg_gen_tps": _rate([(r["tokens"], r["gen_tps"]) for r in rs]),
                     "med_tokens": median(c["tokens"] for c in cs), "med_secs": median(c["secs"] for c in cs),

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/src/tuieval/engine.py

@@ -443,7 +443,7 @@ def save_preset(path, name, labels, packs, repeat, tests=None):
 class Job:
     model: dict
     pack: packs_mod.Pack
-    repeat: int
+    repeat: int             # rounds each question has once it's done (earlier ones included)
     tests: list             # the tests this tier runs (or the ones picked)
     out_path: str
     sampling: dict
@@ -465,6 +465,7 @@ class Job:
     stop_early: bool = False  # stop once the pack's FAIL is certain (verdict.fail_certain)
     first: list = dataclasses.field(default_factory=list)   # test ids round 1 asks first (most failed)
     claimed: bool = False   # its model's server took it on: it can no longer be removed from the run
+    base: int = 0           # rounds already in its result file that a chosen repeat count adds to
 
     @property
     def label(self):
@@ -1148,6 +1149,17 @@ class Engine:
         redo = {t["id"] for t in job.tests}
         return [r for r in rows if r["test"] not in redo]
 
+    def saved_rounds(self, job):
+        """The most rounds any of the job's questions has in its finished result file (0 when the file
+        doesn't match the job's questions and settings). An unfinished run's answers don't count, so
+        resuming one doesn't add more rounds."""
+        data = self._result(job.out_path)
+        if not data or data["pack"]["fingerprint"] != job.pack.fingerprint or not settings_match(data, job.settings):
+            return 0
+        ids = {t["id"] for t in job.tests}
+        rounds = collections.Counter(t for t, _ in {(r["test"], r["repeat"]) for r in data["results"] if r["test"] in ids})
+        return max(rounds.values(), default=0)
+
     def _fill_progress(self, job):
         """Set a job's done/passed/failed and earlier sittings from what's stored."""
         kept, new, job.earlier = self._stored(job)
@@ -1212,10 +1224,13 @@ class Engine:
         return not current and header_matches(head, pack.fingerprint, settings)
 
     # ---- planning
-    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None):
+    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None,
+             total=False):
         """Jobs for models x packs. tier: smoke (3 tests), screen (a spread sample), certify (all
         tests). repeat: times each question is asked; None = the tier default (1 for smoke and
-        screen, the pack's certification repeats for certify). tests: {pack: [test ids]} runs just
+        screen, the pack's certification repeats for certify), counted in total; a number adds that
+        many rounds to the ones already in the result file (a retest starts from none), or with total
+        (a preset's count) is the rounds in all. tests: {pack: [test ids]} runs just
         those tests of a pack instead of the tier's sample (unknown ids raise PackError). Earlier
         matching results are kept and only missing answers are run, unless force; force with
         picked tests reruns those and keeps the pack's other answers. force is True, False, or the
@@ -1240,6 +1255,9 @@ class Engine:
                     job.first = self.most_failed_first(pack)
                 sv = self.serving(m)
                 status = self.result_status(label, name, tier)
+                if repeat and not total and job.merge and not job.redo and not (status or "").startswith("outdated"):
+                    job.base = self.saved_rounds(job)
+                    job.repeat = job.base + repeat
                 job.discards = self._discards(job)
                 missing = [n for n in pack.modules if not importlib.util.find_spec(n)]
                 no_agent = agents.problem(self.cfg, pack, self.root) if pack.kind == "agent" else None
@@ -1273,6 +1291,8 @@ class Engine:
                         job.note = f"{job.done} of {job.total} done earlier, running the rest"
                 if job.status != "waiting":
                     job.discards = []
+                elif job.base:
+                    job.note = "; ".join(filter(None, [job.note, f"×{job.base} → ×{job.repeat}"]))
                 for what, n, why in job.discards:
                     if what == "unfinished":   # (a finished file's reason is already the note)
                         job.note = "; ".join(filter(None, [job.note, f"{n} unfinished answers set aside ({why})"]))

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/src/tuieval/run_evals.py

@@ -101,7 +101,8 @@ class Printer:
             "so the server may be returning cached answers rather than fresh ones", RED)
 
     def on_job_started(self, job, resumed, earlier_s=0, sittings=0):
-        say(f"{job.key}: {job.count} tests x {job.repeat} repeats" + (f", resuming after {resumed}" if resumed else "")
+        reps = f"{job.repeat - job.base} more repeats (×{job.base} → ×{job.repeat})" if job.base else f"{job.repeat} repeats"
+        say(f"{job.key}: {job.count} tests x {reps}" + (f", resuming after {resumed}" if resumed else "")
             + (f" ({earlier_s / 60:.0f} min over {sittings} earlier sitting(s))" if sittings else ""))
 
     def on_request_started(self, job, test, test_id, has_image, repeat, difficulty="unrated"):
@@ -792,8 +793,9 @@ def main(argv=None):
     p.add_argument("--stop-early", action="store_true",
                    help="stop each pack once its FAIL is certain (a critical failure, or a bar out of reach), "
                         "asking first what models fail most (default: each pack's stop_early)")
-    p.add_argument("--repeat", type=int, help="times each question is asked, any tier (default: 1 for smoke and "
-                                                  "screen, the pack's certification count for certify)")
+    p.add_argument("--repeat", type=int, help="rounds to add: each question is asked N more times on top of its "
+                                                  "saved answers (with --retest, N in all). Default: up to 1 for "
+                                                  "smoke and screen, the pack's certification count for certify")
     p.add_argument("--tier", choices=engine.TIERS, default="screen",
                    help="smoke: 3 tests; screen (default): a spread sample; certify: everything, with repeats")
     p.add_argument("--smoke", action="store_const", const="smoke", dest="tier", help="same as --tier smoke")
@@ -838,12 +840,14 @@ def main(argv=None):
         sys.exit("no models to run yet: tuieval add <GGUF path or model id>, or name a hosted one with "
                  "--only openrouter:<model id>" if not models else
                  "no local models; hosted ones (they cost money) run only when named: --only <label>")
+    repeat_total = False
     if args.preset:
         presets = engine.load_presets(os.path.join(os.path.dirname(os.path.abspath(models_file(args.models))),
                                                    "presets.toml"))
         if args.preset not in presets:
             sys.exit(f"unknown preset {args.preset!r}; presets: {', '.join(presets) or 'none (save one in the TUI with p)'}")
         labels, pack_names, preset_repeat, tests = engine.resolve_preset(e.cfg, presets[args.preset], list(e.packs))
+        repeat_total = not args.repeat and bool(preset_repeat)   # a preset's count is the rounds in all
         args.repeat = args.repeat or preset_repeat
     if args.packs:
         pack_names = [x.strip() for x in args.packs.split(",")]
@@ -853,7 +857,8 @@ def main(argv=None):
     if args.tests:
         tests, pack_names = parse_tests(args.tests, pack_names if (args.packs or args.preset) else None, e.packs)
     try:
-        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None)
+        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None,
+                      repeat_total)
     except packs_mod.PackError as err:
         sys.exit(str(err))
 

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/src/tuieval/tui.py

@@ -1025,6 +1025,11 @@ class SetupScreen(Screen):
                 list(self.query_one("#suites", SelectionList).selected), repeat,
                 self.tier(), self.query_one("#force", Checkbox).value)
 
+    def repeat_total(self):
+        """Is the Repeats count the rounds in all? Yes while it's a loaded preset's; a typed one adds rounds."""
+        value = self.query_one("#repeat", Input).value
+        return bool(value) and value == getattr(self, "preset_repeat", None)
+
     def parallel(self):
         """Models at a time for the next run: the field, or blank for this machine's parallel_models."""
         try:
@@ -1088,16 +1093,21 @@ class SetupScreen(Screen):
                        "[b]s[/b] to start." + self.queue_hint())
             return
         picks = self.current_picks(suites)
-        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks)
+        total = self.repeat_total()
+        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks, total=total)
         rerun = not any(j.status == "waiting" for j in jobs)
         if rerun:   # everything has results: Start runs it again, so estimate that
-            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks)
+            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks, total=total)
         run = [j for j in jobs if j.status == "waiting"]
         skipped = [j for j in jobs if j.status == "skipped"]
         secs, notes = self.app.engine.estimate_seconds(jobs)
-        todo = sum(j.total for j in run)
-        reps = sorted({j.repeat for j in jobs})
-        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "")
+        todo = sum(j.total - j.done for j in run)
+        reps = sorted({j.repeat - j.base for j in jobs})
+        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "") \
+            + (" in all (preset)" if total else "")
+        bases = sorted({j.base for j in run if j.base})
+        if bases:   # a chosen count adds to the rounds already answered
+            reps_text += " more" + " [dim](" + ", ".join(f"×{b} → ×{b + repeat}" for b in bases) + ")[/dim]"
         text = (f"[b]{tier.title()}[/b]: [b]{len(labels)}[/b] model(s) × [b]{len(suites)}[/b] pack(s) × "
                 f"[b]{reps_text}[/b] repeat(s) = "
                 f"[b]{todo:,}[/b] answers · about [b]{fmt_secs(secs)}[/b]"
@@ -1158,6 +1168,7 @@ class SetupScreen(Screen):
         for name in sel.get("packs", []):
             if name in e.packs:
                 packs.select(name)
+        self.preset_repeat = str(sel["repeat"]) if sel.get("repeat") and sel.get("repeat_total") else None
         self.query_one("#repeat", Input).value = str(sel["repeat"]) if sel.get("repeat") else ""
         if "parallel" in sel:
             self.query_one("#parallel", Input).value = str(sel["parallel"]) if sel["parallel"] else ""
@@ -1201,10 +1212,11 @@ class SetupScreen(Screen):
             self.notify("Select at least one eval pack and one model.", severity="warning")
             return
         picks = self.current_picks(suites)
-        sel = {"models": labels, "packs": suites, "repeat": repeat, "tier": tier, "force": force, "tests": picks,
-               "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
+        total = self.repeat_total()
+        sel = {"models": labels, "packs": suites, "repeat": repeat, "repeat_total": total, "tier": tier, "force": force,
+               "tests": picks, "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
         e = self.app.engine
-        jobs = e.plan(labels, suites, repeat, tier, True, picks)
+        jobs = e.plan(labels, suites, repeat, tier, True, picks, total=total)
         if not any(j.status == "waiting" for j in jobs):
             reasons = sorted({j.note for j in jobs if j.status == "skipped"})
             self.notify(f"Nothing selected can run: {'; '.join(reasons)}", severity="warning")
@@ -1212,7 +1224,7 @@ class SetupScreen(Screen):
         self.app.save_state({k: v for k, v in sel.items() if k != "models"})   # models aren't remembered
         # Packs whose earlier answers a start would set aside (Retest, everything already done, or
         # other questions or settings): the user picks per pack before anything moves.
-        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force)
+        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force, total)
 
         def go(choices):
             if choices is None:
@@ -1640,7 +1652,7 @@ class PresetsScreen(ModalScreen):
         name = list(self.presets)[event.option_index]
         labels, packs, repeat, tests = engine.resolve_preset(self.app.engine.cfg, self.presets[name],
                                                              list(self.app.engine.packs))
-        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "tests": tests}))
+        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "repeat_total": True, "tests": tests}))
 
     @on(Button.Pressed, "#save")
     @on(Input.Submitted, "#name")
@@ -1765,7 +1777,8 @@ class QueuedRun:
         """(jobs, None), or (None, why it can't run) now."""
         sel = self.sel
         jobs = [j for j in eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], self.retest,
-                                    self.tests(eng), sel.get("stop_early") or None) if j.key not in self.leave]
+                                    self.tests(eng), sel.get("stop_early") or None, sel.get("repeat_total", False))
+                if j.key not in self.leave]
         waiting = [j for j in jobs if j.status == "waiting"]
         if not waiting:
             return None, "every pack left is already done"
@@ -1850,13 +1863,13 @@ class RetestRow:
 CHOICE_TEXT = {"zero": "[b red]Zero[/b red]", "continue": "[b green]Continue[/b green]", "leave": "[dim]Leave out[/dim]"}
 
 
-def retest_rows(eng, labels, packs, repeat, tier, picks, force):
+def retest_rows(eng, labels, packs, repeat, tier, picks, force, total=False):
     """(rows, rerun): the packs Start would set earlier answers aside for, or could. rerun: Retest is
     off but everything selected is done, so Start runs it again. A pack whose questions or settings
     changed can't continue (Zero or Leave out); with Retest every pack with answers starts at Zero,
     without it a pack that can resume continues."""
-    base = eng.plan(labels, packs, repeat, tier, False, picks)
-    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks)}
+    base = eng.plan(labels, packs, repeat, tier, False, picks, total=total)
+    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks, total=total)}
     rerun = not force and not any(j.status == "waiting" for j in base)
     rows = []
     for b in base:
@@ -3474,10 +3487,10 @@ class ResultsScreen(Screen):
         self.pq_summary_args = (tests, models)    # to refit on resize
         rows = compare.compare_models(shared, models)
         # short headings so every column fits; the line above the table says what each one is
-        header = ("Model", "Time", "Passed", "Med tok", "Tokens", "Tok/✓", "Med s", "Prompt t/s", "Gen t/s",
-                  "Fewest", "Fastest", "Cut off")
+        header = ("Model", "Time", "Passed", "Pass rate", "Med tok", "Tokens", "Tok/✓", "Med s", "Prompt t/s",
+                  "Gen t/s", "Fewest", "Fastest", "Cut off")
         best = {k: max((r[k] for r in rows if r[k] is not None), default=None)
-                for k in ("passed", "fewest", "fastest", "avg_pp_tps", "avg_gen_tps")}
+                for k in ("passed", "pass_rate", "fewest", "fastest", "avg_pp_tps", "avg_gen_tps")}
         low = {k: min(r[k] for r in rows if r[k] is not None) if any(r[k] is not None for r in rows) else None
                for k in ("med_tokens", "total_tokens", "tokens_per_pass", "med_secs", "total_secs")}
 
@@ -3492,6 +3505,7 @@ class ResultsScreen(Screen):
                           hi(tt, ("-" if not tt else f"{tt:.1f}s" if tt < 60 else fmt_secs(tt)), "total_secs", "min"),
                           hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max")
                           + (f" [dim]{r['score']:.0%}[/dim]" if r["passed"] < r["n"] and r["score"] else ""),
+                          hi(r["pass_rate"], f"{r['pass_rate']:.0%}", "pass_rate", "max"),
                           hi(r["med_tokens"], f"{r['med_tokens']:,.0f}" if r["med_tokens"] else "-", "med_tokens", "min"),
                           hi(r["total_tokens"], f"{r['total_tokens']:,.0f}", "total_tokens", "min"),
                           hi(r["tokens_per_pass"], f"{r['tokens_per_pass']:,.0f}" if r["tokens_per_pass"] else "-",
@@ -3504,6 +3518,9 @@ class ResultsScreen(Screen):
                           hi(r["fastest"], str(r["fastest"]), "fastest", "max"),
                           (f"[red]{r['cut']}[/red]" if r["cut"] else "0")
                           + (f" [yellow]+{r['over']} over[/yellow]" if r.get("over") else "")])
+        if all(t["cells"][m]["n"] == 1 for t in shared for m in models):   # one answer each: Passed says it all
+            i = header.index("Pass rate")
+            header, cells = header[:i] + header[i + 1:], [c[:i] + c[i + 1:] for c in cells]
         table.add_columns(*header)
         tag_w = max(Text.from_markup(tag).cell_len + 1 for tag in tags.values())
         cells = fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3 - tag_w,
@@ -3524,7 +3541,7 @@ class ResultsScreen(Screen):
                      "shared the machine's speed: compare their times with care.[/yellow]")
         title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
                      "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
-                     "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Fewest / Fastest: questions won; ties count for both. "
+                     "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Pass rate (with repeats): answers right, repeats included (each question weighs the same). Fewest / Fastest: questions won; ties count for both. "
                      "Cut off: answers that hit max_tokens. ×n beside a name: answers per question.[/dim]")
 
     @staticmethod

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev141}/tests/test_tuieval.py

@@ -711,6 +711,117 @@ class Retest(unittest.TestCase):
         self.assertEqual(self.passes("apps"), dict.fromkeys("abcd", True))
 
 
+class AddRepeats(unittest.TestCase):
+    """A chosen repeat count adds rounds to the saved answers (×1 + 2 = ×3); a retest, blank and a
+    resumed run don't add more."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        d = os.path.join(self.ws, "packs", "apps")
+        os.makedirs(d)
+        write(os.path.join(d, "pack.toml"), 'label = "apps"\nscreen = 4\n[certify]\nrepeat = 1\n'
+                                            '[gate]\nmin_accuracy = 0.1\n')
+        write(os.path.join(d, "tests.yaml"), "".join(
+            f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+            "difficulty: easy}\n" for t in "abcd"))
+        port = free_port()
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f'\n[servers.m]\nurl = "http://127.0.0.1:{port}"\n')
+        self.mock = Mock(os.path.join(self.ws, "packs"), "oracle", port=port)
+        tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
+        tuieval(self.ws, "run", "--tier", "certify")
+        self.path = os.path.join(self.ws, "results", "m", "apps.json")
+
+    def tearDown(self):
+        self.mock.stop()
+        self.tmp.cleanup()
+
+    def rounds(self):
+        out = {}
+        for r in json.loads(read(self.path))["results"]:
+            out.setdefault(r["test"], set()).add(r["repeat"])
+        return {t: sorted(v) for t, v in out.items()}
+
+    def test_cli_adds_rounds(self):
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout
+        self.assertIn("×1 → ×3", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+        tuieval(self.ws, "run", "--tier", "certify")                      # blank: the pack's 1, done
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_retest_counts_from_none(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2", "--retest")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+
+    def test_picked_tests_then_top_up(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--tests", "apps:a", "--repeat", "1")
+        self.assertEqual(self.rounds(), {"a": [0, 1], "b": [0], "c": [0], "d": [0]})
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "1")    # the most rounds (2) + 1, for all
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_resumed_run_adds_no_more(self):
+        from tuieval import engine
+        e = engine.Engine(root=self.ws)
+        j = e.plan(["m"], ["apps"], 2, "certify", False, {})[0]
+        self.assertEqual((j.base, j.repeat, j.done), (1, 3, 4))
+        rows = [dict(r, repeat=1) for r in json.loads(read(self.path))["results"] if r["test"] in "ab"]
+        write(self.path + ".partial.jsonl", "".join(json.dumps(l) + "\n" for l in [e._partial_header(j)] + rows))
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout   # the same +2, resumed
+        self.assertIn("resuming after 6", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_preset_repeat_is_a_total(self):
+        from tuieval import engine
+        engine.save_preset(os.path.join(self.ws, "presets.toml"), "two", ["m"], ["apps"], 2)
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))        # ×1 topped up to 2
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")       # already 2: nothing new
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two", "--repeat", "1")   # typed: adds
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def estimate(self, steps):
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp
+            from textual.widgets import SelectionList, Input
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"m"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+            %s
+                    await pilot.pause()
+                    print(str(s.query_one("#estimate").render()))
+            asyncio.run(go())
+        """) % textwrap.indent(textwrap.dedent(steps), " " * 8)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=300,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        return p.stdout + p.stderr
+
+    def test_tui_estimate_says_more(self):
+        self.assertIn("2 more (×1 → ×3) repeat(s) = 8 answers", self.estimate("""
+            s.query_one("#repeat", Input).value = "2"
+        """))
+
+    def test_tui_preset_repeat_is_a_total(self):
+        out = self.estimate("""
+            s.apply_selection({"models": ["m"], "packs": ["apps"], "repeat": 2, "repeat_total": True}, quiet=True)
+            s.query_one("#tier-certify").value = True
+            await pilot.pause()
+            print(str(s.query_one("#estimate").render()))
+            s.query_one("#repeat", Input).value = "3"     # typed over: adds again
+        """)
+        self.assertIn("2 in all (preset) repeat(s) = 4 answers", out)
+        self.assertIn("3 more (×1 → ×4) repeat(s) = 12 answers", out)
+
+
 class Parallel(unittest.TestCase):
     """Several models at a time (parallel_models / --parallel): each runs in its own lane on its own
     port, answers note the models they ran alongside, and the scheduler keeps the memory and
@@ -2340,6 +2451,52 @@ class FirstRun(unittest.TestCase):
             "['once [yellow]×1[/]', 'thrice [dim]×3[/]']",     # Comparison: only the questions both answered
         ], out.stderr[-2000:])
 
+    def test_pass_rate_counts_every_answer(self):
+        from tuieval import compare
+        row = {"suite": "p", "latency": 1000, "tokens": 10, "truncated": False, "gen_tps": 20.0, "prompt_tps": None}
+        oks = {"a": {"q0": [True, True], "q1": [True, False], "q2": [False, False, False, True]},   # ×2, ×2, ×4
+               "b": {"q0": [True], "q1": [False], "q2": [False]},
+               "c": {"q0": [False], "q1": [False], "q2": [False]}}
+        rows = [dict(row, model=m, test=f"p: {q}", ok=ok, repeat=r)
+                for m, qs in oks.items() for q, v in qs.items() for r, ok in enumerate(v)]
+        out = {r["model"]: r for r in compare.compare_models(compare.per_question(rows), list(oks))}
+        self.assertEqual(out["a"]["passed"], 1)                            # strict: right on every repeat
+        self.assertAlmostEqual(out["a"]["pass_rate"], (1 + 0.5 + 0.25) / 3)   # each question weighs the same
+        self.assertAlmostEqual(out["b"]["pass_rate"], 1 / 3)               # ×1: the same as passed / questions
+        self.assertEqual(out["c"]["pass_rate"], 0)
+
+    def test_comparison_shows_pass_rate(self):
+        tuieval(self.ws, "add", "my-model", "--server", "local")
+        code = textwrap.dedent("""
+            import asyncio
+            from rich.text import Text
+            from tuieval import compare
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, TabbedContent
+            oks = {"steady": [[True, True], [True, True]], "shaky": [[True, True], [True, False]]}
+            rows = [{"model": m, "suite": "p", "test": f"p: q{q}", "ok": ok, "latency": 1000, "tokens": 10,
+                     "truncated": False, "gen_tps": 20.0, "prompt_tps": None, "repeat": r}
+                    for m, qs in oks.items() for q, v in enumerate(qs) for r, ok in enumerate(v)]
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 40)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    app.screen.query_one(TabbedContent).active = "tab-per-question"
+                    await pilot.pause(0.3)
+                    app.screen.fill_pq_summary(compare.per_question(rows), list(oks))
+                    await pilot.pause(0.3)
+                    t = app.screen.query_one("#pq-summary", DataTable)
+                    i = [str(c.label) for c in t.ordered_columns].index("Pass rate")
+                    print(str(t.ordered_columns[i - 1].label), [t.get_row_at(n)[i] for n in range(t.row_count)])
+            asyncio.run(go())
+        """)
+        out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                             env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        # right after Passed; the best in bold green
+        self.assertEqual(out.stdout.strip(), "Passed ['[bold green]100%[/bold green]', '75%']", out.stderr[-2000:])
+
     def test_results_open_on_pta_index(self):
         tuieval(self.ws, "add", "my-model", "--server", "local")
         code = textwrap.dedent("""