tuieval 0.2.0.dev137__tar.gz → 0.2.0.dev139__tar.gz

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Repeats (Setup, `tuieval run --repeat N`) add N rounds to the saved answers: ×1 run with 2 gives ×3, not ×2. Retest, a blank count and presets still mean the total.
 - Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
 - Repeats: the PTA index, Per question, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev137
+Version: 0.2.0.dev139
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/docs/models.md

@@ -106,7 +106,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 - **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
-- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`). To steady it, set **Repeats** in Setup (or `tuieval run --repeat N`): N adds that many rounds to the answers already saved, so a `×1` model run with 2 ends up `×3` (with Retest, N in all; left blank, the tier's default count in all). A preset's repeat count is in all too, until you type another.
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev137'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev137')
+__version__ = version = '0.2.0.dev139'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev139')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/src/tuieval/engine.py

@@ -443,7 +443,7 @@ def save_preset(path, name, labels, packs, repeat, tests=None):
 class Job:
     model: dict
     pack: packs_mod.Pack
-    repeat: int
+    repeat: int             # rounds each question has once it's done (earlier ones included)
     tests: list             # the tests this tier runs (or the ones picked)
     out_path: str
     sampling: dict
@@ -465,6 +465,7 @@ class Job:
     stop_early: bool = False  # stop once the pack's FAIL is certain (verdict.fail_certain)
     first: list = dataclasses.field(default_factory=list)   # test ids round 1 asks first (most failed)
     claimed: bool = False   # its model's server took it on: it can no longer be removed from the run
+    base: int = 0           # rounds already in its result file that a chosen repeat count adds to
 
     @property
     def label(self):
@@ -1148,6 +1149,17 @@ class Engine:
         redo = {t["id"] for t in job.tests}
         return [r for r in rows if r["test"] not in redo]
 
+    def saved_rounds(self, job):
+        """The most rounds any of the job's questions has in its finished result file (0 when the file
+        doesn't match the job's questions and settings). An unfinished run's answers don't count, so
+        resuming one doesn't add more rounds."""
+        data = self._result(job.out_path)
+        if not data or data["pack"]["fingerprint"] != job.pack.fingerprint or not settings_match(data, job.settings):
+            return 0
+        ids = {t["id"] for t in job.tests}
+        rounds = collections.Counter(t for t, _ in {(r["test"], r["repeat"]) for r in data["results"] if r["test"] in ids})
+        return max(rounds.values(), default=0)
+
     def _fill_progress(self, job):
         """Set a job's done/passed/failed and earlier sittings from what's stored."""
         kept, new, job.earlier = self._stored(job)
@@ -1212,10 +1224,13 @@ class Engine:
         return not current and header_matches(head, pack.fingerprint, settings)
 
     # ---- planning
-    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None):
+    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None,
+             total=False):
         """Jobs for models x packs. tier: smoke (3 tests), screen (a spread sample), certify (all
         tests). repeat: times each question is asked; None = the tier default (1 for smoke and
-        screen, the pack's certification repeats for certify). tests: {pack: [test ids]} runs just
+        screen, the pack's certification repeats for certify), counted in total; a number adds that
+        many rounds to the ones already in the result file (a retest starts from none), or with total
+        (a preset's count) is the rounds in all. tests: {pack: [test ids]} runs just
         those tests of a pack instead of the tier's sample (unknown ids raise PackError). Earlier
         matching results are kept and only missing answers are run, unless force; force with
         picked tests reruns those and keeps the pack's other answers. force is True, False, or the
@@ -1240,6 +1255,9 @@ class Engine:
                     job.first = self.most_failed_first(pack)
                 sv = self.serving(m)
                 status = self.result_status(label, name, tier)
+                if repeat and not total and job.merge and not job.redo and not (status or "").startswith("outdated"):
+                    job.base = self.saved_rounds(job)
+                    job.repeat = job.base + repeat
                 job.discards = self._discards(job)
                 missing = [n for n in pack.modules if not importlib.util.find_spec(n)]
                 no_agent = agents.problem(self.cfg, pack, self.root) if pack.kind == "agent" else None
@@ -1273,6 +1291,8 @@ class Engine:
                         job.note = f"{job.done} of {job.total} done earlier, running the rest"
                 if job.status != "waiting":
                     job.discards = []
+                elif job.base:
+                    job.note = "; ".join(filter(None, [job.note, f"×{job.base} → ×{job.repeat}"]))
                 for what, n, why in job.discards:
                     if what == "unfinished":   # (a finished file's reason is already the note)
                         job.note = "; ".join(filter(None, [job.note, f"{n} unfinished answers set aside ({why})"]))

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/src/tuieval/run_evals.py

@@ -101,7 +101,8 @@ class Printer:
             "so the server may be returning cached answers rather than fresh ones", RED)
 
     def on_job_started(self, job, resumed, earlier_s=0, sittings=0):
-        say(f"{job.key}: {job.count} tests x {job.repeat} repeats" + (f", resuming after {resumed}" if resumed else "")
+        reps = f"{job.repeat - job.base} more repeats (×{job.base} → ×{job.repeat})" if job.base else f"{job.repeat} repeats"
+        say(f"{job.key}: {job.count} tests x {reps}" + (f", resuming after {resumed}" if resumed else "")
             + (f" ({earlier_s / 60:.0f} min over {sittings} earlier sitting(s))" if sittings else ""))
 
     def on_request_started(self, job, test, test_id, has_image, repeat, difficulty="unrated"):
@@ -792,8 +793,9 @@ def main(argv=None):
     p.add_argument("--stop-early", action="store_true",
                    help="stop each pack once its FAIL is certain (a critical failure, or a bar out of reach), "
                         "asking first what models fail most (default: each pack's stop_early)")
-    p.add_argument("--repeat", type=int, help="times each question is asked, any tier (default: 1 for smoke and "
-                                                  "screen, the pack's certification count for certify)")
+    p.add_argument("--repeat", type=int, help="rounds to add: each question is asked N more times on top of its "
+                                                  "saved answers (with --retest, N in all). Default: up to 1 for "
+                                                  "smoke and screen, the pack's certification count for certify")
     p.add_argument("--tier", choices=engine.TIERS, default="screen",
                    help="smoke: 3 tests; screen (default): a spread sample; certify: everything, with repeats")
     p.add_argument("--smoke", action="store_const", const="smoke", dest="tier", help="same as --tier smoke")
@@ -838,12 +840,14 @@ def main(argv=None):
         sys.exit("no models to run yet: tuieval add <GGUF path or model id>, or name a hosted one with "
                  "--only openrouter:<model id>" if not models else
                  "no local models; hosted ones (they cost money) run only when named: --only <label>")
+    repeat_total = False
     if args.preset:
         presets = engine.load_presets(os.path.join(os.path.dirname(os.path.abspath(models_file(args.models))),
                                                    "presets.toml"))
         if args.preset not in presets:
             sys.exit(f"unknown preset {args.preset!r}; presets: {', '.join(presets) or 'none (save one in the TUI with p)'}")
         labels, pack_names, preset_repeat, tests = engine.resolve_preset(e.cfg, presets[args.preset], list(e.packs))
+        repeat_total = not args.repeat and bool(preset_repeat)   # a preset's count is the rounds in all
         args.repeat = args.repeat or preset_repeat
     if args.packs:
         pack_names = [x.strip() for x in args.packs.split(",")]
@@ -853,7 +857,8 @@ def main(argv=None):
     if args.tests:
         tests, pack_names = parse_tests(args.tests, pack_names if (args.packs or args.preset) else None, e.packs)
     try:
-        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None)
+        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None,
+                      repeat_total)
     except packs_mod.PackError as err:
         sys.exit(str(err))
 

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/src/tuieval/tui.py

@@ -1025,6 +1025,11 @@ class SetupScreen(Screen):
                 list(self.query_one("#suites", SelectionList).selected), repeat,
                 self.tier(), self.query_one("#force", Checkbox).value)
 
+    def repeat_total(self):
+        """Is the Repeats count the rounds in all? Yes while it's a loaded preset's; a typed one adds rounds."""
+        value = self.query_one("#repeat", Input).value
+        return bool(value) and value == getattr(self, "preset_repeat", None)
+
     def parallel(self):
         """Models at a time for the next run: the field, or blank for this machine's parallel_models."""
         try:
@@ -1088,16 +1093,21 @@ class SetupScreen(Screen):
                        "[b]s[/b] to start." + self.queue_hint())
             return
         picks = self.current_picks(suites)
-        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks)
+        total = self.repeat_total()
+        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks, total=total)
         rerun = not any(j.status == "waiting" for j in jobs)
         if rerun:   # everything has results: Start runs it again, so estimate that
-            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks)
+            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks, total=total)
         run = [j for j in jobs if j.status == "waiting"]
         skipped = [j for j in jobs if j.status == "skipped"]
         secs, notes = self.app.engine.estimate_seconds(jobs)
-        todo = sum(j.total for j in run)
-        reps = sorted({j.repeat for j in jobs})
-        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "")
+        todo = sum(j.total - j.done for j in run)
+        reps = sorted({j.repeat - j.base for j in jobs})
+        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "") \
+            + (" in all (preset)" if total else "")
+        bases = sorted({j.base for j in run if j.base})
+        if bases:   # a chosen count adds to the rounds already answered
+            reps_text += " more" + " [dim](" + ", ".join(f"×{b} → ×{b + repeat}" for b in bases) + ")[/dim]"
         text = (f"[b]{tier.title()}[/b]: [b]{len(labels)}[/b] model(s) × [b]{len(suites)}[/b] pack(s) × "
                 f"[b]{reps_text}[/b] repeat(s) = "
                 f"[b]{todo:,}[/b] answers · about [b]{fmt_secs(secs)}[/b]"
@@ -1158,6 +1168,7 @@ class SetupScreen(Screen):
         for name in sel.get("packs", []):
             if name in e.packs:
                 packs.select(name)
+        self.preset_repeat = str(sel["repeat"]) if sel.get("repeat") and sel.get("repeat_total") else None
         self.query_one("#repeat", Input).value = str(sel["repeat"]) if sel.get("repeat") else ""
         if "parallel" in sel:
             self.query_one("#parallel", Input).value = str(sel["parallel"]) if sel["parallel"] else ""
@@ -1201,10 +1212,11 @@ class SetupScreen(Screen):
             self.notify("Select at least one eval pack and one model.", severity="warning")
             return
         picks = self.current_picks(suites)
-        sel = {"models": labels, "packs": suites, "repeat": repeat, "tier": tier, "force": force, "tests": picks,
-               "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
+        total = self.repeat_total()
+        sel = {"models": labels, "packs": suites, "repeat": repeat, "repeat_total": total, "tier": tier, "force": force,
+               "tests": picks, "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
         e = self.app.engine
-        jobs = e.plan(labels, suites, repeat, tier, True, picks)
+        jobs = e.plan(labels, suites, repeat, tier, True, picks, total=total)
         if not any(j.status == "waiting" for j in jobs):
             reasons = sorted({j.note for j in jobs if j.status == "skipped"})
             self.notify(f"Nothing selected can run: {'; '.join(reasons)}", severity="warning")
@@ -1212,7 +1224,7 @@ class SetupScreen(Screen):
         self.app.save_state({k: v for k, v in sel.items() if k != "models"})   # models aren't remembered
         # Packs whose earlier answers a start would set aside (Retest, everything already done, or
         # other questions or settings): the user picks per pack before anything moves.
-        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force)
+        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force, total)
 
         def go(choices):
             if choices is None:
@@ -1640,7 +1652,7 @@ class PresetsScreen(ModalScreen):
         name = list(self.presets)[event.option_index]
         labels, packs, repeat, tests = engine.resolve_preset(self.app.engine.cfg, self.presets[name],
                                                              list(self.app.engine.packs))
-        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "tests": tests}))
+        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "repeat_total": True, "tests": tests}))
 
     @on(Button.Pressed, "#save")
     @on(Input.Submitted, "#name")
@@ -1765,7 +1777,8 @@ class QueuedRun:
         """(jobs, None), or (None, why it can't run) now."""
         sel = self.sel
         jobs = [j for j in eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], self.retest,
-                                    self.tests(eng), sel.get("stop_early") or None) if j.key not in self.leave]
+                                    self.tests(eng), sel.get("stop_early") or None, sel.get("repeat_total", False))
+                if j.key not in self.leave]
         waiting = [j for j in jobs if j.status == "waiting"]
         if not waiting:
             return None, "every pack left is already done"
@@ -1850,13 +1863,13 @@ class RetestRow:
 CHOICE_TEXT = {"zero": "[b red]Zero[/b red]", "continue": "[b green]Continue[/b green]", "leave": "[dim]Leave out[/dim]"}
 
 
-def retest_rows(eng, labels, packs, repeat, tier, picks, force):
+def retest_rows(eng, labels, packs, repeat, tier, picks, force, total=False):
     """(rows, rerun): the packs Start would set earlier answers aside for, or could. rerun: Retest is
     off but everything selected is done, so Start runs it again. A pack whose questions or settings
     changed can't continue (Zero or Leave out); with Retest every pack with answers starts at Zero,
     without it a pack that can resume continues."""
-    base = eng.plan(labels, packs, repeat, tier, False, picks)
-    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks)}
+    base = eng.plan(labels, packs, repeat, tier, False, picks, total=total)
+    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks, total=total)}
     rerun = not force and not any(j.status == "waiting" for j in base)
     rows = []
     for b in base:

{tuieval-0.2.0.dev137 → tuieval-0.2.0.dev139}/tests/test_tuieval.py

@@ -711,6 +711,117 @@ class Retest(unittest.TestCase):
         self.assertEqual(self.passes("apps"), dict.fromkeys("abcd", True))
 
 
+class AddRepeats(unittest.TestCase):
+    """A chosen repeat count adds rounds to the saved answers (×1 + 2 = ×3); a retest, blank and a
+    resumed run don't add more."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        d = os.path.join(self.ws, "packs", "apps")
+        os.makedirs(d)
+        write(os.path.join(d, "pack.toml"), 'label = "apps"\nscreen = 4\n[certify]\nrepeat = 1\n'
+                                            '[gate]\nmin_accuracy = 0.1\n')
+        write(os.path.join(d, "tests.yaml"), "".join(
+            f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+            "difficulty: easy}\n" for t in "abcd"))
+        port = free_port()
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f'\n[servers.m]\nurl = "http://127.0.0.1:{port}"\n')
+        self.mock = Mock(os.path.join(self.ws, "packs"), "oracle", port=port)
+        tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
+        tuieval(self.ws, "run", "--tier", "certify")
+        self.path = os.path.join(self.ws, "results", "m", "apps.json")
+
+    def tearDown(self):
+        self.mock.stop()
+        self.tmp.cleanup()
+
+    def rounds(self):
+        out = {}
+        for r in json.loads(read(self.path))["results"]:
+            out.setdefault(r["test"], set()).add(r["repeat"])
+        return {t: sorted(v) for t, v in out.items()}
+
+    def test_cli_adds_rounds(self):
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout
+        self.assertIn("×1 → ×3", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+        tuieval(self.ws, "run", "--tier", "certify")                      # blank: the pack's 1, done
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_retest_counts_from_none(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2", "--retest")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+
+    def test_picked_tests_then_top_up(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--tests", "apps:a", "--repeat", "1")
+        self.assertEqual(self.rounds(), {"a": [0, 1], "b": [0], "c": [0], "d": [0]})
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "1")    # the most rounds (2) + 1, for all
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_resumed_run_adds_no_more(self):
+        from tuieval import engine
+        e = engine.Engine(root=self.ws)
+        j = e.plan(["m"], ["apps"], 2, "certify", False, {})[0]
+        self.assertEqual((j.base, j.repeat, j.done), (1, 3, 4))
+        rows = [dict(r, repeat=1) for r in json.loads(read(self.path))["results"] if r["test"] in "ab"]
+        write(self.path + ".partial.jsonl", "".join(json.dumps(l) + "\n" for l in [e._partial_header(j)] + rows))
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout   # the same +2, resumed
+        self.assertIn("resuming after 6", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_preset_repeat_is_a_total(self):
+        from tuieval import engine
+        engine.save_preset(os.path.join(self.ws, "presets.toml"), "two", ["m"], ["apps"], 2)
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))        # ×1 topped up to 2
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")       # already 2: nothing new
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two", "--repeat", "1")   # typed: adds
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def estimate(self, steps):
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp
+            from textual.widgets import SelectionList, Input
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"m"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+            %s
+                    await pilot.pause()
+                    print(str(s.query_one("#estimate").render()))
+            asyncio.run(go())
+        """) % textwrap.indent(textwrap.dedent(steps), " " * 8)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=300,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        return p.stdout + p.stderr
+
+    def test_tui_estimate_says_more(self):
+        self.assertIn("2 more (×1 → ×3) repeat(s) = 8 answers", self.estimate("""
+            s.query_one("#repeat", Input).value = "2"
+        """))
+
+    def test_tui_preset_repeat_is_a_total(self):
+        out = self.estimate("""
+            s.apply_selection({"models": ["m"], "packs": ["apps"], "repeat": 2, "repeat_total": True}, quiet=True)
+            s.query_one("#tier-certify").value = True
+            await pilot.pause()
+            print(str(s.query_one("#estimate").render()))
+            s.query_one("#repeat", Input).value = "3"     # typed over: adds again
+        """)
+        self.assertIn("2 in all (preset) repeat(s) = 4 answers", out)
+        self.assertIn("3 more (×1 → ×4) repeat(s) = 12 answers", out)
+
+
 class Parallel(unittest.TestCase):
     """Several models at a time (parallel_models / --parallel): each runs in its own lane on its own
     port, answers note the models they ran alongside, and the scheduler keeps the memory and