tuieval 0.2.0.dev135__tar.gz → 0.2.0.dev139__tar.gz

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/CHANGELOG.md

@@ -2,8 +2,9 @@
 
 ## Unreleased
 
+- Repeats (Setup, `tuieval run --repeat N`) add N rounds to the saved answers: ×1 run with 2 gives ×3, not ×2. Retest, a blank count and presets still mean the total.
 - Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
-- Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
+- Repeats: the PTA index, Per question, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev135
+Version: 0.2.0.dev139
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/docs/models.md

@@ -106,7 +106,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 - **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
-- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`). To steady it, set **Repeats** in Setup (or `tuieval run --repeat N`): N adds that many rounds to the answers already saved, so a `×1` model run with 2 ends up `×3` (with Retest, N in all; left blank, the tier's default count in all). A preset's repeat count is in all too, until you type another.
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev135'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev135')
+__version__ = version = '0.2.0.dev139'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev139')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/src/tuieval/engine.py

@@ -443,7 +443,7 @@ def save_preset(path, name, labels, packs, repeat, tests=None):
 class Job:
     model: dict
     pack: packs_mod.Pack
-    repeat: int
+    repeat: int             # rounds each question has once it's done (earlier ones included)
     tests: list             # the tests this tier runs (or the ones picked)
     out_path: str
     sampling: dict
@@ -465,6 +465,7 @@ class Job:
     stop_early: bool = False  # stop once the pack's FAIL is certain (verdict.fail_certain)
     first: list = dataclasses.field(default_factory=list)   # test ids round 1 asks first (most failed)
     claimed: bool = False   # its model's server took it on: it can no longer be removed from the run
+    base: int = 0           # rounds already in its result file that a chosen repeat count adds to
 
     @property
     def label(self):
@@ -1148,6 +1149,17 @@ class Engine:
         redo = {t["id"] for t in job.tests}
         return [r for r in rows if r["test"] not in redo]
 
+    def saved_rounds(self, job):
+        """The most rounds any of the job's questions has in its finished result file (0 when the file
+        doesn't match the job's questions and settings). An unfinished run's answers don't count, so
+        resuming one doesn't add more rounds."""
+        data = self._result(job.out_path)
+        if not data or data["pack"]["fingerprint"] != job.pack.fingerprint or not settings_match(data, job.settings):
+            return 0
+        ids = {t["id"] for t in job.tests}
+        rounds = collections.Counter(t for t, _ in {(r["test"], r["repeat"]) for r in data["results"] if r["test"] in ids})
+        return max(rounds.values(), default=0)
+
     def _fill_progress(self, job):
         """Set a job's done/passed/failed and earlier sittings from what's stored."""
         kept, new, job.earlier = self._stored(job)
@@ -1212,10 +1224,13 @@ class Engine:
         return not current and header_matches(head, pack.fingerprint, settings)
 
     # ---- planning
-    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None):
+    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None, stop_early=None,
+             total=False):
         """Jobs for models x packs. tier: smoke (3 tests), screen (a spread sample), certify (all
         tests). repeat: times each question is asked; None = the tier default (1 for smoke and
-        screen, the pack's certification repeats for certify). tests: {pack: [test ids]} runs just
+        screen, the pack's certification repeats for certify), counted in total; a number adds that
+        many rounds to the ones already in the result file (a retest starts from none), or with total
+        (a preset's count) is the rounds in all. tests: {pack: [test ids]} runs just
         those tests of a pack instead of the tier's sample (unknown ids raise PackError). Earlier
         matching results are kept and only missing answers are run, unless force; force with
         picked tests reruns those and keeps the pack's other answers. force is True, False, or the
@@ -1240,6 +1255,9 @@ class Engine:
                     job.first = self.most_failed_first(pack)
                 sv = self.serving(m)
                 status = self.result_status(label, name, tier)
+                if repeat and not total and job.merge and not job.redo and not (status or "").startswith("outdated"):
+                    job.base = self.saved_rounds(job)
+                    job.repeat = job.base + repeat
                 job.discards = self._discards(job)
                 missing = [n for n in pack.modules if not importlib.util.find_spec(n)]
                 no_agent = agents.problem(self.cfg, pack, self.root) if pack.kind == "agent" else None
@@ -1273,6 +1291,8 @@ class Engine:
                         job.note = f"{job.done} of {job.total} done earlier, running the rest"
                 if job.status != "waiting":
                     job.discards = []
+                elif job.base:
+                    job.note = "; ".join(filter(None, [job.note, f"×{job.base} → ×{job.repeat}"]))
                 for what, n, why in job.discards:
                     if what == "unfinished":   # (a finished file's reason is already the note)
                         job.note = "; ".join(filter(None, [job.note, f"{n} unfinished answers set aside ({why})"]))

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/src/tuieval/run_evals.py

@@ -101,7 +101,8 @@ class Printer:
             "so the server may be returning cached answers rather than fresh ones", RED)
 
     def on_job_started(self, job, resumed, earlier_s=0, sittings=0):
-        say(f"{job.key}: {job.count} tests x {job.repeat} repeats" + (f", resuming after {resumed}" if resumed else "")
+        reps = f"{job.repeat - job.base} more repeats (×{job.base} → ×{job.repeat})" if job.base else f"{job.repeat} repeats"
+        say(f"{job.key}: {job.count} tests x {reps}" + (f", resuming after {resumed}" if resumed else "")
             + (f" ({earlier_s / 60:.0f} min over {sittings} earlier sitting(s))" if sittings else ""))
 
     def on_request_started(self, job, test, test_id, has_image, repeat, difficulty="unrated"):
@@ -792,8 +793,9 @@ def main(argv=None):
     p.add_argument("--stop-early", action="store_true",
                    help="stop each pack once its FAIL is certain (a critical failure, or a bar out of reach), "
                         "asking first what models fail most (default: each pack's stop_early)")
-    p.add_argument("--repeat", type=int, help="times each question is asked, any tier (default: 1 for smoke and "
-                                                  "screen, the pack's certification count for certify)")
+    p.add_argument("--repeat", type=int, help="rounds to add: each question is asked N more times on top of its "
+                                                  "saved answers (with --retest, N in all). Default: up to 1 for "
+                                                  "smoke and screen, the pack's certification count for certify")
     p.add_argument("--tier", choices=engine.TIERS, default="screen",
                    help="smoke: 3 tests; screen (default): a spread sample; certify: everything, with repeats")
     p.add_argument("--smoke", action="store_const", const="smoke", dest="tier", help="same as --tier smoke")
@@ -838,12 +840,14 @@ def main(argv=None):
         sys.exit("no models to run yet: tuieval add <GGUF path or model id>, or name a hosted one with "
                  "--only openrouter:<model id>" if not models else
                  "no local models; hosted ones (they cost money) run only when named: --only <label>")
+    repeat_total = False
     if args.preset:
         presets = engine.load_presets(os.path.join(os.path.dirname(os.path.abspath(models_file(args.models))),
                                                    "presets.toml"))
         if args.preset not in presets:
             sys.exit(f"unknown preset {args.preset!r}; presets: {', '.join(presets) or 'none (save one in the TUI with p)'}")
         labels, pack_names, preset_repeat, tests = engine.resolve_preset(e.cfg, presets[args.preset], list(e.packs))
+        repeat_total = not args.repeat and bool(preset_repeat)   # a preset's count is the rounds in all
         args.repeat = args.repeat or preset_repeat
     if args.packs:
         pack_names = [x.strip() for x in args.packs.split(",")]
@@ -853,7 +857,8 @@ def main(argv=None):
     if args.tests:
         tests, pack_names = parse_tests(args.tests, pack_names if (args.packs or args.preset) else None, e.packs)
     try:
-        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None)
+        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests, args.stop_early or None,
+                      repeat_total)
     except packs_mod.PackError as err:
         sys.exit(str(err))
 

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/src/tuieval/tui.py

@@ -431,6 +431,15 @@ def trim_names(names, width):
     return trimmed if len(set(trimmed)) == len(set(names)) else list(names)
 
 
+def repeats_tag(counts):
+    """A model's answers per question as markup, as the PTA index shows them: "×3", or "×1-3" when it varies;
+    yellow when it's ×1 (one answer decides each result, the roughest), else dim. "" for no answers."""
+    if not counts:
+        return ""
+    few, most = min(counts), max(counts)
+    return f"[{'yellow' if most == 1 else 'dim'}]×{few}{'' if few == most else f'-{most}'}[/]"
+
+
 def fit_first_column(header, rows, room, floor=24):
     """rows with the first cell trimmed in the middle (mid_trim) so a table of these columns fits room
     cells across, but never below floor cells. Cells are Rich markup; each column adds 2 cells of padding."""
@@ -1016,6 +1025,11 @@ class SetupScreen(Screen):
                 list(self.query_one("#suites", SelectionList).selected), repeat,
                 self.tier(), self.query_one("#force", Checkbox).value)
 
+    def repeat_total(self):
+        """Is the Repeats count the rounds in all? Yes while it's a loaded preset's; a typed one adds rounds."""
+        value = self.query_one("#repeat", Input).value
+        return bool(value) and value == getattr(self, "preset_repeat", None)
+
     def parallel(self):
         """Models at a time for the next run: the field, or blank for this machine's parallel_models."""
         try:
@@ -1079,16 +1093,21 @@ class SetupScreen(Screen):
                        "[b]s[/b] to start." + self.queue_hint())
             return
         picks = self.current_picks(suites)
-        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks)
+        total = self.repeat_total()
+        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks, total=total)
         rerun = not any(j.status == "waiting" for j in jobs)
         if rerun:   # everything has results: Start runs it again, so estimate that
-            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks)
+            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks, total=total)
         run = [j for j in jobs if j.status == "waiting"]
         skipped = [j for j in jobs if j.status == "skipped"]
         secs, notes = self.app.engine.estimate_seconds(jobs)
-        todo = sum(j.total for j in run)
-        reps = sorted({j.repeat for j in jobs})
-        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "")
+        todo = sum(j.total - j.done for j in run)
+        reps = sorted({j.repeat - j.base for j in jobs})
+        reps_text = "/".join(map(str, reps)) + (" (pack default)" if repeat is None and len(reps) > 1 else "") \
+            + (" in all (preset)" if total else "")
+        bases = sorted({j.base for j in run if j.base})
+        if bases:   # a chosen count adds to the rounds already answered
+            reps_text += " more" + " [dim](" + ", ".join(f"×{b} → ×{b + repeat}" for b in bases) + ")[/dim]"
         text = (f"[b]{tier.title()}[/b]: [b]{len(labels)}[/b] model(s) × [b]{len(suites)}[/b] pack(s) × "
                 f"[b]{reps_text}[/b] repeat(s) = "
                 f"[b]{todo:,}[/b] answers · about [b]{fmt_secs(secs)}[/b]"
@@ -1149,6 +1168,7 @@ class SetupScreen(Screen):
         for name in sel.get("packs", []):
             if name in e.packs:
                 packs.select(name)
+        self.preset_repeat = str(sel["repeat"]) if sel.get("repeat") and sel.get("repeat_total") else None
         self.query_one("#repeat", Input).value = str(sel["repeat"]) if sel.get("repeat") else ""
         if "parallel" in sel:
             self.query_one("#parallel", Input).value = str(sel["parallel"]) if sel["parallel"] else ""
@@ -1192,10 +1212,11 @@ class SetupScreen(Screen):
             self.notify("Select at least one eval pack and one model.", severity="warning")
             return
         picks = self.current_picks(suites)
-        sel = {"models": labels, "packs": suites, "repeat": repeat, "tier": tier, "force": force, "tests": picks,
-               "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
+        total = self.repeat_total()
+        sel = {"models": labels, "packs": suites, "repeat": repeat, "repeat_total": total, "tier": tier, "force": force,
+               "tests": picks, "parallel": self.parallel(), "stop_early": self.query_one("#stop-early", Checkbox).value}
         e = self.app.engine
-        jobs = e.plan(labels, suites, repeat, tier, True, picks)
+        jobs = e.plan(labels, suites, repeat, tier, True, picks, total=total)
         if not any(j.status == "waiting" for j in jobs):
             reasons = sorted({j.note for j in jobs if j.status == "skipped"})
             self.notify(f"Nothing selected can run: {'; '.join(reasons)}", severity="warning")
@@ -1203,7 +1224,7 @@ class SetupScreen(Screen):
         self.app.save_state({k: v for k, v in sel.items() if k != "models"})   # models aren't remembered
         # Packs whose earlier answers a start would set aside (Retest, everything already done, or
         # other questions or settings): the user picks per pack before anything moves.
-        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force)
+        rows, rerun = retest_rows(e, labels, suites, repeat, tier, picks, force, total)
 
         def go(choices):
             if choices is None:
@@ -1631,7 +1652,7 @@ class PresetsScreen(ModalScreen):
         name = list(self.presets)[event.option_index]
         labels, packs, repeat, tests = engine.resolve_preset(self.app.engine.cfg, self.presets[name],
                                                              list(self.app.engine.packs))
-        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "tests": tests}))
+        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "repeat_total": True, "tests": tests}))
 
     @on(Button.Pressed, "#save")
     @on(Input.Submitted, "#name")
@@ -1756,7 +1777,8 @@ class QueuedRun:
         """(jobs, None), or (None, why it can't run) now."""
         sel = self.sel
         jobs = [j for j in eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], self.retest,
-                                    self.tests(eng), sel.get("stop_early") or None) if j.key not in self.leave]
+                                    self.tests(eng), sel.get("stop_early") or None, sel.get("repeat_total", False))
+                if j.key not in self.leave]
         waiting = [j for j in jobs if j.status == "waiting"]
         if not waiting:
             return None, "every pack left is already done"
@@ -1841,13 +1863,13 @@ class RetestRow:
 CHOICE_TEXT = {"zero": "[b red]Zero[/b red]", "continue": "[b green]Continue[/b green]", "leave": "[dim]Leave out[/dim]"}
 
 
-def retest_rows(eng, labels, packs, repeat, tier, picks, force):
+def retest_rows(eng, labels, packs, repeat, tier, picks, force, total=False):
     """(rows, rerun): the packs Start would set earlier answers aside for, or could. rerun: Retest is
     off but everything selected is done, so Start runs it again. A pack whose questions or settings
     changed can't continue (Zero or Leave out); with Retest every pack with answers starts at Zero,
     without it a pack that can resume continues."""
-    base = eng.plan(labels, packs, repeat, tier, False, picks)
-    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks)}
+    base = eng.plan(labels, packs, repeat, tier, False, picks, total=total)
+    zero = {j.key: j for j in eng.plan(labels, packs, repeat, tier, True, picks, total=total)}
     rerun = not force and not any(j.status == "waiting" for j in base)
     rows = []
     for b in base:
@@ -3136,7 +3158,10 @@ class ResultsScreen(Screen):
             base = column_name(col.label)
             label = base + SORT_ARROWS[desc] if base == name else base
             if col.label.plain != label:
-                col.label = Text(label)
+                text = col.label.copy()   # keeps its styling, e.g. a model's ×n
+                if len(text.plain) > len(base):   # right_crop(0) would empty it
+                    text.right_crop(len(text.plain) - len(base))
+                col.label = text + SORT_ARROWS[desc] if base == name else text
                 w = cell_len(label)   # room for the arrow (a column keeps the width it was added with)
                 col.content_width = max(col.content_width, w)
                 if not col.auto_width:
@@ -3404,7 +3429,9 @@ class ResultsScreen(Screen):
         here = self.app.engine.machine().id
         table = self.query_one("#per-question", DataTable)
         table.clear(columns=True)
-        table.add_columns(*self.PQ_FIXED, *models)
+        table.add_columns(*self.PQ_FIXED, *(Text.from_markup(
+            f"{rich_escape(m)} {repeats_tag([t['cells'][m]['n'] for t in tests if m in t['cells']])}".rstrip())
+            for m in models))
         self.pq_tests, self.pq_models = {}, models
         for t in tests:
             cells = []
@@ -3427,7 +3454,8 @@ class ResultsScreen(Screen):
             "[green]bold[/green] = fewest tokens among passing answers · [red]cut[/red] = hit max_tokens · "
             "[yellow]over[/yellow] = ran past max_tokens (the provider didn't stop it) · "
             "◐ = pack unfinished (not in scores) · dim seconds = hosted, or measured on another machine; tokens compare "
-            "across machines, seconds don't. ×tokens = most / fewest tokens across models. Enter on a cell opens the "
+            "across machines, seconds don't. ×tokens = most / fewest tokens across models. ×n beside a model: answers "
+            "per question (×1: one answer decides each result, the roughest). Enter on a cell opens the "
             "answers. Click a column name to sort by it (a model's column: by its tokens).[/dim]" + ("\n[yellow]Left out, being rerun with other questions or settings: " + ", ".join(
                 f"{os.path.basename(os.path.dirname(p))} · {os.path.basename(p)[:-5]}" for p in sorted(self.superseded))
                 + ". Only the new run's answers (◐) are shown for these.[/yellow]" if self.superseded else ""))
@@ -3469,6 +3497,7 @@ class ResultsScreen(Screen):
         def hi(v, text, key, good):
             ref = best.get(key) if good == "max" else low.get(key)
             return f"[bold green]{text}[/bold green]" if v is not None and v == ref and len(rows) > 1 else text
+        tags = {m: repeats_tag([t["cells"][m]["n"] for t in shared]) for m in models}
         cells = []
         for r in rows:
             tt = r["total_secs"]
@@ -3489,7 +3518,10 @@ class ResultsScreen(Screen):
                           (f"[red]{r['cut']}[/red]" if r["cut"] else "0")
                           + (f" [yellow]+{r['over']} over[/yellow]" if r.get("over") else "")])
         table.add_columns(*header)
-        table.add_rows(fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3))
+        tag_w = max(Text.from_markup(tag).cell_len + 1 for tag in tags.values())
+        cells = fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3 - tag_w,
+                                 floor=24 - tag_w)   # the column's floor counts the ×n too
+        table.add_rows([[f"{rich_escape(c[0])} {tags[r['model']]}"] + c[1:] for c, r in zip(cells, rows)])
         line = f"[b]Comparison[/b] over the {len(shared)} questions all {len(models)} answered"
         if len(models) == 2:
             h = compare.head_to_head(shared, *models)
@@ -3506,7 +3538,7 @@ class ResultsScreen(Screen):
         title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
                      "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
                      "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Fewest / Fastest: questions won; ties count for both. "
-                     "Cut off: answers that hit max_tokens.[/dim]")
+                     "Cut off: answers that hit max_tokens. ×n beside a name: answers per question.[/dim]")
 
     @staticmethod
     def pq_cell(c, best, here):

{tuieval-0.2.0.dev135 → tuieval-0.2.0.dev139}/tests/test_tuieval.py

@@ -711,6 +711,117 @@ class Retest(unittest.TestCase):
         self.assertEqual(self.passes("apps"), dict.fromkeys("abcd", True))
 
 
+class AddRepeats(unittest.TestCase):
+    """A chosen repeat count adds rounds to the saved answers (×1 + 2 = ×3); a retest, blank and a
+    resumed run don't add more."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        d = os.path.join(self.ws, "packs", "apps")
+        os.makedirs(d)
+        write(os.path.join(d, "pack.toml"), 'label = "apps"\nscreen = 4\n[certify]\nrepeat = 1\n'
+                                            '[gate]\nmin_accuracy = 0.1\n')
+        write(os.path.join(d, "tests.yaml"), "".join(
+            f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+            "difficulty: easy}\n" for t in "abcd"))
+        port = free_port()
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f'\n[servers.m]\nurl = "http://127.0.0.1:{port}"\n')
+        self.mock = Mock(os.path.join(self.ws, "packs"), "oracle", port=port)
+        tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
+        tuieval(self.ws, "run", "--tier", "certify")
+        self.path = os.path.join(self.ws, "results", "m", "apps.json")
+
+    def tearDown(self):
+        self.mock.stop()
+        self.tmp.cleanup()
+
+    def rounds(self):
+        out = {}
+        for r in json.loads(read(self.path))["results"]:
+            out.setdefault(r["test"], set()).add(r["repeat"])
+        return {t: sorted(v) for t, v in out.items()}
+
+    def test_cli_adds_rounds(self):
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout
+        self.assertIn("×1 → ×3", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+        tuieval(self.ws, "run", "--tier", "certify")                      # blank: the pack's 1, done
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_retest_counts_from_none(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2", "--retest")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+
+    def test_picked_tests_then_top_up(self):
+        tuieval(self.ws, "run", "--tier", "certify", "--tests", "apps:a", "--repeat", "1")
+        self.assertEqual(self.rounds(), {"a": [0, 1], "b": [0], "c": [0], "d": [0]})
+        tuieval(self.ws, "run", "--tier", "certify", "--repeat", "1")    # the most rounds (2) + 1, for all
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_resumed_run_adds_no_more(self):
+        from tuieval import engine
+        e = engine.Engine(root=self.ws)
+        j = e.plan(["m"], ["apps"], 2, "certify", False, {})[0]
+        self.assertEqual((j.base, j.repeat, j.done), (1, 3, 4))
+        rows = [dict(r, repeat=1) for r in json.loads(read(self.path))["results"] if r["test"] in "ab"]
+        write(self.path + ".partial.jsonl", "".join(json.dumps(l) + "\n" for l in [e._partial_header(j)] + rows))
+        out = tuieval(self.ws, "run", "--tier", "certify", "--repeat", "2").stdout   # the same +2, resumed
+        self.assertIn("resuming after 6", out)
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def test_preset_repeat_is_a_total(self):
+        from tuieval import engine
+        engine.save_preset(os.path.join(self.ws, "presets.toml"), "two", ["m"], ["apps"], 2)
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))        # ×1 topped up to 2
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two")       # already 2: nothing new
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1]))
+        tuieval(self.ws, "run", "--tier", "certify", "--preset", "two", "--repeat", "1")   # typed: adds
+        self.assertEqual(self.rounds(), dict.fromkeys("abcd", [0, 1, 2]))
+
+    def estimate(self, steps):
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp
+            from textual.widgets import SelectionList, Input
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"m"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+            %s
+                    await pilot.pause()
+                    print(str(s.query_one("#estimate").render()))
+            asyncio.run(go())
+        """) % textwrap.indent(textwrap.dedent(steps), " " * 8)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=300,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        return p.stdout + p.stderr
+
+    def test_tui_estimate_says_more(self):
+        self.assertIn("2 more (×1 → ×3) repeat(s) = 8 answers", self.estimate("""
+            s.query_one("#repeat", Input).value = "2"
+        """))
+
+    def test_tui_preset_repeat_is_a_total(self):
+        out = self.estimate("""
+            s.apply_selection({"models": ["m"], "packs": ["apps"], "repeat": 2, "repeat_total": True}, quiet=True)
+            s.query_one("#tier-certify").value = True
+            await pilot.pause()
+            print(str(s.query_one("#estimate").render()))
+            s.query_one("#repeat", Input).value = "3"     # typed over: adds again
+        """)
+        self.assertIn("2 in all (preset) repeat(s) = 4 answers", out)
+        self.assertIn("3 more (×1 → ×4) repeat(s) = 12 answers", out)
+
+
 class Parallel(unittest.TestCase):
     """Several models at a time (parallel_models / --parallel): each runs in its own lane on its own
     port, answers note the models they ran alongside, and the scheduler keeps the memory and
@@ -1739,7 +1850,7 @@ class PTAIndex(unittest.TestCase):
                     print(t.row_count, str(s.query_one("#fail-count", Static).render()))
                     s.fill_per_question()
                     pq = s.query_one("#per-question", DataTable)
-                    cols = [str(c.label) for c in pq.ordered_columns]
+                    cols = [str(c.label).split(" ×")[0] for c in pq.ordered_columns]
                     print(sorted(str(pq.get_row_at(i)[cols.index("close")]).split()[1] for i in range(pq.row_count)))
             asyncio.run(go())
         """)
@@ -1801,7 +1912,7 @@ class PTAIndex(unittest.TestCase):
                         s = app.screen
                         def state():
                             pq = s.query_one("#per-question", DataTable)
-                            cols = [str(c.label) for c in pq.ordered_columns][len(s.PQ_FIXED):]
+                            cols = [str(c.label).split(" ×")[0] for c in pq.ordered_columns][len(s.PQ_FIXED):]
                             return (sorted(app.pq_models), s.query_one("#pta-pack", Select).value,
                                     str(s.query_one("#pq-models", Button).label), cols, s.query_one("#pq-shared").value)
                         s.query_one("#pq-pack", Select).value = "apps"
@@ -1955,7 +2066,7 @@ class PTAIndex(unittest.TestCase):
                             pq.move_cursor(row=0, column=col)
                             await pilot.press("enter")
                             await pilot.pause(0.3)
-                            opened[str(pq.ordered_columns[col].label.plain)] = app.screen.entries[0]["model"]
+                            opened[pq.ordered_columns[col].label.plain.split(" ×")[0]] = app.screen.entries[0]["model"]
                             app.screen.dismiss()
                             await pilot.pause(0.2)
                         print(json.dumps({"rows": rows, "bars": "P parsimony" in tri, "disagree": [both, alone], "opened": opened,
@@ -2271,6 +2382,7 @@ class FirstRun(unittest.TestCase):
         tuieval(self.ws, "add", "my-model", "--server", "local")
         code = textwrap.dedent("""
             import asyncio
+            from rich.text import Text
             from tuieval import compare
             from tuieval.tui import EvalsApp, ResultsScreen
             from textual.widgets import DataTable, TabbedContent
@@ -2288,14 +2400,56 @@ class FirstRun(unittest.TestCase):
                     app.screen.fill_pq_summary(compare.per_question(rows), models)
                     await pilot.pause(0.3)
                     t = app.screen.query_one("#pq-summary", DataTable)
-                    names = [str(t.get_row_at(i)[0]) for i in range(t.row_count)]
+                    names = [Text.from_markup(t.get_row_at(i)[0]).plain for i in range(t.row_count)]
                     print(t.row_count, t.region.height, t.virtual_size.width <= t.region.width,
-                          all(n.endswith("-q4_k_m-vllm") and "…" in n for n in names), len(set(names)))
+                          all(n.endswith("-q4_k_m-vllm ×1") and "…" in n for n in names), len(set(names)))
+            asyncio.run(go())
+        """)
+        out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                             env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        # every row, no scroll; a trimmed name keeps its ×n
+        self.assertEqual(out.stdout.strip(), "12 13 True True 12", out.stderr[-2000:])
+
+    def test_per_question_shows_repeats(self):
+        tuieval(self.ws, "add", "my-model", "--server", "local")
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, TabbedContent
+            rows = [{"model": m, "suite": "p", "test": f"p: q{q}", "ok": True, "latency": 1000, "tokens": 10,
+                     "truncated": False, "gen_tps": 20.0, "prompt_tps": None, "repeat": r, "score": 1.0}
+                    for m, reps in (("once", 1), ("thrice", 3)) for q in range(2) for r in range(reps)]
+            rows += [dict(rows[-1], test="p: q2", repeat=0)]   # one more question, answered once
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 40)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    s = app.screen
+                    s.rows, s.partial_rows, s.superseded = rows, [], set()
+                    s.query_one(TabbedContent).active = "tab-per-question"
+                    await pilot.pause(0.3)
+                    s.fill_per_question()
+                    await pilot.pause(0.3)
+                    pq = s.query_one("#per-question", DataTable)
+                    heads = [c.label for c in pq.ordered_columns][len(s.PQ_FIXED):]
+                    print([h.plain for h in heads], [str(h.spans[0].style) for h in heads])
+                    app.result_sorts["per-question"] = ("thrice ×1-3", True)   # as a click on its name does
+                    s.sort_table(pq)
+                    head = pq.ordered_columns[-1].label
+                    print(head.plain, str(head.spans[0].style))
+                    t = s.query_one("#pq-summary", DataTable)
+                    print([t.get_row_at(i)[0] for i in range(t.row_count)])
             asyncio.run(go())
         """)
         out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
-        self.assertEqual(out.stdout.strip(), "12 13 True True 12", out.stderr[-2000:])   # every row, no scroll
+        self.assertEqual(out.stdout.strip().splitlines(), [
+            "['once ×1', 'thrice ×1-3'] ['yellow', 'dim']",   # over the questions shown
+            "thrice ×1-3 ▼ dim",                               # sorting keeps the ×n's colour
+            "['once [yellow]×1[/]', 'thrice [dim]×3[/]']",     # Comparison: only the questions both answered
+        ], out.stderr[-2000:])
 
     def test_results_open_on_pta_index(self):
         tuieval(self.ws, "add", "my-model", "--server", "local")