tuieval 0.2.0.dev94__tar.gz → 0.2.0.dev98__tar.gz

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Results, Per question: the model summary shows every picked model without scrolling, and fits the window: shorter column names, and long model names cut in the middle (both ends kept). The PTA index table fits the same way.
+- PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `✂ n%` beside it and a **cut off** column in the table, so overthinking stands apart from wrong answers.
 - Results → Insights (also `tuieval insights`): charts with what to do next: every model against a reference (95% CI), the best model per use case, fine-tunes against their base (`base` in models.toml), why models fail, accuracy against tokens or time, question health, and the Certify runs that would settle the strongest models' verdicts. `[defaults] reference` sets the default reference.
 - A server that stops sending tokens mid-answer for 5 minutes (`stall_timeout_ms`) is a server error: retried, never counted against the model. Keep-alive pings no longer stretch a request past `request_timeout_ms`.
 - The run screen's queue shows what each running model is doing: waiting, thinking or answering with ~tokens so far, and how long it's been quiet, so a model you aren't streaming doesn't look stuck.

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev94
+Version: 0.2.0.dev98
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/docs/models.md

@@ -104,7 +104,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
-- **A accuracy**: the share of answers to those questions that passed.
+- **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `✂ n%` beside the number and a **cut off** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
 

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev94'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev94')
+__version__ = version = '0.2.0.dev98'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev98')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/src/tuieval/pta.py

@@ -47,7 +47,7 @@ def memory_gb(infos):
 
 
 def index(rows, models=None, memory=None):
-    """{"questions": n shared, "models": [{model, P, T, A, tokens, total_s, passed, answers, alongside,
+    """{"questions": n shared, "models": [{model, P, T, A, cut, cut_pct, tokens, total_s, passed, answers, alongside,
     fewest, fastest, tps, ttft, memory_gb}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
     most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
@@ -75,6 +75,9 @@ def index(rows, models=None, memory=None):
                     "total_s": sum(c["secs"] or 0 for c in cells),
                     "passed": sum(c["passed"] for c in cells), "answers": answers,
                     "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    # failed only because max_tokens ran out (overthinking, not wrong yet)
+                    "cut": sum(c["truncated"] for c in cells),
+                    "cut_pct": 100 * sum(c["truncated"] for c in cells) / answers if answers else None,
                     # generation speed and time to first token over the same answers (medians)
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
@@ -118,7 +121,8 @@ def bars(result, width=20, markup=True):
     """Lines with a row per model and a bar each for P, T and A: the at-a-glance view. The P and T bars
     are proportional to tokens and time (the longest is the model that used most; shorter is better),
     with how many times the leanest / fastest model's beside them; the A bar is the share of answers
-    right (longer is better, full only when every answer passed). markup=False gives plain text."""
+    right (longer is better, full only when every answer passed), then the share cut off at max_tokens in
+    orange (▓ in plain text) and "✂ n%" beside it. markup=False gives plain text."""
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
@@ -138,12 +142,18 @@ def bars(result, width=20, markup=True):
                 if v is not None and v < 100:   # full only when every answer passed
                     full = min(full, width - 1)
                 num = "-" if v is None else compare.pct(v / 100)
+                cut = x.get("cut_pct") or 0
+                part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
+                if cut:   # after the accuracy, which stays aligned with the other rows'
+                    num = f"{num:>6}  ✂ {compare.pct(cut / 100)}"
             else:
                 full = _blocks(width, x.get(measure), most[measure])
                 num = times(x.get(shown))
-            bar, rest = "█" * full, "░" * (width - full)
-            bar = f"[{color}]{bar}[/][grey37]{rest}[/]" if markup else bar + rest
-            cells.append(f"{bar} {num:>6}" + " " * (cell_w - width - 7))
+                part = 0
+            bar, cut_bar, rest = "█" * full, "▓" * part, "░" * (width - full - part)
+            bar = (f"[{color}]{bar}[/]" + (f"[#ff8c00]{'█' * part}[/]" if part else "") + f"[grey37]{rest}[/]"
+                   if markup else bar + cut_bar + rest)
+            cells.append(f"{bar} {num:>6}" + " " * max(0, cell_w - width - 1 - max(6, len(num))))
         name = f"{x['model']:<{name_w}}"
         lines.append((f"{rich_escape(name) if markup else name}  " + "".join(cells)).rstrip())
     return lines
@@ -168,10 +178,11 @@ def left_out_lines(result):
 def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
-    header = ["model", "tokens/answer", "time/answer", "answers right", "tok/s", "TTFT s", "memory GB"]
+    header = ["model", "tokens/answer", "time/answer", "answers right", "cut off", "tok/s", "TTFT s", "memory GB"]
     rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
-             f"{x['passed']}/{x['answers']}", _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
+             f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
+             _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
              _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
     return header, rows
 
@@ -204,7 +215,9 @@ def scope_note(result, models):
     return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
-            "(longer is better).")
+            "(longer is better)"
+            + ("; orange ✂: answers cut off at max_tokens, out of tokens rather than wrong (they count as wrong)."
+               if any(x.get("cut") for x in result["models"]) else "."))
 
 
 def for_engine(e, labels=None, packs=None, results_dir=None):

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/src/tuieval/tui.py

@@ -355,6 +355,38 @@ def rich_escape(text):
     return text.replace("[", r"\[")
 
 
+def mid_trim(text, width, head=None):
+    """text cut to width cells with … in the middle, so both ends of a model name show (its family and
+    its server suffix, e.g. -llama / -vllm). head: how many leading characters to keep (default: half)."""
+    if cell_len(text) <= width or width < 5:
+        return text
+    head = width - 1 - width // 2 if head is None else min(head, width - 2)
+    return text[:head] + "…" + text[head - width + 1:]
+
+
+def trim_names(names, width):
+    """names cut to width (mid_trim), still telling them apart: when cutting the middle would make two
+    alike, the cut moves into the start they all share; if that's not enough either, names stay whole."""
+    trimmed = [mid_trim(n, width) for n in names]
+    if len(set(trimmed)) == len(set(names)):
+        return trimmed
+    shared = len(os.path.commonprefix(list(names)))
+    head = width - 1 - max(len(n) - shared for n in names)   # keep every name's end from where they differ
+    if head < 1:
+        return list(names)
+    trimmed = [mid_trim(n, width, head=head) for n in names]
+    return trimmed if len(set(trimmed)) == len(set(names)) else list(names)
+
+
+def fit_first_column(header, rows, room, floor=24):
+    """rows with the first cell trimmed in the middle (mid_trim) so a table of these columns fits room
+    cells across, but never below floor cells. Cells are Rich markup; each column adds 2 cells of padding."""
+    rest = sum(2 + max(cell_len(h), *(Text.from_markup(r[i]).cell_len for r in rows))
+               for i, h in enumerate(header) if i)
+    names = trim_names([r[0] for r in rows], max(floor, room - rest - 2))
+    return [[n] + r[1:] for n, r in zip(names, rows)]
+
+
 class BlockDumper(yaml.SafeDumper):
     """Multi-line strings (reference code, expected text) as readable | blocks, not escaped \\n."""
 
@@ -2981,6 +3013,9 @@ class ResultsScreen(Screen):
         self.query_one("#pta-left-out", Static).update(
             "\n".join(f"[dim]{rich_escape(line)}[/dim]" for line in pta.left_out_lines(result)))
         header, table = pta.table(result)
+        self.pta_shown = bool(table)    # to refit on resize
+        if table:
+            table = fit_first_column(header, table, (self.size.width or self.app.size.width) - 3)
         for i, h in enumerate(header):   # sized to the contents: added before the rows, a column keeps its header's width
             t.add_column(h, width=max(len(h), *(len(r[i]) for r in table)) if table else None)
         t.add_rows(table)
@@ -3065,6 +3100,14 @@ class ResultsScreen(Screen):
                 f"{os.path.basename(os.path.dirname(p))} · {os.path.basename(p)[:-5]}" for p in sorted(self.superseded))
                 + ". Only the new run's answers (◐) are shown for these.[/yellow]" if self.superseded else ""))
 
+    def on_resize(self, event):
+        """Refit the tables whose model names are trimmed to the window's width."""
+        if getattr(self, "pq_summary_args", None):
+            self.fill_pq_summary(*self.pq_summary_args)
+            self.sort_table(self.query_one("#pq-summary", DataTable))
+        if getattr(self, "pta_shown", False):
+            self.fill_pta()
+
     def fill_pq_summary(self, tests, models):
         """One row per shown model over the questions they ALL answered, so totals compare fairly;
         with two models, a head-to-head line."""
@@ -3081,9 +3124,11 @@ class ResultsScreen(Screen):
             title.update("[yellow]These models have no question in common (with the filters above).[/yellow]")
             table.display = False
             return
+        self.pq_summary_args = (tests, models)    # to refit on resize
         rows = compare.compare_models(shared, models)
-        table.add_columns("Model", "Total time", "Passed", "Median tokens", "Total tokens", "Tokens per ✓", "Median s",
-                          "Avg prompt tok/s", "Avg gen tok/s", "Fewest tokens", "Fastest", "Cut off")
+        # short headings so every column fits; the line above the table says what each one is
+        header = ("Model", "Time", "Passed", "Med tok", "Tokens", "Tok/✓", "Med s", "Prompt t/s", "Gen t/s",
+                  "Fewest", "Fastest", "Cut off")
         best = {k: max((r[k] for r in rows if r[k] is not None), default=None)
                 for k in ("passed", "fewest", "fastest", "avg_pp_tps", "avg_gen_tps")}
         low = {k: min(r[k] for r in rows if r[k] is not None) if any(r[k] is not None for r in rows) else None
@@ -3092,9 +3137,10 @@ class ResultsScreen(Screen):
         def hi(v, text, key, good):
             ref = best.get(key) if good == "max" else low.get(key)
             return f"[bold green]{text}[/bold green]" if v is not None and v == ref and len(rows) > 1 else text
+        cells = []
         for r in rows:
             tt = r["total_secs"]
-            table.add_row(r["model"],
+            cells.append([r["model"],
                           hi(tt, ("-" if not tt else f"{tt:.1f}s" if tt < 60 else fmt_secs(tt)), "total_secs", "min"),
                           hi(r["passed"], f"{r['passed']}/{r['n']}", "passed", "max"),
                           hi(r["med_tokens"], f"{r['med_tokens']:,.0f}" if r["med_tokens"] else "-", "med_tokens", "min"),
@@ -3107,7 +3153,9 @@ class ResultsScreen(Screen):
                              "max"),
                           hi(r["fewest"], str(r["fewest"]), "fewest", "max"),
                           hi(r["fastest"], str(r["fastest"]), "fastest", "max"),
-                          f"[red]{r['cut']}[/red]" if r["cut"] else "0")
+                          f"[red]{r['cut']}[/red]" if r["cut"] else "0"])
+        table.add_columns(*header)
+        table.add_rows(fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3))
         line = f"[b]Comparison[/b] over the {len(shared)} questions all {len(models)} answered"
         if len(models) == 2:
             h = compare.head_to_head(shared, *models)
@@ -3121,9 +3169,10 @@ class ResultsScreen(Screen):
         if shared_speed:
             line += (f"\n[yellow]Some answers of {', '.join(shared_speed)} ran alongside other models, which "
                      "shared the machine's speed: compare their times with care.[/yellow]")
-        title.update(line + "  [dim]Fewest tokens / Fastest = questions won; ties count for both. Total tokens / "
-                     "Total time add up each question's median. Avg tok/s = all tokens over all the time they took "
-                     "(cached prompts left out).[/dim]")
+        title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
+                     "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
+                     "they took (cached prompts left out). Fewest / Fastest: questions won; ties count for both. "
+                     "Cut off: answers that hit max_tokens.[/dim]")
 
     @staticmethod
     def pq_cell(c, best, here):
@@ -3277,7 +3326,7 @@ class EvalsApp(App):
     #pq-controls Select { width: 22; margin-right: 1; }
     #pq-models { margin-right: 1; min-width: 20; }
     #pq-compare-title { padding: 0 1; height: auto; }
-    #pq-summary { height: auto; max-height: 10; }
+    #pq-summary { height: auto; }   /* every picked model, no scrolling: the tab scrolls instead */
     #pq-legend { padding: 0 1; height: auto; }
     /* Tabs must read as tabs, not as a line of text: each tab is a button-like chip on its own band,
        the active one filled with the accent colour. (Textual's default: dimmed words on the background.) */

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev98}/tests/test_tuieval.py

@@ -1264,6 +1264,29 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((two_wrong[-2], two_wrong[-1]), ("█████████░", "99.6%"))   # not a full bar
         self.assertEqual((perfect[-2], perfect[-1]), ("██████████", "100%"))
 
+    def test_cut_off_answers_show_in_the_accuracy_bar(self):
+        from tuieval import pta
+
+        def row(model, test, ok, truncated=False):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": 1000,
+                    "tokens": 10, "truncated": truncated, "gen_tps": None, "repeat": 0}
+        rows = ([row("thinker", f"q{i}", i < 6) for i in range(8)]
+                + [row("thinker", f"q{i}", False, truncated=True) for i in (8, 9)]   # out of tokens
+                + [row("plain", f"q{i}", i < 6) for i in range(10)])
+        res = pta.index(rows)
+        by = {x["model"]: x for x in res["models"]}
+        self.assertEqual((by["thinker"]["A"], by["thinker"]["cut"], by["thinker"]["cut_pct"]), (60, 2, 20))
+        self.assertEqual((by["plain"]["cut"], by["plain"]["cut_pct"]), (0, 0))
+        bars = {line.split()[0]: line.split() for line in pta.bars(res, 10, markup=False)[2:]}
+        self.assertEqual(bars["thinker"][-4:], ["██████▓▓░░", "60.0%", "✂", "20.0%"])   # right, cut off, wrong
+        self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # no cut-offs: no ✂
+        self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
+        header, table = pta.table(res)
+        self.assertEqual(header[4], "cut off")
+        self.assertEqual({r[0]: r[4] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertIn("cut off at max_tokens", pta.scope_note(res, ["thinker", "plain"]))
+        self.assertNotIn("cut off", pta.scope_note(pta.index(rows[10:]), ["plain"]))
+
     def test_server_errors_and_models_with_few_answers(self):
         from tuieval import pta
 
@@ -1637,6 +1660,51 @@ class FirstRun(unittest.TestCase):
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60).stdout.strip()
         self.assertEqual(out, "['first'] ['my-model'] smoke False")   # tuning shows once something has run
 
+    def test_trimmed_names_keep_both_ends(self):
+        from tuieval.tui import fit_first_column, mid_trim, trim_names
+        name = "some-author--some-long-model-name-q4_k_m-vllm"
+        self.assertEqual(mid_trim(name, 20), "some-auth…4_k_m-vllm")
+        self.assertEqual(mid_trim("short", 20), "short")
+        rows = [[name, "[green]1/2[/green]", "10"], ["b", "2/2", "1,000"]]
+        fitted = fit_first_column(["Model", "Passed", "Tokens"], rows, 45)   # 2 + 8 + 8 cells for the rest
+        self.assertEqual([len(r[0]) for r in fitted], [27, 1])
+        self.assertEqual(fitted[0][1:], rows[0][1:])
+        self.assertEqual(len(fit_first_column(["Model", "Passed"], rows, 10)[0][0]), 24)   # never below the floor
+        twins = [f"author--model-number-{i}-and-a-long-tail" for i in range(3)]       # alike but in the middle
+        self.assertEqual(trim_names(twins, 20), ["au…0-and-a-long-tail", "au…1-and-a-long-tail",
+                                                 "au…2-and-a-long-tail"])
+        self.assertEqual(trim_names(twins, 10), twins)          # can't keep them apart that short: whole
+
+    def test_per_question_summary_fits_every_model(self):
+        tuieval(self.ws, "add", "my-model", "--server", "local")
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval import compare
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, TabbedContent
+            models = [f"some-author--a-rather-long-model-name-number-{i:02d}-q4_k_m-vllm" for i in range(12)]
+            rows = [{"model": m, "suite": "p", "test": f"p: q{q}", "ok": q % 2 == 0, "latency": 1000, "tokens": 10,
+                     "truncated": False, "gen_tps": 20.0, "prompt_tps": None, "repeat": 0} for m in models for q in range(3)]
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(120, 40)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    app.screen.query_one(TabbedContent).active = "tab-per-question"
+                    await pilot.pause(0.3)
+                    app.screen.fill_pq_summary(compare.per_question(rows), models)
+                    await pilot.pause(0.3)
+                    t = app.screen.query_one("#pq-summary", DataTable)
+                    names = [str(t.get_row_at(i)[0]) for i in range(t.row_count)]
+                    print(t.row_count, t.region.height, t.virtual_size.width <= t.region.width,
+                          all(n.endswith("-q4_k_m-vllm") and "…" in n for n in names), len(set(names)))
+            asyncio.run(go())
+        """)
+        out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                             env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        self.assertEqual(out.stdout.strip(), "12 13 True True 12", out.stderr[-2000:])   # every row, no scroll
+
     def test_results_open_on_pta_index(self):
         tuieval(self.ws, "add", "my-model", "--server", "local")
         code = textwrap.dedent("""