tuieval 0.2.0.dev94__tar.gz → 0.2.0.dev96__tar.gz

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `✂ n%` beside it and a **cut off** column in the table, so overthinking stands apart from wrong answers.
 - Results → Insights (also `tuieval insights`): charts with what to do next: every model against a reference (95% CI), the best model per use case, fine-tunes against their base (`base` in models.toml), why models fail, accuracy against tokens or time, question health, and the Certify runs that would settle the strongest models' verdicts. `[defaults] reference` sets the default reference.
 - A server that stops sending tokens mid-answer for 5 minutes (`stall_timeout_ms`) is a server error: retried, never counted against the model. Keep-alive pings no longer stretch a request past `request_timeout_ms`.
 - The run screen's queue shows what each running model is doing: waiting, thinking or answering with ~tokens so far, and how long it's been quiet, so a model you aren't streaming doesn't look stuck.

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev94
+Version: 0.2.0.dev96
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/docs/models.md

@@ -104,7 +104,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
-- **A accuracy**: the share of answers to those questions that passed.
+- **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `✂ n%` beside the number and a **cut off** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
 

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev94'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev94')
+__version__ = version = '0.2.0.dev96'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev96')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/src/tuieval/pta.py

@@ -47,7 +47,7 @@ def memory_gb(infos):
 
 
 def index(rows, models=None, memory=None):
-    """{"questions": n shared, "models": [{model, P, T, A, tokens, total_s, passed, answers, alongside,
+    """{"questions": n shared, "models": [{model, P, T, A, cut, cut_pct, tokens, total_s, passed, answers, alongside,
     fewest, fastest, tps, ttft, memory_gb}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
     most-answered model}, best accuracy first. models: the ones to compare (default: every model in rows
@@ -75,6 +75,9 @@ def index(rows, models=None, memory=None):
                     "total_s": sum(c["secs"] or 0 for c in cells),
                     "passed": sum(c["passed"] for c in cells), "answers": answers,
                     "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    # failed only because max_tokens ran out (overthinking, not wrong yet)
+                    "cut": sum(c["truncated"] for c in cells),
+                    "cut_pct": 100 * sum(c["truncated"] for c in cells) / answers if answers else None,
                     # generation speed and time to first token over the same answers (medians)
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
@@ -118,7 +121,8 @@ def bars(result, width=20, markup=True):
     """Lines with a row per model and a bar each for P, T and A: the at-a-glance view. The P and T bars
     are proportional to tokens and time (the longest is the model that used most; shorter is better),
     with how many times the leanest / fastest model's beside them; the A bar is the share of answers
-    right (longer is better, full only when every answer passed). markup=False gives plain text."""
+    right (longer is better, full only when every answer passed), then the share cut off at max_tokens in
+    orange (▓ in plain text) and "✂ n%" beside it. markup=False gives plain text."""
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
@@ -138,12 +142,18 @@ def bars(result, width=20, markup=True):
                 if v is not None and v < 100:   # full only when every answer passed
                     full = min(full, width - 1)
                 num = "-" if v is None else compare.pct(v / 100)
+                cut = x.get("cut_pct") or 0
+                part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
+                if cut:   # after the accuracy, which stays aligned with the other rows'
+                    num = f"{num:>6}  ✂ {compare.pct(cut / 100)}"
             else:
                 full = _blocks(width, x.get(measure), most[measure])
                 num = times(x.get(shown))
-            bar, rest = "█" * full, "░" * (width - full)
-            bar = f"[{color}]{bar}[/][grey37]{rest}[/]" if markup else bar + rest
-            cells.append(f"{bar} {num:>6}" + " " * (cell_w - width - 7))
+                part = 0
+            bar, cut_bar, rest = "█" * full, "▓" * part, "░" * (width - full - part)
+            bar = (f"[{color}]{bar}[/]" + (f"[#ff8c00]{'█' * part}[/]" if part else "") + f"[grey37]{rest}[/]"
+                   if markup else bar + cut_bar + rest)
+            cells.append(f"{bar} {num:>6}" + " " * max(0, cell_w - width - 1 - max(6, len(num))))
         name = f"{x['model']:<{name_w}}"
         lines.append((f"{rich_escape(name) if markup else name}  " + "".join(cells)).rstrip())
     return lines
@@ -168,10 +178,11 @@ def left_out_lines(result):
 def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
-    header = ["model", "tokens/answer", "time/answer", "answers right", "tok/s", "TTFT s", "memory GB"]
+    header = ["model", "tokens/answer", "time/answer", "answers right", "cut off", "tok/s", "TTFT s", "memory GB"]
     rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
-             f"{x['passed']}/{x['answers']}", _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
+             f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
+             _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
              _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
     return header, rows
 
@@ -204,7 +215,9 @@ def scope_note(result, models):
     return (f"Compared on the {result['questions']} questions all {len(result['models'])} model(s) answered. "
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
-            "(longer is better).")
+            "(longer is better)"
+            + ("; orange ✂: answers cut off at max_tokens, out of tokens rather than wrong (they count as wrong)."
+               if any(x.get("cut") for x in result["models"]) else "."))
 
 
 def for_engine(e, labels=None, packs=None, results_dir=None):

{tuieval-0.2.0.dev94 → tuieval-0.2.0.dev96}/tests/test_tuieval.py

@@ -1264,6 +1264,29 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((two_wrong[-2], two_wrong[-1]), ("█████████░", "99.6%"))   # not a full bar
         self.assertEqual((perfect[-2], perfect[-1]), ("██████████", "100%"))
 
+    def test_cut_off_answers_show_in_the_accuracy_bar(self):
+        from tuieval import pta
+
+        def row(model, test, ok, truncated=False):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": 1000,
+                    "tokens": 10, "truncated": truncated, "gen_tps": None, "repeat": 0}
+        rows = ([row("thinker", f"q{i}", i < 6) for i in range(8)]
+                + [row("thinker", f"q{i}", False, truncated=True) for i in (8, 9)]   # out of tokens
+                + [row("plain", f"q{i}", i < 6) for i in range(10)])
+        res = pta.index(rows)
+        by = {x["model"]: x for x in res["models"]}
+        self.assertEqual((by["thinker"]["A"], by["thinker"]["cut"], by["thinker"]["cut_pct"]), (60, 2, 20))
+        self.assertEqual((by["plain"]["cut"], by["plain"]["cut_pct"]), (0, 0))
+        bars = {line.split()[0]: line.split() for line in pta.bars(res, 10, markup=False)[2:]}
+        self.assertEqual(bars["thinker"][-4:], ["██████▓▓░░", "60.0%", "✂", "20.0%"])   # right, cut off, wrong
+        self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # no cut-offs: no ✂
+        self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
+        header, table = pta.table(res)
+        self.assertEqual(header[4], "cut off")
+        self.assertEqual({r[0]: r[4] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertIn("cut off at max_tokens", pta.scope_note(res, ["thinker", "plain"]))
+        self.assertNotIn("cut off", pta.scope_note(pta.index(rows[10:]), ["plain"]))
+
     def test_server_errors_and_models_with_few_answers(self):
         from tuieval import pta