tuieval 0.2.0.dev98__tar.gz → 0.2.0.dev99__tar.gz

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/CHANGELOG.md

@@ -3,7 +3,7 @@
 ## Unreleased
 
 - Results, Per question: the model summary shows every picked model without scrolling, and fits the window: shorter column names, and long model names cut in the middle (both ends kept). The PTA index table fits the same way.
-- PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `✂ n%` beside it and a **cut off** column in the table, so overthinking stands apart from wrong answers.
+- PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `(n% truncated)` beside it and a **truncated** column in the table, so overthinking stands apart from wrong answers.
 - Results → Insights (also `tuieval insights`): charts with what to do next: every model against a reference (95% CI), the best model per use case, fine-tunes against their base (`base` in models.toml), why models fail, accuracy against tokens or time, question health, and the Certify runs that would settle the strongest models' verdicts. `[defaults] reference` sets the default reference.
 - A server that stops sending tokens mid-answer for 5 minutes (`stall_timeout_ms`) is a server error: retried, never counted against the model. Keep-alive pings no longer stretch a request past `request_timeout_ms`.
 - The run screen's queue shows what each running model is doing: waiting, thinking or answering with ~tokens so far, and how long it's been quiet, so a model you aren't streaming doesn't look stuck.

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev98
+Version: 0.2.0.dev99
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/docs/models.md

@@ -104,7 +104,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
-- **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `✂ n%` beside the number and a **cut off** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
+- **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
 

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev98'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev98')
+__version__ = version = '0.2.0.dev99'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev99')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/src/tuieval/pta.py

@@ -122,7 +122,7 @@ def bars(result, width=20, markup=True):
     are proportional to tokens and time (the longest is the model that used most; shorter is better),
     with how many times the leanest / fastest model's beside them; the A bar is the share of answers
     right (longer is better, full only when every answer passed), then the share cut off at max_tokens in
-    orange (▓ in plain text) and "✂ n%" beside it. markup=False gives plain text."""
+    orange (▓ in plain text) and "(n% truncated)" beside it. markup=False gives plain text."""
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
@@ -145,7 +145,7 @@ def bars(result, width=20, markup=True):
                 cut = x.get("cut_pct") or 0
                 part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
                 if cut:   # after the accuracy, which stays aligned with the other rows'
-                    num = f"{num:>6}  ✂ {compare.pct(cut / 100)}"
+                    num = f"{num:>6}  ({compare.pct(cut / 100)} truncated)"
             else:
                 full = _blocks(width, x.get(measure), most[measure])
                 num = times(x.get(shown))
@@ -178,7 +178,7 @@ def left_out_lines(result):
 def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
-    header = ["model", "tokens/answer", "time/answer", "answers right", "cut off", "tok/s", "TTFT s", "memory GB"]
+    header = ["model", "tokens/answer", "time/answer", "answers right", "truncated", "tok/s", "TTFT s", "memory GB"]
     rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
              f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
@@ -216,8 +216,8 @@ def scope_note(result, models):
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
             "(longer is better)"
-            + ("; orange ✂: answers cut off at max_tokens, out of tokens rather than wrong (they count as wrong)."
-               if any(x.get("cut") for x in result["models"]) else "."))
+            + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
+               "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else "."))
 
 
 def for_engine(e, labels=None, packs=None, results_dir=None):

{tuieval-0.2.0.dev98 → tuieval-0.2.0.dev99}/tests/test_tuieval.py

@@ -1278,14 +1278,14 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((by["thinker"]["A"], by["thinker"]["cut"], by["thinker"]["cut_pct"]), (60, 2, 20))
         self.assertEqual((by["plain"]["cut"], by["plain"]["cut_pct"]), (0, 0))
         bars = {line.split()[0]: line.split() for line in pta.bars(res, 10, markup=False)[2:]}
-        self.assertEqual(bars["thinker"][-4:], ["██████▓▓░░", "60.0%", "✂", "20.0%"])   # right, cut off, wrong
-        self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # no cut-offs: no ✂
+        self.assertEqual(bars["thinker"][-4:], ["██████▓▓░░", "60.0%", "(20.0%", "truncated)"])   # right, cut off, wrong
+        self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # none truncated: nothing beside it
         self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
         header, table = pta.table(res)
-        self.assertEqual(header[4], "cut off")
+        self.assertEqual(header[4], "truncated")
         self.assertEqual({r[0]: r[4] for r in table}, {"thinker": "2/10", "plain": "0/10"})
-        self.assertIn("cut off at max_tokens", pta.scope_note(res, ["thinker", "plain"]))
-        self.assertNotIn("cut off", pta.scope_note(pta.index(rows[10:]), ["plain"]))
+        self.assertIn("truncated", pta.scope_note(res, ["thinker", "plain"]))
+        self.assertNotIn("truncated", pta.scope_note(pta.index(rows[10:]), ["plain"]))
 
     def test_server_errors_and_models_with_few_answers(self):
         from tuieval import pta