tuieval 0.2.0.dev121__tar.gz → 0.2.0.dev123__tar.gz

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Answers that used more tokens than the `max_tokens` sent (a provider that didn't stop them) are marked over budget: `(n over budget)` in the PTA index, an **over budget** column, `over` in Per question and a note in the answer view. `[defaults] over_budget = "fail"` fails them instead. Works on stored results.
 - Fix: Results → Per question opened the answers of the model one column to the right of the cell picked, and crashed on the last model's column.
 - Fix: `v` on the run screen switches to the last model still running when the stream shows one that has finished.
 - Setup opens with no models ticked: at launch, after New run, and after Back from a run or tune. Packs and options are still remembered.

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev121
+Version: 0.2.0.dev123
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/docs/models.md

@@ -105,6 +105,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **P parsimony (tokens)**: the tokens used, reasoning and answer, on every question compared (each question at its median over repeats). Beside the bar: how many times the leanest model's, e.g. `2.4×` (`1.0×` is the leanest, ★ in the table's **tokens/answer**). Unlike time it doesn't depend on your machine: it's how much a model says to get to an answer, which is what you pay for on an API and what fills its context.
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
+- **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
 
@@ -207,6 +208,7 @@ A workspace's `models.toml` only needs what differs from these. `[defaults]` fil
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 # reference = "<label>"        # the model Results → Insights compares against by default
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev121'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev121')
+__version__ = version = '0.2.0.dev123'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev123')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/builtin.toml

@@ -6,6 +6,7 @@
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
 

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/client.py

@@ -42,6 +42,26 @@ def server_error_row(rec):
     return not rec.get("pass") and (is_server_error(rec.get("reason")) or is_unavailable(rec.get("reason")))
 
 
+OVER_BUDGET_MODES = ("flag", "fail")
+
+
+def mark_over_budget(records, limit, mode="flag"):
+    """Answers that used more tokens than the max_tokens sent and weren't cut off there: some providers
+    don't stop thinking at the limit. Each gets `over_budget` = the limit; with mode "fail", a passing
+    one fails. Estimated token counts are never flagged."""
+    if not limit:
+        return records
+    for r in records:
+        if r.get("finish") == "length" or r.get("tokens_estimated") or (r.get("completion_tokens") or 0) <= limit:
+            continue
+        r["over_budget"] = limit
+        if mode == "fail" and r.get("pass"):
+            r.update({"pass": False, "score": 0.0, "severity": None,
+                      "reason": f"OVER BUDGET: {r['completion_tokens']:,} tokens, over the {limit:,} limit sent; "
+                                + r.get("reason", "")})
+    return records
+
+
 class Cancelled(Exception):
     pass
 

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/compare.py

@@ -51,6 +51,8 @@ def _load_native(data):
             "ok": bool(r["pass"]), "score": float(r["score"]), "reason": r["reason"],
             "error": r["reason"].startswith(("server returned", "connection error")),
             "truncated": r.get("finish") == "length", "output": r.get("answer") or "",
+            # tokens past the max_tokens sent, which the provider didn't stop (client.mark_over_budget)
+            "over_budget": r.get("over_budget"),
             "tokens": r.get("completion_tokens") or 0, "reasoning_tokens": r.get("reasoning_tokens"),
             # a reused (cached) prompt makes time-to-first-token and prompt speed look faster than they are
             "answer_tokens": r.get("answer_tokens"), "ttft": None if r.get("cached_tokens") else r.get("ttft_s"),
@@ -67,13 +69,16 @@ def _load_native(data):
 
 def load(paths):
     """(rows, infos): one row per answer; one info per (model, result file)."""
+    from . import engine
     rows, infos = [], []
+    mode = over_budget_mode(paths)
     for path in paths:
         with open(path) as f:
             data = json.load(f)
         if data.get("format") != "evals/1":
             continue
         data["results"] = [r for r in data["results"] if not client.server_error_row(r)]
+        client.mark_over_budget(data["results"], engine.sent_max_tokens(data), mode)
         r, i = _load_native(data)
         for row, rec in zip(r, data["results"]):
             row.update(raw=rec, path=path, pack_fp=data["pack"]["fingerprint"])
@@ -93,17 +98,31 @@ def load(paths):
     return rows, infos
 
 
+def _configs(paths):
+    """The models.toml configs of the workspaces these results/<label>/<pack>.json are from (unreadable
+    ones left out)."""
+    from . import engine
+    out = []
+    for root in sorted({os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(p)))) for p in paths}):
+        try:
+            out.append(engine.load_config(os.path.join(root, "models.toml")))
+        except Exception:
+            continue
+    return out
+
+
+def over_budget_mode(paths):
+    """[defaults] over_budget of the workspace(s) the results are from: "fail" if any says so."""
+    return "fail" if any(c["defaults"]["over_budget"] == "fail" for c in _configs(paths)) else "flag"
+
+
 def configured_hosted(paths):
     """(labels, label prefixes) models.toml says are hosted, in the workspaces these results/<label>/
     <pack>.json are from: its hosted models, and the prefix of models named at run time on a hosted
     server (e.g. "or-"). Covers hosted models whose results aren't loaded."""
     from . import engine
     labels, prefixes = set(), set()
-    for root in {os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(p)))) for p in paths}:
-        try:
-            cfg = engine.load_config(os.path.join(root, "models.toml"))
-        except Exception:
-            continue    # no readable models.toml there: the results' own marks still apply
+    for cfg in _configs(paths):   # no readable models.toml: the results' own marks still apply
         labels |= {m["label"] for m in cfg["models"] if engine.hosted(cfg, m)}
         prefixes |= {engine.remote_label(n, s, "") for n, s in cfg["servers"].items()
                      if s.get("any_model") and engine.hosted(cfg, {"server": n})}
@@ -125,6 +144,7 @@ def load_partials(results_dir=None):
     <pack>.json.partial.jsonl, marked partial. They never count toward scores or verdicts."""
     results_dir = results_dir or workspace.path("results")
     rows = []
+    mode = over_budget_mode([os.path.join(results_dir, "x", "x.json")])
     for path in sorted(glob.glob(os.path.join(results_dir, "*", "*" + PARTIAL_SUFFIX))):
         label = os.path.basename(os.path.dirname(path))
         if label in ("smoke", "archive"):
@@ -137,6 +157,9 @@ def load_partials(results_dir=None):
         if len(lines) < 2:
             continue
         head, recs = lines[0], [r for r in lines[1:] if not client.server_error_row(r)]
+        # the limit sent: recorded in the header (older ones: the model's own)
+        client.mark_over_budget(recs, head.get("max_tokens") or ((head.get("settings") or {}).get("sampling") or {})
+                                .get("max_tokens"), mode)
         data = {"model": {"label": label}, "pack": {"name": os.path.basename(path)[:-len(PARTIAL_SUFFIX)],
                                                     "fingerprint": head.get("pack_fingerprint")},
                 "run": {}, "results": recs}
@@ -222,7 +245,9 @@ def scorecard(rows, rng=None):
     models = sorted({r["model"] for r in rows})
     suites = sorted({r["suite"] for r in rows})
     extra = dict(graders.COLUMNS)   # columns your graders add (graders.scorecard_column)
-    header = ["model"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc", "rep"]
+    over = any(r.get("over_budget") for r in rows)   # a column only when some answer ran past max_tokens
+    header = ["model"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
+                                   *(["over budget"] if over else []), "rep"]
     table = []
     for m in models:
         mine = [r for r in rows if r["model"] == m]
@@ -243,6 +268,7 @@ def scorecard(rows, rng=None):
                 cell = "error"
             line.append("-" if cell is None else str(cell))
         line += [str(sum(r["error"] for r in mine)), str(sum(r["truncated"] for r in mine)),
+                 *([str(sum(bool(r.get("over_budget")) for r in mine))] if over else []),
                  str(sum(repetitive(r["output"]) for r in mine))]
         table.append(line)
     return header, table
@@ -339,6 +365,7 @@ def per_question(rows):
         for c in t["cells"].values():
             v = sorted(c["rows"], key=lambda r: r.get("repeat", 0))
             c.update(rows=v, n=len(v), passed=sum(r["ok"] for r in v), truncated=sum(r["truncated"] for r in v),
+                     over=sum(bool(r.get("over_budget")) for r in v),
                      partial=any(r.get("partial") for r in v),
                      tokens=median(r["tokens"] or None for r in v), secs=median(r["latency"] / 1000 or None for r in v),
                      tps=median(r["gen_tps"] for r in v))
@@ -386,7 +413,8 @@ def compare_models(tests, models):
                     "med_tokens": median(c["tokens"] for c in cs), "med_secs": median(c["secs"] for c in cs),
                     "total_secs": sum(c["secs"] or 0 for c in cs) or None,
                     "tokens_per_pass": total / passed if passed else None,
-                    "fewest": fewest, "fastest": fastest, "cut": sum(c["truncated"] for c in cs)})
+                    "fewest": fewest, "fastest": fastest, "cut": sum(c["truncated"] for c in cs),
+                    "over": sum(c.get("over", 0) for c in cs)})
     return out
 
 

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/engine.py

@@ -107,6 +107,9 @@ def load_config(path):
         cfg = tomllib.load(f)
     builtin = builtin_config()
     cfg["defaults"] = {**builtin["defaults"], **cfg.get("defaults", {})}
+    if cfg["defaults"]["over_budget"] not in client.OVER_BUDGET_MODES:
+        raise ConfigError(f"{path}: [defaults] over_budget must be one of "
+                          f"{', '.join(map(repr, client.OVER_BUDGET_MODES))}, not {cfg['defaults']['over_budget']!r}")
     cfg.setdefault("sampling", builtin["sampling"])
     own = cfg.get("servers", {})
     cfg["servers"] = {**builtin["servers"], **own}
@@ -528,6 +531,11 @@ def read_result(path):
     return data
 
 
+def sent_max_tokens(data):
+    """The max_tokens a result file's requests were sent with: the pack's own, else the model's."""
+    return (data.get("pack") or {}).get("max_tokens") or (data.get("settings") or {}).get("max_tokens")
+
+
 def server_error(log, start):
     """Why a server stopped while loading: the last error line it wrote since start, when it
     names one (some servers refuse a context that does not fit with one)."""
@@ -925,9 +933,13 @@ class Engine:
         return cached[1]
 
     def records(self, label, pack_name):
-        """Stored result rows for a model and pack, or [] (smoke runs excluded)."""
+        """Stored result rows for a model and pack, or [] (smoke runs excluded), as scored: copies, with
+        answers past the max_tokens sent marked (and failed, with [defaults] over_budget = "fail")."""
         data = self._result(self.result_path(label, pack_name))
-        return data["results"] if data else []
+        if not data:
+            return []
+        return client.mark_over_budget([dict(r) for r in data["results"]], sent_max_tokens(data),
+                                       self.cfg["defaults"]["over_budget"])
 
     def certify_repeat(self, pack):
         return pack.certify_repeat or self.cfg["defaults"]["repeat"]
@@ -1126,7 +1138,7 @@ class Engine:
 
     def _partial_header(self, job):
         return {"pack_fingerprint": job.pack.fingerprint, "settings_fingerprint": settings_fingerprint(job.settings),
-                "settings": job.settings}
+                "settings": job.settings, "max_tokens": job.pack.max_tokens or job.sampling["max_tokens"]}
 
     def _header_ok(self, head, job):
         return header_matches(head, job.pack.fingerprint, job.settings)

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/insights.py

@@ -278,7 +278,7 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith("TRUNCATED"):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/pta.py

@@ -78,6 +78,8 @@ def index(rows, models=None, memory=None):
                     # failed only because max_tokens ran out (overthinking, not wrong yet)
                     "cut": sum(c["truncated"] for c in cells),
                     "cut_pct": 100 * sum(c["truncated"] for c in cells) / answers if answers else None,
+                    # ran past the max_tokens sent: the provider didn't stop it, so it had more room than asked
+                    "over": sum(c.get("over", 0) for c in cells),
                     # generation speed and time to first token over the same answers (medians)
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
@@ -147,6 +149,9 @@ def bars(result, width=20, markup=True, click=None):
                 part = min(width - full, max(1, round(width * cut / 100))) if cut else 0
                 if cut:   # after the accuracy, which stays aligned with the other rows'
                     num = f"{num:>6}  ({compare.pct(cut / 100)} truncated)"
+                if x.get("over"):
+                    note = f"({x['over']} over budget)"
+                    num = (f"{num}  " if cut else f"{num:>6}  ") + (f"[yellow]{note}[/yellow]" if markup else note)
             else:
                 full = _blocks(width, x.get(measure), most[measure])
                 num = times(x.get(shown))
@@ -183,10 +188,13 @@ def left_out_lines(result):
 def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
-    header = ["model", "tokens/answer", "time/answer", "answers right", "truncated", "tok/s", "TTFT s", "memory GB"]
+    over = any(x.get("over") for x in result["models"])   # a column only when some answer ran past max_tokens
+    header = ["model", "tokens/answer", "time/answer", "answers right", "truncated", *(["over budget"] if over else []),
+              "tok/s", "TTFT s", "memory GB"]
     rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
              f"{x['passed']}/{x['answers']}", f"{x.get('cut') or 0}/{x['answers']}",
+             *([f"{x.get('over') or 0}/{x['answers']}"] if over else []),
              _num(x.get("tps"), ".0f"), _num(x.get("ttft"), ".1f"),
              _num(x.get("memory_gb"), ".1f")] for x in result["models"]]
     return header, rows
@@ -222,7 +230,10 @@ def scope_note(result, models):
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
             "(longer is better)"
             + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
-               "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else "."))
+               "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else ".")
+            + (" Over budget: answers that used more tokens than the max_tokens sent, because the provider didn't "
+               "stop them, so the model had more room than the others (over_budget = \"fail\" in models.toml's defaults fails them)."
+               if any(x.get("over") for x in result["models"]) else ""))
 
 
 def for_engine(e, labels=None, packs=None, results_dir=None):

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/src/tuieval/tui.py

@@ -610,7 +610,9 @@ class AnswerScreen(ModalScreen):
         split = ", ".join(f"{k} {rec[k + '_tokens']:,}" for k in ("reasoning", "answer")
                           if rec.get(k + "_tokens") is not None)
         facts = [f"finish {finish}",
-                 (f"{toks:,} tokens" + (f" ({split})" if split else "")) if toks else "",
+                 (f"{toks:,} tokens" + (f" ({split})" if split else "")
+                  + (f" [yellow]over the {rec['over_budget']:,} limit sent (the provider didn't stop it)[/yellow]"
+                     if rec.get("over_budget") else "")) if toks else "",
                  f"prompt {rec['prompt_tokens']:,} tokens" if rec.get("prompt_tokens") else "",
                  (f"TTFT {rec['ttft_s']:.1f}s" + (f" [yellow]⟲ {rec['cached_tokens']} cached[/yellow]"
                                                    if rec.get("cached_tokens") else "")) if rec.get("ttft_s") is not None else "",
@@ -3317,6 +3319,7 @@ class ResultsScreen(Screen):
         self.query_one("#pq-legend", Static).update(
             f"[dim]{len(tests)} questions. Each cell: result (passes/answers if repeated), median tokens, median seconds. "
             "[green]bold[/green] = fewest tokens among passing answers · [red]cut[/red] = hit max_tokens · "
+            "[yellow]over[/yellow] = ran past max_tokens (the provider didn't stop it) · "
             "◐ = pack unfinished (not in scores) · dim seconds = hosted, or measured on another machine; tokens compare "
             "across machines, seconds don't. ×tokens = most / fewest tokens across models. Enter on a cell opens the "
             "answers. Click a column name to sort by it (a model's column: by its tokens).[/dim]" + ("\n[yellow]Left out, being rerun with other questions or settings: " + ", ".join(
@@ -3376,7 +3379,8 @@ class ResultsScreen(Screen):
                              "max"),
                           hi(r["fewest"], str(r["fewest"]), "fewest", "max"),
                           hi(r["fastest"], str(r["fastest"]), "fastest", "max"),
-                          f"[red]{r['cut']}[/red]" if r["cut"] else "0"])
+                          (f"[red]{r['cut']}[/red]" if r["cut"] else "0")
+                          + (f" [yellow]+{r['over']} over[/yellow]" if r.get("over") else "")])
         table.add_columns(*header)
         table.add_rows(fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3))
         line = f"[b]Comparison[/b] over the {len(shared)} questions all {len(models)} answered"
@@ -3413,6 +3417,7 @@ class ResultsScreen(Screen):
         if remote:
             secs = f"[dim]{secs}[/dim]"
         return (f"{mark} {tok_s} {secs}" + (" [red]cut[/red]" if c["truncated"] else "")
+                + (" [yellow]over[/yellow]" if c.get("over") else "")
                 + (" [cyan]◐[/cyan]" if c["partial"] else ""))
 
     PQ_FIXED = ("Test", "Pack", "Level", "Added", "×tokens")   # Per question's columns before the models

{tuieval-0.2.0.dev121 → tuieval-0.2.0.dev123}/tests/test_tuieval.py

@@ -2421,6 +2421,81 @@ def fake_gguf(path, embedding=5120, size=0, kv_heads=4):
         f.write(out + b"\0" * max(0, size - len(out)))
 
 
+class OverBudget(unittest.TestCase):
+    """Answers that used more tokens than the max_tokens sent (a provider that didn't stop them) are
+    marked when results are read: flagged, or failed with [defaults] over_budget = "fail"."""
+
+    def setUp(self):
+        from tuieval import packs
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        d = os.path.join(self.ws, "packs", "p")
+        os.makedirs(d)
+        write(os.path.join(d, "pack.toml"), 'label = "p"\nmax_tokens = 100\n[gate]\nmin_accuracy = 0.5\n')
+        write(os.path.join(d, "tests.yaml"), "".join(f"- id: q{i}\n  input: q{i}\n  expected: 7\n" for i in range(4)))
+        rec = lambda i, ok, toks, **kw: {"test": f"q{i}", "description": f"q{i}", "pass": ok, "score": float(ok),
+                                         "reason": "ok" if ok else "wrong", "finish": "stop", "repeat": 0,
+                                         "completion_tokens": toks, **kw}
+        self.path = os.path.join(self.ws, "results", "m", "p.json")
+        os.makedirs(os.path.dirname(self.path))
+        with open(self.path, "w") as f:
+            json.dump({"format": "evals/1", "model": {"label": "m"},
+                       "pack": {"name": "p", "fingerprint": packs.load_pack(d).fingerprint, "max_tokens": 100},
+                       "run": {}, "settings": {"max_tokens": 16384},
+                       "results": [rec(0, True, 150),                              # past the pack's 100: over
+                                   rec(1, True, 90),
+                                   rec(2, False, 100, finish="length"),            # cut off: truncated, not over
+                                   rec(3, True, 500, tokens_estimated=True)]}, f)  # an estimate: never flagged
+
+    def tearDown(self):
+        self.tmp.cleanup()
+
+    def set_mode(self, mode):
+        path = os.path.join(self.ws, "models.toml")
+        write(path, read(path).replace("[defaults]\n", f'[defaults]\nover_budget = "{mode}"\n', 1))
+
+    def test_flag_marks_copies_and_shows_counts(self):
+        from tuieval import compare, engine, pta
+        e = engine.Engine(root=self.ws)
+        recs = e.records("m", "p")
+        self.assertEqual([r.get("over_budget") for r in recs], [100, None, None, None])
+        self.assertEqual([r["pass"] for r in recs], [True, True, False, True])
+        self.assertNotIn("over_budget", e._result(self.path)["results"][0])   # what a rerun writes back stays as stored
+        self.assertNotIn("over_budget", read(self.path))
+        rows, _ = compare.load([self.path])
+        self.assertEqual([r["over_budget"] for r in rows], [100, None, None, None])
+        cell = compare.per_question(rows)[0]["cells"]["m"]
+        self.assertEqual((cell["over"], cell["passed"]), (1, 1))
+        header, table = pta.table(pta.index(rows))
+        self.assertEqual(table[0][header.index("over budget")], "1/4")
+        self.assertIn("(1 over budget)", "\n".join(pta.bars(pta.index(rows), markup=False)))
+        self.assertIn("over budget", compare.scorecard(rows)[0])
+
+    def test_fail_mode_fails_them(self):
+        from tuieval import compare, engine
+        self.set_mode("fail")   # records() is what verdicts are computed from
+        e = engine.Engine(root=self.ws)
+        recs = e.records("m", "p")
+        self.assertEqual([r["pass"] for r in recs], [False, True, False, True])
+        self.assertTrue(recs[0]["reason"].startswith("OVER BUDGET: 150 tokens, over the 100 limit sent; ok"))
+        rows, _ = compare.load([self.path])
+        self.assertEqual([r["ok"] for r in rows], [False, True, False, True])
+        self.assertIn('"pass": true', read(self.path))                       # the stored grade is unchanged
+
+    def test_partials_and_bad_setting(self):
+        from tuieval import compare, engine
+        part = os.path.join(self.ws, "results", "m", "q.json.partial.jsonl")
+        with open(part, "w") as f:
+            f.write(json.dumps({"pack_fingerprint": "x", "max_tokens": 50, "settings": {"sampling": {"max_tokens": 999}}}) + "\n")
+            f.write(json.dumps({"test": "a", "description": "a", "pass": True, "score": 1.0, "reason": "ok",
+                                "finish": "stop", "completion_tokens": 60}) + "\n")
+        self.assertEqual([r["over_budget"] for r in compare.load_partials(os.path.join(self.ws, "results"))], [50])
+        self.set_mode("sometimes")
+        with self.assertRaisesRegex(engine.ConfigError, "over_budget must be one of"):
+            engine.Engine(root=self.ws)
+
+
 class FitCheck(unittest.TestCase):
     def test_default_follows_the_ctx_placeholder(self):
         from tuieval import engine