tuieval 0.2.0.dev131__tar.gz → 0.2.0.dev135__tar.gz

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
+- Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev131
+Version: 0.2.0.dev135
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/docs/models.md

@@ -106,6 +106,7 @@ The **PTA index** puts parsimony (tokens), time and accuracy side by side, as a
 - **T time**: the total time to answer those questions, as how many times the fastest model's, e.g. `28×` (★ in **time/answer**).
 - **A accuracy**: the share of answers to those questions that passed. Answers cut off at `max_tokens` count as wrong, but show in orange after the green, with `(n% truncated)` beside the number and a **truncated** column in the table: those failures are a model out of tokens (overthinking), not a wrong answer yet. A pack's own `max_tokens` gives it more room.
 - **Score**: partial credit, averaged over the same answers. Graders that can tell how close an answer came give it: the code grader the share of checks passed (5 of 6 is 83%, where accuracy counts it wrong), agent packs the share of hidden tests, the tool-call and RAG graders part marks. It shows beside the accuracy when the two differ, and in a **score** column. Order the models by it with the **By score** dropdown in the tab or `tuieval pta --sort score`; verdicts still use accuracy only.
+- **Repeats**: beside each name, how many times the model answered each question compared (`×3`, or `×1-3` when it varies), also a **repeats** column here and in `tuieval compare`. A `×1` model (in yellow) answered each question once, so its accuracy is rougher: one answer more or less right moves it by 100 / questions points. Setup shows the same beside each pack in a model's results (`✓coding×3`).
 - **Over budget**: some providers don't stop a model at `max_tokens` (thinking goes on past it), so the answer isn't cut off but the model had more room than the others. Answers that used more tokens than were sent as `max_tokens` show as `(n over budget)` beside the accuracy, an **over budget** column, `over` in Results → Per question and a note in the answer view. They're only flagged; `over_budget = "fail"` in `[defaults]` counts them as failed instead (stored results aren't changed, so switching back restores them).
 
 The P and T bars are as long as the tokens and time used: the model that used the most fills its bar, the others in proportion (at least one block), so **shorter is better**. The A bar is the share of answers right, so **longer is better**; it's full only when every answer passed. Each heading says which way is better.
@@ -197,6 +198,10 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
 | `headers` | extra HTTP headers |
 
+## Answers stuck in a loop
+
+A model sometimes gets stuck repeating one short piece of text (a closing tag, "wait, ", a line) until it runs out of `max_tokens`, which can take many minutes and fails anyway. tuieval watches the reasoning and the answer as they stream: once either ends in one unit of up to 64 characters repeated back to back over the last `loop_stop_chars` characters (2000 by default; units without a letter or digit, like `-----`, don't count), it ends the request there. The answer fails with `LOOP: the answer kept repeating '…'`, its finish is `loop` (the answer view says so), Results → Insights counts it as cut off and `tuieval compare` in **rep**. Set `loop_stop_chars = 0` in `[defaults]` to let loops run to the end, as before.
+
 ## Built-in servers and defaults
 
 A workspace's `models.toml` only needs what differs from these. `[defaults]` fill in key by key. `[sampling]` applies only when `models.toml` has no `[sampling]` section at all, because sampling is part of every result's fingerprint. A server you define replaces the built-in one with the same name; to change one setting, copy the whole block into `models.toml` and edit it. `tuieval add` adds a `[servers.<name>]` block with just a `url` when it finds a model on a server that's already running.
@@ -209,6 +214,8 @@ A workspace's `models.toml` only needs what differs from these. `[defaults]` fil
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 # reference = "<label>"        # the model Results → Insights compares against by default
 # agent = "<name>"             # the [agents.<name>] that runs agent packs (see docs/agent-packs.md)

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev131'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev131')
+__version__ = version = '0.2.0.dev135'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev135')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/builtin.toml

@@ -6,6 +6,8 @@
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/client.py

@@ -9,9 +9,12 @@ answer together with the numbers that matter for choosing a local model:
   prompt_tps     prompt processing speed, tokens/s (llama.cpp reports it)
   prompt_tokens, completion_tokens, and — when the split can be known — reasoning_tokens and
   answer_tokens (tokens_estimated=True means the split was estimated from text length)
+  loop           set when the request was ended early because the model got stuck repeating itself
+                 ({"stream": "reasoning" or "answer", "unit": the repeated text}; finish is "loop")
 """
 import http.client
 import json
+import re
 import socket
 import time
 import urllib.parse
@@ -119,17 +122,37 @@ def count_tokens(base_url, text):
         return None
 
 
+LOOP_MAX_UNIT = 64   # longest repeated unit looked for (a short line, a closing tag, a phrase)
+LOOP_EVERY = 256     # chars of new text between checks
+
+
+def loop_unit(text, min_chars):
+    """The text the end of `text` repeats, exactly and back to back, over its last min_chars or more (a unit of
+    at most LOOP_MAX_UNIT chars with a letter or digit in it, so rulers like '-----' don't count), else None."""
+    if not min_chars or len(text) < min_chars:
+        return None
+    tail = text[-min_chars:]
+    for n in range(1, LOOP_MAX_UNIT + 1):
+        if tail[n:] == tail[:-n]:          # the tail has period n: one unit over and over
+            unit = tail[-n:]
+            return unit if re.search(r"[A-Za-z0-9]", unit) else None
+    return None
+
+
 def _stalled(stall_s):
     return f"server error: stalled, no tokens for {stall_s:.0f} s after it started answering"
 
 
-def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None):
+def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None,
+                loop_chars=None):
     """POST {base_url}/v1/chat/completions with streaming. Returns a dict (see module docstring).
 
     on_delta(kind, text) is called for "reasoning" and "answer" text as it arrives.
     timeout bounds the whole request, keep-alive comments (OpenRouter's ": OPENROUTER PROCESSING")
     included; running out counts against the model, like any slow answer. stall_s: once tokens have
     started, this long without one is the server stalling (a server error, never the model's answer).
+    loop_chars: when the reasoning or the answer ends in one short unit repeated over this many chars, the
+    model is stuck in a loop that would run to max_tokens: the request ends there (finish "loop").
     """
     stream = stream or Stream()
     body = dict(body, stream=True, stream_options={"include_usage": True})
@@ -138,6 +161,7 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
     out = {"answer": "", "reasoning": "", "tool_calls": [], "finish": None, "usage": {}, "timings": {},
            "ttft_s": None, "total_s": None, "error": None}
     reasoning, answer, tools = [], [], {}
+    tails, unchecked = {"reasoning": "", "answer": ""}, {"reasoning": 0, "answer": 0}   # for loop_chars
     done = False                      # [DONE] seen
     start = last = time.time()
     try:
@@ -195,6 +219,17 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
                 if text:
                     answer.append(text)
                     on_delta and on_delta("answer", text)
+                for kind, piece in (("reasoning", think), ("answer", text)):
+                    if loop_chars and piece:
+                        tails[kind] = (tails[kind] + piece)[-loop_chars:]
+                        unchecked[kind] += len(piece)
+                        if unchecked[kind] >= LOOP_EVERY:
+                            unchecked[kind] = 0
+                            unit = loop_unit(tails[kind], loop_chars)
+                            if unit:
+                                out["finish"], out["loop"] = "loop", {"stream": kind, "unit": unit}
+            if out.get("loop"):
+                break
         if not out["error"] and not done and out["finish"] is None:
             # the stream just ended: a server that crashed or dropped the connection, not the model's answer
             out["error"] = "connection error: the stream ended before the server finished (no finish reason)"

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/compare.py

@@ -52,6 +52,7 @@ def _load_native(data):
             # a timeout is the model too slow (client.is_server_error), not an error; older results called it one
             "error": r["reason"].startswith(("server returned", "connection error")) and "timed out" not in r["reason"],
             "truncated": r.get("finish") == "length", "output": r.get("answer") or "",
+            "loop": r.get("finish") == "loop",   # stopped live: it kept repeating itself (client.loop_unit)
             # tokens past the max_tokens sent, which the provider didn't stop (client.mark_over_budget)
             "over_budget": r.get("over_budget"),
             "tokens": r.get("completion_tokens") or 0, "reasoning_tokens": r.get("reasoning_tokens"),
@@ -248,12 +249,14 @@ def scorecard(rows, rng=None):
     suites = sorted({r["suite"] for r in rows})
     extra = dict(graders.COLUMNS)   # columns your graders add (graders.scorecard_column)
     over = any(r.get("over_budget") for r in rows)   # a column only when some answer ran past max_tokens
-    header = ["model"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
+    header = ["model", "repeats"] + suites + ["overall", "95% CI", "mean score", "all repeats", *extra, "errors", "trunc",
                                    *(["over budget"] if over else []), "rep"]
     table = []
     for m in models:
         mine = [r for r in rows if r["model"] == m]
-        line = [m]
+        per_test = collections.Counter(r["test"] for r in mine)   # answers per question: the repeats
+        few, most = min(per_test.values()), max(per_test.values())
+        line = [m, f"×{few}" if few == most else f"×{few}-{most}"]
         for s in suites:
             sub = [r for r in mine if r["suite"] == s]
             line.append(f"{sum(r['ok'] for r in sub)}/{len(sub)}" if sub else "-")
@@ -271,7 +274,7 @@ def scorecard(rows, rng=None):
             line.append("-" if cell is None else str(cell))
         line += [str(sum(r["error"] for r in mine)), str(sum(r["truncated"] for r in mine)),
                  *([str(sum(bool(r.get("over_budget")) for r in mine))] if over else []),
-                 str(sum(repetitive(r["output"]) for r in mine))]
+                 str(sum(bool(r.get("loop")) or repetitive(r["output"]) for r in mine))]
         table.append(line)
     return header, table
 
@@ -678,8 +681,9 @@ def main(argv):
     for note in settings_notes(infos):
         print("NOTE:", note)
     print_table(*scorecard(rows, rng))
-    print("\nmean score = average score, with partial credit; all repeats = tests passed on every repeat.")
-    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern.")
+    print("\nrepeats = answers per question (×1: one answer each, the roughest accuracy).")
+    print("mean score = average score, with partial credit; all repeats = tests passed on every repeat.")
+    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern (or was stopped live as a loop).")
     header, table = by_difficulty(rows)
     if any(r.get("difficulty") in LEVELS for r in rows):
         print("\nPass rate by difficulty:")

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/engine.py

@@ -989,6 +989,14 @@ class Engine:
         need = {(t["id"], r) for t in pack.tests for r in range(self.certify_repeat(pack))}
         return "certified" if need <= keys else "screened"
 
+    def result_repeats(self, label, pack_name, tier="certify"):
+        """How many times each answered question of a stored result was asked: (fewest, most), or None."""
+        data = self._result(self.result_path(label, pack_name, tier))
+        if not data or not data["results"]:
+            return None
+        per_test = collections.Counter(r["test"] for r in data["results"])
+        return min(per_test.values()), max(per_test.values())
+
     def _partial_lines(self, path):
         """(header, answers) of an unfinished run's .partial.jsonl, cached like _result;
         (None, []) if there is none."""
@@ -2392,6 +2400,7 @@ class Engine:
                   earlier_s=sum(s.get("wall_s") or 0 for s in job.earlier), sittings=len(job.earlier))
         timeout = self.cfg["defaults"]["request_timeout_ms"] / 1000
         stall_s = (self.cfg["defaults"].get("stall_timeout_ms") or 0) / 1000 or None
+        loop_chars = int(self.cfg["defaults"].get("loop_stop_chars") or 0) or None
         headers = self.request_headers(job.model)
         # a hosted API has no /tokenize to split reasoning from answer tokens (it reports the split)
         tokenize_url = None if self.cfg["servers"][job.model["server"]].get("api_key_env") else base_url
@@ -2421,7 +2430,7 @@ class Engine:
                                                      lambda kind, text: self.emit("delta", stream=kind, text=text,
                                                                                   label=job.label),
                                                      timeout=timeout, stream=self._stream, headers=headers,
-                                                     stall_s=stall_s)
+                                                     stall_s=stall_s, loop_chars=loop_chars)
                         except client.Cancelled:
                             self._check()
                             raise Cancelled()
@@ -2458,8 +2467,8 @@ class Engine:
                     if res["error"]:
                         g = {"pass": False, "score": 0.0, "reason": res["error"]}
                     else:
-                        g = graders.grade(test, res["answer"],
-                                          {"finish": res["finish"], "tool_calls": res["tool_calls"]})
+                        g = graders.grade(test, res["answer"], {"finish": res["finish"], "tool_calls": res["tool_calls"],
+                                                                "loop": res.get("loop")})
                     log_name = self._log_request(job, test, res, meta)
                     rec = {"test": test["id"], "description": test["description"], "category": test.get("category", ""),
                            "difficulty": test.get("difficulty", "unrated"),
@@ -2467,6 +2476,7 @@ class Engine:
                            "repeat": rep, "pass": g["pass"], "score": g["score"], "reason": g["reason"],
                            "severity": g.get("severity"), "answer": res["answer"], "tool_calls": res["tool_calls"],
                            "finish": res["finish"], "reasoning_chars": len(res["reasoning"]),
+                           **({"loop": res["loop"]} if res.get("loop") else {}),
                            "log": log_name,   # logs/live/<log>: the reasoning, which the result doesn't keep
                            "answered_at": dates.now(),
                            **({"checks": g["checks"]} if "checks" in g else {}),
@@ -2590,7 +2600,8 @@ def regrade(path, packs_dir=None):
             continue
         if r["reason"].startswith(("server returned", "connection error")):
             continue
-        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", [])})
+        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", []),
+                                           "loop": r.get("loop")})
         r.update({"pass": g["pass"], "score": g["score"], "reason": g["reason"], "severity": g.get("severity"),
                   "critical_test": bool(t.get("critical_trial")), "group": t.get("group")})
         r.pop("checks", None)

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/graders/__init__.py

@@ -137,6 +137,10 @@ def grade(test, answer, meta=None):
         return result(False, f"unknown grader {name!r}; known: {', '.join(sorted(REGISTRY))}")
     if meta.get("finish") == "length":
         return result(False, "TRUNCATED: hit max_tokens before finishing")
+    if meta.get("finish") == "loop":
+        loop = meta.get("loop") or {}
+        return result(False, f"LOOP: the {loop.get('stream', 'answer')} kept repeating "
+                             f"{loop.get('unit', '')[:40]!r}; stopped early (it would have run to max_tokens)")
     text = strip_thinking(answer)
     if not text and not meta.get("tool_calls"):
         return result(False, "Empty answer")

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/insights.py

@@ -280,7 +280,7 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit")):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit", "LOOP")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/pta.py

@@ -15,6 +15,9 @@ the others in proportion, at least one block), so shorter is better; the A bar:
 P and T also carry a 0-100 score (100 for the best, POINTS_PER_DOUBLING points less for every
 doubling), used to order models with the same accuracy.
 
+Beside each name, the repeats: how many times it answered each of those questions ("×3", or "×1-3" when
+it varies). A ×1 model's accuracy rests on one answer per question, so it is rougher than a ×3 one's.
+
 Models are compared on the questions all of them answered, so one that skipped some never looks
 faster. Server and connection errors aren't answers. By default a model with fewer than half as many
 answers as the most-answered model is left out (listed, so it can be picked), or it would shrink the
@@ -54,7 +57,7 @@ SORTS = ("accuracy", "score")
 
 def index(rows, models=None, memory=None, sort="accuracy"):
     """{"questions": n shared, "models": [{model, P, T, A, S, cut, cut_pct, tokens, total_s, passed, answers, alongside,
-    fewest, fastest, tps, ttft, memory_gb}],
+    fewest, fastest, tps, ttft, memory_gb, repeats: (fewest, most) answers per question}],
     "left_out": [(model, questions answered)], "no_answers": [model], "most": questions answered by the
     most-answered model}, best accuracy (sort="accuracy") or score (sort="score") first. S: the mean
 score (partial credit) in %. models: the ones to compare (default: every model in rows
@@ -92,6 +95,7 @@ score (partial credit) in %. models: the ones to compare (default: every model i
                     "tps": compare.median(r.get("gen_tps") for c in cells for r in c["rows"]),
                     "ttft": compare.median(r.get("ttft") for c in cells for r in c["rows"]),
                     "memory_gb": (memory or {}).get(m),
+                    "repeats": (min(c["n"] for c in cells), max(c["n"] for c in cells)) if cells else None,
                     # answers that ran while other models were served shared the machine's speed
                     "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
     fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
@@ -116,6 +120,14 @@ LETTERS = (("P", "parsimony (tokens)", "▼ shorter is better", "tokens", "token
            ("A", "accuracy", "▲ longer is better", "A", "A", "green"))
 
 
+def repeats(x):
+    """How many times a model answered each question compared: "×3", "×1-3", or "" when unknown."""
+    r = x.get("repeats")
+    if not r:
+        return ""
+    return f"×{r[0]}" if r[0] == r[1] else f"×{r[0]}-{r[1]}"
+
+
 def times(v):
     """A multiple of the best: "1.0×", "2.4×", "28×"."""
     return "-" if v is None else f"{v:.1f}×" if v < 9.95 else f"{v:.0f}×"
@@ -138,6 +150,8 @@ def bars(result, width=20, markup=True, click=None):
     if not result["models"]:
         return []
     name_w = max(len(x["model"]) for x in result["models"])
+    rep_w = max(len(repeats(x)) for x in result["models"])
+    name_w += rep_w + 1 if rep_w else 0   # the repeats sit after the name, in its column
     # each column: the bar, a space, "100%" or "9.8×" (6), then a gap; at least as wide as its heading
     cell_w = max(width + 9, *(len(t) + 2 for k, name, better, *_ in LETTERS for t in (f"{k} {name}", better)))
     pad = " " * (name_w + 2)
@@ -173,11 +187,14 @@ def bars(result, width=20, markup=True, click=None):
             bar = (f"[{color}]{bar}[/]" + (f"[#ff8c00]{'█' * part}[/]" if part else "") + f"[grey37]{rest}[/]"
                    if markup else bar + cut_bar + rest)
             cells.append(f"{bar} {num:>6}" + " " * max(0, cell_w - width - 1 - max(6, len(num))))
-        name = f"{x['model']:<{name_w}}"
+        tag = repeats(x)
+        name = f"{x['model'] + (' ' + tag if tag else ''):<{name_w}}"
         if markup:
-            pad_w = name_w - len(x["model"])
-            name = (f"[@click={click}({row})]{rich_escape(x['model'])}[/]" + " " * pad_w if click
-                    else rich_escape(name))
+            pad_w = name_w - len(x["model"]) - (len(tag) + 1 if tag else 0)
+            # a single answer per question in yellow: its accuracy is the roughest
+            tag = f" [{'yellow' if x['repeats'][1] == 1 else 'dim'}]{tag}[/]" if tag else ""
+            name = ((f"[@click={click}({row})]{rich_escape(x['model'])}[/]" if click else rich_escape(x["model"]))
+                    + tag + " " * pad_w)
         lines.append((f"{name}  " + "".join(cells)).rstrip())
     return lines
 
@@ -202,9 +219,9 @@ def table(result):
     """(header, rows) for a plain table of the index."""
     n = result["questions"] or 1
     over = any(x.get("over") for x in result["models"])   # a column only when some answer ran past max_tokens
-    header = ["model", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
+    header = ["model", "repeats", "tokens/answer", "time/answer", "answers right", "score", "truncated", *(["over budget"] if over else []),
               "tok/s", "TTFT s", "memory GB"]
-    rows = [[x["model"], f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
+    rows = [[x["model"], repeats(x) or "-", f"{round(x['tokens'] / n):,}" + (" ★" if x.get("fewest") else ""),
              fmt_secs(x["total_s"] / n) + (" ★" if x.get("fastest") else ""),
              f"{x['passed']}/{x['answers']}", "-" if x.get("S") is None else compare.pct(x["S"] / 100),
              f"{x.get('cut') or 0}/{x['answers']}",
@@ -243,6 +260,10 @@ def scope_note(result, models):
             "P parsimony (tokens) and T time: bars as long as the tokens / time used (shorter is better), "
             "and how many times the leanest / fastest model's (1.0×, ★ in the table). A accuracy: answers right "
             "(longer is better); score: partial credit, e.g. 5 of 6 checks passed is 83% (shown when it differs)"
+            + (f"; ×n beside a name: answers per question. ×1 models answered each question once, so one "
+               f"answer more or less right moves their accuracy by {100 / result['questions']:.0f} points; "
+               "repeats steady it" if len({x.get("repeats") for x in result["models"]}) > 1
+               or any(x.get("repeats") and x["repeats"][1] == 1 for x in result["models"]) else "")
             + ("; orange, (n% truncated): answers cut off at max_tokens. They count as wrong, but the model ran "
                "out of tokens rather than answering wrong." if any(x.get("cut") for x in result["models"]) else ".")
             + (" Over budget: answers that used more tokens than the max_tokens sent, because the provider didn't "

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/src/tuieval/tui.py

@@ -624,7 +624,8 @@ class AnswerScreen(ModalScreen):
         quant = f"  [magenta]{rec['quantization']}[/magenta]" if rec.get("quantization") else ""
         finish = rec.get("finish") or "?"
         limit = f"this pack's max_tokens, {pack.max_tokens:,}" if pack and pack.max_tokens else "max_tokens"
-        finish = f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length" else finish
+        finish = (f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length"
+                  else f"[bold red]loop (stopped: it kept repeating itself)[/]" if finish == "loop" else finish)
         toks = rec.get("completion_tokens")
         split = ", ".join(f"{k} {rec[k + '_tokens']:,}" for k in ("reasoning", "answer")
                           if rec.get(k + "_tokens") is not None)
@@ -937,7 +938,10 @@ class SetupScreen(Screen):
         for n in e.packs:
             st = e.result_status(label, n)
             if st:
-                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}")
+                reps = e.result_repeats(label, n) if st in ("certified", "screened") else None
+                times = ("" if not reps else f"[dim]×{reps[0]}[/dim]" if reps[0] == reps[1]
+                         else f"[dim]×{reps[0]}-{reps[1]}[/dim]")
+                packs.append(f"{mark.get(st, '[yellow]~[/yellow]')}{n}{times}")
                 if st.startswith("outdated"):
                     why.setdefault(st.removeprefix("outdated: "), []).append(n)
         if why:  # each reason once, not once per pack
@@ -951,7 +955,7 @@ class SetupScreen(Screen):
         hidden = (f"  [cyan]{' '.join(tags)}[/cyan]" if tags else "") + hidden
         detail.update(f"[b]{label}[/b]  [dim]{m['server']} · {m.get('model') or ''}[/dim]"
                       f"{serving_badge(e, m, not e.has_results())}{hidden}\n{verdicts}\n"
-                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial):[/dim] " + "  ".join(packs)
+                      + ("[dim]results (✓ certified ◐ screened ~ outdated … partial; ×n answers per question):[/dim] " + "  ".join(packs)
                          if packs else "[dim]no results yet[/dim]"))
 
     @on(Checkbox.Changed, "#show-hidden")

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/tests/mock_server.py

@@ -13,6 +13,7 @@ Streams like llama.cpp: reasoning_content then content deltas, usage and timings
 PROCESSING"), or of silence with --silent; --stall-before S: the same before the first token.
 --die-after N: the process exits after N stream chunks (0: right after the headers), with no finish
 reason or [DONE], like a server that crashes mid-request.
+--loop TEXT: after the answer, TEXT 5000 times over (a model stuck in a loop).
 An answer longer than the request's max_tokens (one token per word) is cut off with finish
 "length". --max-output N also serves an OpenRouter-style endpoints list with that output limit.
 """
@@ -55,7 +56,7 @@ def reply_for(test, mode):
 class Handler(BaseHTTPRequestHandler):
     answers, mode, model, delay, max_output = {}, "oracle", "mock", 0.0, None
     stall, stall_before, silent = 0.0, 0.0, False
-    die_after = None
+    die_after, loop = None, None
     hold, answered = None, 0   # while the hold file exists, answers after the first wait (a run that stays going)
 
     def log_message(self, *a):
@@ -94,6 +95,8 @@ class Handler(BaseHTTPRequestHandler):
         if self.mode == "unavailable":   # like OpenRouter with no provider serving the model
             return self._json({"error": {"message": f"No endpoints found for {self.model}.", "code": 404}}, 404)
         text, calls = reply_for(test, self.mode)
+        if self.loop:
+            text += self.loop * 5000
         finish = "tool_calls" if calls else "stop"
         if req.get("max_tokens") and len(text.split()) > req["max_tokens"]:
             text, finish = " ".join(text.split()[:req["max_tokens"]]), "length"
@@ -162,11 +165,12 @@ def main(argv=None):
     p.add_argument("--silent", action="store_true")
     p.add_argument("--die-after", type=int)
     p.add_argument("--hold", help="while this file exists, every answer after the first waits")
+    p.add_argument("--loop")
     a, _ = p.parse_known_args(argv)   # other flags (e.g. speed knobs under test) are accepted and ignored
     Handler.answers, Handler.mode, Handler.model, Handler.delay = load_answers(a.packs), a.mode, a.model, a.delay
     Handler.max_output = a.max_output
     Handler.stall, Handler.stall_before, Handler.silent = a.stall, a.stall_before, a.silent
-    Handler.die_after, Handler.hold = a.die_after, a.hold
+    Handler.die_after, Handler.hold, Handler.loop = a.die_after, a.hold, a.loop
     srv = Server(("127.0.0.1", a.port), Handler)
     print(f"mock server on 127.0.0.1:{a.port} ({a.mode}, {len(Handler.answers)} known questions)", flush=True)
     try:

{tuieval-0.2.0.dev131 → tuieval-0.2.0.dev135}/tests/test_tuieval.py

@@ -425,12 +425,12 @@ class PickTests(unittest.TestCase):
             self.mock.stop()
         self.tmp.cleanup()
 
-    def server(self, mode):
+    def server(self, mode, *flags):
         """(Re)start the mock on the same port: oracle answers right, wrong answers wrong."""
         first = self.mock is None
         if self.mock:
             self.mock.stop()
-        self.mock = Mock(os.path.join(self.ws, "packs"), mode, port=self.port)
+        self.mock = Mock(os.path.join(self.ws, "packs"), mode, self.port, 0.0, *flags)
         if first:
             tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
 
@@ -445,6 +445,20 @@ class PickTests(unittest.TestCase):
         return subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                               env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip()
 
+    def test_a_looping_answer_is_stopped_and_failed(self):
+        self.server("oracle", "--loop", "</think>\n")   # the right answer, then stuck
+        tuieval(self.ws, "run", "--tests", "apps:a")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual((rec["finish"], rec["pass"], rec["loop"]["stream"]), ("loop", False, "answer"))
+        self.assertTrue(rec["reason"].startswith("LOOP:"), rec["reason"])
+        self.assertLess(len(rec["answer"]), 3000)
+        path = os.path.join(self.ws, "models.toml")                      # off: it runs to the end, as before
+        write(path, read(path).replace("[defaults]\n", "[defaults]\nloop_stop_chars = 0\n", 1))
+        tuieval(self.ws, "run", "--tests", "apps:a", "--force")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual(rec["finish"], "stop")
+        self.assertNotIn("loop", rec)
+
     def test_pick(self):
         from tuieval import packs
         p = packs.load_pack(os.path.join(self.ws, "packs", "apps"))
@@ -502,6 +516,9 @@ class PickTests(unittest.TestCase):
         self.assertIn("done earlier", out)                                       # nothing missing: not re-asked
         tuieval(self.ws, "run", "--tests", "apps:d", "--tier", "certify")
         self.assertEqual(self.status(), "certified")
+        code = "from tuieval import engine; print(engine.Engine().result_repeats('m', 'apps'))"
+        self.assertEqual(subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,   # ✓×1 on Setup
+                                        env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip(), "(1, 1)")
 
         self.server("wrong")
         tuieval(self.ws, "run", "--tests", "apps:b", "--force")
@@ -1555,7 +1572,7 @@ class PTAIndex(unittest.TestCase):
     scale, and accuracy, all over the questions every compared model answered."""
 
     def test_index_compares_shared_questions(self):
-        from tuieval import pta
+        from tuieval import compare, pta
 
         def row(model, test, ok, secs, repeat=0):
             return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": secs * 1000,
@@ -1574,9 +1591,9 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual((by["slow"]["P"], by["fast"]["P"]), (100, 70))          # 4x the tokens: two doublings
         self.assertEqual((pta.score(2, 1), pta.score(1000, 1)), (85, 0))
         header, table = pta.table(res)
-        self.assertEqual(header[1:4], ["tokens/answer", "time/answer", "answers right"])
+        self.assertEqual(header[1:5], ["repeats", "tokens/answer", "time/answer", "answers right"])
         cells = {r[0]: r for r in table}
-        self.assertEqual((cells["slow"][1], cells["fast"][1:3]), ("10 ★", ["40", "2.0s ★"]))   # per answer
+        self.assertEqual((cells["slow"][2], cells["fast"][2:4]), ("10 ★", ["40", "2.0s ★"]))   # per answer
         bars = pta.bars(res, 10, markup=False)
         self.assertEqual(bars[0].split(), ["P", "parsimony", "(tokens)", "T", "time", "A", "accuracy"])
         self.assertEqual(bars[1].split(), ["▼", "shorter", "is", "better"] * 2 + ["▲", "longer", "is", "better"])
@@ -1585,8 +1602,13 @@ class PTAIndex(unittest.TestCase):
                          (fast.index("██████████"), fast.index("███░░░░░░░")))       # its heading
         # P and T bars as long as tokens and time (shorter is better): fast used 4x the tokens of slow but
         # took 4 s against 13 s, so it has the longest token bar and the shortest time bar
-        self.assertEqual(fast.split(), ["fast", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
-        self.assertEqual(slow.split()[1:5], ["██░░░░░░░░", "1.0×", "██████████", "3.2×"])
+        self.assertEqual(fast.split(), ["fast", "×1", "██████████", "4.0×", "███░░░░░░░", "1.0×", "█████░░░░░", "50.0%"])
+        self.assertEqual(slow.split()[1:6], ["×1-2", "██░░░░░░░░", "1.0×", "██████████", "3.2×"])   # ×: answers per question
+        self.assertEqual((cells["slow"][1], cells["fast"][1]), ("×1-2", "×1"))
+        self.assertIn("×1 models answered each question once", pta.scope_note(res, ["slow", "fast"]))
+        full = [dict(r, score=float(r["ok"]), error=False, output="") for r in rows]
+        header, table = compare.scorecard(full)
+        self.assertEqual({r[0]: r[header.index("repeats")] for r in table}, {"slow": "×1-2", "fast": "×1"})
         self.assertEqual((pta.times(1), pta.times(9.84), pta.times(27.9)), ("1.0×", "9.8×", "28×"))
         self.assertEqual((by["fast"]["A"], by["slow"]["A"]), (50, 100))
         self.assertEqual([x["model"] for x in res["models"]], ["slow", "fast"])   # best accuracy first
@@ -1618,8 +1640,8 @@ class PTAIndex(unittest.TestCase):
         self.assertEqual(bars["plain"][-2:], ["██████░░░░", "60.0%"])                  # none truncated: nothing beside it
         self.assertIn("[#ff8c00]██[/]", pta.bars(res, 10)[2] + pta.bars(res, 10)[3])
         header, table = pta.table(res)
-        self.assertEqual(header[4:6], ["score", "truncated"])
-        self.assertEqual({r[0]: r[5] for r in table}, {"thinker": "2/10", "plain": "0/10"})
+        self.assertEqual(header[5:7], ["score", "truncated"])
+        self.assertEqual({r[0]: r[6] for r in table}, {"thinker": "2/10", "plain": "0/10"})
         self.assertIn("truncated", pta.scope_note(res, ["thinker", "plain"]))
         self.assertNotIn("truncated", pta.scope_note(pta.index(rows[10:]), ["plain"]))
 
@@ -1949,7 +1971,7 @@ class PTAIndex(unittest.TestCase):
             self.assertGreaterEqual(len(got["opened"]), 2)
             self.assertEqual(got["opened"], {m: m for m in got["opened"]})   # a cell opens its column's model
             self.assertEqual(got["tab"], "tab-pta")                  # Results opens on the PTA index
-            self.assertEqual([(r[0], r[3]) for r in got["rows"]][0], ("good", "4/4"))
+            self.assertEqual([(r[0], r[4]) for r in got["rows"]][0], ("good", "4/4"))
             s = got["sorts"]   # a click on a column name sorts by it, a second click reverses
             self.assertEqual((s["up"], s["down"], s["kept"]), (["hosted", "good"], ["good", "hosted"], ["good", "hosted"]))
             self.assertIn("answers right ▼", s["labels"])
@@ -3180,6 +3202,32 @@ class StallGuard(unittest.TestCase):
         self.assertTrue(client.is_server_error(res["error"]), res["error"])
         self.assertIsNone(res["finish"])
 
+    def test_a_loop_is_stopped_early(self):
+        from tuieval import client, graders
+        res, took = self.ask("--loop", "</think>\n", timeout=30, loop_chars=2000)
+        self.assertIsNone(res["error"])
+        self.assertEqual((res["finish"], res["loop"]["stream"]), ("loop", "answer"))
+        self.assertEqual(sorted(res["loop"]["unit"]), sorted("</think>\n"))   # the unit, from wherever it was cut
+        self.assertLess(len(res["answer"]), 3000)                               # 45,000 chars were coming
+        g = graders.grade({"grader": "answer", "answer": "42"}, res["answer"], {"finish": "loop", "loop": res["loop"]})
+        self.assertFalse(g["pass"])
+        self.assertTrue(g["reason"].startswith("LOOP: the answer kept repeating"), g["reason"])
+        res, _ = self.ask("--loop", "</think>\n", timeout=30)                   # off: the whole answer
+        self.assertEqual(res["finish"], "stop")
+        self.assertGreater(len(res["answer"]), 40000)
+
+    def test_what_counts_as_a_loop(self):
+        from tuieval import client
+        prose = " ".join(f"step {i}: check the value and move on." for i in range(200))
+        self.assertIsNone(client.loop_unit(prose, 2000))
+        self.assertEqual(client.loop_unit("intro " + "wait, " * 400, 2000), "wait, ")
+        self.assertEqual(client.loop_unit("x" * 2500, 2000), "x")
+        self.assertIsNone(client.loop_unit("| --- " * 500, 2000))           # a ruler: no letter or digit
+        self.assertIsNone(client.loop_unit("wait, " * 300, 2000))           # shorter than min_chars
+        line = "a sentence that is a good deal longer than sixty-four chars in all.\n"
+        self.assertIsNone(client.loop_unit(line * 40, 2000))                # unit too long to call a loop
+        self.assertIsNone(client.loop_unit("wait, " * 400, 0))              # off
+
     def test_finish_without_done_is_an_answer(self):
         res, _ = self.ask("--die-after", "99", timeout=30)    # every chunk, then gone before [DONE]
         self.assertIsNone(res["error"])