tuieval 0.2.0.dev133__tar.gz → 0.2.0.dev137__tar.gz

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## Unreleased
 
-- Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
+- Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
+- Repeats: the PTA index, Per question, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.
 - Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev133
+Version: 0.2.0.dev137
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/docs/models.md

@@ -198,6 +198,10 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
 | `headers` | extra HTTP headers |
 
+## Answers stuck in a loop
+
+A model sometimes gets stuck repeating one short piece of text (a closing tag, "wait, ", a line) until it runs out of `max_tokens`, which can take many minutes and fails anyway. tuieval watches the reasoning and the answer as they stream: once either ends in one unit of up to 64 characters repeated back to back over the last `loop_stop_chars` characters (2000 by default; units without a letter or digit, like `-----`, don't count), it ends the request there. The answer fails with `LOOP: the answer kept repeating '…'`, its finish is `loop` (the answer view says so), Results → Insights counts it as cut off and `tuieval compare` in **rep**. Set `loop_stop_chars = 0` in `[defaults]` to let loops run to the end, as before.
+
 ## Built-in servers and defaults
 
 A workspace's `models.toml` only needs what differs from these. `[defaults]` fill in key by key. `[sampling]` applies only when `models.toml` has no `[sampling]` section at all, because sampling is part of every result's fingerprint. A server you define replaces the built-in one with the same name; to change one setting, copy the whole block into `models.toml` and edit it. `tuieval add` adds a `[servers.<name>]` block with just a `url` when it finds a model on a server that's already running.
@@ -210,6 +214,8 @@ A workspace's `models.toml` only needs what differs from these. `[defaults]` fil
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 # reference = "<label>"        # the model Results → Insights compares against by default
 # agent = "<name>"             # the [agents.<name>] that runs agent packs (see docs/agent-packs.md)

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev133'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev133')
+__version__ = version = '0.2.0.dev137'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev137')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/builtin.toml

@@ -6,6 +6,8 @@
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/client.py

@@ -9,9 +9,12 @@ answer together with the numbers that matter for choosing a local model:
   prompt_tps     prompt processing speed, tokens/s (llama.cpp reports it)
   prompt_tokens, completion_tokens, and — when the split can be known — reasoning_tokens and
   answer_tokens (tokens_estimated=True means the split was estimated from text length)
+  loop           set when the request was ended early because the model got stuck repeating itself
+                 ({"stream": "reasoning" or "answer", "unit": the repeated text}; finish is "loop")
 """
 import http.client
 import json
+import re
 import socket
 import time
 import urllib.parse
@@ -119,17 +122,37 @@ def count_tokens(base_url, text):
         return None
 
 
+LOOP_MAX_UNIT = 64   # longest repeated unit looked for (a short line, a closing tag, a phrase)
+LOOP_EVERY = 256     # chars of new text between checks
+
+
+def loop_unit(text, min_chars):
+    """The text the end of `text` repeats, exactly and back to back, over its last min_chars or more (a unit of
+    at most LOOP_MAX_UNIT chars with a letter or digit in it, so rulers like '-----' don't count), else None."""
+    if not min_chars or len(text) < min_chars:
+        return None
+    tail = text[-min_chars:]
+    for n in range(1, LOOP_MAX_UNIT + 1):
+        if tail[n:] == tail[:-n]:          # the tail has period n: one unit over and over
+            unit = tail[-n:]
+            return unit if re.search(r"[A-Za-z0-9]", unit) else None
+    return None
+
+
 def _stalled(stall_s):
     return f"server error: stalled, no tokens for {stall_s:.0f} s after it started answering"
 
 
-def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None):
+def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None,
+                loop_chars=None):
     """POST {base_url}/v1/chat/completions with streaming. Returns a dict (see module docstring).
 
     on_delta(kind, text) is called for "reasoning" and "answer" text as it arrives.
     timeout bounds the whole request, keep-alive comments (OpenRouter's ": OPENROUTER PROCESSING")
     included; running out counts against the model, like any slow answer. stall_s: once tokens have
     started, this long without one is the server stalling (a server error, never the model's answer).
+    loop_chars: when the reasoning or the answer ends in one short unit repeated over this many chars, the
+    model is stuck in a loop that would run to max_tokens: the request ends there (finish "loop").
     """
     stream = stream or Stream()
     body = dict(body, stream=True, stream_options={"include_usage": True})
@@ -138,6 +161,7 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
     out = {"answer": "", "reasoning": "", "tool_calls": [], "finish": None, "usage": {}, "timings": {},
            "ttft_s": None, "total_s": None, "error": None}
     reasoning, answer, tools = [], [], {}
+    tails, unchecked = {"reasoning": "", "answer": ""}, {"reasoning": 0, "answer": 0}   # for loop_chars
     done = False                      # [DONE] seen
     start = last = time.time()
     try:
@@ -195,6 +219,17 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
                 if text:
                     answer.append(text)
                     on_delta and on_delta("answer", text)
+                for kind, piece in (("reasoning", think), ("answer", text)):
+                    if loop_chars and piece:
+                        tails[kind] = (tails[kind] + piece)[-loop_chars:]
+                        unchecked[kind] += len(piece)
+                        if unchecked[kind] >= LOOP_EVERY:
+                            unchecked[kind] = 0
+                            unit = loop_unit(tails[kind], loop_chars)
+                            if unit:
+                                out["finish"], out["loop"] = "loop", {"stream": kind, "unit": unit}
+            if out.get("loop"):
+                break
         if not out["error"] and not done and out["finish"] is None:
             # the stream just ended: a server that crashed or dropped the connection, not the model's answer
             out["error"] = "connection error: the stream ended before the server finished (no finish reason)"

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/compare.py

@@ -52,6 +52,7 @@ def _load_native(data):
             # a timeout is the model too slow (client.is_server_error), not an error; older results called it one
             "error": r["reason"].startswith(("server returned", "connection error")) and "timed out" not in r["reason"],
             "truncated": r.get("finish") == "length", "output": r.get("answer") or "",
+            "loop": r.get("finish") == "loop",   # stopped live: it kept repeating itself (client.loop_unit)
             # tokens past the max_tokens sent, which the provider didn't stop (client.mark_over_budget)
             "over_budget": r.get("over_budget"),
             "tokens": r.get("completion_tokens") or 0, "reasoning_tokens": r.get("reasoning_tokens"),
@@ -273,7 +274,7 @@ def scorecard(rows, rng=None):
             line.append("-" if cell is None else str(cell))
         line += [str(sum(r["error"] for r in mine)), str(sum(r["truncated"] for r in mine)),
                  *([str(sum(bool(r.get("over_budget")) for r in mine))] if over else []),
-                 str(sum(repetitive(r["output"]) for r in mine))]
+                 str(sum(bool(r.get("loop")) or repetitive(r["output"]) for r in mine))]
         table.append(line)
     return header, table
 
@@ -682,7 +683,7 @@ def main(argv):
     print_table(*scorecard(rows, rng))
     print("\nrepeats = answers per question (×1: one answer each, the roughest accuracy).")
     print("mean score = average score, with partial credit; all repeats = tests passed on every repeat.")
-    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern.")
+    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern (or was stopped live as a loop).")
     header, table = by_difficulty(rows)
     if any(r.get("difficulty") in LEVELS for r in rows):
         print("\nPass rate by difficulty:")

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/engine.py

@@ -2400,6 +2400,7 @@ class Engine:
                   earlier_s=sum(s.get("wall_s") or 0 for s in job.earlier), sittings=len(job.earlier))
         timeout = self.cfg["defaults"]["request_timeout_ms"] / 1000
         stall_s = (self.cfg["defaults"].get("stall_timeout_ms") or 0) / 1000 or None
+        loop_chars = int(self.cfg["defaults"].get("loop_stop_chars") or 0) or None
         headers = self.request_headers(job.model)
         # a hosted API has no /tokenize to split reasoning from answer tokens (it reports the split)
         tokenize_url = None if self.cfg["servers"][job.model["server"]].get("api_key_env") else base_url
@@ -2429,7 +2430,7 @@ class Engine:
                                                      lambda kind, text: self.emit("delta", stream=kind, text=text,
                                                                                   label=job.label),
                                                      timeout=timeout, stream=self._stream, headers=headers,
-                                                     stall_s=stall_s)
+                                                     stall_s=stall_s, loop_chars=loop_chars)
                         except client.Cancelled:
                             self._check()
                             raise Cancelled()
@@ -2466,8 +2467,8 @@ class Engine:
                     if res["error"]:
                         g = {"pass": False, "score": 0.0, "reason": res["error"]}
                     else:
-                        g = graders.grade(test, res["answer"],
-                                          {"finish": res["finish"], "tool_calls": res["tool_calls"]})
+                        g = graders.grade(test, res["answer"], {"finish": res["finish"], "tool_calls": res["tool_calls"],
+                                                                "loop": res.get("loop")})
                     log_name = self._log_request(job, test, res, meta)
                     rec = {"test": test["id"], "description": test["description"], "category": test.get("category", ""),
                            "difficulty": test.get("difficulty", "unrated"),
@@ -2475,6 +2476,7 @@ class Engine:
                            "repeat": rep, "pass": g["pass"], "score": g["score"], "reason": g["reason"],
                            "severity": g.get("severity"), "answer": res["answer"], "tool_calls": res["tool_calls"],
                            "finish": res["finish"], "reasoning_chars": len(res["reasoning"]),
+                           **({"loop": res["loop"]} if res.get("loop") else {}),
                            "log": log_name,   # logs/live/<log>: the reasoning, which the result doesn't keep
                            "answered_at": dates.now(),
                            **({"checks": g["checks"]} if "checks" in g else {}),
@@ -2598,7 +2600,8 @@ def regrade(path, packs_dir=None):
             continue
         if r["reason"].startswith(("server returned", "connection error")):
             continue
-        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", [])})
+        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", []),
+                                           "loop": r.get("loop")})
         r.update({"pass": g["pass"], "score": g["score"], "reason": g["reason"], "severity": g.get("severity"),
                   "critical_test": bool(t.get("critical_trial")), "group": t.get("group")})
         r.pop("checks", None)

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/graders/__init__.py

@@ -137,6 +137,10 @@ def grade(test, answer, meta=None):
         return result(False, f"unknown grader {name!r}; known: {', '.join(sorted(REGISTRY))}")
     if meta.get("finish") == "length":
         return result(False, "TRUNCATED: hit max_tokens before finishing")
+    if meta.get("finish") == "loop":
+        loop = meta.get("loop") or {}
+        return result(False, f"LOOP: the {loop.get('stream', 'answer')} kept repeating "
+                             f"{loop.get('unit', '')[:40]!r}; stopped early (it would have run to max_tokens)")
     text = strip_thinking(answer)
     if not text and not meta.get("tool_calls"):
         return result(False, "Empty answer")

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/insights.py

@@ -280,7 +280,7 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit")):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit", "LOOP")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/src/tuieval/tui.py

@@ -431,6 +431,15 @@ def trim_names(names, width):
     return trimmed if len(set(trimmed)) == len(set(names)) else list(names)
 
 
+def repeats_tag(counts):
+    """A model's answers per question as markup, as the PTA index shows them: "×3", or "×1-3" when it varies;
+    yellow when it's ×1 (one answer decides each result, the roughest), else dim. "" for no answers."""
+    if not counts:
+        return ""
+    few, most = min(counts), max(counts)
+    return f"[{'yellow' if most == 1 else 'dim'}]×{few}{'' if few == most else f'-{most}'}[/]"
+
+
 def fit_first_column(header, rows, room, floor=24):
     """rows with the first cell trimmed in the middle (mid_trim) so a table of these columns fits room
     cells across, but never below floor cells. Cells are Rich markup; each column adds 2 cells of padding."""
@@ -624,7 +633,8 @@ class AnswerScreen(ModalScreen):
         quant = f"  [magenta]{rec['quantization']}[/magenta]" if rec.get("quantization") else ""
         finish = rec.get("finish") or "?"
         limit = f"this pack's max_tokens, {pack.max_tokens:,}" if pack and pack.max_tokens else "max_tokens"
-        finish = f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length" else finish
+        finish = (f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length"
+                  else f"[bold red]loop (stopped: it kept repeating itself)[/]" if finish == "loop" else finish)
         toks = rec.get("completion_tokens")
         split = ", ".join(f"{k} {rec[k + '_tokens']:,}" for k in ("reasoning", "answer")
                           if rec.get(k + "_tokens") is not None)
@@ -3135,7 +3145,10 @@ class ResultsScreen(Screen):
             base = column_name(col.label)
             label = base + SORT_ARROWS[desc] if base == name else base
             if col.label.plain != label:
-                col.label = Text(label)
+                text = col.label.copy()   # keeps its styling, e.g. a model's ×n
+                if len(text.plain) > len(base):   # right_crop(0) would empty it
+                    text.right_crop(len(text.plain) - len(base))
+                col.label = text + SORT_ARROWS[desc] if base == name else text
                 w = cell_len(label)   # room for the arrow (a column keeps the width it was added with)
                 col.content_width = max(col.content_width, w)
                 if not col.auto_width:
@@ -3403,7 +3416,9 @@ class ResultsScreen(Screen):
         here = self.app.engine.machine().id
         table = self.query_one("#per-question", DataTable)
         table.clear(columns=True)
-        table.add_columns(*self.PQ_FIXED, *models)
+        table.add_columns(*self.PQ_FIXED, *(Text.from_markup(
+            f"{rich_escape(m)} {repeats_tag([t['cells'][m]['n'] for t in tests if m in t['cells']])}".rstrip())
+            for m in models))
         self.pq_tests, self.pq_models = {}, models
         for t in tests:
             cells = []
@@ -3426,7 +3441,8 @@ class ResultsScreen(Screen):
             "[green]bold[/green] = fewest tokens among passing answers · [red]cut[/red] = hit max_tokens · "
             "[yellow]over[/yellow] = ran past max_tokens (the provider didn't stop it) · "
             "◐ = pack unfinished (not in scores) · dim seconds = hosted, or measured on another machine; tokens compare "
-            "across machines, seconds don't. ×tokens = most / fewest tokens across models. Enter on a cell opens the "
+            "across machines, seconds don't. ×tokens = most / fewest tokens across models. ×n beside a model: answers "
+            "per question (×1: one answer decides each result, the roughest). Enter on a cell opens the "
             "answers. Click a column name to sort by it (a model's column: by its tokens).[/dim]" + ("\n[yellow]Left out, being rerun with other questions or settings: " + ", ".join(
                 f"{os.path.basename(os.path.dirname(p))} · {os.path.basename(p)[:-5]}" for p in sorted(self.superseded))
                 + ". Only the new run's answers (◐) are shown for these.[/yellow]" if self.superseded else ""))
@@ -3468,6 +3484,7 @@ class ResultsScreen(Screen):
         def hi(v, text, key, good):
             ref = best.get(key) if good == "max" else low.get(key)
             return f"[bold green]{text}[/bold green]" if v is not None and v == ref and len(rows) > 1 else text
+        tags = {m: repeats_tag([t["cells"][m]["n"] for t in shared]) for m in models}
         cells = []
         for r in rows:
             tt = r["total_secs"]
@@ -3488,7 +3505,10 @@ class ResultsScreen(Screen):
                           (f"[red]{r['cut']}[/red]" if r["cut"] else "0")
                           + (f" [yellow]+{r['over']} over[/yellow]" if r.get("over") else "")])
         table.add_columns(*header)
-        table.add_rows(fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3))
+        tag_w = max(Text.from_markup(tag).cell_len + 1 for tag in tags.values())
+        cells = fit_first_column(header, cells, (self.size.width or self.app.size.width) - 3 - tag_w,
+                                 floor=24 - tag_w)   # the column's floor counts the ×n too
+        table.add_rows([[f"{rich_escape(c[0])} {tags[r['model']]}"] + c[1:] for c, r in zip(cells, rows)])
         line = f"[b]Comparison[/b] over the {len(shared)} questions all {len(models)} answered"
         if len(models) == 2:
             h = compare.head_to_head(shared, *models)
@@ -3505,7 +3525,7 @@ class ResultsScreen(Screen):
         title.update(line + "  [dim]Time / Tokens: each question's median, added up. Med tok / Med s: median "
                      "per answer. Tok/✓: tokens per answer right. Prompt t/s / Gen t/s: all tokens over all the time "
                      "they took (cached prompts left out). Passed: questions right on every repeat, then the score (partial credit, e.g. checks passed) when some failed. Fewest / Fastest: questions won; ties count for both. "
-                     "Cut off: answers that hit max_tokens.[/dim]")
+                     "Cut off: answers that hit max_tokens. ×n beside a name: answers per question.[/dim]")
 
     @staticmethod
     def pq_cell(c, best, here):

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/tests/mock_server.py

@@ -13,6 +13,7 @@ Streams like llama.cpp: reasoning_content then content deltas, usage and timings
 PROCESSING"), or of silence with --silent; --stall-before S: the same before the first token.
 --die-after N: the process exits after N stream chunks (0: right after the headers), with no finish
 reason or [DONE], like a server that crashes mid-request.
+--loop TEXT: after the answer, TEXT 5000 times over (a model stuck in a loop).
 An answer longer than the request's max_tokens (one token per word) is cut off with finish
 "length". --max-output N also serves an OpenRouter-style endpoints list with that output limit.
 """
@@ -55,7 +56,7 @@ def reply_for(test, mode):
 class Handler(BaseHTTPRequestHandler):
     answers, mode, model, delay, max_output = {}, "oracle", "mock", 0.0, None
     stall, stall_before, silent = 0.0, 0.0, False
-    die_after = None
+    die_after, loop = None, None
     hold, answered = None, 0   # while the hold file exists, answers after the first wait (a run that stays going)
 
     def log_message(self, *a):
@@ -94,6 +95,8 @@ class Handler(BaseHTTPRequestHandler):
         if self.mode == "unavailable":   # like OpenRouter with no provider serving the model
             return self._json({"error": {"message": f"No endpoints found for {self.model}.", "code": 404}}, 404)
         text, calls = reply_for(test, self.mode)
+        if self.loop:
+            text += self.loop * 5000
         finish = "tool_calls" if calls else "stop"
         if req.get("max_tokens") and len(text.split()) > req["max_tokens"]:
             text, finish = " ".join(text.split()[:req["max_tokens"]]), "length"
@@ -162,11 +165,12 @@ def main(argv=None):
     p.add_argument("--silent", action="store_true")
     p.add_argument("--die-after", type=int)
     p.add_argument("--hold", help="while this file exists, every answer after the first waits")
+    p.add_argument("--loop")
     a, _ = p.parse_known_args(argv)   # other flags (e.g. speed knobs under test) are accepted and ignored
     Handler.answers, Handler.mode, Handler.model, Handler.delay = load_answers(a.packs), a.mode, a.model, a.delay
     Handler.max_output = a.max_output
     Handler.stall, Handler.stall_before, Handler.silent = a.stall, a.stall_before, a.silent
-    Handler.die_after, Handler.hold = a.die_after, a.hold
+    Handler.die_after, Handler.hold, Handler.loop = a.die_after, a.hold, a.loop
     srv = Server(("127.0.0.1", a.port), Handler)
     print(f"mock server on 127.0.0.1:{a.port} ({a.mode}, {len(Handler.answers)} known questions)", flush=True)
     try:

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev137}/tests/test_tuieval.py

@@ -425,12 +425,12 @@ class PickTests(unittest.TestCase):
             self.mock.stop()
         self.tmp.cleanup()
 
-    def server(self, mode):
+    def server(self, mode, *flags):
         """(Re)start the mock on the same port: oracle answers right, wrong answers wrong."""
         first = self.mock is None
         if self.mock:
             self.mock.stop()
-        self.mock = Mock(os.path.join(self.ws, "packs"), mode, port=self.port)
+        self.mock = Mock(os.path.join(self.ws, "packs"), mode, self.port, 0.0, *flags)
         if first:
             tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
 
@@ -445,6 +445,20 @@ class PickTests(unittest.TestCase):
         return subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                               env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip()
 
+    def test_a_looping_answer_is_stopped_and_failed(self):
+        self.server("oracle", "--loop", "</think>\n")   # the right answer, then stuck
+        tuieval(self.ws, "run", "--tests", "apps:a")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual((rec["finish"], rec["pass"], rec["loop"]["stream"]), ("loop", False, "answer"))
+        self.assertTrue(rec["reason"].startswith("LOOP:"), rec["reason"])
+        self.assertLess(len(rec["answer"]), 3000)
+        path = os.path.join(self.ws, "models.toml")                      # off: it runs to the end, as before
+        write(path, read(path).replace("[defaults]\n", "[defaults]\nloop_stop_chars = 0\n", 1))
+        tuieval(self.ws, "run", "--tests", "apps:a", "--force")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual(rec["finish"], "stop")
+        self.assertNotIn("loop", rec)
+
     def test_pick(self):
         from tuieval import packs
         p = packs.load_pack(os.path.join(self.ws, "packs", "apps"))
@@ -1725,7 +1739,7 @@ class PTAIndex(unittest.TestCase):
                     print(t.row_count, str(s.query_one("#fail-count", Static).render()))
                     s.fill_per_question()
                     pq = s.query_one("#per-question", DataTable)
-                    cols = [str(c.label) for c in pq.ordered_columns]
+                    cols = [str(c.label).split(" ×")[0] for c in pq.ordered_columns]
                     print(sorted(str(pq.get_row_at(i)[cols.index("close")]).split()[1] for i in range(pq.row_count)))
             asyncio.run(go())
         """)
@@ -1787,7 +1801,7 @@ class PTAIndex(unittest.TestCase):
                         s = app.screen
                         def state():
                             pq = s.query_one("#per-question", DataTable)
-                            cols = [str(c.label) for c in pq.ordered_columns][len(s.PQ_FIXED):]
+                            cols = [str(c.label).split(" ×")[0] for c in pq.ordered_columns][len(s.PQ_FIXED):]
                             return (sorted(app.pq_models), s.query_one("#pta-pack", Select).value,
                                     str(s.query_one("#pq-models", Button).label), cols, s.query_one("#pq-shared").value)
                         s.query_one("#pq-pack", Select).value = "apps"
@@ -1941,7 +1955,7 @@ class PTAIndex(unittest.TestCase):
                             pq.move_cursor(row=0, column=col)
                             await pilot.press("enter")
                             await pilot.pause(0.3)
-                            opened[str(pq.ordered_columns[col].label.plain)] = app.screen.entries[0]["model"]
+                            opened[pq.ordered_columns[col].label.plain.split(" ×")[0]] = app.screen.entries[0]["model"]
                             app.screen.dismiss()
                             await pilot.pause(0.2)
                         print(json.dumps({"rows": rows, "bars": "P parsimony" in tri, "disagree": [both, alone], "opened": opened,
@@ -2257,6 +2271,7 @@ class FirstRun(unittest.TestCase):
         tuieval(self.ws, "add", "my-model", "--server", "local")
         code = textwrap.dedent("""
             import asyncio
+            from rich.text import Text
             from tuieval import compare
             from tuieval.tui import EvalsApp, ResultsScreen
             from textual.widgets import DataTable, TabbedContent
@@ -2274,14 +2289,56 @@ class FirstRun(unittest.TestCase):
                     app.screen.fill_pq_summary(compare.per_question(rows), models)
                     await pilot.pause(0.3)
                     t = app.screen.query_one("#pq-summary", DataTable)
-                    names = [str(t.get_row_at(i)[0]) for i in range(t.row_count)]
+                    names = [Text.from_markup(t.get_row_at(i)[0]).plain for i in range(t.row_count)]
                     print(t.row_count, t.region.height, t.virtual_size.width <= t.region.width,
-                          all(n.endswith("-q4_k_m-vllm") and "…" in n for n in names), len(set(names)))
+                          all(n.endswith("-q4_k_m-vllm ×1") and "…" in n for n in names), len(set(names)))
             asyncio.run(go())
         """)
         out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
-        self.assertEqual(out.stdout.strip(), "12 13 True True 12", out.stderr[-2000:])   # every row, no scroll
+        # every row, no scroll; a trimmed name keeps its ×n
+        self.assertEqual(out.stdout.strip(), "12 13 True True 12", out.stderr[-2000:])
+
+    def test_per_question_shows_repeats(self):
+        tuieval(self.ws, "add", "my-model", "--server", "local")
+        code = textwrap.dedent("""
+            import asyncio
+            from tuieval.tui import EvalsApp, ResultsScreen
+            from textual.widgets import DataTable, TabbedContent
+            rows = [{"model": m, "suite": "p", "test": f"p: q{q}", "ok": True, "latency": 1000, "tokens": 10,
+                     "truncated": False, "gen_tps": 20.0, "prompt_tps": None, "repeat": r, "score": 1.0}
+                    for m, reps in (("once", 1), ("thrice", 3)) for q in range(2) for r in range(reps)]
+            rows += [dict(rows[-1], test="p: q2", repeat=0)]   # one more question, answered once
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 40)) as pilot:
+                    await pilot.pause()
+                    app.push_screen(ResultsScreen())
+                    await pilot.pause(0.5)
+                    s = app.screen
+                    s.rows, s.partial_rows, s.superseded = rows, [], set()
+                    s.query_one(TabbedContent).active = "tab-per-question"
+                    await pilot.pause(0.3)
+                    s.fill_per_question()
+                    await pilot.pause(0.3)
+                    pq = s.query_one("#per-question", DataTable)
+                    heads = [c.label for c in pq.ordered_columns][len(s.PQ_FIXED):]
+                    print([h.plain for h in heads], [str(h.spans[0].style) for h in heads])
+                    app.result_sorts["per-question"] = ("thrice ×1-3", True)   # as a click on its name does
+                    s.sort_table(pq)
+                    head = pq.ordered_columns[-1].label
+                    print(head.plain, str(head.spans[0].style))
+                    t = s.query_one("#pq-summary", DataTable)
+                    print([t.get_row_at(i)[0] for i in range(t.row_count)])
+            asyncio.run(go())
+        """)
+        out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                             env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60)
+        self.assertEqual(out.stdout.strip().splitlines(), [
+            "['once ×1', 'thrice ×1-3'] ['yellow', 'dim']",   # over the questions shown
+            "thrice ×1-3 ▼ dim",                               # sorting keeps the ×n's colour
+            "['once [yellow]×1[/]', 'thrice [dim]×3[/]']",     # Comparison: only the questions both answered
+        ], out.stderr[-2000:])
 
     def test_results_open_on_pta_index(self):
         tuieval(self.ws, "add", "my-model", "--server", "local")
@@ -3188,6 +3245,32 @@ class StallGuard(unittest.TestCase):
         self.assertTrue(client.is_server_error(res["error"]), res["error"])
         self.assertIsNone(res["finish"])
 
+    def test_a_loop_is_stopped_early(self):
+        from tuieval import client, graders
+        res, took = self.ask("--loop", "</think>\n", timeout=30, loop_chars=2000)
+        self.assertIsNone(res["error"])
+        self.assertEqual((res["finish"], res["loop"]["stream"]), ("loop", "answer"))
+        self.assertEqual(sorted(res["loop"]["unit"]), sorted("</think>\n"))   # the unit, from wherever it was cut
+        self.assertLess(len(res["answer"]), 3000)                               # 45,000 chars were coming
+        g = graders.grade({"grader": "answer", "answer": "42"}, res["answer"], {"finish": "loop", "loop": res["loop"]})
+        self.assertFalse(g["pass"])
+        self.assertTrue(g["reason"].startswith("LOOP: the answer kept repeating"), g["reason"])
+        res, _ = self.ask("--loop", "</think>\n", timeout=30)                   # off: the whole answer
+        self.assertEqual(res["finish"], "stop")
+        self.assertGreater(len(res["answer"]), 40000)
+
+    def test_what_counts_as_a_loop(self):
+        from tuieval import client
+        prose = " ".join(f"step {i}: check the value and move on." for i in range(200))
+        self.assertIsNone(client.loop_unit(prose, 2000))
+        self.assertEqual(client.loop_unit("intro " + "wait, " * 400, 2000), "wait, ")
+        self.assertEqual(client.loop_unit("x" * 2500, 2000), "x")
+        self.assertIsNone(client.loop_unit("| --- " * 500, 2000))           # a ruler: no letter or digit
+        self.assertIsNone(client.loop_unit("wait, " * 300, 2000))           # shorter than min_chars
+        line = "a sentence that is a good deal longer than sixty-four chars in all.\n"
+        self.assertIsNone(client.loop_unit(line * 40, 2000))                # unit too long to call a loop
+        self.assertIsNone(client.loop_unit("wait, " * 400, 0))              # off
+
     def test_finish_without_done_is_an_answer(self):
         res, _ = self.ask("--die-after", "99", timeout=30)    # every chunk, then gone before [DONE]
         self.assertIsNone(res["error"])