tuieval 0.2.0.dev133__tar.gz → 0.2.0.dev135__tar.gz

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Answers stuck in a loop (one short unit repeated over 2000 chars, in the reasoning or the answer) are stopped as they stream and failed as `LOOP`, instead of running to max_tokens. `[defaults] loop_stop_chars` sets the length; 0 turns it off.
 - Repeats: the PTA index, `tuieval compare` and Setup show how many times each model answered each question (`×3`, `×1-3`), so single-run results stand out.
 - Partial credit: the PTA index shows each model's score (e.g. 5 of 6 checks passed is 83%) beside accuracy, and can be ordered by it (By score, `tuieval pta --sort score`). Why models fail splits failures into near miss, partly right and wrong; Failures shows each one's kind and checks and filters by kind; Per question cells show `✗ 5/6`. Verdicts are unchanged.
 - Results: picking a pack (PTA index or Per question) compares the models that answered all of it; the Models button can change that, and All packs goes back to the earlier pick.

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev133
+Version: 0.2.0.dev135
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/docs/models.md

@@ -198,6 +198,10 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
 | `headers` | extra HTTP headers |
 
+## Answers stuck in a loop
+
+A model sometimes gets stuck repeating one short piece of text (a closing tag, "wait, ", a line) until it runs out of `max_tokens`, which can take many minutes and fails anyway. tuieval watches the reasoning and the answer as they stream: once either ends in one unit of up to 64 characters repeated back to back over the last `loop_stop_chars` characters (2000 by default; units without a letter or digit, like `-----`, don't count), it ends the request there. The answer fails with `LOOP: the answer kept repeating '…'`, its finish is `loop` (the answer view says so), Results → Insights counts it as cut off and `tuieval compare` in **rep**. Set `loop_stop_chars = 0` in `[defaults]` to let loops run to the end, as before.
+
 ## Built-in servers and defaults
 
 A workspace's `models.toml` only needs what differs from these. `[defaults]` fill in key by key. `[sampling]` applies only when `models.toml` has no `[sampling]` section at all, because sampling is part of every result's fingerprint. A server you define replaces the built-in one with the same name; to change one setting, copy the whole block into `models.toml` and edit it. `tuieval add` adds a `[servers.<name>]` block with just a `url` when it finds a model on a server that's already running.
@@ -210,6 +214,8 @@ A workspace's `models.toml` only needs what differs from these. `[defaults]` fil
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 # reference = "<label>"        # the model Results → Insights compares against by default
 # agent = "<name>"             # the [agents.<name>] that runs agent packs (see docs/agent-packs.md)

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev133'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev133')
+__version__ = version = '0.2.0.dev135'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev135')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/builtin.toml

@@ -6,6 +6,8 @@
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+loop_stop_chars = 2000         # an answer (or reasoning) ending in one short unit repeated over this many chars is
+                               # stuck in a loop: stopped there and failed as LOOP, not left to run to max_tokens (0: off)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/client.py

@@ -9,9 +9,12 @@ answer together with the numbers that matter for choosing a local model:
   prompt_tps     prompt processing speed, tokens/s (llama.cpp reports it)
   prompt_tokens, completion_tokens, and — when the split can be known — reasoning_tokens and
   answer_tokens (tokens_estimated=True means the split was estimated from text length)
+  loop           set when the request was ended early because the model got stuck repeating itself
+                 ({"stream": "reasoning" or "answer", "unit": the repeated text}; finish is "loop")
 """
 import http.client
 import json
+import re
 import socket
 import time
 import urllib.parse
@@ -119,17 +122,37 @@ def count_tokens(base_url, text):
         return None
 
 
+LOOP_MAX_UNIT = 64   # longest repeated unit looked for (a short line, a closing tag, a phrase)
+LOOP_EVERY = 256     # chars of new text between checks
+
+
+def loop_unit(text, min_chars):
+    """The text the end of `text` repeats, exactly and back to back, over its last min_chars or more (a unit of
+    at most LOOP_MAX_UNIT chars with a letter or digit in it, so rulers like '-----' don't count), else None."""
+    if not min_chars or len(text) < min_chars:
+        return None
+    tail = text[-min_chars:]
+    for n in range(1, LOOP_MAX_UNIT + 1):
+        if tail[n:] == tail[:-n]:          # the tail has period n: one unit over and over
+            unit = tail[-n:]
+            return unit if re.search(r"[A-Za-z0-9]", unit) else None
+    return None
+
+
 def _stalled(stall_s):
     return f"server error: stalled, no tokens for {stall_s:.0f} s after it started answering"
 
 
-def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None):
+def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None,
+                loop_chars=None):
     """POST {base_url}/v1/chat/completions with streaming. Returns a dict (see module docstring).
 
     on_delta(kind, text) is called for "reasoning" and "answer" text as it arrives.
     timeout bounds the whole request, keep-alive comments (OpenRouter's ": OPENROUTER PROCESSING")
     included; running out counts against the model, like any slow answer. stall_s: once tokens have
     started, this long without one is the server stalling (a server error, never the model's answer).
+    loop_chars: when the reasoning or the answer ends in one short unit repeated over this many chars, the
+    model is stuck in a loop that would run to max_tokens: the request ends there (finish "loop").
     """
     stream = stream or Stream()
     body = dict(body, stream=True, stream_options={"include_usage": True})
@@ -138,6 +161,7 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
     out = {"answer": "", "reasoning": "", "tool_calls": [], "finish": None, "usage": {}, "timings": {},
            "ttft_s": None, "total_s": None, "error": None}
     reasoning, answer, tools = [], [], {}
+    tails, unchecked = {"reasoning": "", "answer": ""}, {"reasoning": 0, "answer": 0}   # for loop_chars
     done = False                      # [DONE] seen
     start = last = time.time()
     try:
@@ -195,6 +219,17 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
                 if text:
                     answer.append(text)
                     on_delta and on_delta("answer", text)
+                for kind, piece in (("reasoning", think), ("answer", text)):
+                    if loop_chars and piece:
+                        tails[kind] = (tails[kind] + piece)[-loop_chars:]
+                        unchecked[kind] += len(piece)
+                        if unchecked[kind] >= LOOP_EVERY:
+                            unchecked[kind] = 0
+                            unit = loop_unit(tails[kind], loop_chars)
+                            if unit:
+                                out["finish"], out["loop"] = "loop", {"stream": kind, "unit": unit}
+            if out.get("loop"):
+                break
         if not out["error"] and not done and out["finish"] is None:
             # the stream just ended: a server that crashed or dropped the connection, not the model's answer
             out["error"] = "connection error: the stream ended before the server finished (no finish reason)"

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/compare.py

@@ -52,6 +52,7 @@ def _load_native(data):
             # a timeout is the model too slow (client.is_server_error), not an error; older results called it one
             "error": r["reason"].startswith(("server returned", "connection error")) and "timed out" not in r["reason"],
             "truncated": r.get("finish") == "length", "output": r.get("answer") or "",
+            "loop": r.get("finish") == "loop",   # stopped live: it kept repeating itself (client.loop_unit)
             # tokens past the max_tokens sent, which the provider didn't stop (client.mark_over_budget)
             "over_budget": r.get("over_budget"),
             "tokens": r.get("completion_tokens") or 0, "reasoning_tokens": r.get("reasoning_tokens"),
@@ -273,7 +274,7 @@ def scorecard(rows, rng=None):
             line.append("-" if cell is None else str(cell))
         line += [str(sum(r["error"] for r in mine)), str(sum(r["truncated"] for r in mine)),
                  *([str(sum(bool(r.get("over_budget")) for r in mine))] if over else []),
-                 str(sum(repetitive(r["output"]) for r in mine))]
+                 str(sum(bool(r.get("loop")) or repetitive(r["output"]) for r in mine))]
         table.append(line)
     return header, table
 
@@ -682,7 +683,7 @@ def main(argv):
     print_table(*scorecard(rows, rng))
     print("\nrepeats = answers per question (×1: one answer each, the roughest accuracy).")
     print("mean score = average score, with partial credit; all repeats = tests passed on every repeat.")
-    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern.")
+    print("trunc = hit max_tokens; rep = output ends in a degenerate repeated pattern (or was stopped live as a loop).")
     header, table = by_difficulty(rows)
     if any(r.get("difficulty") in LEVELS for r in rows):
         print("\nPass rate by difficulty:")

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/engine.py

@@ -2400,6 +2400,7 @@ class Engine:
                   earlier_s=sum(s.get("wall_s") or 0 for s in job.earlier), sittings=len(job.earlier))
         timeout = self.cfg["defaults"]["request_timeout_ms"] / 1000
         stall_s = (self.cfg["defaults"].get("stall_timeout_ms") or 0) / 1000 or None
+        loop_chars = int(self.cfg["defaults"].get("loop_stop_chars") or 0) or None
         headers = self.request_headers(job.model)
         # a hosted API has no /tokenize to split reasoning from answer tokens (it reports the split)
         tokenize_url = None if self.cfg["servers"][job.model["server"]].get("api_key_env") else base_url
@@ -2429,7 +2430,7 @@ class Engine:
                                                      lambda kind, text: self.emit("delta", stream=kind, text=text,
                                                                                   label=job.label),
                                                      timeout=timeout, stream=self._stream, headers=headers,
-                                                     stall_s=stall_s)
+                                                     stall_s=stall_s, loop_chars=loop_chars)
                         except client.Cancelled:
                             self._check()
                             raise Cancelled()
@@ -2466,8 +2467,8 @@ class Engine:
                     if res["error"]:
                         g = {"pass": False, "score": 0.0, "reason": res["error"]}
                     else:
-                        g = graders.grade(test, res["answer"],
-                                          {"finish": res["finish"], "tool_calls": res["tool_calls"]})
+                        g = graders.grade(test, res["answer"], {"finish": res["finish"], "tool_calls": res["tool_calls"],
+                                                                "loop": res.get("loop")})
                     log_name = self._log_request(job, test, res, meta)
                     rec = {"test": test["id"], "description": test["description"], "category": test.get("category", ""),
                            "difficulty": test.get("difficulty", "unrated"),
@@ -2475,6 +2476,7 @@ class Engine:
                            "repeat": rep, "pass": g["pass"], "score": g["score"], "reason": g["reason"],
                            "severity": g.get("severity"), "answer": res["answer"], "tool_calls": res["tool_calls"],
                            "finish": res["finish"], "reasoning_chars": len(res["reasoning"]),
+                           **({"loop": res["loop"]} if res.get("loop") else {}),
                            "log": log_name,   # logs/live/<log>: the reasoning, which the result doesn't keep
                            "answered_at": dates.now(),
                            **({"checks": g["checks"]} if "checks" in g else {}),
@@ -2598,7 +2600,8 @@ def regrade(path, packs_dir=None):
             continue
         if r["reason"].startswith(("server returned", "connection error")):
             continue
-        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", [])})
+        g = graders.grade(t, r["answer"], {"finish": r["finish"], "tool_calls": r.get("tool_calls", []),
+                                           "loop": r.get("loop")})
         r.update({"pass": g["pass"], "score": g["score"], "reason": g["reason"], "severity": g.get("severity"),
                   "critical_test": bool(t.get("critical_trial")), "group": t.get("group")})
         r.pop("checks", None)

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/graders/__init__.py

@@ -137,6 +137,10 @@ def grade(test, answer, meta=None):
         return result(False, f"unknown grader {name!r}; known: {', '.join(sorted(REGISTRY))}")
     if meta.get("finish") == "length":
         return result(False, "TRUNCATED: hit max_tokens before finishing")
+    if meta.get("finish") == "loop":
+        loop = meta.get("loop") or {}
+        return result(False, f"LOOP: the {loop.get('stream', 'answer')} kept repeating "
+                             f"{loop.get('unit', '')[:40]!r}; stopped early (it would have run to max_tokens)")
     text = strip_thinking(answer)
     if not text and not meta.get("tool_calls"):
         return result(False, "Empty answer")

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/insights.py

@@ -280,7 +280,7 @@ def kind(r):
         return "right"
     if r.get("severity") == "critical":
         return "critical"
-    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit")):
+    if r["truncated"] or r["reason"].startswith(("TRUNCATED", "OVER BUDGET", "time limit", "LOOP")):
         return "cut off"
     if r["reason"].startswith(FORMAT):
         return "no answer / format"

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/src/tuieval/tui.py

@@ -624,7 +624,8 @@ class AnswerScreen(ModalScreen):
         quant = f"  [magenta]{rec['quantization']}[/magenta]" if rec.get("quantization") else ""
         finish = rec.get("finish") or "?"
         limit = f"this pack's max_tokens, {pack.max_tokens:,}" if pack and pack.max_tokens else "max_tokens"
-        finish = f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length" else finish
+        finish = (f"[bold red]{finish} (cut off at {limit})[/]" if finish == "length"
+                  else f"[bold red]loop (stopped: it kept repeating itself)[/]" if finish == "loop" else finish)
         toks = rec.get("completion_tokens")
         split = ", ".join(f"{k} {rec[k + '_tokens']:,}" for k in ("reasoning", "answer")
                           if rec.get(k + "_tokens") is not None)

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/tests/mock_server.py

@@ -13,6 +13,7 @@ Streams like llama.cpp: reasoning_content then content deltas, usage and timings
 PROCESSING"), or of silence with --silent; --stall-before S: the same before the first token.
 --die-after N: the process exits after N stream chunks (0: right after the headers), with no finish
 reason or [DONE], like a server that crashes mid-request.
+--loop TEXT: after the answer, TEXT 5000 times over (a model stuck in a loop).
 An answer longer than the request's max_tokens (one token per word) is cut off with finish
 "length". --max-output N also serves an OpenRouter-style endpoints list with that output limit.
 """
@@ -55,7 +56,7 @@ def reply_for(test, mode):
 class Handler(BaseHTTPRequestHandler):
     answers, mode, model, delay, max_output = {}, "oracle", "mock", 0.0, None
     stall, stall_before, silent = 0.0, 0.0, False
-    die_after = None
+    die_after, loop = None, None
     hold, answered = None, 0   # while the hold file exists, answers after the first wait (a run that stays going)
 
     def log_message(self, *a):
@@ -94,6 +95,8 @@ class Handler(BaseHTTPRequestHandler):
         if self.mode == "unavailable":   # like OpenRouter with no provider serving the model
             return self._json({"error": {"message": f"No endpoints found for {self.model}.", "code": 404}}, 404)
         text, calls = reply_for(test, self.mode)
+        if self.loop:
+            text += self.loop * 5000
         finish = "tool_calls" if calls else "stop"
         if req.get("max_tokens") and len(text.split()) > req["max_tokens"]:
             text, finish = " ".join(text.split()[:req["max_tokens"]]), "length"
@@ -162,11 +165,12 @@ def main(argv=None):
     p.add_argument("--silent", action="store_true")
     p.add_argument("--die-after", type=int)
     p.add_argument("--hold", help="while this file exists, every answer after the first waits")
+    p.add_argument("--loop")
     a, _ = p.parse_known_args(argv)   # other flags (e.g. speed knobs under test) are accepted and ignored
     Handler.answers, Handler.mode, Handler.model, Handler.delay = load_answers(a.packs), a.mode, a.model, a.delay
     Handler.max_output = a.max_output
     Handler.stall, Handler.stall_before, Handler.silent = a.stall, a.stall_before, a.silent
-    Handler.die_after, Handler.hold = a.die_after, a.hold
+    Handler.die_after, Handler.hold, Handler.loop = a.die_after, a.hold, a.loop
     srv = Server(("127.0.0.1", a.port), Handler)
     print(f"mock server on 127.0.0.1:{a.port} ({a.mode}, {len(Handler.answers)} known questions)", flush=True)
     try:

{tuieval-0.2.0.dev133 → tuieval-0.2.0.dev135}/tests/test_tuieval.py

@@ -425,12 +425,12 @@ class PickTests(unittest.TestCase):
             self.mock.stop()
         self.tmp.cleanup()
 
-    def server(self, mode):
+    def server(self, mode, *flags):
         """(Re)start the mock on the same port: oracle answers right, wrong answers wrong."""
         first = self.mock is None
         if self.mock:
             self.mock.stop()
-        self.mock = Mock(os.path.join(self.ws, "packs"), mode, port=self.port)
+        self.mock = Mock(os.path.join(self.ws, "packs"), mode, self.port, 0.0, *flags)
         if first:
             tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
 
@@ -445,6 +445,20 @@ class PickTests(unittest.TestCase):
         return subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
                               env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip()
 
+    def test_a_looping_answer_is_stopped_and_failed(self):
+        self.server("oracle", "--loop", "</think>\n")   # the right answer, then stuck
+        tuieval(self.ws, "run", "--tests", "apps:a")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual((rec["finish"], rec["pass"], rec["loop"]["stream"]), ("loop", False, "answer"))
+        self.assertTrue(rec["reason"].startswith("LOOP:"), rec["reason"])
+        self.assertLess(len(rec["answer"]), 3000)
+        path = os.path.join(self.ws, "models.toml")                      # off: it runs to the end, as before
+        write(path, read(path).replace("[defaults]\n", "[defaults]\nloop_stop_chars = 0\n", 1))
+        tuieval(self.ws, "run", "--tests", "apps:a", "--force")
+        rec = json.loads(read(os.path.join(self.ws, "results", "m", "apps.json")))["results"][0]
+        self.assertEqual(rec["finish"], "stop")
+        self.assertNotIn("loop", rec)
+
     def test_pick(self):
         from tuieval import packs
         p = packs.load_pack(os.path.join(self.ws, "packs", "apps"))
@@ -3188,6 +3202,32 @@ class StallGuard(unittest.TestCase):
         self.assertTrue(client.is_server_error(res["error"]), res["error"])
         self.assertIsNone(res["finish"])
 
+    def test_a_loop_is_stopped_early(self):
+        from tuieval import client, graders
+        res, took = self.ask("--loop", "</think>\n", timeout=30, loop_chars=2000)
+        self.assertIsNone(res["error"])
+        self.assertEqual((res["finish"], res["loop"]["stream"]), ("loop", "answer"))
+        self.assertEqual(sorted(res["loop"]["unit"]), sorted("</think>\n"))   # the unit, from wherever it was cut
+        self.assertLess(len(res["answer"]), 3000)                               # 45,000 chars were coming
+        g = graders.grade({"grader": "answer", "answer": "42"}, res["answer"], {"finish": "loop", "loop": res["loop"]})
+        self.assertFalse(g["pass"])
+        self.assertTrue(g["reason"].startswith("LOOP: the answer kept repeating"), g["reason"])
+        res, _ = self.ask("--loop", "</think>\n", timeout=30)                   # off: the whole answer
+        self.assertEqual(res["finish"], "stop")
+        self.assertGreater(len(res["answer"]), 40000)
+
+    def test_what_counts_as_a_loop(self):
+        from tuieval import client
+        prose = " ".join(f"step {i}: check the value and move on." for i in range(200))
+        self.assertIsNone(client.loop_unit(prose, 2000))
+        self.assertEqual(client.loop_unit("intro " + "wait, " * 400, 2000), "wait, ")
+        self.assertEqual(client.loop_unit("x" * 2500, 2000), "x")
+        self.assertIsNone(client.loop_unit("| --- " * 500, 2000))           # a ruler: no letter or digit
+        self.assertIsNone(client.loop_unit("wait, " * 300, 2000))           # shorter than min_chars
+        line = "a sentence that is a good deal longer than sixty-four chars in all.\n"
+        self.assertIsNone(client.loop_unit(line * 40, 2000))                # unit too long to call a loop
+        self.assertIsNone(client.loop_unit("wait, " * 400, 0))              # off
+
     def test_finish_without_done_is_an_answer(self):
         res, _ = self.ask("--die-after", "99", timeout=30)    # every chunk, then gone before [DONE]
         self.assertIsNone(res["error"])