tuieval 0.2.0.dev90__tar.gz → 0.2.0.dev94__tar.gz

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## Unreleased
 
+- Results → Insights (also `tuieval insights`): charts with what to do next: every model against a reference (95% CI), the best model per use case, fine-tunes against their base (`base` in models.toml), why models fail, accuracy against tokens or time, question health, and the Certify runs that would settle the strongest models' verdicts. `[defaults] reference` sets the default reference.
+- A server that stops sending tokens mid-answer for 5 minutes (`stall_timeout_ms`) is a server error: retried, never counted against the model. Keep-alive pings no longer stretch a request past `request_timeout_ms`.
+- The run screen's queue shows what each running model is doing: waiting, thinking or answering with ~tokens so far, and how long it's been quiet, so a model you aren't streaming doesn't look stuck.
 - Results, Per question: the side-by-side summary shows any number of models (it stopped at 8).
 - `max_tokens` in pack.toml sets one pack's output budget; changing it reruns only that pack. Models whose context or endpoint can't fit it are skipped for that pack.
 - The run screen's queue keeps the running model in view, unless you scroll it yourself (scrolling back to it follows again).

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev90
+Version: 0.2.0.dev94
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/docs/models.md

@@ -40,6 +40,7 @@ In the TUI, `a` adds a model and `m` scans your model folders. Each model is one
 | `max_context` (other servers) | the context the server provides; packs whose prompts need more are skipped |
 | `kv_type` (llama) | e.g. `"q8_0"`: a quantized KV cache, which makes it a different model (give it its own label) |
 | `tools = false` | packs that need tool calling are skipped |
+| `base` | for a fine-tune: the label of its base model, compared in Results → Insights |
 
 Results record the settings each model actually ran with, and the results screen notes any differences between models.
 
@@ -109,6 +110,20 @@ The P and T bars are as long as the tokens and time used: the model that used th
 
 Models are compared on the questions all of them answered, so one that skipped some never looks faster. Server and connection errors aren't answers: a model with only those is listed as having none. A model with fewer than half as many answers as the most-answered one is left out (and listed), or it would shrink the shared questions for everyone. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`: models you pick are always compared.
 
+## Insights
+
+Results → **Insights** (also `tuieval insights`) turns the results into charts, each with plain lines on what to do:
+
+1. **Who's worth keeping**: every model against a reference model, paired on the questions both answered: the difference in accuracy points with its 95% CI, and better, tie or worse. Pick the reference in the tab or with `--reference`; set a default with `reference = "<label>"` under `[defaults]` in `models.toml`. Without one, the most accurate model is the reference.
+2. **Best model per use case**: a heatmap of accuracy per model and use case (pack group); ★ marks the best among models that answered most of it.
+3. **Fine-tunes against their base model**: for each model with `base = "<label>"` in its `[[models]]` block, the difference per pack and the question categories (a test's `category`) it gained or lost most.
+4. **Why models fail**: each model's answers split into right, wrong, cut off at max_tokens, no answer or bad format, code that crashed or hung, and critical failures.
+5. **Accuracy against cost**: accuracy against median tokens (or seconds) per question, over the questions every model answered; green models are the ones no other model beats on both. With hosted and local models together, the best local trade-offs get their own line.
+6. **Question health**: questions every model fails (check them), every model passes (they don't separate models), and the ones that split models most.
+7. **Next runs**: the Certify runs that would settle the open verdicts of the strongest models, with their time and command.
+
+Models are the ones picked with **Models** (shared with the other tabs), else every model with at least half as many answers as the most-answered one. Server and connection errors aren't answers.
+
 ## Tuning
 
 `tuieval tune <model>` (or `t` on the setup screen, for the ticked models) finds the fastest speed flags for a model on this machine:
@@ -191,6 +206,8 @@ A workspace's `models.toml` only needs what differs from these. `[defaults]` fil
 [defaults]
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
+stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
+# reference = "<label>"        # the model Results → Insights compares against by default
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
 

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev90'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev90')
+__version__ = version = '0.2.0.dev94'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev94')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/src/tuieval/builtin.toml

@@ -5,6 +5,7 @@
 [defaults]
 repeat = 3
 request_timeout_ms = 1800000   # per request; long thinking runs need it
+stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
 

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/src/tuieval/cli.py

@@ -15,6 +15,7 @@ Start here
 Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --reference M, --failures)
   tuieval pta                 the PTA index: parsimony (tokens), time and accuracy per model, as bars
+  tuieval insights            charts with what to do next: vs a reference, per use case, why models fail, …
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
   tuieval items               tests that don't separate models or look broken

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/src/tuieval/client.py

@@ -99,10 +99,17 @@ def count_tokens(base_url, text):
         return None
 
 
-def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None):
+def _stalled(stall_s):
+    return f"server error: stalled, no tokens for {stall_s:.0f} s after it started answering"
+
+
+def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, headers=None, stall_s=None):
     """POST {base_url}/v1/chat/completions with streaming. Returns a dict (see module docstring).
 
     on_delta(kind, text) is called for "reasoning" and "answer" text as it arrives.
+    timeout bounds the whole request, keep-alive comments (OpenRouter's ": OPENROUTER PROCESSING")
+    included; running out counts against the model, like any slow answer. stall_s: once tokens have
+    started, this long without one is the server stalling (a server error, never the model's answer).
     """
     stream = stream or Stream()
     body = dict(body, stream=True, stream_options={"include_usage": True})
@@ -111,7 +118,7 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
     out = {"answer": "", "reasoning": "", "tool_calls": [], "finish": None, "usage": {}, "timings": {},
            "ttft_s": None, "total_s": None, "error": None}
     reasoning, answer, tools = [], [], {}
-    start = time.time()
+    start = last = time.time()
     try:
         conn.request("POST", prefix + "/v1/chat/completions", body=json.dumps(body),
                      headers={"Content-Type": "application/json", "Authorization": "Bearer none", **(headers or {})})
@@ -123,6 +130,13 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
             out["error"] = f"server returned HTTP {r.status}: {r.read()[:300].decode('utf-8', 'replace')}"
             return out
         for raw in r:
+            now = time.time()
+            if now - start > timeout:
+                out["error"] = f"connection error: timed out after {timeout:.0f} s in all"
+                break
+            if stall_s and out["ttft_s"] is not None and now - last > stall_s:
+                out["error"] = _stalled(stall_s)
+                break
             line = raw.decode("utf-8", "replace").strip()
             if not line.startswith("data:"):
                 continue
@@ -147,8 +161,12 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
                     fn = tc.get("function") or {}
                     slot["name"] += fn.get("name") or ""
                     slot["arguments"] += fn.get("arguments") or ""
-                if (think or text or delta.get("tool_calls")) and out["ttft_s"] is None:
-                    out["ttft_s"] = time.time() - start
+                if think or text or delta.get("tool_calls"):
+                    last = time.time()
+                    if out["ttft_s"] is None:
+                        out["ttft_s"] = last - start
+                        if stall_s:      # a server that goes silent mid-answer: don't wait out the whole timeout
+                            stream.sock.settimeout(min(stall_s, timeout))
                 if think:
                     reasoning.append(think)
                     on_delta and on_delta("reasoning", think)
@@ -158,7 +176,8 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
     except (OSError, http.client.HTTPException, ValueError) as e:
         if stream.cancelled:
             raise Cancelled() from None
-        out["error"] = f"connection error: {e!r}"
+        stalled = stall_s and out["ttft_s"] is not None and isinstance(e, TimeoutError)
+        out["error"] = _stalled(stall_s) if stalled else f"connection error: {e!r}"
     finally:
         conn.close()
         out["total_s"] = time.time() - start

{tuieval-0.2.0.dev90 → tuieval-0.2.0.dev94}/src/tuieval/engine.py

@@ -2229,6 +2229,7 @@ class Engine:
         self.emit("job_started", job=job, resumed=len(done_keys),
                   earlier_s=sum(s.get("wall_s") or 0 for s in job.earlier), sittings=len(job.earlier))
         timeout = self.cfg["defaults"]["request_timeout_ms"] / 1000
+        stall_s = (self.cfg["defaults"].get("stall_timeout_ms") or 0) / 1000 or None
         headers = self.request_headers(job.model)
         # a hosted API has no /tokenize to split reasoning from answer tokens (it reports the split)
         tokenize_url = None if self.cfg["servers"][job.model["server"]].get("api_key_env") else base_url
@@ -2253,7 +2254,8 @@ class Engine:
                         body = self._body(job, test, rep)
                         res = client.stream_chat(base_url, body,
                                                  lambda kind, text: self.emit("delta", stream=kind, text=text, label=job.label),
-                                                 timeout=timeout, stream=self._stream, headers=headers)
+                                                 timeout=timeout, stream=self._stream, headers=headers,
+                                                 stall_s=stall_s)
                     except client.Cancelled:
                         self._check()
                         raise Cancelled()