tuieval 0.2.0.dev34__tar.gz → 0.2.0.dev42__tar.gz

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- PTA index: privacy, time (total, relative to the fastest) and accuracy per model, as a triangle in Results, `tuieval pta` and the report.
 - Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
 - More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev34
+Version: 0.2.0.dev42
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -62,6 +62,14 @@ Once you've added your own packs and models, the setup screen looks like this (p
 
 ![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
+While a run goes, you watch each question's reasoning and answer stream in:
+
+![tuieval's run screen with a model's reasoning streaming in for a coding question](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-run-reasoning.png)
+
+Afterwards, every answer keeps its reasoning. Enter on a result opens it with the grade and each check, the full reasoning and the answer; `[` and `]` step through the others:
+
+![one answer in full: the grade and its checks, the model's reasoning and its code](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-answer-reasoning.png)
+
 ## What you get
 
 - **A TUI** to pick packs and models, watch reasoning and answers stream live with TTFT, tokens/s, memory and a running score, and browse results.
@@ -156,6 +164,7 @@ tuieval selftest                              # check every test's reference and
 tuieval items                                 # tests that don't separate models or look broken
 tuieval capture logs/live/<file> --pack X     # turn a real failure into a new test
 tuieval regrade                               # re-score stored answers after changing a grader
+tuieval regrade --only <model> --packs X      # just one model's results (and packs)
 tuieval machines                              # this machine and others, fit and tuning per model
 tuieval tune <model>                          # fastest speed flags for a model on this machine
 tuieval export pi <model>                     # serve it in the pi coding agent with those flags

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/README.md

@@ -36,6 +36,14 @@ Once you've added your own packs and models, the setup screen looks like this (p
 
 ![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
+While a run goes, you watch each question's reasoning and answer stream in:
+
+![tuieval's run screen with a model's reasoning streaming in for a coding question](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-run-reasoning.png)
+
+Afterwards, every answer keeps its reasoning. Enter on a result opens it with the grade and each check, the full reasoning and the answer; `[` and `]` step through the others:
+
+![one answer in full: the grade and its checks, the model's reasoning and its code](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-answer-reasoning.png)
+
 ## What you get
 
 - **A TUI** to pick packs and models, watch reasoning and answers stream live with TTFT, tokens/s, memory and a running score, and browse results.
@@ -130,6 +138,7 @@ tuieval selftest                              # check every test's reference and
 tuieval items                                 # tests that don't separate models or look broken
 tuieval capture logs/live/<file> --pack X     # turn a real failure into a new test
 tuieval regrade                               # re-score stored answers after changing a grader
+tuieval regrade --only <model> --packs X      # just one model's results (and packs)
 tuieval machines                              # this machine and others, fit and tuning per model
 tuieval tune <model>                          # fastest speed flags for a model on this machine
 tuieval export pi <model>                     # serve it in the pi coding agent with those flags

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/docs/models.md

@@ -84,6 +84,16 @@ By default a run serves one model at a time. On a machine with room for more, se
 - In the TUI, one model's answers stream at a time: `v` switches to the next, `k` skips the one streaming. `tuieval run` prints one line per answer instead of streaming.
 - Only one run uses the machine at a time across tuieval windows; models at a time applies within a run.
 
+## PTA index
+
+The **PTA index** puts privacy, time and accuracy side by side, 0-100 each, as a radar triangle (Results → PTA index, `tuieval pta`, and a section in `tuieval report`). Each model is a triangle reaching each corner as far as its score: bigger means better all round. There's no combined number; verdicts and critical failures stay in Production readiness.
+
+- **P privacy**: 100 when prompts stay on machines you control (tuieval starts the server, its `url` is this machine, or the server has `private = true`, e.g. your own box on the network); 0 for a hosted API.
+- **T time**: the total time to answer every question compared (each question at its median over repeats), relative to the fastest model: `100 × fastest total ÷ this model's total`. Adding a faster model lowers the others.
+- **A accuracy**: the share of answers to those questions that passed.
+
+Models are compared on the questions all of them answered, so one that skipped some never looks faster. Pick the models and pack to compare in the TUI, or use `--only` and `--packs`.
+
 ## Tuning
 
 `tuieval tune <model>` (or `t` on the setup screen, for the ticked models) finds the fastest speed flags for a model on this machine:
@@ -130,6 +140,7 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 |---|---|
 | `cmd` | command that starts the server (no `cmd` = an already-running server at `url`). Placeholders: `{model}`, `{served_name}`, `{port}`, `{mmproj}`, `{ctx}`, `{kv_type}`, `{root}` (the workspace), and the tune placeholders. An argument `env:NAME=value` sets an environment variable instead. |
 | `url` | an already-running server's base URL |
+| `private` | `true` for a server on a machine you control (e.g. your own box on the network): its models score 100 for privacy in the PTA index |
 | `port` | the port `cmd` serves on |
 | `cwd` | folder to start `cmd` in |
 | `model_is_path` | `model` is a file or folder: check it exists before starting |

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev34'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev34')
+__version__ = version = '0.2.0.dev42'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev42')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/src/tuieval/cli.py

@@ -14,11 +14,12 @@ Start here
 
 Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
+  tuieval pta                 the PTA index: privacy, time and accuracy per model, as a triangle
   tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
   tuieval items               tests that don't separate models or look broken
   tuieval capture LOG --pack  turn a real failure (logs/live/*.txt) into a new test
-  tuieval regrade             re-score stored answers after changing a grader
+  tuieval regrade             re-score stored answers after changing a grader (--only, --packs)
   tuieval list                the models tuieval knows, hidden ones separately (--packs: the packs)
   tuieval remove MODEL        remove models, or every hidden one (--hidden); archived in removed/
 

tuieval-0.2.0.dev42/src/tuieval/pta.py

@@ -0,0 +1,185 @@
+"""PTA index: privacy, time and accuracy per model, 0-100 each, drawn as a radar triangle.
+
+    P  privacy   100 when prompts stay on machines you control: tuieval starts the server, its URL
+                 is this machine, or the server is marked private = true in models.toml. 0 otherwise
+                 (a hosted API: there's no partial privacy once prompts leave).
+    T  time      the total time to answer every question compared (each question at its median over
+                 repeats), relative to the fastest model: 100 x fastest total / this model's total.
+    A  accuracy  the share of answers to those questions that passed.
+
+Models are compared on the questions all of them answered, so one that skipped some never looks
+faster. There's no combined number: the three stay side by side, and verdicts and critical
+failures stay where they are (verdict.py).
+"""
+import urllib.parse
+
+from . import compare
+
+LOCAL_HOSTS = {"localhost", "127.0.0.1", "::1", "0.0.0.0"}
+# one colour per model on the triangle; more models than colours are listed but not drawn
+COLORS = ["cyan", "magenta", "yellow", "green", "red", "blue", "bright_white", "bright_cyan"]
+
+
+def privacy(cfg, label):
+    """100 if a model's prompts stay on machines you control, else 0 (see the module docstring);
+    None for a model that's no longer in models.toml (where it ran isn't known)."""
+    m = next((m for m in cfg["models"] if m["label"] == label), None)
+    server = cfg["servers"].get(m["server"], {}) if m else {}
+    if not server:
+        return None
+    if server.get("cmd") or server.get("private"):
+        return 100
+    host = urllib.parse.urlparse(server.get("url", "")).hostname or ""
+    return 100 if host.lower() in LOCAL_HOSTS else 0
+
+
+def index(rows, privacy_of, models=None):
+    """{"questions": n shared, "models": [{model, P, T, A, total_s, passed, answers, alongside}]},
+    best accuracy first. models: the ones to compare (default: every model in rows)."""
+    tests = compare.per_question(rows)
+    models = sorted(models or {m for t in tests for m in t["cells"]})
+    shared = [t for t in tests if models and all(m in t["cells"] for m in models)]
+    out = []
+    for m in models:
+        cells = [t["cells"][m] for t in shared]
+        answers = sum(c["n"] for c in cells)
+        out.append({"model": m, "P": privacy_of(m), "total_s": sum(c["secs"] or 0 for c in cells),
+                    "passed": sum(c["passed"] for c in cells), "answers": answers,
+                    "A": 100 * sum(c["passed"] for c in cells) / answers if answers else None,
+                    # answers that ran while other models were served shared the machine's speed
+                    "alongside": any(r.get("alongside") for c in cells for r in c["rows"])})
+    fastest = min((x["total_s"] for x in out if x["total_s"] > 0), default=None)
+    for x in out:
+        x["T"] = 100 * fastest / x["total_s"] if fastest and x["total_s"] > 0 else None
+    out.sort(key=lambda x: (-(x["A"] or 0), -(x["T"] or 0), x["model"]))
+    return {"questions": len(shared), "models": out}
+
+
+# ---------------------------------------------------------------- the triangle (braille, no dependencies)
+_DOTS = ((0x01, 0x08), (0x02, 0x10), (0x04, 0x20), (0x40, 0x80))   # braille dot bits by (row, column)
+
+
+class _Canvas:
+    """Braille cells, 2 dots wide and 4 tall each; every cell takes the colour of the last line through it."""
+    def __init__(self, cols, rows):
+        self.cols, self.rows = cols, rows
+        self.bits = [[0] * cols for _ in range(rows)]
+        self.color = [[None] * cols for _ in range(rows)]
+
+    def dot(self, x, y, color):
+        cx, cy = int(x) // 2, int(y) // 4
+        if 0 <= cx < self.cols and 0 <= cy < self.rows:
+            self.bits[cy][cx] |= _DOTS[int(y) % 4][int(x) % 2]
+            self.color[cy][cx] = color
+
+    def line(self, a, b, color, dotted=False):
+        steps = int(max(abs(b[0] - a[0]), abs(b[1] - a[1]))) or 1
+        for i in range(steps + 1):
+            if dotted and i % 3:
+                continue
+            self.dot(a[0] + (b[0] - a[0]) * i / steps, a[1] + (b[1] - a[1]) * i / steps, color)
+
+    def text(self):
+        lines = []
+        for bits, colors in zip(self.bits, self.color):
+            out = []
+            for b, c in zip(bits, colors):
+                ch = chr(0x2800 + b) if b else " "
+                out.append(f"[{c}]{ch}[/]" if b and c else ch)
+            lines.append("".join(out).rstrip())
+        return lines
+
+
+def triangle(result, width=60):
+    """The radar triangle as Rich markup lines: P at the top, T bottom left, A bottom right; a
+    model's triangle reaches each corner as far as its score (bigger = better all round)."""
+    cols = max(20, width)
+    w = cols * 2 - 2                            # dots across
+    h = int(w * 0.866)                          # an equilateral triangle (braille dots are about square)
+    rows = h // 4 + 1
+    top, left, right = (w / 2, 0), (0, h), (w, h)
+    center = (w / 2, h * 2 / 3)
+    c = _Canvas(cols, rows)
+
+    def at(corner, score):
+        return (center[0] + (corner[0] - center[0]) * score / 100, center[1] + (corner[1] - center[1]) * score / 100)
+
+    for corner in (top, left, right):           # axes, the 50% ring, then the frame
+        c.line(center, corner, "grey37", dotted=True)
+    half = [at(k, 50) for k in (top, left, right)]
+    for a, b in zip(half, half[1:] + half[:1]):
+        c.line(a, b, "grey37", dotted=True)
+    for a, b in ((top, left), (left, right), (right, top)):
+        c.line(a, b, "grey70")
+    for i, x in enumerate(result["models"][:len(COLORS)][::-1]):   # the best ends on top
+        color = COLORS[len(result["models"][:len(COLORS)]) - 1 - i]
+        pts = [at(k, s or 0) for k, s in ((top, x["P"]), (left, x["T"]), (right, x["A"]))]
+        for a, b in zip(pts, pts[1:] + pts[:1]):
+            c.line(a, b, color)
+    lines = c.text()
+    pad = " " * max(0, cols // 2 - 5)
+    return ([f"{pad}[b]P[/b] privacy"] + lines
+            + [f"[b]T[/b] time{' ' * max(1, cols - 15)}[b]A[/b] accuracy"])
+
+
+def legend(result):
+    """One Rich markup line per model: its colour on the triangle and its three scores."""
+    out = []
+    for i, x in enumerate(result["models"]):
+        mark = f"[{COLORS[i]}]■[/]" if i < len(COLORS) else " "
+        out.append(f"{mark} {x['model']}  P {_score(x['P'])} · T {_score(x['T'])} · A {_score(x['A'])}"
+                   + ("  [yellow](some answers ran alongside other models)[/yellow]" if x["alongside"] else ""))
+    return out
+
+
+def _score(v):
+    return "-" if v is None else f"{v:.0f}"
+
+
+def table(result):
+    """(header, rows) for a plain table of the index."""
+    header = ["model", "P privacy", "T time", "A accuracy", "total time", "answers right"]
+    rows = [[x["model"], _score(x["P"]), _score(x["T"]), _score(x["A"]), fmt_total(x["total_s"]),
+             f"{x['passed']}/{x['answers']}"] for x in result["models"]]
+    return header, rows
+
+
+def fmt_total(secs):
+    if not secs:
+        return "-"
+    if secs < 60:
+        return f"{secs:.1f}s"
+    secs = round(secs)
+    h, rem = divmod(secs, 3600)
+    return f"{h}h{rem // 60:02d}m" if h else f"{rem // 60}m{rem % 60:02d}s" if rem >= 60 else f"{rem}s"
+
+
+def scope_note(result, models):
+    """What the scores compare, in a sentence."""
+    if not result["questions"]:
+        return "These models have no question in common yet, so there's nothing to compare."
+    return (f"Compared on the {result['questions']} questions all {len(models)} model(s) answered. "
+            "T is relative to the fastest total, so adding a faster model lowers the others.")
+
+
+def for_engine(e, labels=None, packs=None, results_dir=None):
+    """(result, models) from a workspace's finished results (results on their way out excluded)."""
+    paths = [p for p in compare.default_paths(results_dir or e.results_dir) if not e.superseded(p)]
+    rows, _ = compare.load(paths) if paths else ([], [])
+    if packs:
+        rows = [r for r in rows if r["suite"] in packs]
+    models = sorted({r["model"] for r in rows} & set(labels) if labels else {r["model"] for r in rows})
+    return index(rows, lambda m: privacy(e.cfg, m), models), models
+
+
+def markdown(result, models):
+    """The index as a markdown section (tuieval report)."""
+    lines = ["## PTA index", "", scope_note(result, models), ""]
+    if not result["questions"]:
+        return lines
+    header, rows = table(result)
+    lines += ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]
+    lines += ["| " + " | ".join(r) + " |" for r in rows]
+    if any(x["alongside"] for x in result["models"]):
+        lines += ["", "Some answers ran alongside other models, which shared the machine's speed."]
+    return lines

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/src/tuieval/run_evals.py

@@ -21,6 +21,7 @@
     tuieval list                                  # the models tuieval knows; hidden ones listed separately
     tuieval remove --hidden                       # clean up hidden models (archived in removed/)
     tuieval regrade                               # re-score stored answers with the current graders
+    tuieval regrade --only my-model --packs coding   # just one model's results (and packs)
     tuieval machines                              # this machine, known machines, fit and tuning per model
     tuieval tune my-model                         # find the fastest speed flags for a model on this machine
     tuieval tune --untuned                        # tune every model that has no profile here yet
@@ -270,9 +271,22 @@ def cmd_remove(argv):
 def cmd_regrade(argv):
     p = argparse.ArgumentParser(prog="tuieval regrade", description="Re-score stored answers with current graders.")
     p.add_argument("paths", nargs="*", help="result files (default: all in results/)")
+    p.add_argument("--only", help="comma-separated model labels")
+    p.add_argument("--packs", help="comma-separated packs")
     a = p.parse_args(argv)
     from . import compare
-    for path in a.paths or compare.default_paths():
+    paths = a.paths or compare.default_paths()
+    for flag, value, part in (("--only", a.only, lambda q: os.path.basename(os.path.dirname(q))),
+                              ("--packs", a.packs, lambda q: os.path.basename(q)[:-len(".json")])):
+        if not value:
+            continue
+        wanted = [v.strip() for v in value.split(",") if v.strip()]
+        known = sorted({part(q) for q in paths})
+        missing = [v for v in wanted if v not in known]
+        if missing:
+            sys.exit(f"{flag}: no results for {', '.join(missing)}. With results: {', '.join(known) or 'none'}")
+        paths = [q for q in paths if part(q) in wanted]
+    for path in paths:
         try:
             before, after = engine.regrade(path)
             print(f"{os.path.relpath(path)}: {before} -> {after} passed")
@@ -368,9 +382,43 @@ def report_markdown(e, labels=None):
                     lines.append("  - by difficulty: " + ", ".join(levels))
                 for ex in ev.get("critical_examples", []):
                     lines.append(f"  - critical: {ex}")
+    from . import pta
+    lines += [""] + pta.markdown(*pta.for_engine(e, labels))
     return "\n".join(lines) + "\n"
 
 
+def cmd_pta(argv):
+    p = argparse.ArgumentParser(prog="tuieval pta",
+                                description="The PTA index: privacy, time and accuracy per model (0-100 each), as a "
+                                            "triangle and a table, over the questions all the models answered.")
+    p.add_argument("--only", help="comma-separated model labels (default: every model with results)")
+    p.add_argument("--packs", help="comma-separated packs (default: all)")
+    p.add_argument("--width", type=int, default=48, help="triangle width in characters (default 48)")
+    p.add_argument("--models")
+    p.add_argument("--results-dir")
+    a = p.parse_args(argv)
+    from rich.console import Console
+    from . import pta
+    e = engine.Engine(models_path=a.models, results_dir=a.results_dir)
+    labels = [x.strip() for x in a.only.split(",")] if a.only else None
+    packs = [x.strip() for x in a.packs.split(",")] if a.packs else None
+    result, models = pta.for_engine(e, labels, packs)
+    out = Console(highlight=False)
+    if not models:
+        sys.exit("no finished results to compare" + (" for those models or packs" if labels or packs else ""))
+    out.print(pta.scope_note(result, models))
+    if not result["questions"]:
+        return
+    out.print()
+    for line in pta.triangle(result, a.width) + [""] + pta.legend(result):
+        out.print(line)
+    header, rows = pta.table(result)
+    widths = [max(len(h), *(len(r[i]) for r in rows)) for i, h in enumerate(header)]
+    out.print()
+    for line in [header] + rows:
+        out.print("  ".join(c.ljust(w) for c, w in zip(line, widths)).rstrip(), markup=False)
+
+
 def cmd_report(argv):
     p = argparse.ArgumentParser(prog="tuieval report", description="Write the readiness report as markdown.")
     p.add_argument("-o", "--output", help="default: reports/readiness-<date>.md")
@@ -652,7 +700,7 @@ def cmd_export(argv):
 COMMANDS = {"add": cmd_add, "list": cmd_list, "scan": cmd_scan, "regrade": cmd_regrade,
             "verdict": cmd_verdict, "report": cmd_report, "items": cmd_items, "selftest": cmd_selftest,
             "capture": cmd_capture, "history": cmd_history, "machines": cmd_machines, "tune": cmd_tune,
-            "export": cmd_export, "remove": cmd_remove}
+            "export": cmd_export, "remove": cmd_remove, "pta": cmd_pta}
 
 
 def parse_tests(text, pack_names, known):

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/src/tuieval/tui.py

@@ -181,7 +181,8 @@ def fmt_secs(secs):
 
 
 class StreamView(TextArea):
-    """Read-only, word-wrapped text that grows as tokens stream in. Scroll it with the mouse."""
+    """Read-only, word-wrapped text that grows as tokens stream in. Scroll it with the mouse; it follows
+    new text only while scrolled to the bottom."""
 
     can_focus = False
 
@@ -189,14 +190,28 @@ class StreamView(TextArea):
 
     def __init__(self, **kw):
         super().__init__(read_only=True, soft_wrap=True, show_cursor=False, max_checkpoints=1, **kw)
+        self.trim_pending = False
 
     def append(self, text, follow=True):
+        # Follow only when already at the bottom: scrolled up to read, the view stays put.
+        follow = follow and self.scroll_y >= self.max_scroll_y - 1
         self.insert(text, self.document.end)
         if len(self.text) > self.MAX_CHARS:
-            self.load_text("…" + self.text[-self.MAX_CHARS // 2:])
+            if follow:
+                self.trim()
+            elif not self.trim_pending:   # once the new text is laid out, so the place is measured right
+                self.trim_pending = True
+                self.call_after_refresh(self.trim)
         if follow:
             self.scroll_end(animate=False)
 
+    def trim(self):
+        """Drop the oldest text, keeping the view the same distance from the end (the text that's left)."""
+        self.trim_pending = False
+        from_end = self.max_scroll_y - self.scroll_y
+        self.load_text("…" + self.text[-self.MAX_CHARS // 2:])
+        self.call_after_refresh(lambda: self.scroll_to(y=max(0, self.max_scroll_y - from_end), animate=False))
+
 
 # ------------------------------------------------------------------ answer detail
 def answer_entry(row):
@@ -320,6 +335,7 @@ Opens on [b]Per question[/b]. [b]Production readiness[/b] gives one verdict per
 and what's missing when it's INCONCLUSIVE. The other tabs are the evidence:
   Scorecard              accuracy, critical failures and truncation per model and pack
   Speed & tokens         time and tokens per answer (★ = nothing beats it on both accuracy and time)
+  PTA index              privacy, total time and accuracy per model (0-100 each), as a triangle
   Per question           every question, model by model
   Is the difference real? whether one model is really better than another, or it's noise
   Tests that separate    the questions that tell models apart
@@ -2350,6 +2366,19 @@ class ResultsScreen(Screen):
                              "On another machine, times are measured there if the model ran there, otherwise "
                              "projected from each answer's token counts and that machine's tuned speeds.[/dim]")
                 yield DataTable(id="speed", zebra_stripes=True)
+            with TabPane("PTA index", id="tab-pta"):
+                with Horizontal(id="pta-controls"):
+                    yield Select([("All packs", "")], id="pta-pack", allow_blank=False, value="")
+                    yield Button("Models: all", id="pta-models")
+                yield Static("[dim]P privacy: 100 when prompts stay on machines you control (local, or a server "
+                             "marked private = true), 0 for a hosted API. T time: total time for every question "
+                             "compared, relative to the fastest model (100). A accuracy: answers right. Bigger "
+                             "triangle = better all round; verdicts and critical failures are in Production "
+                             "readiness.[/dim]")
+                yield Static(id="pta-note")
+                with Horizontal(id="pta-body"):
+                    yield Static(id="pta-triangle")
+                    yield DataTable(id="pta", zebra_stripes=True, cursor_type="none")
             with TabPane("Per question", id="tab-per-question"):
                 with Horizontal(id="pq-controls"):
                     yield Select([("All packs", "")], id="pq-pack", allow_blank=False, value="")
@@ -2445,6 +2474,7 @@ class ResultsScreen(Screen):
             info.update("No results yet. Run an eval first." + banner)
             self.rows, self.infos = [], []
             self.fill_per_question()
+            self.fill_pta()
             return
         rows, infos = compare.load(paths)
         notes = compare.settings_notes(infos)
@@ -2486,6 +2516,7 @@ class ResultsScreen(Screen):
             for test, flags, rates in compare.items(rows):
                 qt.add_row(", ".join(flags), test, *[f"{100 * rates[m]:.0f}%" if m in rates else "-" for m in models])
         self.fill_per_question()
+        self.fill_pta()
         self.fill_readiness(results_dir)
 
     @on(DataTable.RowSelected, "#failures")
@@ -2509,6 +2540,7 @@ class ResultsScreen(Screen):
             self.fill_per_question()
 
     @on(Button.Pressed, "#pq-models")
+    @on(Button.Pressed, "#pta-models")
     def pick_models(self):
         counts = {}
         for t in getattr(self, "pq_all_tests", []):
@@ -2526,8 +2558,51 @@ class ResultsScreen(Screen):
                     self.query_one("#pq-shared", Checkbox).value = True
             # after the picker has closed: a table rebuilt under a modal keeps its header-only widths
             self.call_after_refresh(self.fill_per_question)
+            self.call_after_refresh(self.fill_pta)
         self.app.push_screen(ModelPickScreen(models, counts, self.app.pq_models), picked)
 
+    @on(Select.Changed, "#pta-pack")
+    def pta_changed(self):
+        if getattr(self, "rows", None) is not None:
+            self.fill_pta()
+
+    def fill_pta(self):
+        """The PTA index (pta.py) of the picked models (Models button, shared with Per question) on
+        the finished results, over the questions all of them answered."""
+        from . import pta
+        e = self.app.engine
+        rows = [r for r in self.rows if r.get("path") not in self.superseded]
+        packs = sorted({r["suite"] for r in rows})
+        sel = self.query_one("#pta-pack", Select)
+        options = [("All packs", "")] + [(e.packs[p].label if p in e.packs else p, p) for p in packs]
+        if [v for _, v in options] != getattr(self, "pta_pack_values", None):
+            self.pta_pack_values = [v for _, v in options]
+            current = sel.value
+            with sel.prevent(Select.Changed):
+                sel.set_options(options)
+                sel.value = current if current in self.pta_pack_values else ""
+        if sel.value:
+            rows = [r for r in rows if r["suite"] == sel.value]
+        present = {r["model"] for r in rows}
+        chosen = {m for m in self.app.pq_models if m in present}
+        self.query_one("#pta-models", Button).label = f"Models: {len(chosen)} picked" if chosen else "Models: all"
+        models = sorted(chosen or present)
+        t = self.query_one("#pta", DataTable)
+        t.clear(columns=True)
+        if not models:
+            self.query_one("#pta-note", Static).update("No finished results to compare yet.")
+            self.query_one("#pta-triangle", Static).update("")
+            return
+        result = pta.index(rows, lambda m: pta.privacy(e.cfg, m), models)
+        extra = (f"  [yellow]{len(pta.COLORS)} models are drawn at most; pick fewer with Models.[/yellow]"
+                 if len(models) > len(pta.COLORS) else "")
+        self.query_one("#pta-note", Static).update(pta.scope_note(result, models) + extra)
+        self.query_one("#pta-triangle", Static).update(
+            "\n".join(pta.triangle(result, 48) + [""] + pta.legend(result)) if result["questions"] else "")
+        header, table = pta.table(result)
+        t.add_columns(*header)
+        t.add_rows(table)
+
     def fill_per_question(self):
         # a pack being rerun under new questions or settings: its old result file is on its way out,
         # so it isn't compared (it moves to history/ when the new run of that pack finishes)
@@ -2831,6 +2906,11 @@ class EvalsApp(App):
     #results-info { width: 1fr; height: auto; }
     #toggle-smoke { width: auto; min-width: 32; }
     #pq-controls { height: 3; }
+    #pta-controls { height: 3; }
+    #pta-controls Select { width: 28; margin-right: 1; }
+    #pta-body { height: auto; }
+    #pta-triangle { width: auto; padding: 1 2 0 0; }
+    #pta { width: 1fr; height: auto; margin-top: 1; }
     #pq-controls Checkbox { width: auto; }
     #pq-controls Select { width: 28; margin-right: 1; }
     #pq-models { margin-right: 1; min-width: 20; }

{tuieval-0.2.0.dev34 → tuieval-0.2.0.dev42}/tests/test_tuieval.py

@@ -46,10 +46,10 @@ def free_port():
 
 
 class Mock:
-    def __init__(self, packs, mode, port=None):
+    def __init__(self, packs, mode, port=None, delay=0.0):
         self.port = port or free_port()
         self.proc = subprocess.Popen([sys.executable, os.path.join(HERE, "mock_server.py"), "--port", str(self.port),
-                                      "--packs", packs, "--mode", mode], stdout=subprocess.DEVNULL)
+                                      "--packs", packs, "--mode", mode, "--delay", str(delay)], stdout=subprocess.DEVNULL)
         for _ in range(100):
             try:
                 urllib.request.urlopen(f"http://127.0.0.1:{self.port}/v1/models", timeout=1)
@@ -412,6 +412,21 @@ class Parallel(unittest.TestCase):
             self.assertFalse(any("ran_alongside" in r for r in data["results"]))
             self.assertNotIn("loaded_alongside", data["run"])
 
+    def test_regrade_one_model(self):
+        tuieval(self.ws, "run", "--only", "a,b", "--tier", "certify")
+        tests = os.path.join(self.ws, "packs", "apps", "tests.yaml")
+        write(tests, read(tests).replace("expected: 1", "expected: 2"))   # a grader change: every answer now wrong
+        out = tuieval(self.ws, "regrade", "--only", "a", "--packs", "apps").stdout
+        self.assertIn("6 -> 0 passed", out)
+        self.assertNotIn(os.path.join("b", "apps.json"), out)
+        self.assertEqual(sum(r["pass"] for r in self.results("a")["results"]), 0)
+        self.assertEqual(sum(r["pass"] for r in self.results("b")["results"]), 6)   # left as it was
+        for args, name in ((["--only", "a,nope"], "nope"), (["--packs", "nope"], "nope")):
+            p = tuieval(self.ws, "regrade", *args, check=False)
+            self.assertNotEqual(p.returncode, 0)
+            self.assertIn(f"no results for {name}", p.stderr)
+        self.assertEqual(sum(r["pass"] for r in self.results("b")["results"]), 6)   # a typo regrades nothing
+
     def test_side_by_side_on_their_own_ports(self):
         out = tuieval(self.ws, "run", "--only", "a,b", "--tier", "certify", "--parallel", "2").stdout
         self.assertIn("up to 2 models at a time", out)
@@ -676,6 +691,102 @@ class AddToRun(unittest.TestCase):
         self.assertEqual(out["second"], [["a/other", "done"]])                  # queued: its own run after
 
 
+class PTAIndex(unittest.TestCase):
+    """The PTA index: privacy from where a model runs, time relative to the fastest total, accuracy,
+    all over the questions every compared model answered."""
+
+    def test_privacy(self):
+        from tuieval import pta
+        cfg = {"servers": {"llama": {"cmd": ["llama-server"]}, "lm": {"url": "http://localhost:1234/v1"},
+                           "box": {"url": "http://192.168.1.20:8000/v1"}, "mine": {"url": "http://192.168.1.20:8000/v1",
+                                                                                   "private": True},
+                           "or": {"url": "https://openrouter.ai/api/v1"}},
+               "models": [{"label": l, "server": l} for l in ("llama", "lm", "box", "mine", "or")]}
+        self.assertEqual([pta.privacy(cfg, l) for l in ("llama", "lm", "box", "mine", "or", "gone")],
+                         [100, 100, 0, 100, 0, None])
+
+    def test_index_compares_shared_questions(self):
+        from tuieval import pta
+
+        def row(model, test, ok, secs, repeat=0):
+            return {"model": model, "suite": "p", "test": f"p: {test}", "ok": ok, "latency": secs * 1000,
+                    "tokens": 10, "truncated": False, "gen_tps": None, "repeat": repeat}
+        rows = [row("fast", "q1", True, 2), row("fast", "q2", False, 2),
+                row("slow", "q1", True, 8), row("slow", "q2", True, 4), row("slow", "q2", True, 6, repeat=1),
+                row("slow", "q3", True, 100)]               # only slow answered q3: not compared
+        res = pta.index(rows, lambda m: 100 if m == "slow" else 0)
+        self.assertEqual(res["questions"], 2)
+        by = {x["model"]: x for x in res["models"]}
+        self.assertEqual(by["fast"]["total_s"], 4)
+        self.assertEqual(by["slow"]["total_s"], 13)      # q1 8 + q2 median of 4 and 6
+        self.assertEqual(by["fast"]["T"], 100)
+        self.assertAlmostEqual(by["slow"]["T"], 100 * 4 / 13)
+        self.assertEqual((by["fast"]["A"], by["slow"]["A"]), (50, 100))
+        self.assertEqual([x["model"] for x in res["models"]], ["slow", "fast"])   # best accuracy first
+        lines = pta.triangle(res, 40)
+        self.assertIn("P[/b] privacy", lines[0])
+        self.assertTrue(any("[cyan]" in l for l in lines) and any("[magenta]" in l for l in lines))
+
+    def test_cli_report_and_tui(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            d = os.path.join(ws, "packs", "apps")
+            os.makedirs(d)
+            write(os.path.join(d, "pack.toml"), 'label = "apps"\n[certify]\nrepeat = 1\n[gate]\nmin_accuracy = 0.1\n')
+            write(os.path.join(d, "tests.yaml"), "".join(
+                f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+                "difficulty: easy}\n" for t in "abcd"))
+            right = Mock(os.path.join(ws, "packs"), "oracle")
+            wrong = Mock(os.path.join(ws, "packs"), "wrong", delay=0.3)
+            try:
+                with open(os.path.join(ws, "models.toml"), "a") as f:
+                    f.write(f'\n[servers.here]\nurl = "http://127.0.0.1:{right.port}"\n'
+                            f'\n[servers.there]\nurl = "http://127.0.0.1:{wrong.port}"\n')
+                tuieval(ws, "add", "mock", "--server", "here", "--label", "good")
+                tuieval(ws, "add", "mock", "--server", "there", "--label", "hosted")
+                tuieval(ws, "run", "--tier", "certify")
+            finally:
+                right.stop()
+                wrong.stop()
+            path = os.path.join(ws, "models.toml")    # as if "there" were a hosted API
+            write(path, read(path).replace(f"http://127.0.0.1:{wrong.port}", "https://api.example.com/v1"))
+            out = tuieval(ws, "pta").stdout
+            self.assertIn("Compared on the 4 questions all 2 model(s) answered", out)
+            table = {l.split()[0]: l.split()[1:4] for l in out.splitlines() if l.startswith(("good ", "hosted "))}
+            self.assertEqual(table["good"], ["100", "100", "100"])
+            self.assertEqual(table["hosted"][0], "0")
+            self.assertLess(int(table["hosted"][1]), 100)   # slower in total
+            self.assertEqual(table["hosted"][2], "0")
+            self.assertIn("P privacy", out)
+            self.assertIn("4/4", tuieval(ws, "pta", "--only", "good").stdout)
+            report = os.path.join(tmp, "r.md")
+            tuieval(ws, "report", "-o", report)
+            self.assertIn("## PTA index", read(report))
+            code = textwrap.dedent("""
+                import asyncio, json
+                from tuieval.tui import EvalsApp, ResultsScreen
+                from textual.widgets import DataTable, Static
+                async def go():
+                    app = EvalsApp({})
+                    async with app.run_test(size=(180, 60)) as pilot:
+                        await pilot.pause()
+                        app.push_screen(ResultsScreen())
+                        await pilot.pause(0.5)
+                        t = app.screen.query_one("#pta", DataTable)
+                        rows = [[str(c) for c in t.get_row_at(i)] for i in range(t.row_count)]
+                        tri = str(app.screen.query_one("#pta-triangle", Static).render())
+                        print(json.dumps({"rows": rows, "triangle": "P privacy" in tri}))
+                asyncio.run(go())
+            """)
+            p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=300,
+                               env=dict(os.environ, TUIEVAL_HOME=ws, TUIEVAL_DETECT_PORTS=""))
+            got = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+            self.assertIsInstance(got, dict, got)
+            self.assertTrue(got["triangle"])
+            self.assertEqual([r[:4] for r in got["rows"]][0], ["good", "100", "100", "100"])
+
+
 class FirstRun(unittest.TestCase):
     """A new user's first run fails fast with the fix, never hangs or shows a bare Python error."""
 
@@ -752,6 +863,45 @@ class FirstRun(unittest.TestCase):
                              env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""), timeout=60).stdout.strip()
         self.assertEqual(out, "ResultsScreen tab-per-question")
 
+    def test_stream_follows_only_at_the_bottom(self):
+        code = textwrap.dedent("""
+            import asyncio
+            from textual.app import App
+            from tuieval.tui import StreamView
+            class A(App):
+                def compose(self):
+                    yield StreamView()
+            async def go():
+                app = A()
+                async with app.run_test(size=(60, 12)) as pilot:
+                    s = app.query_one(StreamView)
+                    for i in range(100):
+                        s.append(f"line {i}\\n")
+                    await pilot.pause()
+                    s.scroll_to(y=20, animate=False)
+                    await pilot.pause()
+                    for i in range(100):
+                        s.append("more\\n")
+                        await pilot.pause()
+                    held = s.scroll_y
+                    s.scroll_end(animate=False)
+                    await pilot.pause()
+                    s.append("last\\n")
+                    await pilot.pause()
+                    following = s.scroll_y == s.max_scroll_y
+                    s.MAX_CHARS = len(s.text) + 100        # the next text trims the oldest
+                    s.scroll_to(y=s.max_scroll_y - 30, animate=False)
+                    await pilot.pause()
+                    top = s.document.get_line(s.scroll_y)
+                    s.append("x\\n" * 60)
+                    await pilot.pause()
+                    await pilot.pause()
+                    print(held, following, s.text.startswith("…"), s.document.get_line(s.scroll_y) == top)
+            asyncio.run(go())
+        """)
+        out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=60).stdout.strip()
+        self.assertEqual(out, "20 True True True")   # scrolled up stays put, also through a trim; bottom follows
+
     def test_doctor(self):
         tuieval(self.ws, "add", "mock", "--server", "local", "--label", "wanted")
         p = tuieval(self.ws, "doctor", check=False, ports=self.port)