tuieval 0.2.0.dev123__tar.gz → 0.2.0.dev126__tar.gz

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Fix: a request that runs out of time says `time limit: timed out after N s`, not `connection error`, and `tuieval compare` counts it against the model like the run does (it was left out as an error).
+- Agent packs (`kind = "agent"`): each test is a small project with a task, worked on by a command-line agent you define in `[agents]`, using the model tuieval serves, and graded by the task's visible and hidden tests. Results show the agent's stats, diff and transcript; `tuieval compare` adds an Agent tasks table; `tuieval selftest` checks each task's reference and wrong fixes. See docs/agent-packs.md.
 - Answers that used more tokens than the `max_tokens` sent (a provider that didn't stop them) are marked over budget: `(n over budget)` in the PTA index, an **over budget** column, `over` in Per question and a note in the answer view. `[defaults] over_budget = "fail"` fails them instead. Works on stored results.
 - Fix: Results → Per question opened the answers of the model one column to the right of the cell picked, and crashed on the last model's column.
 - Fix: `v` on the run screen switches to the last model still running when the stream shows one that has finished.

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev123
+Version: 0.2.0.dev126
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -111,6 +111,8 @@ packs/support-bot/
 
 Built-in graders: `answer` (a number or word on an `ANSWER:` line, or a correct refusal when the data isn't there), `rag` (grounded answers with citations), `reply` (free-form replies against rules), `tool_call` (the right tool with the right arguments, or rightly none) and `code` (Python run against hidden tests). Your own grader is one Python file in the workspace's `graders/` folder.
 
+**Agent packs** test models inside a coding agent: each test is a small project with a task. An agent you choose works on it, using the model tuieval serves, and the project's tests grade the result. See [docs/agent-packs.md](docs/agent-packs.md).
+
 `tuieval new-pack <name> --grader <grader>` creates a pack with working examples for any grader. **See [docs/writing-packs.md](docs/writing-packs.md)** for every field, gates, critical failures, difficulty labels, custom graders and how to write tests that separate good models from weak ones.
 
 ## The workspace
@@ -186,7 +188,7 @@ tuieval help
 
 ## Safety
 
-⚠️ The `code` grader executes model-written code on your machine. Run code packs inside a container or VM with no credentials in the environment.
+⚠️ The `code` grader executes model-written code on your machine, and an agent pack's agent runs commands as you. Run code and agent packs inside a container or VM with no credentials in the environment.
 
 ## Development
 

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/README.md

@@ -85,6 +85,8 @@ packs/support-bot/
 
 Built-in graders: `answer` (a number or word on an `ANSWER:` line, or a correct refusal when the data isn't there), `rag` (grounded answers with citations), `reply` (free-form replies against rules), `tool_call` (the right tool with the right arguments, or rightly none) and `code` (Python run against hidden tests). Your own grader is one Python file in the workspace's `graders/` folder.
 
+**Agent packs** test models inside a coding agent: each test is a small project with a task. An agent you choose works on it, using the model tuieval serves, and the project's tests grade the result. See [docs/agent-packs.md](docs/agent-packs.md).
+
 `tuieval new-pack <name> --grader <grader>` creates a pack with working examples for any grader. **See [docs/writing-packs.md](docs/writing-packs.md)** for every field, gates, critical failures, difficulty labels, custom graders and how to write tests that separate good models from weak ones.
 
 ## The workspace
@@ -160,7 +162,7 @@ tuieval help
 
 ## Safety
 
-⚠️ The `code` grader executes model-written code on your machine. Run code packs inside a container or VM with no credentials in the environment.
+⚠️ The `code` grader executes model-written code on your machine, and an agent pack's agent runs commands as you. Run code and agent packs inside a container or VM with no credentials in the environment.
 
 ## Development
 

tuieval-0.2.0.dev126/docs/agent-packs.md

@@ -0,0 +1,142 @@
+# Agent packs
+
+An ordinary pack sends one prompt and grades one answer. That ranks models fairly, but it can't show how a model does inside a coding agent: reading files, editing them, running the tests, reacting to failures and deciding when it's done. An **agent pack** tests that.
+
+Each test is a small project with a task. tuieval serves the model as for any other pack, hands a scratch copy of the project to a command-line agent you choose, and grades what the agent leaves behind with tests. Results show up beside your other packs.
+
+## How a task runs
+
+1. **Copy the project.** The task's `repo/` is copied to a scratch folder and committed to git, so the agent's changes can be shown as a diff.
+2. **Run the agent.** Its command runs in that folder. Every line it prints shows live in the run screen.
+3. **Stop it at the time limit.** The agent is stopped at the task's time limit (`minutes`). It keeps to the turn limit (`max_turns`) itself.
+4. **Grade the folder** with pytest. Two sets of tests run:
+   - the task's **original** visible tests. If the agent edits or deletes them, it gains nothing, and the edit is recorded;
+   - its **hidden** tests, which the agent never sees.
+
+   The task passes when every test passes. Its score is the share of hidden tests passed, so a near miss is visible.
+
+Server failures never count against the model. If the model's server stops, or the agent reports that the server failed, the task is asked again. If the server keeps failing, the pack stops, like any pack whose server keeps failing.
+
+## The pack
+
+```
+packs/agentic-coding/
+  pack.toml
+  tasks/
+    fix-fees/
+      task.md          the instruction the agent gets
+      task.toml        optional: description, category, difficulty, minutes, max_turns, visible_tests
+      repo/            the project the agent works on, with its visible tests (tests/ by default)
+      hidden/          tests the agent never sees, copied in only for grading
+      solution/        optional: a reference fix, copied over repo/ (a .delete file lists files to remove)
+      wrong/<name>/    optional: plausible wrong fixes that must fail
+```
+
+```toml
+# pack.toml
+label = "Agentic coding"
+group = "Coding"
+kind = "agent"            # task folders run by an agent
+agent = "my-agent"        # optional: which [agents.<name>]; default: [defaults] agent
+python = "agentic/.venv/bin/python"   # optional: the Python the tasks run with (relative to the workspace)
+minutes = 15              # optional: default time limit per task
+max_turns = 40            # optional: default turn limit per task
+min_context = 32768       # optional: context a model needs for this pack (default 32768)
+
+[certify]
+repeat = 1
+[gate]
+min_accuracy = 0.0
+```
+
+```toml
+# tasks/fix-fees/task.toml
+description = "Fees counted twice"
+category = "bug fix"
+difficulty = "medium"
+minutes = 15
+max_turns = 40
+```
+
+- **Python:** the pack's Python must have pytest installed, plus whatever the tasks import. Without a `python` setting, tuieval's own interpreter is used (`pipx inject tuieval pytest` adds pytest to a pipx install).
+- **Fingerprint:** it covers each task's `task.md`, its limits and every file under `repo/` and `hidden/`. Editing any of them reruns that pack. The description, category and difficulty are labels only.
+- **Selftest:** `tuieval selftest <pack>` checks three things for every task: the untouched `repo/` fails, `solution/` passes, and every `wrong/<name>/` fails. A task that passes untouched, or a wrong fix that passes, can't tell good agents from bad ones.
+
+## The agent
+
+Define the agent in `models.toml`. Its command is any program; `{root}` is your workspace.
+
+```toml
+[agents.my-agent]
+cmd = ["python3", "{root}/agents/my_agent.py"]
+
+[defaults]
+agent = "my-agent"
+```
+
+The agent's name and command are part of an agent pack's settings, so changing the command reruns the pack. The agent's version isn't: each answer records it, and `tuieval compare` marks a pack that ran on more than one version as MIXED.
+
+Most agents need a small **adapter**: a script that reads what tuieval provides, starts the agent with it, and writes the stats file. pi, Aider, Codex CLI, OpenCode and similar agents can all be wrapped this way. The adapter gets everything from its environment:
+
+| Variable | Contents |
+|---|---|
+| `TUIEVAL_BASE_URL` | OpenAI-compatible URL of the model (ends in `/v1`) |
+| `TUIEVAL_MODEL_ID` | the model name to send |
+| `TUIEVAL_API_KEY` | the API key for hosted models; empty for local ones |
+| `TUIEVAL_PROMPT_FILE` | the task's instruction |
+| `TUIEVAL_SAMPLING` | the model's sampling settings as JSON (temperature, top_p, max_tokens, reasoning_effort, …) |
+| `TUIEVAL_CONTEXT` | the model's context size in tokens, if known |
+| `TUIEVAL_MAX_TURNS` | the task's turn limit |
+| `TUIEVAL_TASK` | the task's id |
+| `TUIEVAL_STATS_FILE` | where to write the stats (below) |
+
+- **Working folder:** the scratch copy of the task.
+- **PATH:** starts with the pack's Python, so `python` and `pytest` work.
+- **Exit code:** ignored. Only the tests decide.
+
+**Stats file.** It's optional; without it, tuieval records only the time taken. Any of these keys may be written:
+
+```json
+{"agent": "my-agent 1.4.2", "turns": 14, "input_tokens": 182340, "output_tokens": 9120,
+ "tool_calls": 22, "tool_errors": 1, "ran_tests": true, "stop": "done",
+ "final_message": "Fixed: the ledger subtracted the fee twice."}
+```
+
+- **`tool_errors`:** count failed edits and malformed tool calls, not commands that exit nonzero.
+- **`stop`:** `done`, `turn limit` or `error`.
+- **`server_error`:** if the model's server failed rather than the model, write this key with the error text. The task isn't counted and is asked again.
+
+A minimal adapter:
+
+```python
+#!/usr/bin/env python3
+import json, os, subprocess
+
+prompt = open(os.environ["TUIEVAL_PROMPT_FILE"]).read()
+proc = subprocess.run(["my-agent", "--base-url", os.environ["TUIEVAL_BASE_URL"],
+                       "--model", os.environ["TUIEVAL_MODEL_ID"], "--max-turns", os.environ["TUIEVAL_MAX_TURNS"],
+                       "--print", prompt], capture_output=True, text=True)
+print(proc.stdout)                      # shows in the run screen and the transcript
+with open(os.environ["TUIEVAL_STATS_FILE"], "w") as f:
+    json.dump({"agent": "my-agent", "stop": "done" if proc.returncode == 0 else "error",
+               "final_message": proc.stdout.strip().splitlines()[-1] if proc.stdout.strip() else ""}, f)
+```
+
+For a live view in the run screen, print a line per turn and tool call as they happen rather than at the end.
+
+## Results
+
+Agent tasks appear wherever other answers do: Per question, Failures, the scorecard and the PTA index.
+
+- **Answer view:**
+  - the agent's stats: version, turns, tokens, tool errors, and whether it ran the tests;
+  - its final message and its changes as a diff;
+  - the full transcript, with the test output;
+  - under **What's checked**, the visible and hidden tests.
+- **`tuieval compare`:** adds an **Agent tasks** table per model and pack. It shows tasks passed, median minutes and turns, output tokens, tool errors, how often the tests were run, limits hit, tests edited, and the agent version.
+
+`tuieval regrade` leaves agent results alone: they're graded when they run, so retest the pack instead.
+
+## Safety
+
+⚠️ The agent runs commands as you, with your permissions. It works in a scratch folder, but nothing stops it from reaching outside that folder. Run agent packs in a container or VM with no credentials in the environment. The hidden tests and reference fixes live in your pack folder, so an agent that searched your disk could find them. The transcript would show it.

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/docs/models.md

@@ -210,6 +210,7 @@ request_timeout_ms = 1800000   # per request; long thinking runs need it
 stall_timeout_ms = 300000      # no tokens this long once answering began: the server stalled (retried, not counted)
 over_budget = "flag"           # answers past the max_tokens sent (a provider that didn't stop them): "flag" or "fail"
 # reference = "<label>"        # the model Results → Insights compares against by default
+# agent = "<name>"             # the [agents.<name>] that runs agent packs (see docs/agent-packs.md)
 ready_timeout_s = 1200         # how long a model may take to load before giving up
 model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
 

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/docs/writing-packs.md

@@ -15,6 +15,8 @@ Two shortcuts to a first set of real questions:
 - **From a spreadsheet:** `tuieval new-pack quiz --from questions.csv` makes one test per row from a `question` and an `answer` column (optional: `id`, `category`, `difficulty`, `tolerance`, and `wrong` with known-wrong answers separated by `|`). Numbers are checked with the tolerance, words case-insensitively, and `NOT_AVAILABLE` means the right answer is "the question doesn't say".
 - **Drafted by a model:** `tuieval new-pack support-bot --grader reply --about "refund questions for a shoe store"` also writes `DRAFT-PROMPT.md`. Paste it into the strongest model you have, save the YAML it writes as `tests.yaml`, and run `tuieval selftest`: it catches drafted tests whose expected answer is wrong or whose grader can't tell a mistake from the right answer. Read the questions anyway; you know your domain better than the model.
 
+An **agent pack** (`kind = "agent"`) has task folders instead of questions: a coding agent works on each one, and tests grade the result. See [agent-packs.md](agent-packs.md).
+
 To try a different set of questions, copy a pack (`cp -r packs/support-bot packs/support-bot-v2`), edit the copy, and pick whichever you want in the TUI.
 
 ## Fingerprints

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev123'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev123')
+__version__ = version = '0.2.0.dev126'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev126')
 
 __commit_id__ = commit_id = None

tuieval-0.2.0.dev126/src/tuieval/agents.py

@@ -0,0 +1,353 @@
+"""Agent packs: coding tasks a command-line agent works on, graded by tests.
+
+models.toml says how to start each agent (any program; {root} is the workspace):
+
+    [agents.my-agent]
+    cmd = ["python3", "{root}/agents/my_agent.py"]
+
+    [defaults]
+    agent = "my-agent"      # for agent packs that don't name one (pack.toml agent = "...")
+
+tuieval serves the model as for any pack, then for each task:
+
+  1. copies the task's repo/ to a scratch folder, a git repository, so the agent's changes show as a diff
+  2. runs the agent's command there; every line it prints shows live in the run screen
+  3. stops it at the task's time limit (task.toml minutes); the agent keeps to max_turns itself
+  4. grades the folder with pytest: the task's original visible tests (editing or deleting them gains
+     the agent nothing, and is recorded) plus its hidden tests, which the agent never sees
+
+The agent learns the rest from its environment:
+
+  TUIEVAL_BASE_URL     OpenAI-compatible URL of the model (…/v1)
+  TUIEVAL_MODEL_ID     the model name to send
+  TUIEVAL_API_KEY      the API key for hosted models ("" for local ones)
+  TUIEVAL_PROMPT_FILE  the task's instruction
+  TUIEVAL_SAMPLING     the model's sampling settings as JSON (temperature, max_tokens, reasoning_effort, …)
+  TUIEVAL_CONTEXT      the model's context size (tokens), if known
+  TUIEVAL_MAX_TURNS    the task's turn limit
+  TUIEVAL_STATS_FILE   where it may write its stats as JSON when it finishes (all optional):
+                         agent ("name version"), turns, input_tokens, output_tokens, tool_calls,
+                         tool_errors (failed edits, bad tool calls; not commands that exit nonzero),
+                         ran_tests, stop ("done", "turn limit" or "error"), final_message, and
+                         server_error: the model's server failed, not the model (the task isn't counted
+                         and is asked again)
+
+PATH starts with the pack's Python (pack.toml python, else tuieval's), so `python` runs the tasks'
+code and tests. The agent's exit code decides nothing; the tests do.
+
+The agent runs as you, with your permissions, in the scratch folder: there is no sandbox.
+"""
+import filecmp
+import json
+import os
+import queue
+import re
+import shutil
+import signal
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import xml.etree.ElementTree as ET
+
+PROMPT_TAIL = "\n\nYou're working in a project in the current directory. Run Python with `python`.\n"
+GRADE_TIMEOUT_S = 300
+DIFF_KEEP = 20_000      # characters of the diff kept in the result (the whole diff is in the log)
+HIDDEN_RE = re.compile(r"\.?_hidden\b")   # junit names of the hidden tests (a collection error starts with ".")
+STATS_KEYS = ("agent", "turns", "input_tokens", "output_tokens", "tool_calls", "tool_errors", "ran_tests", "stop",
+              "final_message", "server_error")
+
+
+class AgentError(Exception):
+    pass
+
+
+class ServerGone(Exception):
+    """The model's server stopped while the agent worked: not the agent's fault."""
+
+
+def agent_for(cfg, pack):
+    """(name, [agents.<name>] table or None) for an agent pack."""
+    name = pack.agent or cfg.get("defaults", {}).get("agent", "")
+    return name, cfg.get("agents", {}).get(name) if name else None
+
+
+def identity(cfg, pack):
+    """What a result's settings record about the agent: its name and command (a different command
+    makes earlier results outdated; a new version of the same agent doesn't, it's shown per answer)."""
+    name, a = agent_for(cfg, pack)
+    return {"name": name, "cmd": list((a or {}).get("cmd", []))}
+
+
+def python_for(pack, root):
+    """The Python an agent pack's tasks and tests run with."""
+    if not pack.python:
+        return sys.executable
+    p = os.path.expanduser(pack.python)
+    return p if os.path.isabs(p) else os.path.join(root, p)
+
+
+_PYTEST = {}
+
+
+def problem(cfg, pack, root):
+    """Why an agent pack can't run here (a note for a skipped job), or None."""
+    name, a = agent_for(cfg, pack)
+    if not name:
+        return 'no agent: add [agents.<name>] to models.toml and [defaults] agent = "<name>"'
+    if not a or not a.get("cmd"):
+        return f"agent {name!r} isn't defined: add [agents.{name}] with a cmd to models.toml"
+    py = python_for(pack, root)
+    if py not in _PYTEST:
+        try:
+            _PYTEST[py] = subprocess.run([py, "-c", "import pytest"], capture_output=True, timeout=60).returncode == 0
+        except (OSError, subprocess.TimeoutExpired):
+            _PYTEST[py] = False
+    if not _PYTEST[py]:
+        return f"pytest isn't installed in {py} (pack.toml python), which grades the tasks"
+    return None
+
+
+def env_for(python, extra):
+    """The environment an agent and its tests run in: the pack's Python first on PATH."""
+    env = {k: v for k, v in os.environ.items() if not k.startswith(("PYTHON", "VIRTUAL_ENV", "CONDA_PREFIX"))}
+    env["PATH"] = os.path.dirname(python) + os.pathsep + env.get("PATH", "/usr/bin:/bin")
+    venv = os.path.dirname(os.path.dirname(python))
+    if os.path.isfile(os.path.join(venv, "pyvenv.cfg")):
+        env["VIRTUAL_ENV"] = venv
+    env["MPLBACKEND"] = "Agg"
+    env.update(extra)
+    return env
+
+
+def _git(work, *args):
+    try:
+        return subprocess.run(["git", "-c", "user.name=tuieval", "-c", "user.email=tuieval@localhost",
+                               "-c", "commit.gpgsign=false", *args], cwd=work, capture_output=True, text=True,
+                              timeout=60)
+    except (OSError, subprocess.TimeoutExpired):
+        return None
+
+
+def prepare(test, parent):
+    """Copy a task's repo/ into parent/work as a fresh git repository; returns the folder."""
+    work = os.path.join(parent, "work")
+    shutil.copytree(os.path.join(test["task_dir"], "repo"), work,
+                    ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache", ".git"))
+    if _git(work, "init", "-q") is not None:
+        with open(os.path.join(work, ".git", "info", "exclude"), "a") as f:
+            f.write("__pycache__/\n.pytest_cache/\n")
+        _git(work, "add", "-A")
+        _git(work, "commit", "-qm", "task")
+    return work
+
+
+def diff(work):
+    """The agent's changes as a patch (empty without git)."""
+    if not os.path.isdir(os.path.join(work, ".git")):
+        return ""
+    _git(work, "add", "-A")
+    out = _git(work, "diff", "--cached", "--stat", "-p")
+    return out.stdout if out and out.returncode == 0 else ""
+
+
+def _kill(proc):
+    for sig in (signal.SIGTERM, signal.SIGKILL):
+        try:
+            os.killpg(proc.pid, sig)
+        except (ProcessLookupError, PermissionError):
+            return
+        try:
+            proc.wait(timeout=5)
+            return
+        except subprocess.TimeoutExpired:
+            continue
+
+
+def run(cmd, work, env, limit_s, on_line, check, alive, log):
+    """Run the agent in work until it exits or limit_s passes. Every output line goes to on_line and
+    the log file. check() may raise (a cancel): the agent is stopped first. Returns (stop, seconds,
+    exit code); stop is "exited" or "time limit". Raises ServerGone when alive() turns False."""
+    t0 = time.monotonic()
+    proc = subprocess.Popen(cmd, cwd=work, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT, start_new_session=True)
+    lines = queue.Queue()
+
+    def read():
+        for raw in iter(proc.stdout.readline, b""):
+            lines.put(raw.decode("utf-8", "replace"))
+        lines.put(None)
+    threading.Thread(target=read, daemon=True).start()
+    stop = "exited"
+    try:
+        while True:
+            check()
+            if not alive():
+                raise ServerGone("the model's server stopped while the agent worked")
+            if time.monotonic() - t0 > limit_s:
+                stop = "time limit"
+                break
+            try:
+                line = lines.get(timeout=0.5)
+            except queue.Empty:
+                continue
+            if line is None:
+                break
+            log.write(line)
+            on_line(line)
+    finally:
+        if proc.poll() is None:
+            _kill(proc)
+        else:   # exited: its own children may still run
+            try:
+                os.killpg(proc.pid, signal.SIGKILL)
+            except (ProcessLookupError, PermissionError):
+                pass
+    return stop, round(time.monotonic() - t0, 1), proc.wait()
+
+
+def read_stats(path):
+    """The stats the agent wrote (only the known keys), or {}."""
+    try:
+        with open(path) as f:
+            data = json.load(f)
+    except (OSError, ValueError):
+        return {}
+    return {k: data[k] for k in STATS_KEYS if k in data} if isinstance(data, dict) else {}
+
+
+def tampered(test, work):
+    """Visible test files the agent changed or removed."""
+    src = os.path.join(test["task_dir"], "repo", test["visible_tests"])
+    out = []
+    for dirpath, _, files in os.walk(src):
+        for fn in sorted(files):
+            if fn.endswith(".py"):
+                a = os.path.join(dirpath, fn)
+                rel = os.path.relpath(a, os.path.join(test["task_dir"], "repo"))
+                b = os.path.join(work, rel)
+                if not os.path.isfile(b) or not filecmp.cmp(a, b, shallow=False):
+                    out.append(rel)
+    return sorted(out)
+
+
+def _cases(xml_path):
+    """{test name: "pass" | "fail" | "skip"} from a pytest junit file."""
+    try:
+        root = ET.parse(xml_path).getroot()
+    except (OSError, ET.ParseError):
+        return {}
+    out = {}
+    for case in root.iter("testcase"):
+        name = f'{case.get("classname")}.{case.get("name")}'
+        bad = case.find("failure") is not None or case.find("error") is not None
+        out[name] = "skip" if case.find("skipped") is not None else "fail" if bad else "pass"
+    return out
+
+
+def grade(test, work, python):
+    """Run the task's original visible tests and its hidden tests on a copy of work.
+    Returns {pass, score, visible, hidden, failed, output}; score is the share of hidden tests passed."""
+    with tempfile.TemporaryDirectory(prefix="tuieval-grade-") as tmp:
+        g = os.path.join(tmp, "work")
+        shutil.copytree(work, g, ignore=shutil.ignore_patterns(".git", "__pycache__", ".pytest_cache"))
+        vis_src = os.path.join(test["task_dir"], "repo", test["visible_tests"])
+        vis = os.path.join(g, test["visible_tests"])
+        has_visible = os.path.isdir(vis_src)
+        if os.path.isdir(vis):
+            shutil.rmtree(vis)
+        if has_visible:
+            shutil.copytree(vis_src, vis)
+        shutil.copytree(os.path.join(test["task_dir"], "hidden"), os.path.join(g, "_hidden"))
+        xml = os.path.join(tmp, "junit.xml")
+        cmd = [python, "-m", "pytest", "-q", "-p", "no:cacheprovider", "-o", "addopts=",
+               "--continue-on-collection-errors", f"--junitxml={xml}",
+               *([test["visible_tests"]] if has_visible else []), "_hidden"]
+        try:
+            p = subprocess.run(cmd, cwd=g, env=env_for(python, {}), capture_output=True, text=True,
+                               timeout=GRADE_TIMEOUT_S)
+        except subprocess.TimeoutExpired:
+            return {"pass": False, "score": 0.0, "visible": "0/0", "hidden": "0/0", "failed": [],
+                    "output": f"the tests ran over {GRADE_TIMEOUT_S} s"}
+        cases = _cases(xml)
+    hidden = {k: v for k, v in cases.items() if HIDDEN_RE.match(k)}
+    visible = {k: v for k, v in cases.items() if not HIDDEN_RE.match(k)}
+    failed = [k for k, v in cases.items() if v == "fail"]
+    passed_hidden = sum(v == "pass" for v in hidden.values())
+
+    def frac(c):
+        return f'{sum(v == "pass" for v in c.values())}/{len(c)}'
+    # a module that no longer imports shows as a collection error: a nonzero exit, never a pass
+    ok = bool(hidden) and not failed and p.returncode == 0
+    return {"pass": ok, "score": round(passed_hidden / len(hidden), 4) if hidden else 0.0,
+            "visible": frac(visible), "hidden": frac(hidden), "failed": failed,
+            "output": (p.stdout[-6000:] + p.stderr[-2000:]).strip()}
+
+
+def reason(g, stop, edited, exit_code, stats):
+    """One line about a graded task, for the answer's reason."""
+    parts = [f"hidden tests {g['hidden']}, visible {g['visible']}"]
+    if g["failed"]:
+        names = [re.sub(r"^\.?(_hidden|tests)\.", "", f) for f in g["failed"]]
+        parts.append("failed: " + ", ".join(names[:6]) + (f" and {len(names) - 6} more" if len(names) > 6 else ""))
+    elif not g["pass"] and g["output"]:
+        parts.append(g["output"].splitlines()[-1][:200])
+    if stop == "time limit":
+        parts.append("stopped at the time limit")
+    elif stats.get("stop") == "turn limit":
+        parts.append("stopped at the turn limit")
+    elif stats.get("stop") == "error":
+        parts.append("the agent stopped with an error")
+    elif exit_code not in (0, None) and not stats:
+        parts.append(f"the agent exited with code {exit_code}")
+    if edited:
+        parts.append("edited the tests: " + ", ".join(edited))
+    return "; ".join(parts)
+
+
+def check_task(test, python):
+    """Selftest errors for one task: the untouched repo must fail, solution/ must pass, each
+    wrong/<name>/ must fail. (A task without solution/ is only checked to fail untouched.)"""
+    errors = []
+    with tempfile.TemporaryDirectory(prefix="tuieval-selftest-") as tmp:
+        def variant(name, overlay_dir=None):
+            work = os.path.join(tmp, name)
+            shutil.copytree(os.path.join(test["task_dir"], "repo"), work,
+                            ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache", ".git"))
+            if overlay_dir:
+                overlay(overlay_dir, work)
+            return grade(test, work, python)
+        if variant("base")["pass"]:
+            errors.append(f"{test['id']}: the untouched repo already passes its tests")
+        sol = os.path.join(test["task_dir"], "solution")
+        if os.path.isdir(sol):
+            g = variant("solution", sol)
+            if not g["pass"]:
+                errors.append(f"{test['id']}: solution/ fails: {g['failed'][:4] or g['output'][-300:]}")
+        wrong = os.path.join(test["task_dir"], "wrong")
+        for w in sorted(os.listdir(wrong)) if os.path.isdir(wrong) else []:
+            if os.path.isdir(os.path.join(wrong, w)) and variant(f"wrong-{w}", os.path.join(wrong, w))["pass"]:
+                errors.append(f"{test['id']}: wrong/{w} passes, so the tests can't catch that mistake")
+    return errors
+
+
+def overlay(src, dst):
+    """Copy src's files over dst; a .delete file in src lists paths (one per line) to remove from dst."""
+    for dirpath, dirnames, files in os.walk(src):
+        dirnames[:] = [d for d in dirnames if d != "__pycache__"]
+        for fn in files:
+            if fn == ".delete":
+                continue
+            rel = os.path.relpath(os.path.join(dirpath, fn), src)
+            os.makedirs(os.path.dirname(os.path.join(dst, rel)), exist_ok=True)
+            shutil.copy2(os.path.join(dirpath, fn), os.path.join(dst, rel))
+    listing = os.path.join(src, ".delete")
+    if os.path.isfile(listing):
+        with open(listing) as f:
+            for line in f.read().split():
+                target = os.path.join(dst, line)
+                if os.path.isdir(target):
+                    shutil.rmtree(target)
+                elif os.path.exists(target):
+                    os.remove(target)

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/src/tuieval/client.py

@@ -153,7 +153,7 @@ def stream_chat(base_url, body, on_delta=None, timeout=1800, stream=None, header
         for raw in r:
             now = time.time()
             if now - start > timeout:
-                out["error"] = f"connection error: timed out after {timeout:.0f} s in all"
+                out["error"] = f"time limit: timed out after {timeout:.0f} s in all"
                 break
             if stall_s and out["ttft_s"] is not None and now - last > stall_s:
                 out["error"] = _stalled(stall_s)

{tuieval-0.2.0.dev123 → tuieval-0.2.0.dev126}/src/tuieval/compare.py

@@ -49,7 +49,8 @@ def _load_native(data):
             "model": m["label"], "suite": pack["name"], "test": f"{pack['name']}: {r['test']}",
             "name": f"{pack['name']}: {r['description']}",
             "ok": bool(r["pass"]), "score": float(r["score"]), "reason": r["reason"],
-            "error": r["reason"].startswith(("server returned", "connection error")),
+            # a timeout is the model too slow (client.is_server_error), not an error; older results called it one
+            "error": r["reason"].startswith(("server returned", "connection error")) and "timed out" not in r["reason"],
             "truncated": r.get("finish") == "length", "output": r.get("answer") or "",
             # tokens past the max_tokens sent, which the provider didn't stop (client.mark_over_budget)
             "over_budget": r.get("over_budget"),
@@ -63,6 +64,7 @@ def _load_native(data):
             "machine": r.get("machine") or run.get("machine"), "prompt_tokens": r.get("prompt_tokens"),
             # other models served at the same time (parallel_models): its speed was shared with them
             "alongside": r.get("ran_alongside") or [],
+            "agent": r.get("agent"),   # an agent pack's task: the agent's stats (see agents.py)
         })
     return rows, info
 
@@ -573,6 +575,34 @@ def items(rows):
     return out
 
 
+def agent_table(rows):
+    """(header, table) summing up agent-pack tasks per model and pack, or None without any. A pack run
+    by more than one agent version is marked, since a new version can change how models do."""
+    groups = collections.defaultdict(list)
+    for r in rows:
+        if r.get("agent"):
+            groups[(r["model"], r["suite"])].append(r)
+    if not groups:
+        return None
+    header = ["model", "pack", "passed", "med min", "med turns", "out tokens", "tool errors", "ran tests",
+              "limits hit", "tests edited", "agent"]
+    table = []
+    for (model, suite), rs in sorted(groups.items()):
+        a = [r["agent"] for r in rs]
+        turns = [x["turns"] for x in a if x.get("turns") is not None]
+        out_tokens = sum(x.get("output_tokens") or 0 for x in a)
+        versions = sorted({x.get("version") or "?" for x in a})
+        table.append([model, suite, f"{sum(r['ok'] for r in rs)}/{len(rs)}",
+                      f"{median([r['latency'] / 60000 for r in rs]):.1f}", f"{median(turns):.0f}" if turns else "-",
+                      f"{out_tokens:,}" if out_tokens else "-",
+                      str(sum(x.get("tool_errors") or 0 for x in a)),
+                      f"{sum(bool(x.get('ran_tests')) for x in a)}/{len(a)}",
+                      str(sum(x.get("stop") in ("time limit", "turn limit") for x in a)),
+                      str(sum(bool(x.get("tests_edited")) for x in a)),
+                      " + ".join(versions) + (" (MIXED)" if len(versions) > 1 else "")])
+    return header, table
+
+
 def print_table(header, table):
     widths = [max(len(str(x)) for x in col) for col in zip(header, *table)]
     fmt = "  ".join("{:<%d}" % w for w in widths)
@@ -629,6 +659,10 @@ def main(argv):
     if any(r.get("difficulty") in LEVELS for r in rows):
         print("\nPass rate by difficulty:")
         print_table(header, table)
+    agents_summary = agent_table(rows)
+    if agents_summary:
+        print("\nAgent tasks (agent packs; MIXED = answers from more than one agent version):")
+        print_table(*agents_summary)
     machine = argv[argv.index("--machine") + 1] if "--machine" in argv else None
     if "--speed" in flags or machine:
         speeds = {}