tuieval 0.2.0.dev10__tar.gz → 0.2.0.dev14__tar.gz

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Run only some tests of a pack: `e` in the TUI, or `tuieval run --tests`. Runs add up; `--force` reruns just the picked tests. Presets keep the pick.
 - `tuieval tune` scores options on projected full-length answers (`[tune] answer_tokens`, default 1024), so generation speed counts as in real runs.
 - `tuieval tune` reports an option's memory cost instead of rejecting it; it rejects only on critical memory pressure or over `[tune] swap_limit_mb`.
 - Warm-start tuning recognises fine-tunes whose GGUF describes the MTP draft layer differently (e.g. listing KV heads per layer) as the same model shape, so they start from an already-tuned sibling instead of tuning in full.

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev10
+Version: 0.2.0.dev14
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -32,7 +32,7 @@ Run the same eval packs against local models (llama.cpp GGUFs, LM Studio, Ollama
 
 tuieval measures **accuracy, speed and token use** together, and gives a **PASS / FAIL / INCONCLUSIVE** verdict per use case. Grading is automatic; there's no LLM judge.
 
-Public benchmarks leak into training data and You know your workload better than anyone. tuieval ships with **no built-in benchmark**.
+You know your workload better than anyone. tuieval ships with **no built-in benchmark**.
 Instead you build *eval packs* from what you actually do: questions with checkable answers, in your domain, with your rules
 Then you get an answer to the questions that matter:
 
@@ -125,7 +125,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 ## The TUI
 
-1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results).
+1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
@@ -137,6 +137,7 @@ Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scan
 ```bash
 tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
+tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/README.md

@@ -6,7 +6,7 @@ Run the same eval packs against local models (llama.cpp GGUFs, LM Studio, Ollama
 
 tuieval measures **accuracy, speed and token use** together, and gives a **PASS / FAIL / INCONCLUSIVE** verdict per use case. Grading is automatic; there's no LLM judge.
 
-Public benchmarks leak into training data and You know your workload better than anyone. tuieval ships with **no built-in benchmark**.
+You know your workload better than anyone. tuieval ships with **no built-in benchmark**.
 Instead you build *eval packs* from what you actually do: questions with checkable answers, in your domain, with your rules
 Then you get an answer to the questions that matter:
 
@@ -99,7 +99,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 ## The TUI
 
-1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results).
+1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
@@ -111,6 +111,7 @@ Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scan
 ```bash
 tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
+tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev10'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev10')
+__version__ = version = '0.2.0.dev14'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev14')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/src/tuieval/engine.py

@@ -38,6 +38,7 @@ import mimetypes
 import os
 import random
 import re
+import shutil
 import signal
 import socket
 import statistics
@@ -358,6 +359,7 @@ def load_presets(path):
         tags = ["moe"]            # models with any of these tags …
         models = ["foo", "bar"]   # … and/or these labels (omit both for every model)
         repeat = 3
+        tests = { coding = ["parse-dates"] }   # optional: only these tests of a pack
     """
     if not os.path.isfile(path):
         return {}
@@ -366,19 +368,24 @@ def load_presets(path):
 
 
 def resolve_preset(cfg, preset, pack_names):
-    """(labels, packs, repeat) for a preset; unknown names are dropped."""
+    """(labels, packs, repeat, tests) for a preset; unknown names are dropped. tests: {pack: [test
+    ids]} for packs where only some tests run."""
     labels = [m["label"] for m in cfg["models"]]
     if preset.get("models") or preset.get("tags"):
         want, tags = set(preset.get("models", [])), set(preset.get("tags", []))
         labels = [m["label"] for m in cfg["models"] if m["label"] in want or tags & set(m["tags"])]
     packs = [p for p in preset.get("packs", pack_names) if p in pack_names]
-    return labels, packs, preset.get("repeat")
+    tests = {p: list(ids) for p, ids in (preset.get("tests") or {}).items() if p in packs and ids}
+    return labels, packs, preset.get("repeat"), tests
 
 
-def save_preset(path, name, labels, packs, repeat):
+def save_preset(path, name, labels, packs, repeat, tests=None):
     """Add or replace one preset, keeping the rest of the file as written."""
+    tests = {p: ids for p, ids in (tests or {}).items() if p in packs and ids}
     block = (f"[{name}]\nmodels = [{', '.join(json.dumps(l) for l in labels)}]\n"
-             f"packs = [{', '.join(json.dumps(p) for p in packs)}]\n" + (f"repeat = {int(repeat)}\n" if repeat else ""))
+             f"packs = [{', '.join(json.dumps(p) for p in packs)}]\n" + (f"repeat = {int(repeat)}\n" if repeat else "")
+             + ("tests = { " + ", ".join(f"{json.dumps(p)} = [{', '.join(json.dumps(i) for i in ids)}]"
+                                         for p, ids in tests.items()) + " }\n" if tests else ""))
     text = open(path).read() if os.path.isfile(path) else \
         "# Saved selections for the TUI (p) and tuieval run --preset <name>. See engine.load_presets.\n"
     pattern = re.compile(rf"^\[{re.escape(name)}\]\n(?:(?!\[).*\n?)*", re.MULTILINE)
@@ -394,12 +401,13 @@ class Job:
     model: dict
     pack: packs_mod.Pack
     repeat: int
-    tests: list             # the tests this tier runs
+    tests: list             # the tests this tier runs (or the ones picked)
     out_path: str
     sampling: dict
     tier: str = "certify"
     settings: dict = dataclasses.field(default_factory=dict)  # sampling + serving identity (fingerprinted)
     merge: bool = True      # keep matching earlier results and only run what's missing
+    redo: bool = False      # rerun the picked tests, keeping the pack's other answers
     status: str = "waiting"  # waiting, loading, running, done, failed, skipped
     note: str = ""
     discards: list = dataclasses.field(default_factory=list)  # [(what, answers, why)] set aside on start
@@ -868,7 +876,7 @@ class Engine:
         if job.merge:
             data = self._result(job.out_path)
             if data and data["pack"]["fingerprint"] == job.pack.fingerprint and settings_match(data, job.settings):
-                kept = data["results"]
+                kept = self._kept(job, data["results"])
                 if kept:
                     earlier = self._file_sittings(data["run"])
         head, new = self._partial_lines(job.out_path + ".partial.jsonl")
@@ -877,6 +885,14 @@ class Engine:
         saved = self._saved_sittings(job)
         return kept, new, earlier + saved + self._rebuild_sittings(new, saved + earlier)
 
+    @staticmethod
+    def _kept(job, rows):
+        """A finished file's rows that stay: all of them, less the picked tests' when redoing them."""
+        if not job.redo:
+            return rows
+        redo = {t["id"] for t in job.tests}
+        return [r for r in rows if r["test"] not in redo]
+
     def _fill_progress(self, job):
         """Set a job's done/passed/failed and earlier sittings from what's stored."""
         kept, new, job.earlier = self._stored(job)
@@ -927,11 +943,13 @@ class Engine:
         return not current and header_matches(head, pack.fingerprint, settings)
 
     # ---- planning
-    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False):
+    def plan(self, labels, pack_names, repeat=None, tier="screen", force=False, tests=None):
         """Jobs for models x packs. tier: smoke (3 tests), screen (a spread sample), certify (all
         tests). repeat: times each question is asked; None = the tier default (1 for smoke and
-        screen, the pack's certification repeats for certify). Earlier matching results are kept
-        and only missing answers are run, unless force."""
+        screen, the pack's certification repeats for certify). tests: {pack: [test ids]} runs just
+        those tests of a pack instead of the tier's sample (unknown ids raise PackError). Earlier
+        matching results are kept and only missing answers are run, unless force; force with
+        picked tests reruns those and keeps the pack's other answers."""
         if tier not in TIERS:
             raise ValueError(f"tier must be one of {TIERS}")
         jobs = []
@@ -940,8 +958,10 @@ class Engine:
             for name in pack_names:
                 pack = self.packs[name]
                 reps = repeat or self.default_repeat(pack, tier)
-                job = Job(m, pack, reps, pack.select(tier), self.result_path(label, name, tier),
-                          effective_sampling(self.cfg, m), tier=tier, merge=not force, settings=self.settings(m))
+                picked = (tests or {}).get(name)
+                job = Job(m, pack, reps, pack.pick(picked) if picked else pack.select(tier),
+                          self.result_path(label, name, tier), effective_sampling(self.cfg, m), tier=tier,
+                          merge=not force or bool(picked), redo=force and bool(picked), settings=self.settings(m))
                 sv = self.serving(m)
                 status = self.result_status(label, name, tier)
                 job.discards = self._discards(job)
@@ -958,7 +978,7 @@ class Engine:
                 elif missing:
                     job.status, job.note = "skipped", f"{', '.join(missing)} not installed in tuieval's Python"
                 elif status and status.startswith("outdated"):
-                    job.merge, job.note = False, status.split(": ", 1)[-1] + ", rerunning"
+                    job.merge, job.redo, job.note = False, False, status.split(": ", 1)[-1] + ", rerunning"
                     if not force:   # a rerun under the new settings already under way resumes
                         self._fill_progress(job)
                         if job.done:
@@ -1656,7 +1676,7 @@ class Engine:
             data = self._result(job.out_path)
             if data and data["pack"]["fingerprint"] == job.pack.fingerprint and \
                     settings_match(data, job.settings):
-                kept = data["results"]
+                kept = self._kept(job, data["results"])
                 if kept:
                     job.earlier = self._file_sittings(data["run"])
         new = []
@@ -1789,18 +1809,19 @@ class Engine:
         job.note = f"{job.passed}/{job.done} passed"
         self.emit("job_done", job=job)
 
-    def _archive(self, job):
-        """Keep a result file that is about to be replaced (rerun or outdated) in history/."""
+    def _archive(self, job, keep=False):
+        """Keep a result file that is about to be replaced (rerun or outdated) in history/; keep:
+        copy it (some of its answers stay in the new file)."""
         if job.tier == "smoke" or not os.path.isfile(job.out_path):
             return
         hist = os.path.join(os.path.dirname(job.out_path), "history")
         os.makedirs(hist, exist_ok=True)
         stamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(os.path.getmtime(job.out_path)))
-        os.replace(job.out_path, os.path.join(hist, f"{job.pack.name}-{stamp}.json"))
+        (shutil.copy2 if keep else os.replace)(job.out_path, os.path.join(hist, f"{job.pack.name}-{stamp}.json"))
 
     def _write_result(self, job, records, run_info):
-        if not job.merge:
-            self._archive(job)
+        if not job.merge or job.redo:
+            self._archive(job, keep=job.redo)
         m = job.model
         data = {
             "format": RESULT_FORMAT,

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/src/tuieval/packs.py

@@ -55,6 +55,15 @@ class Pack:
         """Python modules the pack needs (the `needs` entries that aren't capabilities)."""
         return [n for n in self.needs if n not in CAPABILITIES]
 
+    def pick(self, ids):
+        """The tests with these ids, in the pack's order. Unknown ids raise PackError."""
+        known = {t["id"] for t in self.tests}
+        bad = [i for i in ids if i not in known]
+        if bad:
+            raise PackError(f"pack {self.name!r} has no test {', '.join(map(repr, bad))}; "
+                            f"its tests are: {', '.join(t['id'] for t in self.tests)}")
+        return [t for t in self.tests if t["id"] in set(ids)]
+
     def select(self, tier):
         """The tests a tier runs: smoke = first 3; screen = `screen` tests spread evenly over
         categories (one variant per group); certify = all."""

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/src/tuieval/run_evals.py

@@ -10,6 +10,7 @@
     tuieval capture logs/live/<file> --pack X     # turn a real failure into a new test
     tuieval run --only my-model --packs my-pack,other-pack
     tuieval run --tags moe --packs my-pack        # models tagged "moe"
+    tuieval run --packs my-pack --tests a,b       # only these tests of a pack (or --tests my-pack:a,other:b)
     tuieval run --preset nightly                  # a selection saved in the TUI (presets.toml)
     tuieval run --tier smoke --only my-model      # 3 tests per pack, 1 repeat: check paths and flags
     tuieval run --tier smoke --only openrouter:qwen/qwen3-32b   # any OpenRouter model, no models.toml edit
@@ -36,6 +37,7 @@ import sys
 import threading
 
 from . import engine
+from . import packs as packs_mod
 from . import workspace
 
 BOLD, DIM, RED, GREEN, CYAN, YELLOW, RESET = "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[36m", "\033[33m", "\033[0m"
@@ -605,6 +607,27 @@ COMMANDS = {"add": cmd_add, "list": cmd_list, "scan": cmd_scan, "regrade": cmd_r
             "export": cmd_export, "remove": cmd_remove}
 
 
+def parse_tests(text, pack_names, known):
+    """({pack: [test ids]}, packs to run) from --tests. A bare id needs exactly one pack chosen;
+    pack:id names its pack, and without --packs the packs named are the ones that run."""
+    tests = {}
+    for item in filter(None, (x.strip() for x in text.split(","))):
+        pack, _, tid = item.rpartition(":")
+        if not pack:
+            if not pack_names or len(pack_names) != 1:
+                sys.exit(f"--tests {item}: say which pack, as pack:{item} (or choose one pack with --packs)")
+            pack = pack_names[0]
+        if pack not in known:
+            sys.exit(f"--tests {item}: unknown pack {pack!r}; packs are {', '.join(known)}")
+        tests.setdefault(pack, []).append(tid)
+    if pack_names is None:
+        pack_names = list(tests)
+    missing = [p for p in tests if p not in pack_names]
+    if missing:
+        sys.exit(f"--tests names pack(s) {missing} that aren't in --packs")
+    return tests, pack_names
+
+
 def main(argv=None):
     """tuieval run [options] (argv without the "run")."""
     argv = sys.argv[1:] if argv is None else argv
@@ -615,7 +638,9 @@ def main(argv=None):
     p.add_argument("--tags", help="comma-separated tags; models having any of them")
     p.add_argument("--preset", help="a saved selection from presets.toml")
     p.add_argument("--packs", help="comma-separated packs (default: all)")
-    p.add_argument("--force", action="store_true", help="rerun packs that already have current results")
+    p.add_argument("--tests", help="comma-separated test ids to run instead of the tier's sample: id with one "
+                                   "pack in --packs, else pack:id (and --packs defaults to those packs)")
+    p.add_argument("--force", action="store_true", help="rerun packs (or the --tests) that already have current results")
     p.add_argument("--repeat", type=int, help="times each question is asked, any tier (default: 1 for smoke and "
                                                   "screen, the pack's certification count for certify)")
     p.add_argument("--tier", choices=engine.TIERS, default="screen",
@@ -642,7 +667,7 @@ def main(argv=None):
         labels = [l for l in labels if want & set(e.model(l)["tags"])]
         if not labels:
             sys.exit(f"no models tagged {sorted(want)}")
-    pack_names = list(e.packs)
+    pack_names, tests = list(e.packs), {}
     if not pack_names:
         sys.exit("no packs in this workspace yet: tuieval new-pack <name> creates one (see docs/writing-packs.md)")
     if args.preset:
@@ -650,14 +675,19 @@ def main(argv=None):
                                                    "presets.toml"))
         if args.preset not in presets:
             sys.exit(f"unknown preset {args.preset!r}; presets: {', '.join(presets) or 'none (save one in the TUI with p)'}")
-        labels, pack_names, preset_repeat = engine.resolve_preset(e.cfg, presets[args.preset], list(e.packs))
+        labels, pack_names, preset_repeat, tests = engine.resolve_preset(e.cfg, presets[args.preset], list(e.packs))
         args.repeat = args.repeat or preset_repeat
     if args.packs:
         pack_names = [x.strip() for x in args.packs.split(",")]
         bad = [x for x in pack_names if x not in e.packs]
         if bad:
             sys.exit(f"unknown pack(s) {bad}; packs are {', '.join(e.packs)}")
-    jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force)
+    if args.tests:
+        tests, pack_names = parse_tests(args.tests, pack_names if (args.packs or args.preset) else None, e.packs)
+    try:
+        jobs = e.plan(labels, pack_names, args.repeat, args.tier, args.force, tests)
+    except packs_mod.PackError as err:
+        sys.exit(str(err))
 
     if args.dry_run:
         seen = set()

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/src/tuieval/tui.py

@@ -21,6 +21,7 @@ import argparse
 import glob
 import json
 import os
+import re
 import threading
 import time
 
@@ -398,6 +399,7 @@ class SetupScreen(Screen):
         Binding("a", "add_model", "Add model"),
         Binding("m", "scan", "Scan for models"),
         Binding("p", "presets", "Presets"),
+        Binding("e", "pick_tests", "Pick tests"),
         Binding("t", "tune", "Tune speed"),
         Binding("r", "results", "Results"),
         Binding("x", "hide_model", "Hide/unhide"),
@@ -426,7 +428,7 @@ class SetupScreen(Screen):
         yield Static(id="machine")
         with Horizontal(id="pickers"):
             with Vertical(classes="picker"):
-                yield Label("[b]Eval packs[/b]  (space to toggle)")
+                yield Label("[b]Eval packs[/b]  (space to toggle · e picks tests)")
                 yield SelectionList(id="suites")
             with Vertical(classes="picker"):
                 yield Label(id="models-legend")
@@ -455,6 +457,7 @@ class SetupScreen(Screen):
 
     def on_mount(self):
         self.selected_models = set()
+        self.picks = {}    # pack -> test ids to run instead of the tier's usual tests
         self.refresh_machine()
         self.refresh_packs()
         self.refresh_models()
@@ -471,14 +474,19 @@ class SetupScreen(Screen):
     def refresh_packs(self):
         e = self.app.engine
         packs = self.query_one("#suites", SelectionList)
-        keep = set(packs.selected)
+        keep, at = set(packs.selected), packs.highlighted
         packs.clear_options()
         for name, pk in e.packs.items():
+            picked = self.current_picks().get(name)
+            count = (f"[b cyan]{len(picked)}/{len(pk.tests)} tests picked[/]" if picked
+                     else f"[dim]{len(pk.tests)} tests[/dim]")
             needs = f"  [magenta]{', '.join(pk.needs)}[/magenta]" if pk.needs else ""
             mix = {lvl: sum(t.get("difficulty") == lvl for t in pk.tests) for lvl in ("easy", "medium", "hard")}
             mixed = " ".join(f"[{DIFF_STYLE[l]}]{n}{l[0].upper()}[/]" for l, n in mix.items() if n)
-            packs.add_option(Selection(f"[dim]{pk.group} ·[/dim] {pk.label}  [dim]{len(pk.tests)} tests[/dim] {mixed}{needs}",
+            packs.add_option(Selection(f"[dim]{pk.group} ·[/dim] {pk.label}  {count} {mixed}{needs}",
                                        name, name in keep))
+        if at is not None and at < packs.option_count:
+            packs.highlighted = at
         if not e.packs:
             for i, line in enumerate(NO_PACKS):
                 packs.add_option(Selection(line, f"\x00nopacks:{i}", disabled=True))
@@ -637,6 +645,38 @@ class SetupScreen(Screen):
                 list(self.query_one("#suites", SelectionList).selected), repeat,
                 self.tier(), self.query_one("#force", Checkbox).value)
 
+    def current_picks(self, suites=None):
+        """{pack: picked test ids} for packs that still have those tests (and, given suites, are ticked)."""
+        e = self.app.engine
+        out = {}
+        for name, ids in getattr(self, "picks", {}).items():
+            pk = e.packs.get(name)
+            known = [i for i in ids if pk and any(t["id"] == i for t in pk.tests)]
+            if known and (suites is None or name in suites):
+                out[name] = known
+        return out
+
+    def action_pick_tests(self):
+        """Choose which tests of the highlighted pack run (none ticked: the tier's usual tests)."""
+        packs = self.query_one("#suites", SelectionList)
+        opt = packs.highlighted_option
+        if opt is None or opt.value not in self.app.engine.packs:
+            self.notify("Highlight a pack first.", severity="warning")
+            return
+        name = opt.value
+
+        def done(ids):
+            if ids is None:
+                return
+            if ids:
+                self.picks[name] = ids
+                packs.select(name)
+            else:
+                self.picks.pop(name, None)
+            self.refresh_packs()
+            self.update_estimate()
+        self.app.push_screen(PickTestsScreen(self.app.engine.packs[name], self.current_picks().get(name, [])), done)
+
     @on(SelectionList.SelectedChanged)
     @on(Input.Changed)
     @on(Checkbox.Changed)
@@ -658,10 +698,11 @@ class SetupScreen(Screen):
         if not labels or not suites:
             est.update("[dim]Select at least one eval pack and one model.[/dim]" + self.queue_hint())
             return
-        jobs = self.app.engine.plan(labels, suites, repeat, tier, force)
+        picks = self.current_picks(suites)
+        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks)
         rerun = not any(j.status == "waiting" for j in jobs)
         if rerun:   # everything has results: Start runs it again, so estimate that
-            jobs = self.app.engine.plan(labels, suites, repeat, tier, force=True)
+            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks)
         run = [j for j in jobs if j.status == "waiting"]
         skipped = [j for j in jobs if j.status == "skipped"]
         secs, notes = self.app.engine.estimate_seconds(jobs)
@@ -672,7 +713,10 @@ class SetupScreen(Screen):
                 f"[b]{reps_text}[/b] repeat(s) = "
                 f"[b]{todo:,}[/b] answers · about [b]{fmt_secs(secs)}[/b]"
                 + (f" [dim](rough: {'; '.join(notes)})[/dim]" if notes else ""))
-        if tier == "screen":
+        if picks:
+            text += f"\n[cyan]Picked tests only in {', '.join(e.packs[p].label for p in picks)}[/cyan]" \
+                    "[dim] (e changes; a pack with picked tests can't PASS until all its tests have run)[/dim]"
+        elif tier == "screen":
             text += "\n[dim]Screening can only say FAIL or promising; PASS needs Certify.[/dim]"
         finished = [j for j in jobs if j.status == "done"]
         if finished:
@@ -708,6 +752,8 @@ class SetupScreen(Screen):
             return
         e = self.app.engine
         self.selected_models = {l for l in sel.get("models", []) if any(m["label"] == l for m in e.cfg["models"])}
+        self.picks = {p: list(ids) for p, ids in (sel.get("tests") or {}).items()}
+        self.refresh_packs()
         packs = self.query_one("#suites", SelectionList)
         packs.deselect_all()
         for name in sel.get("packs", []):
@@ -726,13 +772,14 @@ class SetupScreen(Screen):
 
     def action_presets(self):
         labels, packs, repeat, _, _ = self.settings()
+        tests = self.current_picks(packs)
 
         def done(result):
             if result and result[0] == "load":
                 self.apply_selection(result[1])
             elif result and result[0] == "saved":
                 self.notify(f"Saved preset '{result[1]}'. Also usable as tuieval run --preset {result[1]}")
-        self.app.push_screen(PresetsScreen(labels, packs, repeat), done)
+        self.app.push_screen(PresetsScreen(labels, packs, repeat, tests), done)
 
     def action_scan(self):
         def done(added):
@@ -750,13 +797,14 @@ class SetupScreen(Screen):
         if not labels or not suites:
             self.notify("Select at least one eval pack and one model.", severity="warning")
             return
-        sel = {"models": labels, "packs": suites, "repeat": repeat, "tier": tier, "force": force}
-        jobs = self.app.engine.plan(labels, suites, repeat, tier, force)
+        picks = self.current_picks(suites)
+        sel = {"models": labels, "packs": suites, "repeat": repeat, "tier": tier, "force": force, "tests": picks}
+        jobs = self.app.engine.plan(labels, suites, repeat, tier, force, picks)
         rerun = False
         if not any(j.status == "waiting" for j in jobs):
             # Everything selected already has results: Start means run it again (earlier results
             # are kept in history/). Packs skipped for another reason (no vision, …) stay skipped.
-            jobs = self.app.engine.plan(labels, suites, repeat, tier, force=True)
+            jobs = self.app.engine.plan(labels, suites, repeat, tier, True, picks)
             if not any(j.status == "waiting" for j in jobs):
                 reasons = sorted({j.note for j in jobs if j.status == "skipped"})
                 self.notify(f"Nothing selected can run: {'; '.join(reasons)}", severity="warning")
@@ -1102,9 +1150,9 @@ class PresetsScreen(ModalScreen):
     """Load a saved selection, or save the current one (presets.toml)."""
     BINDINGS = [Binding("escape", "cancel", "Cancel")]
 
-    def __init__(self, labels, packs, repeat):
+    def __init__(self, labels, packs, repeat, tests=None):
         super().__init__()
-        self.current = (labels, packs, repeat)
+        self.current = (labels, packs, repeat, tests or {})
 
     def compose(self) -> ComposeResult:
         with Vertical(id="dialog"):
@@ -1123,7 +1171,8 @@ class PresetsScreen(ModalScreen):
         for name, pr in self.presets.items():
             who = ", ".join(pr.get("models", [])) or ("tags " + ", ".join(pr["tags"]) if pr.get("tags") else "all models")
             opts.add_option(f"[b]{name}[/b]  [dim]{', '.join(pr.get('packs', ['all packs']))} · {who}"
-                            f"{' · repeat ' + str(pr['repeat']) if pr.get('repeat') else ''}[/dim]")
+                            f"{' · repeat ' + str(pr['repeat']) if pr.get('repeat') else ''}"
+                            f"{' · picked tests in ' + ', '.join(pr['tests']) if pr.get('tests') else ''}[/dim]")
         if not self.presets:
             opts.add_option("[dim]No presets yet: select models and packs, then save them here.[/dim]")
             opts.disabled = True
@@ -1131,14 +1180,15 @@ class PresetsScreen(ModalScreen):
     @on(OptionList.OptionSelected, "#presets")
     def load(self, event):
         name = list(self.presets)[event.option_index]
-        labels, packs, repeat = engine.resolve_preset(self.app.engine.cfg, self.presets[name], list(self.app.engine.packs))
-        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat}))
+        labels, packs, repeat, tests = engine.resolve_preset(self.app.engine.cfg, self.presets[name],
+                                                             list(self.app.engine.packs))
+        self.dismiss(("load", {"models": labels, "packs": packs, "repeat": repeat, "tests": tests}))
 
     @on(Button.Pressed, "#save")
     @on(Input.Submitted, "#name")
     def save(self):
         name = self.query_one("#name", Input).value.strip()
-        labels, packs, repeat = self.current
+        labels, packs, repeat, tests = self.current
         err = self.query_one("#error", Static)
         if not name or not engine.LABEL_RE.match(name):
             err.update("[red]Use lowercase letters, digits, '.', '_' or '-'.[/red]")
@@ -1146,7 +1196,7 @@ class PresetsScreen(ModalScreen):
         if not labels or not packs:
             err.update("[red]Select at least one model and one pack first.[/red]")
             return
-        engine.save_preset(self.app.presets_path, name, labels, packs, repeat)
+        engine.save_preset(self.app.presets_path, name, labels, packs, repeat, tests)
         self.dismiss(("saved", name))
 
     @on(Button.Pressed, "#cancel")
@@ -1215,13 +1265,24 @@ class QueuedRun:
         self.jobs = jobs
         self.est = eng.estimate_seconds(jobs)[0]
 
+    def tests(self, eng):
+        """The picked tests that are still in their packs (a pack edited while queued drops the rest)."""
+        out = {}
+        for p, ids in (self.sel.get("tests") or {}).items():
+            pk = eng.packs.get(p)
+            known = [i for i in ids if pk and any(t["id"] == i for t in pk.tests)]
+            if known:
+                out[p] = known
+        return out
+
     def line(self):
         return f"{self.title}  [dim]~{fmt_secs(self.est)} · queued {time.strftime('%H:%M', time.localtime(self.queued))}[/dim]"
 
     def make(self, eng):
         """(RunScreen, None), or (None, why it can't start) when its turn comes."""
         sel = self.sel
-        jobs = eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], sel["force"] or self.rerun)
+        jobs = eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], sel["force"] or self.rerun,
+                        self.tests(eng))
         waiting = [j for j in jobs if j.status == "waiting"]
         if not waiting:
             return None, "nothing left to run (earlier runs finished it)"
@@ -1287,6 +1348,138 @@ class ConfirmScreen(ModalScreen):
         self.dismiss(event.button.id == "yes")
 
 
+def _words(text):
+    return re.findall(r"[a-z0-9]+", str(text).lower())
+
+
+def test_note(t):
+    """What a test's row shows after its id: the description's words the id doesn't already say,
+    and the category unless the id or description already names it."""
+    desc, ids = str(t.get("description") or ""), [w for w in _words(t["id"]) if not re.fullmatch(r"v\d+", w)]
+    note = desc
+    tokens = list(re.finditer(r"[a-z0-9]+", desc.lower()))
+    if set(w.group() for w in tokens) <= set(_words(t["id"])):
+        note = ""                                   # the id in other words
+    elif len(ids) > 1:                              # "normal entry #1, pretty JSON" for normal-entry-1-v1
+        n = 0
+        while n < len(tokens) and n < len(ids) and tokens[n].group() == ids[n]:
+            n += 1
+        if n == len(ids):
+            note = desc[tokens[n - 1].end():].strip(" ,;:·-#")
+    cat = str(t.get("category") or "")
+    said = " ".join(_words(t["id"]) + _words(note))
+    if cat and " ".join(_words(cat)) not in said:
+        note = f"{note} · {cat}" if note else cat
+    return note
+
+
+class PickTestsScreen(ModalScreen):
+    """Pick which tests of one pack run. Dismisses with the ticked ids in pack order (empty = the
+    tier's usual tests), or None to keep the current pick."""
+    BINDINGS = [Binding("escape", "cancel", "Cancel"),
+                Binding("space", "toggle", "Tick", show=False),
+                Binding("a", "toggle_shown", "Tick all shown")]
+
+    def __init__(self, pack, chosen):
+        super().__init__()
+        self.pack, self.chosen, self.shown = pack, set(chosen), []
+
+    def compose(self) -> ComposeResult:
+        with Vertical(id="dialog", classes="wide"):
+            yield Label(f"[b]Pick tests: {rich_escape(self.pack.label)}[/b]\n[dim]space ticks · a ticks all shown · "
+                        "type to filter, enter to go to the list · none ticked = the tier's usual tests[/dim]")
+            yield Input(placeholder="filter by id, description or category", id="pick-filter")
+            yield OptionList(id="pick-tests")
+            yield Static(id="pick-count")
+            with Horizontal(classes="dialog-buttons"):
+                yield Button("Use these", id="pick-apply", variant="success")
+                yield Button("Clear (usual tests)", id="pick-clear")
+                yield Button("Cancel", id="pick-cancel")
+
+    def on_mount(self):
+        self.fill("")
+        self.query_one("#pick-tests").focus()
+
+    def row(self, t):
+        on = t["id"] in self.chosen
+        mark = "[b green]\\[✓][/]" if on else "[dim]\\[ ][/dim]"
+        tid = f"[b]{rich_escape(t['id'])}[/b]" if on else rich_escape(t["id"])
+        note = test_note(t)
+        return f"{mark} {diff_badge(t.get('difficulty', 'unrated'))} {tid}" + (f"  [dim]{rich_escape(note)}[/dim]" if note else "")
+
+    def fill(self, text):
+        ol = self.query_one("#pick-tests", OptionList)
+        ol.clear_options()
+        text = text.lower()
+        self.shown = [t for t in self.pack.tests
+                      if text in " ".join(str(t.get(k, "")) for k in ("id", "description", "category")).lower()]
+        ol.add_options([Option(self.row(t), id=t["id"]) for t in self.shown])
+        if self.shown:
+            ol.highlighted = 0
+        self.count()
+
+    def count(self):
+        shown = f"  [dim]({len(self.shown)} shown)[/dim]" if len(self.shown) != len(self.pack.tests) else ""
+        self.query_one("#pick-count", Static).update(
+            (f"[b]{len(self.chosen)}[/b] of {len(self.pack.tests)} ticked" if self.chosen
+             else "[dim]None ticked: the tier's usual tests run.[/dim]") + shown)
+
+    def redraw(self, indexes):
+        ol = self.query_one("#pick-tests", OptionList)
+        for i in indexes:
+            ol.replace_option_prompt_at_index(i, self.row(self.shown[i]))
+        self.count()
+
+    def action_toggle(self):
+        ol = self.query_one("#pick-tests", OptionList)
+        if not ol.has_focus or ol.highlighted is None:
+            return
+        self.chosen.symmetric_difference_update({self.shown[ol.highlighted]["id"]})
+        self.redraw([ol.highlighted])
+
+    @on(OptionList.OptionSelected, "#pick-tests")
+    def selected(self, event):   # enter on a row ticks it too
+        self.chosen.symmetric_difference_update({event.option.id})
+        self.redraw([event.option_index])
+
+    def action_toggle_shown(self):
+        """Tick every shown test, or untick them all when they're all ticked."""
+        if self.query_one("#pick-filter", Input).has_focus:
+            return
+        ids = {t["id"] for t in self.shown}
+        if ids <= self.chosen:
+            self.chosen -= ids
+        else:
+            self.chosen |= ids
+        self.redraw(range(len(self.shown)))
+
+    def check_action(self, action, parameters):
+        # while typing in the filter, space and a are text
+        if action in ("toggle", "toggle_shown") and self.query_one("#pick-filter", Input).has_focus:
+            return None
+        return True
+
+    @on(Input.Changed, "#pick-filter")
+    def filter_changed(self, event):
+        self.fill(event.value)
+
+    @on(Input.Submitted, "#pick-filter")
+    def to_list(self):
+        self.query_one("#pick-tests").focus()
+
+    @on(Button.Pressed, "#pick-apply")
+    def apply(self):
+        self.dismiss([t["id"] for t in self.pack.tests if t["id"] in self.chosen])
+
+    @on(Button.Pressed, "#pick-clear")
+    def clear(self):
+        self.dismiss([])
+
+    @on(Button.Pressed, "#pick-cancel")
+    def action_cancel(self):
+        self.dismiss(None)
+
+
 class ModelPickScreen(ModalScreen):
     """Pick which models Per question compares. Dismisses with a set of labels (empty = all) or None."""
     BINDINGS = [Binding("escape", "cancel", "Cancel")]
@@ -2331,7 +2524,7 @@ class EvalsApp(App):
     #dialog { width: 80; height: auto; max-height: 90%; padding: 1 2; border: thick $primary; background: $surface; }
     #dialog.wide { width: 120; }
     #dialog SelectionList, #dialog OptionList { height: auto; max-height: 20; }
-    PresetsScreen, ScanScreen, SessionsScreen, ChoiceScreen { align: center middle; }
+    PresetsScreen, ScanScreen, SessionsScreen, ChoiceScreen, PickTestsScreen { align: center middle; }
     #dialog Input, #dialog Select { margin-bottom: 1; }
     .dialog-buttons { height: 3; margin-top: 1; }
     .dialog-buttons Button { margin-right: 2; }

{tuieval-0.2.0.dev10 → tuieval-0.2.0.dev14}/tests/test_tuieval.py

@@ -43,8 +43,8 @@ def free_port():
 
 
 class Mock:
-    def __init__(self, packs, mode):
-        self.port = free_port()
+    def __init__(self, packs, mode, port=None):
+        self.port = port or free_port()
         self.proc = subprocess.Popen([sys.executable, os.path.join(HERE, "mock_server.py"), "--port", str(self.port),
                                       "--packs", packs, "--mode", mode], stdout=subprocess.DEVNULL)
         for _ in range(100):
@@ -209,6 +209,13 @@ class Units(unittest.TestCase):
             write(log, "ok\nerror: runtime bootstrap failed [x]: context does not fit\n")
             self.assertEqual(engine.server_error(log, 0), "context does not fit")
 
+    def test_test_note_skips_what_the_id_says(self):
+        from tuieval.tui import test_note
+        self.assertEqual(test_note({"id": "normal-entry-1-v2", "category": "normal entry",
+                                    "description": "normal entry #1, compact JSON, shuffled keys"}), "compact JSON, shuffled keys")
+        self.assertEqual(test_note({"id": "stale-quote", "description": "stale quote", "category": "data"}), "data")
+        self.assertEqual(test_note({"id": "todo", "description": "To-do list app", "category": ""}), "To-do list app")
+
     def test_short_note(self):
         from tuieval import tui
         self.assertEqual(tui.short_note("server exited with code 1 while loading: bad file"), "didn't start: bad file")
@@ -225,6 +232,106 @@ class Units(unittest.TestCase):
             self.assertEqual(p.modules, ["some_missing_module"])
 
 
+class PickTests(unittest.TestCase):
+    """Running only some tests of a pack: partial runs add up, --force redoes just the picked ones."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        for name in ("apps", "other"):
+            d = os.path.join(self.ws, "packs", name)
+            os.makedirs(d)
+            write(os.path.join(d, "pack.toml"), f'label = "{name}"\nscreen = 1\n[certify]\nrepeat = 1\n'
+                                                '[gate]\nmin_accuracy = 0.1\n')
+            write(os.path.join(d, "tests.yaml"), "".join(
+                f"- {{id: {t}, input: {name} question {t}, expected: 1, reference: 'ANSWER: 1', wrong: ['ANSWER: 2'], "
+                "difficulty: easy}\n" for t in ("a", "b", "c", "d")))
+        self.mock, self.port = None, free_port()
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f'\n[servers.m]\nurl = "http://127.0.0.1:{self.port}"\n')
+
+    def tearDown(self):
+        if self.mock:
+            self.mock.stop()
+        self.tmp.cleanup()
+
+    def server(self, mode):
+        """(Re)start the mock on the same port: oracle answers right, wrong answers wrong."""
+        first = self.mock is None
+        if self.mock:
+            self.mock.stop()
+        self.mock = Mock(os.path.join(self.ws, "packs"), mode, port=self.port)
+        if first:
+            tuieval(self.ws, "add", "mock", "--server", "m", "--label", "m")
+
+    def rows(self, pack="apps"):
+        path = os.path.join(self.ws, "results", "m", f"{pack}.json")
+        if not os.path.isfile(path):
+            return {}
+        return {r["test"]: r["pass"] for r in json.loads(read(path))["results"]}
+
+    def status(self, pack="apps"):
+        code = f"from tuieval import engine; print(engine.Engine().result_status('m', '{pack}'))"
+        return subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                              env=dict(os.environ, TUIEVAL_HOME=self.ws)).stdout.strip()
+
+    def test_pick(self):
+        from tuieval import packs
+        p = packs.load_pack(os.path.join(self.ws, "packs", "apps"))
+        self.assertEqual([t["id"] for t in p.pick(["c", "a"])], ["a", "c"])   # the pack's order
+        with self.assertRaises(packs.PackError) as cm:
+            p.pick(["a", "zz"])
+        self.assertIn("'zz'", str(cm.exception))
+        self.assertIn("a, b, c, d", str(cm.exception))
+
+    def test_partial_runs_add_up_and_force_redoes_only_the_picked(self):
+        self.server("oracle")
+        tuieval(self.ws, "run", "--packs", "apps", "--tests", "b")
+        self.assertEqual(self.rows(), {"b": True})
+        self.assertEqual(self.rows("other"), {})
+        tuieval(self.ws, "run", "--tests", "apps:a,apps:c", "--tier", "certify")   # --packs from --tests
+        self.assertEqual(self.rows(), {"a": True, "b": True, "c": True})
+        self.assertEqual(self.status(), "screened")
+        out = tuieval(self.ws, "run", "--tests", "apps:b").stdout
+        self.assertIn("done earlier", out)                                       # nothing missing: not re-asked
+        tuieval(self.ws, "run", "--tests", "apps:d", "--tier", "certify")
+        self.assertEqual(self.status(), "certified")
+
+        self.server("wrong")
+        tuieval(self.ws, "run", "--tests", "apps:b", "--force")
+        self.assertEqual(self.rows(), {"a": True, "b": False, "c": True, "d": True})
+        hist = os.listdir(os.path.join(self.ws, "results", "m", "history"))
+        self.assertTrue(any(f.startswith("apps-") for f in hist), hist)
+        self.assertEqual(self.status(), "certified")
+
+    def test_cli_errors(self):
+        self.server("oracle")
+        for args, msg in ((["--tests", "a"], "say which pack"),
+                          (["--packs", "apps,other", "--tests", "a"], "say which pack"),
+                          (["--tests", "nope:a"], "unknown pack 'nope'"),
+                          (["--packs", "apps", "--tests", "zz"], "has no test 'zz'"),
+                          (["--packs", "other", "--tests", "apps:a"], "aren't in --packs")):
+            p = tuieval(self.ws, "run", *args, check=False)
+            self.assertNotEqual(p.returncode, 0, args)
+            self.assertIn(msg, p.stderr, args)
+        self.assertEqual(self.rows(), {})
+
+    def test_presets_keep_picked_tests(self):
+        from tuieval import engine
+        path = os.path.join(self.ws, "presets.toml")
+        engine.save_preset(path, "quick", ["m"], ["apps", "other"], None, {"apps": ["b", "d"], "gone": ["x"]})
+        presets = engine.load_presets(path)
+        self.assertEqual(presets["quick"]["tests"], {"apps": ["b", "d"]})
+        cfg = {"models": [{"label": "m", "tags": []}]}
+        self.assertEqual(engine.resolve_preset(cfg, presets["quick"], ["apps", "other"]),
+                         (["m"], ["apps", "other"], None, {"apps": ["b", "d"]}))
+        self.server("oracle")
+        tuieval(self.ws, "run", "--preset", "quick")
+        self.assertEqual(self.rows(), {"b": True, "d": True})
+        self.assertEqual(len(self.rows("other")), 1)     # no pick: the Screen sample (screen = 1)
+
+
 def fake_gguf(path, embedding=5120, size=0, kv_heads=4):
     """A GGUF header with just the keys machines.read_gguf reads, padded to size bytes. kv_heads: a
     number, or one per layer (a list, as some GGUFs store it)."""