tuieval 0.2.0.dev100__tar.gz → 0.2.0.dev103__tar.gz

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Run screen: `x` removes the highlighted pack from the run going if it hasn't started (the reverse of adding it); a model left with nothing isn't started.
+- `k` (skip model) asks first, naming the model. Yes/no dialogs start on No; `y` answers yes, `n` or Esc no.
 - Fix: hosted models no longer count as running alongside local ones (they don't share the machine), so a one-model-at-a-time setup no longer gets the "ran alongside other models" note. Older results are read the same way.
 - Results, Per question: the model summary shows every picked model without scrolling, and fits the window: shorter column names, and long model names cut in the middle (both ends kept). The PTA index table fits the same way.
 - PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `(n% truncated)` beside it and a **truncated** column in the table, so overthinking stands apart from wrong answers.

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev100
+Version: 0.2.0.dev103
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -143,7 +143,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model (after a yes), `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run. `x` takes a pack that hasn't started out of the run; `w` lists queued runs (`del` removes one).
 4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures; click a column name to sort. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/README.md

@@ -117,7 +117,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model (after a yes), `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run. `x` takes a pack that hasn't started out of the run; `w` lists queued runs (`del` removes one).
 4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures; click a column name to sort. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev100'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev100')
+__version__ = version = '0.2.0.dev103'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev103')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/src/tuieval/engine.py

@@ -457,6 +457,7 @@ class Job:
     earlier: list = dataclasses.field(default_factory=list)  # earlier sittings (see Engine._sittings)
     stop_early: bool = False  # stop once the pack's FAIL is certain (verdict.fail_certain)
     first: list = dataclasses.field(default_factory=list)   # test ids round 1 asks first (most failed)
+    claimed: bool = False   # its model's server took it on: it can no longer be removed from the run
 
     @property
     def label(self):
@@ -1728,6 +1729,33 @@ class Engine:
             r["cond"].notify_all()
         return added, None if added else "everything picked is already in the run"
 
+    def removable(self, job):
+        """Why a job of the run going can't be removed (None: it can): only a pack not yet started."""
+        if job.status in ("done", "failed", "skipped"):
+            return "it has finished"
+        if job.claimed or job.status == "running":
+            return "it's running (k skips the model)"
+        return None
+
+    def remove_jobs(self, keys):
+        """Take jobs not yet started (removable) out of the run going, the reverse of add_jobs: they never
+        run, and a model left with none isn't started. Returns the jobs removed."""
+        removed = []
+        with self._run_lock:
+            r = self._run
+            if not r or not r["open"]:
+                return removed
+            for j in [j for j in r["jobs"] if j.key in keys and self.removable(j) is None]:
+                j.status, j.note = "removed", "removed from the run"
+                r["jobs"].remove(j)
+                for _, group in r["pending"]:
+                    if j in group:
+                        group.remove(j)
+                removed.append(j)
+            r["pending"][:] = [g for g in r["pending"] if g[1]]   # the lanes hold this very list
+            r["cond"].notify_all()
+        return removed
+
     def _run_lanes(self, parallel, parallel_hosted=4):
         """Run each model's jobs in its own lane: up to `parallel` local models at a time, in order (the
         next one starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
@@ -2021,6 +2049,10 @@ class Engine:
             failed_in_row = 0
             for j in jobs:
                 self._check()
+                with self._run_lock:
+                    if j.status == "removed":   # taken out of the run while its model ran another pack
+                        continue
+                    j.claimed = True
                 try:
                     self._run_job(j, base_url, run_info, alive=info["alive"])
                 except BaseException as ex:

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/src/tuieval/tui.py

@@ -1828,16 +1828,33 @@ class RetestScreen(ModalScreen):
 
 
 class ConfirmScreen(ModalScreen):
-    def __init__(self, question):
+    """Yes or no; No has the focus, so Enter by reflex changes nothing. y: yes; n or Esc: no
+    (and no_keys, e.g. the key that opened it, pressed again)."""
+    BINDINGS = [Binding("y", "answer(True)", "Yes", show=False), Binding("n", "answer(False)", "No", show=False),
+                Binding("escape", "answer(False)", "No", show=False)]
+
+    def __init__(self, question, no_keys=()):
         super().__init__()
         self.question = question
+        self.no_keys = no_keys
 
     def compose(self) -> ComposeResult:
         with Vertical(id="dialog"):
             yield Label(self.question)
             with Horizontal(classes="dialog-buttons"):
-                yield Button("Yes", id="yes", variant="error")
-                yield Button("No", id="no")
+                yield Button("Yes (y)", id="yes", variant="error")
+                yield Button("No (n)", id="no")
+
+    def on_mount(self):
+        self.query_one("#no", Button).focus()
+
+    def on_key(self, event):
+        if event.key in self.no_keys:
+            event.stop()
+            self.dismiss(False)
+
+    def action_answer(self, yes):
+        self.dismiss(yes)
 
     @on(Button.Pressed)
     def answer(self, event):
@@ -2114,6 +2131,7 @@ class RunScreen(Screen):
     # priority: these work whichever pane has focus (the text panes would otherwise swallow keys)
     BINDINGS = [
         Binding("k", "skip_model", "Skip model", priority=True),
+        Binding("x", "remove_job", "Remove from run", priority=True),
         Binding("c", "cancel_all", "Cancel all", priority=True),
         Binding("f", "toggle_follow", "Pause scroll", priority=True),
         Binding("v", "watch_next", "Next model", priority=True),
@@ -2124,7 +2142,7 @@ class RunScreen(Screen):
         Binding("q", "back", "Back", show=False, priority=True),
     ]
     kind = "Run"
-    LIVE_ONLY = {"skip_model", "cancel_all", "toggle_follow"}
+    LIVE_ONLY = {"skip_model", "remove_job", "cancel_all", "toggle_follow"}
     DONE_ONLY = set()
 
     def __init__(self, jobs, parallel=None):
@@ -2180,6 +2198,8 @@ class RunScreen(Screen):
     def check_action(self, action, parameters):
         if action == "watch_next":
             return self.running and self.side_by_side
+        if action == "remove_job":   # only while something is left to remove
+            return self.running and any(self.app.engine.removable(j) is None for j in self.jobs)
         if action in self.LIVE_ONLY:
             return self.running
         if action in self.DONE_ONLY:
@@ -2219,7 +2239,7 @@ class RunScreen(Screen):
         with Vertical(id="top"):
             yield Static(id="overall")
             yield ProgressBar(id="bar", show_eta=False)
-            yield DataTable(id="queue", cursor_type="none", zebra_stripes=True)
+            yield DataTable(id="queue", cursor_type="row", zebra_stripes=True)
         yield Static(id="current")
         with Horizontal(id="done-bar"):
             yield Button("New run  [n]", id="done-new", variant="success")
@@ -2365,10 +2385,12 @@ class RunScreen(Screen):
                 self.total += j.total
             self.query_one("#bar", ProgressBar).update(total=max(self.total, 1))
             self.event(f"added to this run: {', '.join(j.key for j in d['jobs'])}")
+            self.refresh_bindings()
             self.update_overall()
         elif kind in ("job_started", "job_update", "job_done"):
             job = d["job"]
             self.update_job(job)
+            self.refresh_bindings()   # x shows while a pack can still be removed
             if kind == "job_started" and d.get("resumed"):
                 earlier = (f" ({fmt_secs(d['earlier_s'])} over {d['sittings']} earlier sitting(s))"
                            if d.get("sittings") else "")
@@ -2611,13 +2633,63 @@ class RunScreen(Screen):
 
     # -- actions
     def action_skip_model(self):
-        if self.running:
-            if self.side_by_side and self.watch:
-                self.app.engine.skip_model(self.watch)
-                self.notify(f"Skipping {self.watch}; the others keep going…")
+        if not self.running:
+            return
+        one = self.watch if self.side_by_side and self.watch else None
+        names = [one] if one else self.active()
+        if not names:
+            return
+
+        def skip(yes):
+            if not yes or not self.running:
+                return
+            if one:
+                self.app.engine.skip_model(one)
+                self.notify(f"Skipping {one}; the others keep going…")
             else:
                 self.app.engine.skip_model()
                 self.notify("Skipping the current model…")
+        self.app.push_screen(ConfirmScreen(
+            f"Skip {', '.join(names)}? Answers so far are kept (a later run picks up from them); the rest "
+            f"of {'its' if len(names) == 1 else 'their'} questions in this run are skipped.", no_keys=("k",)), skip)
+
+    def action_remove_job(self):
+        """Take the highlighted pack out of the run if it hasn't started (the reverse of adding it)."""
+        if not self.running:
+            return
+        q = self.query_one("#queue", DataTable)
+        try:
+            key = q.coordinate_to_cell_key(q.cursor_coordinate).row_key.value
+        except Exception:
+            key = None
+        job = next((j for j in self.jobs if j.key == key), None)
+        if job is None:
+            self.notify("Highlight a pack in the queue (click it or use ↑/↓ there), then press x.")
+            return
+        why = self.app.engine.removable(job)
+        if why:
+            self.notify(f"{job.label} · {job.pack.label} can't be removed: {why}.")
+            return
+
+        def remove(yes):
+            if not yes:
+                return
+            removed = self.app.engine.remove_jobs({job.key})
+            if not removed:
+                self.notify(f"{job.label} · {job.pack.label} started meanwhile, so it stays (k skips the model).")
+                return
+            for j in removed:
+                q.remove_row(j.key)
+                self.total -= j.total
+            self.query_one("#bar", ProgressBar).update(total=max(self.total, 1))
+            self.event(f"removed from this run: {', '.join(j.key for j in removed)}")
+            self.update_overall()
+            self.refresh_bindings()
+            self.app.refresh_live()
+            self.notify(f"Removed from the run: {job.label} · {job.pack.label}.")
+        self.app.push_screen(ConfirmScreen(
+            f"Remove {job.label} · {job.pack.label} from this run? It hasn't started, so nothing is lost; "
+            "Start can add it again later.", no_keys=("x",)), remove)
 
     def action_watch_next(self):
         """Stream the next model running alongside (from its next answer)."""
@@ -3489,7 +3561,7 @@ class EvalsApp(App):
             return
         run.handle("jobs_added", {"jobs": added})
         models = sorted({j.label for j in added})
-        self.notify(f"Added {len(added)} pack run(s) to the run going: {', '.join(models)}. w shows it.", timeout=10)
+        self.notify(f"Added {len(added)} pack run(s) to the run going: {', '.join(models)}. x removes a pack not started yet.", timeout=10)
         self.refresh_live()
 
     def enqueue(self, item):

{tuieval-0.2.0.dev100 → tuieval-0.2.0.dev103}/tests/test_tuieval.py

@@ -1030,6 +1030,141 @@ class AddToRun(unittest.TestCase):
         loads = [d["label"] for k, d in events if k == "model_loading"]
         self.assertEqual(loads.count("a"), 2)                                 # a's new pack: after it ended
 
+    def test_engine_removes_jobs_not_started(self):
+        from tuieval import engine
+        events = []
+        e = engine.Engine(lambda kind, **d: events.append((kind, d)), root=self.ws)
+        jobs = e.plan(["a"], ["apps", "other"], None, "certify", False, {})
+        t = threading.Thread(target=e.run, args=(jobs, 1))
+        t.start()
+        deadline = time.time() + 180   # CI machines can take long to start servers
+        while not any(k == "request_done" for k, _ in events) and time.time() < deadline:
+            time.sleep(0.05)
+        added, _ = e.add_jobs(e.plan(["b"], ["apps"], None, "certify", False, {}))
+        self.assertEqual([j.key for j in added], ["b/apps"])
+        self.assertEqual(e.remove_jobs({"a/apps"}), [])                      # answering: k skips it instead
+        # a's next pack (its server is up) and b (never started) both go
+        self.assertEqual(sorted(j.key for j in e.remove_jobs({"a/other", "b/apps"})), ["a/other", "b/apps"])
+        self.finish(e, t)
+        self.assertEqual([(j.key, j.status) for j in jobs], [("a/apps", "done")])
+        self.assertNotIn(("model_loading", "b"), [(k, d.get("label")) for k, d in events])
+        self.assertFalse(os.path.exists(os.path.join(self.ws, "results", "a", "other.json")))
+        self.assertEqual(e.remove_jobs({"a/apps"}), [])                      # the run is over
+
+    def test_tui_removes_a_pack_from_the_run(self):
+        path = os.path.join(self.ws, "models.toml")
+        write(path, read(path).replace('"--delay", "0.2"', '"--delay", "3"'))   # still running while it's asked
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, ConfirmScreen
+            from textual.widgets import SelectionList, DataTable
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.query_one("#suites", SelectionList).select("other")
+                    s.selected_models = {"a", "b"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+                    await pilot.pause()
+                    s.action_start()
+                    deadline = time.time() + 180     # CI machines can take long to start servers
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run, out = app.screen, {}
+                    q = run.query_one("#queue", DataTable)
+                    total = run.total
+                    q.move_cursor(row=q.get_row_index("b/apps"))
+                    await pilot.press("x")
+                    await pilot.pause()
+                    out["asked"] = isinstance(app.screen, ConfirmScreen) and app.screen.focused.id == "no"
+                    await pilot.press("n")
+                    await pilot.pause()
+                    out["kept"] = [j.key for j in run.jobs]
+                    q.move_cursor(row=q.get_row_index("b/apps"))
+                    await pilot.press("x")
+                    await pilot.pause()
+                    await pilot.press("y")
+                    await pilot.pause()
+                    out["after"] = [[j.key for j in run.jobs], list(map(str, (k.value for k in q.rows))),
+                                    total - run.total]
+                    q.move_cursor(row=q.get_row_index("a/apps"))      # running: stays, nothing asked
+                    await pilot.press("x")
+                    await pilot.pause()
+                    out["running"] = [isinstance(app.screen, ConfirmScreen), "a/apps" in [j.key for j in run.jobs]]
+                    app.engine.cancel()
+                    deadline = time.time() + 60
+                    while time.time() < deadline and run.running:
+                        await pilot.pause(0.2)
+                    print(json.dumps(out))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertTrue(out["asked"])
+        self.assertIn("b/apps", out["kept"])
+        jobs, rows, fewer = out["after"]
+        self.assertNotIn("b/apps", jobs)
+        self.assertNotIn("b/apps", rows)
+        self.assertEqual(fewer, 4)                     # its 4 questions are no longer counted
+        self.assertEqual(out["running"], [False, True])
+
+    def test_tui_asks_before_skipping_a_model(self):
+        path = os.path.join(self.ws, "models.toml")
+        write(path, read(path).replace('"--delay", "0.2"', '"--delay", "3"'))   # still running while it's asked
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, ConfirmScreen
+            from textual.widgets import SelectionList
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"a"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+                    await pilot.pause()
+                    s.action_start()
+                    deadline = time.time() + 180     # CI machines can take long to start servers
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run, out = app.screen, {}
+                    for keys in (["k", "enter"], ["k", "k"], ["k", "n"], ["k", "escape"]):   # each one: no
+                        await pilot.press(keys[0])
+                        await pilot.pause()
+                        out.setdefault("asked", []).append(isinstance(app.screen, ConfirmScreen)
+                                                           and app.screen.focused.id == "no")
+                        await pilot.press(keys[1])
+                        await pilot.pause()
+                    out["kept"] = [run.running, app.screen is run, [j.status for j in run.jobs]]
+                    await pilot.press("k")
+                    await pilot.pause()
+                    await pilot.press("y")
+                    deadline = time.time() + 120
+                    while time.time() < deadline and run.running:
+                        await pilot.pause(0.2)
+                    out["skipped"] = [j.status for j in run.jobs]
+                    print(json.dumps(out))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertEqual(out["asked"], [True] * 4)
+        self.assertEqual(out["kept"], [True, True, ["running"]])
+        self.assertEqual(out["skipped"], ["skipped"])
+
     def test_tui_adds_or_queues_while_running(self):
         code = textwrap.dedent("""
             import asyncio, json, time