tuieval 0.2.0.dev28__tar.gz → 0.2.0.dev32__tar.gz

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 - Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
+- More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.
 - Less to take in at first: until something has run, the setup screen leaves out context sizes, tuning state and the Tune key; Presets and Hide are off the footer (`?` lists every key); `tuieval help` groups commands into Start here, Day to day and Advanced.
 - Results lead with a plain summary per model ("ready for Support; not decided yet for Coding → run Certify to finish"), in the TUI, `tuieval verdict` (with the command to run next) and the report; the statistics follow.

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev28
+Version: 0.2.0.dev32
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -137,7 +137,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/README.md

@@ -111,7 +111,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev28'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev28')
+__version__ = version = '0.2.0.dev32'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev32')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/src/tuieval/engine.py

@@ -698,6 +698,8 @@ class Engine:
         self._lanes_lock = threading.Lock()
         self._procs = []
         self._stop_lock = threading.Lock()
+        self._run_lock = threading.Lock()
+        self._run = None          # the run going: {"jobs", "pending", "open", "machine", "cond"} (add_jobs)
         self._header_cache = {}
         self._serving = {}
         self.tuning_dir = os.path.join(root, "tuning")
@@ -1552,29 +1554,38 @@ class Engine:
     # ---- running
     def run(self, jobs, parallel=None):
         """Run the jobs (blocking). Jobs of the same model share one server start. parallel: how many
-        models are served at a time (default: parallel_models for this machine)."""
+        models are served at a time (default: parallel_models for this machine). add_jobs() can add
+        more while it runs; they run after the ones before them."""
         self._cancel.clear()
         self.emit("queue_started", jobs=jobs)
         models = sorted({j.label for j in jobs if j.status == "waiting"})
+        machine = bool(jobs) and self.needs_machine(jobs)
         lock = (self.machine_lock(f"{jobs[0].tier} run: {', '.join(models[:3])}"
                                   + (f" +{len(models) - 3}" if len(models) > 3 else ""))
-                if jobs and self.needs_machine(jobs) else contextlib.nullcontext())
+                if machine else contextlib.nullcontext())
+        order = []
+        for j in jobs:
+            if j.label not in order:
+                order.append(j.label)
+        groups = [(label, [j for j in jobs if j.label == label and j.status == "waiting"]) for label in order]
+        with self._run_lock:
+            self._run = {"jobs": jobs, "pending": [g for g in groups if g[1]], "open": True, "machine": machine,
+                         "cond": threading.Condition(self._run_lock)}
         try:
             with lock:
-                order = []
-                for j in jobs:
-                    if j.label not in order:
-                        order.append(j.label)
-                groups = [(label, [j for j in jobs if j.label == label and j.status == "waiting"]) for label in order]
-                self._run_lanes([g for g in groups if g[1]], max(1, int(parallel or self.parallel_models())))
+                self._run_lanes(max(1, int(parallel or self.parallel_models())))
             if self._cancel.is_set():
                 raise Cancelled()
         except Cancelled:
+            with self._run_lock:
+                self._run["open"] = False
             for j in jobs:
                 if j.status in ("waiting", "loading", "running"):
                     j.status, j.note = "skipped", "cancelled (progress kept)" if j.done else "cancelled"
                     self.emit("job_done", job=j)
         finally:
+            with self._run_lock:
+                self._run["open"] = False
             changes = []
             if any(j.status == "done" and j.started and j.tier != "smoke" for j in jobs):
                 try:
@@ -1586,12 +1597,36 @@ class Engine:
                 self.emit("verdicts", changes=changes)
             self.emit("queue_done", jobs=jobs, cancelled=self._cancel.is_set())
 
-    def _run_lanes(self, groups, parallel):
+    def add_jobs(self, new):
+        """Add planned jobs to the run going on, after its other jobs. Returns (added, why not): a
+        pack already waiting or running in it isn't added twice, and nothing is added once the run is
+        finishing or cancelled, or when it holds no machine lock (hosted only) and these need one.
+        Runs on the caller's thread, so the caller shows what was added (no event)."""
+        with self._run_lock:
+            r = self._run
+            if not r or not r["open"] or self._cancel.is_set():
+                return [], "the run is finishing"
+            if not r["machine"] and self.needs_machine(new):
+                return [], "the run going uses only hosted models, and these need this machine"
+            busy = {j.key for j in r["jobs"] if j.status in ("waiting", "loading", "running")}
+            added = [j for j in new if j.status == "waiting" and j.key not in busy]
+            r["jobs"].extend(added)
+            for j in added:   # a model still waiting takes them on; otherwise it comes again at the end
+                group = next((g for g in r["pending"] if g[0] == j.label), None)
+                if group:
+                    group[1].append(j)
+                else:
+                    r["pending"].append((j.label, [j]))
+            r["cond"].notify_all()
+        return added, None if added else "everything picked is already in the run"
+
+    def _run_lanes(self, parallel):
         """Run each model's jobs in its own lane, up to `parallel` at a time, in order: the next
         model starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
-        next to the ones running (memory_need_gb; an unknown need is left to the setting)."""
-        pending, running, told = list(groups), {}, set()
-        cond = threading.Condition()
+        next to the ones running (memory_need_gb; an unknown need is left to the setting). Lanes
+        wait for jobs added meanwhile (add_jobs) until every model is done."""
+        pending, running, told = self._run["pending"], {}, set()
+        cond = self._run["cond"]
 
         def blocked(label):
             """Why the next model can't start now (None: it can)."""
@@ -1614,20 +1649,29 @@ class Engine:
 
         def lane():
             while True:
-                with cond:
-                    while True:
-                        if self._cancel.is_set() or not pending:
-                            return
-                        label = pending[0][0]
-                        why = blocked(label)
-                        if why is None:
-                            label, mine = pending.pop(0)
-                            running[label] = self.memory_need_gb(self.model(label))
-                            break
-                        if why and label not in told:
-                            told.add(label)
-                            self.emit("model_waiting", label=label, message=f"{label} waits: {why}")
+                mine, note = None, None
+                with cond:   # events go out after the lock is released: the UI may be waiting for it (add_jobs)
+                    if self._cancel.is_set() or not pending and not running:
+                        self._run["open"] = False   # nothing more can be added
+                        cond.notify_all()
+                        return
+                    # in order, except that a model added again while it runs waits for itself to end
+                    # without holding up the ones behind it
+                    nxt = next((i for i, g in enumerate(pending) if g[0] not in running), None)
+                    label = pending[nxt][0] if nxt is not None else None
+                    why = blocked(label) if label else ""   # none can start: a model still runs, more may come
+                    if why is None:
+                        label, mine = pending.pop(nxt)
+                        running[label] = self.memory_need_gb(self.model(label))
+                    elif why and label not in told:
+                        told.add(label)
+                        note = f"{label} waits: {why}"
+                    else:
                         cond.wait(1.0)
+                if note:
+                    self.emit("model_waiting", label=label, message=note)
+                if mine is None:
+                    continue
                 try:
                     self._run_lane(label, mine)
                 finally:
@@ -1635,10 +1679,10 @@ class Engine:
                         running.pop(label, None)
                         cond.notify_all()
 
-        if parallel == 1 or len(groups) == 1:
+        if parallel == 1:
             lane()
             return
-        threads = [threading.Thread(target=lane, name=f"lane-{i + 1}") for i in range(min(parallel, len(groups)))]
+        threads = [threading.Thread(target=lane, name=f"lane-{i + 1}") for i in range(parallel)]
         for t in threads:
             t.start()
         for t in threads:

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/src/tuieval/tui.py

@@ -294,7 +294,7 @@ HELP = {
 parallel_models in models.toml, default 1). Answers note the models they ran alongside.
 
 [b]Keys[/b]
-  s  start (or queue it behind a run that's going)     r  results
+  s  start (with a run going: add to it, or queue it)   r  results
   a  add a model (a GGUF, a model id, openrouter:<id>)  m  scan model folders for new GGUFs
   e  pick which tests of the highlighted pack run       p  save or load a selection (preset)
   t  tune the ticked models' speed flags here (after   x  hide or unhide a model
@@ -310,7 +310,7 @@ with the model's reasoning and answer as they stream, then recent results with t
 
 [b]Keys[/b]
   k  skip the current model          c  cancel everything (finished answers are kept; it resumes next time)
-  f  pause or resume auto-scroll     r  results     n  set up a new run (this one keeps going)
+  f  pause or resume auto-scroll     r  results     n  more evals: add them to this run or queue them
   w  runs and queue                  esc  back to Setup (the run keeps going)
   v  with several models at a time: stream the next one (k skips the one streaming)""",
     "ResultsScreen": """[b]Results[/b]
@@ -878,7 +878,8 @@ class SetupScreen(Screen):
         """Says what Start will do when something is already running, here or in another window."""
         app = self.app
         if app.is_busy():
-            return (f"\n[cyan]{app.busy_text()}: Start adds this to the queue as #{len(app.queue) + 1} "
+            add = "add this to the run going or " if isinstance(app.active_session, RunScreen) else ""
+            return (f"\n[cyan]{app.busy_text()}: Start lets you {add}queue it as #{len(app.queue) + 1} "
                     "(w shows the queue).[/cyan]")
         holder = app.engine.machine_holder()
         if holder:
@@ -1464,8 +1465,8 @@ class QueuedRun:
     def line(self):
         return f"{self.title}  [dim]~{fmt_secs(self.est)} · queued {time.strftime('%H:%M', time.localtime(self.queued))}[/dim]"
 
-    def make(self, eng):
-        """(RunScreen, None), or (None, why it can't start) when its turn comes."""
+    def plan(self, eng):
+        """(jobs, None), or (None, why it can't run) now."""
         sel = self.sel
         jobs = eng.plan(sel["models"], sel["packs"], sel["repeat"], sel["tier"], sel["force"] or self.rerun,
                         self.tests(eng))
@@ -1476,7 +1477,12 @@ class QueuedRun:
         if new:
             return None, (f"it would now start over {', '.join(new)}, which wasn't confirmed when it was queued. "
                           "Start it again from Setup to see why")
-        return RunScreen(jobs, sel.get("parallel")), None
+        return jobs, None
+
+    def make(self, eng):
+        """(RunScreen, None), or (None, why it can't start) when its turn comes."""
+        jobs, problem = self.plan(eng)
+        return (RunScreen(jobs, self.sel.get("parallel")), None) if jobs else (None, problem)
 
 
 class QueuedTune:
@@ -1807,7 +1813,7 @@ class RunScreen(Screen):
         Binding("c", "cancel_all", "Cancel all", priority=True),
         Binding("f", "toggle_follow", "Pause scroll", priority=True),
         Binding("v", "watch_next", "Next model", priority=True),
-        Binding("n", "new_run", "New run", priority=True),
+        Binding("n", "new_run", "More evals", priority=True),
         Binding("r", "results", "Results", priority=True),
         Binding("w", "runs", "Runs", priority=True),
         Binding("escape", "back", "Setup", priority=True),
@@ -1815,7 +1821,7 @@ class RunScreen(Screen):
     ]
     kind = "Run"
     LIVE_ONLY = {"skip_model", "cancel_all", "toggle_follow"}
-    DONE_ONLY = {"new_run"}
+    DONE_ONLY = set()
 
     def __init__(self, jobs, parallel=None):
         super().__init__()
@@ -2027,6 +2033,16 @@ class RunScreen(Screen):
             if self.watched(kind, d):
                 self.req_started = None
             self.update_overall()
+        elif kind == "jobs_added":
+            q = self.query_one("#queue", DataTable)
+            for j in d["jobs"]:
+                if j.key not in q.rows:
+                    q.add_row(j.label, j.pack.label, "", "", "", "", "", "", key=j.key)
+                self.update_job(j)
+                self.total += j.total
+            self.query_one("#bar", ProgressBar).update(total=max(self.total, 1))
+            self.event(f"added to this run: {', '.join(j.key for j in d['jobs'])}")
+            self.update_overall()
         elif kind in ("job_started", "job_update", "job_done"):
             job = d["job"]
             self.update_job(job)
@@ -2262,6 +2278,9 @@ class RunScreen(Screen):
 
     @on(Button.Pressed, "#done-new")
     def action_new_run(self):
+        if self.running:
+            self.notify("Pick models and packs, then Start: add them to this run or queue them after it.",
+                        timeout=8)
         self.app.back_to_setup()  # Setup still holds the last selection
         self.app.screen.query_one("#models").focus()
 
@@ -2923,6 +2942,40 @@ class EvalsApp(App):
                 self.notify("Everything selected had results; running it again.")
             self.add_session(screen)
             return
+        run = self.active_session
+        if item.kind == "Run" and isinstance(run, RunScreen) and run.running \
+                and item.sel["tier"] != run.jobs[0].tier:
+            self.notify(f"The run going is {run.jobs[0].tier.title()} and this is {item.sel['tier'].title()}, "
+                        "so it can't join it.", timeout=10)
+        elif item.kind == "Run" and isinstance(run, RunScreen) and run.running:
+            def picked(choice):
+                if choice == "add":
+                    self.add_to_run(run, item)
+                elif choice == "queue":
+                    self.enqueue(item)
+            what = f"{', '.join(item.sel['models'][:4])}{' …' if len(item.sel['models']) > 4 else ''} · " \
+                   f"{', '.join(item.sel['packs'][:4])}{' …' if len(item.sel['packs']) > 4 else ''}"
+            self.push_screen(ChoiceScreen(
+                f"A run is going: {run.live_text()}.\n\nAdd {what} to it? Its models run after the ones "
+                "already in it.\nOr queue it as a separate run that starts when this one ends.",
+                [("add", "Add to this run", "success"), ("queue", f"Queue it as #{len(self.queue) + 1}", "primary"),
+                 ("keep", "Cancel", "default")]), picked)
+            return
+        self.enqueue(item)
+
+    def add_to_run(self, run, item):
+        jobs, problem = item.plan(self.engine)
+        added, why = self.engine.add_jobs(jobs) if jobs else ([], problem)
+        if not added:
+            self.notify(f"Not added: {why}. Queued it instead.", severity="warning", timeout=10)
+            self.enqueue(item)
+            return
+        run.handle("jobs_added", {"jobs": added})
+        models = sorted({j.label for j in added})
+        self.notify(f"Added {len(added)} pack run(s) to the run going: {', '.join(models)}. w shows it.", timeout=10)
+        self.refresh_live()
+
+    def enqueue(self, item):
         ahead = self.busy_text()
         self.queue.append(item)
         self.notify(f"Queued #{len(self.queue)}: {item.title}. It starts by itself after: {ahead}. "

{tuieval-0.2.0.dev28 → tuieval-0.2.0.dev32}/tests/test_tuieval.py

@@ -522,6 +522,160 @@ class Parallel(unittest.TestCase):
         self.assertEqual(sum(r["pass"] for r in self.results("b")["results"]), 6)
 
 
+class AddToRun(unittest.TestCase):
+    """More evals while a run is going: added to it (they run after its other jobs) or queued after it."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        for name in ("apps", "other"):
+            d = os.path.join(self.ws, "packs", name)
+            os.makedirs(d)
+            write(os.path.join(d, "pack.toml"), f'label = "{name}"\n[certify]\nrepeat = 1\n[gate]\nmin_accuracy = 0.1\n')
+            write(os.path.join(d, "tests.yaml"), "".join(
+                f"- {{id: {t}, input: {name} question {t}, expected: 1, reference: 'ANSWER: 1', difficulty: easy}}\n"
+                for t in "abcd"))
+        mock = [sys.executable, os.path.join(HERE, "mock_server.py"), "--port", "{port}", "--packs",
+                os.path.join(self.ws, "packs"), "--model", "{served_name}", "--delay", "0.2"]
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f"\n[servers.mk]\ncmd = {json.dumps(mock)}\nport = {free_port()}\n")
+            for label in "ab":
+                f.write(f'\n[[models]]\nlabel = "{label}"\nserver = "mk"\nmodel = "{label}"\n')
+
+    def tearDown(self):
+        self.tmp.cleanup()
+
+    def finish(self, e, t, limit=480):
+        """Wait for a run thread; CI machines can take half a minute per server start. One that's
+        still going after `limit` is cancelled, so it can't run into the next test."""
+        t.join(limit)
+        if t.is_alive():
+            e.cancel()
+            t.join(120)
+            self.fail(f"the run didn't finish in {limit}s")
+
+    def test_engine_adds_jobs_to_the_run_going(self):
+        from tuieval import engine
+        events = []
+        e = engine.Engine(lambda kind, **d: events.append((kind, d)), root=self.ws)
+        jobs = e.plan(["a"], ["apps"], None, "certify", False, {})
+        self.assertEqual(e.add_jobs(e.plan(["b"], ["apps"], None, "certify", False, {})),
+                         ([], "the run is finishing"))                     # no run yet
+        t = threading.Thread(target=e.run, args=(jobs,))
+        t.start()
+        deadline = time.time() + 180   # CI machines can take long to start servers
+        while not any(k == "request_done" for k, _ in events) and time.time() < deadline:
+            time.sleep(0.05)
+        added, why = e.add_jobs(e.plan(["b", "a"], ["apps", "other"], None, "certify", False, {}))
+        self.assertIsNone(why)
+        self.assertEqual(sorted(j.key for j in added), ["a/other", "b/apps", "b/other"])   # a/apps is running
+        self.assertEqual(e.add_jobs(e.plan(["b"], ["apps"], None, "certify", False, {}))[1],
+                         "everything picked is already in the run")
+        self.finish(e, t)
+        self.assertEqual([j.status for j in jobs], ["done"] * 4)
+        self.assertEqual(jobs[0].key, "a/apps")
+        started = [d["job"].key for k, d in events if k == "job_started"]
+        self.assertEqual(started[0], "a/apps")
+        self.assertEqual(e.add_jobs(e.plan(["b"], ["apps"], None, "certify", True, {}))[1], "the run is finishing")
+        queue_done = [d for k, d in events if k == "queue_done"]
+        self.assertEqual(len(queue_done[0]["jobs"]), 4)
+
+    def test_added_models_use_a_free_lane(self):
+        from tuieval import engine
+        path = os.path.join(self.ws, "models.toml")    # slower answers: a is surely still answering when b is added
+        write(path, read(path).replace('"--delay", "0.2"', '"--delay", "1.0"'))
+        events = []
+        e = engine.Engine(lambda kind, **d: events.append((kind, d)), root=self.ws)
+        jobs = e.plan(["a"], ["apps"], None, "certify", False, {})
+        t = threading.Thread(target=e.run, args=(jobs, 2))
+        t.start()
+        deadline = time.time() + 180   # CI machines can take long to start servers
+        while not any(k == "request_done" for k, _ in events) and time.time() < deadline:
+            time.sleep(0.05)
+        added, why = e.add_jobs(e.plan(["a", "b"], ["other", "apps"], None, "certify", False, {}))
+        self.assertEqual(sorted(j.key for j in added), ["a/other", "b/apps", "b/other"])
+        self.finish(e, t)
+        self.assertEqual([j.status for j in jobs], ["done"] * 4)
+        order = [(k, d.get("label") or d["job"].key) for k, d in events if k in ("model_loading", "job_done")]
+        self.assertLess(order.index(("model_loading", "b")), order.index(("job_done", "a/apps")), order)  # next to a
+        a_other = json.loads(read(os.path.join(self.ws, "results", "a", "other.json")))["results"]
+        self.assertFalse(any("a" in (r.get("ran_alongside") or []) for r in a_other))
+        loads = [d["label"] for k, d in events if k == "model_loading"]
+        self.assertEqual(loads.count("a"), 2)                                 # a's new pack: after it ended
+
+    def test_tui_adds_or_queues_while_running(self):
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, SetupScreen, ChoiceScreen
+            from textual.widgets import SelectionList
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"a"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+                    await pilot.pause()
+                    s.action_start()
+                    deadline = time.time() + 180     # CI machines can take long to start servers
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run = app.screen
+                    await pilot.press("n")                       # more evals while it runs
+                    await pilot.pause()
+                    on_setup = isinstance(app.screen, SetupScreen)
+                    app.screen.selected_models = {"b"}
+                    app.screen.refresh_models()
+                    app.screen.action_start()
+                    await pilot.pause()
+                    asked = isinstance(app.screen, ChoiceScreen)
+                    app.screen.dismiss("add")
+                    await pilot.pause()
+                    app.screen.query_one("#suites", SelectionList).deselect("apps")
+                    app.screen.query_one("#suites", SelectionList).select("other")
+                    app.screen.selected_models = {"a"}
+                    app.screen.refresh_models()
+                    app.screen.action_start()
+                    await pilot.pause()
+                    app.screen.dismiss("queue")
+                    await pilot.pause()
+                    app.screen.query_one("#tier-smoke").value = True    # another tier can't join: queued
+                    await pilot.pause()
+                    app.screen.action_start()
+                    await pilot.pause()
+                    other_tier = not isinstance(app.screen, ChoiceScreen)
+                    queued = len(app.queue)
+                    deadline = time.time() + 300
+                    while time.time() < deadline:
+                        await pilot.pause(0.2)
+                        if not run.running and not app.queue and not app.active_session:
+                            break
+                    second = app.sessions[1]
+                    print(json.dumps({"on_setup": on_setup, "asked": asked, "queued": queued, "other_tier": other_tier,
+                                      "run": [(j.key, j.status) for j in run.jobs], "total": run.total,
+                                      "sessions": len(app.sessions),
+                                      "second": [(j.key, j.status) for j in second.jobs]}))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertTrue(out["on_setup"])
+        self.assertTrue(out["asked"])
+        self.assertTrue(out["other_tier"])
+        self.assertEqual(out["queued"], 2)
+        self.assertEqual(out["run"], [["a/apps", "done"], ["b/apps", "done"]])   # added to the run going
+        self.assertEqual(out["total"], 8)
+        self.assertEqual(out["sessions"], 3)
+        self.assertEqual(out["second"], [["a/other", "done"]])                  # queued: its own run after
+
+
 class FirstRun(unittest.TestCase):
     """A new user's first run fails fast with the fix, never hangs or shows a bare Python error."""