tuieval 0.2.0.dev79__tar.gz → 0.2.0.dev86__tar.gz

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## Unreleased
 
+- Start while a run is going skips models and packs already running or queued (it says which); picking only those adds nothing. Retest and picked questions still go through.
+- Fix: something queued just as a run ended (while its server was stopping) now starts instead of waiting forever.
+- Hosted models (OpenRouter and other APIs) run beside local ones instead of waiting for a slot: up to `parallel_hosted` at a time (default 4, `--parallel-hosted N`). Local models can be added to a hosted-only run.
 - Early stop: `tuieval run --stop-early`, **Stop a pack once it fails** in Setup, or `stop_early = true` in a pack.toml ends a pack once its FAIL is certain (a critical failure, or an accuracy or truncation bar out of reach), asking first the questions models fail most. Verdicts also say FAIL when a partial run can no longer pass.
 - `tuieval compare --reference <model>`: every other model paired against one on the questions both answered, per pack and overall: the difference with a 95% CI, better/tie/worse, critical failures and tokens vs the reference. `--packs` limits any compare to some packs.
 - Fix: a question description or grader reason containing `[` no longer crashes the run screen.

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev79
+Version: 0.2.0.dev86
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/docs/models.md

@@ -71,7 +71,7 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 - **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`, or `rtx4090-128gb` on a Linux machine with a discrete GPU; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
 - **Fit check (llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory (a Mac's unified memory, or a discrete GPU's VRAM), capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
-- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free: default 4 on a Mac, 1.5 in a discrete GPU's VRAM), `gpu_memory_gb` (the GPU memory to size models against, when detection gets it wrong or you want to hold some back), `gpu_residency_gb` (see the stall guard) and `parallel_models` (see below).
+- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free: default 4 on a Mac, 1.5 in a discrete GPU's VRAM), `gpu_memory_gb` (the GPU memory to size models against, when detection gets it wrong or you want to hold some back), `gpu_residency_gb` (see the stall guard), `parallel_models` and `parallel_hosted` (see below).
 
 ### Discrete GPUs (Linux: Nvidia, AMD)
 
@@ -89,6 +89,7 @@ Windows isn't supported (WSL2 works like Linux).
 
 By default a run serves one model at a time. On a machine with room for more, set `parallel_models = 2` (or more) under `[machines.<id>]`, type a number in **Models at a time** on the TUI's setup screen (blank uses the machine's setting), or pass `tuieval run --parallel N`.
 
+- **Hosted models** (OpenRouter and other APIs) don't use this machine, so they don't count against this: up to `parallel_hosted` of them (default 4, under `[machines.<id>]`, or `tuieval run --parallel-hosted N`) run beside the local ones, and one never waits for a local model to finish. A server is hosted when tuieval doesn't start it and its url isn't this machine; set `hosted = true` (or `false`) on a server to say otherwise. A hosted-only run doesn't hold the machine; if local models are added to it, it takes the machine when the first of them starts.
 - Models start in queue order, each in its own lane. The next one waits while its fit-check memory estimate wouldn't fit next to the ones running (without one, e.g. a server that sizes its own memory, the setting decides). Contexts are never shrunk to make room.
 - A second model on the same server gets a free port, so a server's `cmd` must take `{port}`.
 - A server with `before_start` (which may stop other servers) always runs on its own. Tuning is always one model at a time.
@@ -291,5 +292,6 @@ Optional sections:
 # memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
 # gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
 # parallel_models = 2          # models a run serves at a time (default 1)
+# parallel_hosted = 4          # hosted models (APIs) a run serves beside them (default 4)
 ```
 

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev79'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev79')
+__version__ = version = '0.2.0.dev86'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev86')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/src/tuieval/engine.py

@@ -47,6 +47,7 @@ import subprocess
 import threading
 import time
 import tomllib
+import urllib.parse
 import urllib.request
 
 from . import client
@@ -703,6 +704,7 @@ class Engine:
         self._procs = []
         self._stop_lock = threading.Lock()
         self._run_lock = threading.Lock()
+        self._machine_take = threading.Lock()   # one lane takes the machine lock for a hosted-only run
         self._run = None          # the run going: {"jobs", "pending", "open", "machine", "cond"} (add_jobs)
         self._header_cache = {}
         self._serving = {}
@@ -1258,6 +1260,36 @@ class Engine:
         except (TypeError, ValueError):
             return 1
 
+    def parallel_hosted(self):
+        """How many hosted models (APIs) a run serves at a time, besides the local ones: models.toml
+        [machines.<id>] parallel_hosted (default 4). They don't use this machine's GPU or memory."""
+        try:
+            return max(1, int(self.machine_settings().get("parallel_hosted", 4)))
+        except (TypeError, ValueError):
+            return 4
+
+    def models_at_once(self, labels, parallel=None, parallel_hosted=None):
+        """How many of these models run at the same time: local ones up to parallel (parallel_models),
+        hosted ones beside them up to parallel_hosted."""
+        labels = set(labels)
+        hosted = sum(self.hosted(self.model(l)) for l in labels)
+        return (min(parallel or self.parallel_models(), len(labels) - hosted)
+                + min(parallel_hosted or self.parallel_hosted(), hosted))
+
+    def hosted(self, m):
+        """A model served by an API elsewhere: tuieval doesn't start its server and its url isn't this
+        machine (or its server sets hosted = true). A server you started yourself on localhost still
+        counts as local (it uses the GPU)."""
+        if m.get("remote"):
+            return True
+        server = self.cfg["servers"][m["server"]]
+        if "hosted" in server:   # say so for a server the url doesn't give away (e.g. a local proxy to an API)
+            return bool(server["hosted"])
+        if server.get("cmd"):
+            return False
+        host = urllib.parse.urlparse(server.get("url") or "").hostname or ""
+        return bool(host) and host not in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+
     def memory_need_gb(self, m):
         """GPU memory a model takes while served here: 0 for one we don't start (a hosted API or a
         server already running), the fit check's estimate for a local GGUF, else None (unknown)."""
@@ -1616,17 +1648,17 @@ class Engine:
         return any(self.cfg["servers"][j.model["server"]].get("cmd") for j in jobs if j.status == "waiting")
 
     # ---- running
-    def run(self, jobs, parallel=None):
+    def run(self, jobs, parallel=None, parallel_hosted=None):
         """Run the jobs (blocking). Jobs of the same model share one server start. parallel: how many
-        models are served at a time (default: parallel_models for this machine). add_jobs() can add
-        more while it runs; they run after the ones before them."""
+        local models are served at a time (default: parallel_models for this machine); hosted models
+        (APIs) run beside them, up to parallel_hosted at a time (default: parallel_hosted). add_jobs()
+        can add more while it runs; they run after the ones before them."""
         self._cancel.clear()
         self.emit("queue_started", jobs=jobs)
         models = sorted({j.label for j in jobs if j.status == "waiting"})
         machine = bool(jobs) and self.needs_machine(jobs)
-        lock = (self.machine_lock(f"{jobs[0].tier} run: {', '.join(models[:3])}"
-                                  + (f" +{len(models) - 3}" if len(models) > 3 else ""))
-                if machine else contextlib.nullcontext())
+        what = f"{jobs[0].tier} run: {', '.join(models[:3])}" + (f" +{len(models) - 3}" if len(models) > 3 else "")
+        stack = contextlib.ExitStack()   # holds the machine lock; a hosted-only run takes it if local models are added
         order = []
         for j in jobs:
             if j.label not in order:
@@ -1634,10 +1666,13 @@ class Engine:
         groups = [(label, [j for j in jobs if j.label == label and j.status == "waiting"]) for label in order]
         with self._run_lock:
             self._run = {"jobs": jobs, "pending": [g for g in groups if g[1]], "open": True, "machine": machine,
-                         "cond": threading.Condition(self._run_lock)}
+                         "cond": threading.Condition(self._run_lock), "stack": stack, "what": what}
         try:
-            with lock:
-                self._run_lanes(max(1, int(parallel or self.parallel_models())))
+            with stack:
+                if machine:
+                    stack.enter_context(self.machine_lock(what))
+                self._run_lanes(max(1, int(parallel or self.parallel_models())),
+                                max(1, int(parallel_hosted or self.parallel_hosted())))
             if self._cancel.is_set():
                 raise Cancelled()
         except Cancelled:
@@ -1664,14 +1699,13 @@ class Engine:
     def add_jobs(self, new):
         """Add planned jobs to the run going on, after its other jobs. Returns (added, why not): a
         pack already waiting or running in it isn't added twice, and nothing is added once the run is
-        finishing or cancelled, or when it holds no machine lock (hosted only) and these need one.
+        finishing or cancelled. Local models added to a hosted-only run take the machine lock when the
+        first of them starts.
         Runs on the caller's thread, so the caller shows what was added (no event)."""
         with self._run_lock:
             r = self._run
             if not r or not r["open"] or self._cancel.is_set():
                 return [], "the run is finishing"
-            if not r["machine"] and self.needs_machine(new):
-                return [], "the run going uses only hosted models, and these need this machine"
             busy = {j.key for j in r["jobs"] if j.status in ("waiting", "loading", "running")}
             added = [j for j in new if j.status == "waiting" and j.key not in busy]
             r["jobs"].extend(added)
@@ -1684,33 +1718,51 @@ class Engine:
             r["cond"].notify_all()
         return added, None if added else "everything picked is already in the run"
 
-    def _run_lanes(self, parallel):
-        """Run each model's jobs in its own lane, up to `parallel` at a time, in order: the next
-        model starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
-        next to the ones running (memory_need_gb; an unknown need is left to the setting). Lanes
-        wait for jobs added meanwhile (add_jobs) until every model is done."""
+    def _run_lanes(self, parallel, parallel_hosted=4):
+        """Run each model's jobs in its own lane: up to `parallel` local models at a time, in order (the
+        next one starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
+        next to the ones running (memory_need_gb; an unknown need is left to the setting)), and beside
+        them up to `parallel_hosted` hosted models, which need none of this machine. Lanes wait for jobs
+        added meanwhile (add_jobs) until every model is done."""
         pending, running, told = self._run["pending"], {}, set()
         cond = self._run["cond"]
+        hosted = {}
+
+        def is_hosted(label):
+            if label not in hosted:
+                hosted[label] = self.hosted(self.model(label))
+            return hosted[label]
 
         def blocked(label):
             """Why the next model can't start now (None: it can)."""
-            if not running:
+            if is_hosted(label):
+                return "" if sum(map(is_hosted, running)) >= parallel_hosted else None
+            local = [l for l in running if not is_hosted(l)]
+            if not local:
                 return None
-            if len(running) >= parallel:
+            if len(local) >= parallel:
                 return ""
             m = self.model(label)
-            if self.runs_alone(m) or any(self.runs_alone(self.model(l)) for l in running):
-                alone = label if self.runs_alone(m) else next(l for l in running if self.runs_alone(self.model(l)))
+            if self.runs_alone(m) or any(self.runs_alone(self.model(l)) for l in local):
+                alone = label if self.runs_alone(m) else next(l for l in local if self.runs_alone(self.model(l)))
                 return (f"{alone}'s server runs a before_start step that may stop other servers, so it runs "
                         "on its own")
             need = self.memory_need_gb(m)
-            used = sum(n for n in running.values() if n)
+            used = sum(running[l] or 0 for l in local)
             room = self.memory_available_gb()
             if need and used + need > room:
-                return (f"needs ~{need:.0f} GB; {', '.join(running)} take ~{used:.0f} of {room:.0f} GB, so it "
+                return (f"needs ~{need:.0f} GB; {', '.join(local)} take ~{used:.0f} of {room:.0f} GB, so it "
                         "starts when there's room")
             return None
 
+        def take_machine():
+            """A hosted-only run that was given local models takes the machine lock before the first
+            of them starts (it may wait for another tuieval window, like any run)."""
+            with self._machine_take:
+                if not self._run["machine"]:
+                    self._run["stack"].enter_context(self.machine_lock(self._run["what"]))
+                    self._run["machine"] = True
+
         def lane():
             while True:
                 mine, note = None, None
@@ -1720,33 +1772,41 @@ class Engine:
                         cond.notify_all()
                         return
                     # in order, except that a model added again while it runs waits for itself to end
-                    # without holding up the ones behind it
-                    nxt = next((i for i, g in enumerate(pending) if g[0] not in running), None)
-                    label = pending[nxt][0] if nxt is not None else None
-                    why = blocked(label) if label else ""   # none can start: a model still runs, more may come
-                    if why is None:
-                        label, mine = pending.pop(nxt)
-                        running[label] = self.memory_need_gb(self.model(label))
-                    elif why and label not in told:
-                        told.add(label)
-                        note = f"{label} waits: {why}"
-                    else:
-                        cond.wait(1.0)
+                    # without holding up the ones behind it; the first waiting hosted model doesn't wait
+                    # for local ones (and the first local one doesn't wait for hosted ones)
+                    waiting = [i for i, g in enumerate(pending) if g[0] not in running]
+                    first = {}
+                    for i in waiting:
+                        first.setdefault(is_hosted(pending[i][0]), i)
+                    why, label = "", None
+                    for i in sorted(first.values()):
+                        why = blocked(pending[i][0])
+                        if why is None:
+                            label, mine = pending.pop(i)
+                            running[label] = self.memory_need_gb(self.model(label))
+                            break
+                    if mine is None:
+                        label = pending[min(first.values())][0] if first else None
+                        if why and label and label not in told:
+                            told.add(label)
+                            note = f"{label} waits: {why}"
+                        else:
+                            cond.wait(1.0)
                 if note:
                     self.emit("model_waiting", label=label, message=note)
                 if mine is None:
                     continue
                 try:
+                    if not is_hosted(label) and self.cfg["servers"][mine[0].model["server"]].get("cmd"):
+                        take_machine()
                     self._run_lane(label, mine)
                 finally:
                     with cond:
                         running.pop(label, None)
                         cond.notify_all()
 
-        if parallel == 1:
-            lane()
-            return
-        threads = [threading.Thread(target=lane, name=f"lane-{i + 1}") for i in range(parallel)]
+        # lanes for hosted models too, even if none is queued yet: add_jobs may bring some
+        threads = [threading.Thread(target=lane, name=f"lane-{i + 1}") for i in range(parallel + parallel_hosted)]
         for t in threads:
             t.start()
         for t in threads:

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/src/tuieval/run_evals.py

@@ -17,6 +17,7 @@
     tuieval run --tier smoke --only openrouter:qwen/qwen3-32b   # any OpenRouter model, no models.toml edit
     tuieval run --dry-run                         # print the plan and server commands only
     tuieval run --parallel 2                      # serve two models at a time (parallel_models in models.toml)
+    tuieval run --parallel-hosted 6               # hosted models at a time beside them (default 4)
     tuieval add ~/models/New-Model-Q4_K_M.gguf    # register a model (--vision/--mmproj, --no-think, --tags)
     tuieval scan                                  # GGUFs in model_dirs that aren't registered yet
     tuieval list                                  # the models tuieval knows; hidden ones listed separately
@@ -755,6 +756,9 @@ def main(argv=None):
     p.add_argument("--smoke", action="store_const", const="smoke", dest="tier", help="same as --tier smoke")
     p.add_argument("--parallel", type=int, metavar="N",
                    help="models served at a time (default: parallel_models for this machine in models.toml, else 1)")
+    p.add_argument("--parallel-hosted", type=int, metavar="N",
+                   help="hosted models (APIs) served at a time beside the local ones (default: parallel_hosted "
+                        "in models.toml, else 4)")
     p.add_argument("--dry-run", action="store_true", help="print the plan and commands without running")
     p.add_argument("--brief", action="store_true", help="end with one line per model instead of the scorecards")
     p.add_argument("--models", default=None, help="default: the workspace's models.toml")
@@ -769,6 +773,9 @@ def main(argv=None):
     if args.parallel is not None and args.parallel < 1:
         sys.exit("--parallel must be 1 or more")
     parallel = args.parallel or e.parallel_models()
+    if args.parallel_hosted is not None and args.parallel_hosted < 1:
+        sys.exit("--parallel-hosted must be 1 or more")
+    parallel_hosted = args.parallel_hosted or e.parallel_hosted()
     models = e.cfg["models"]
     labels = [m["label"] for m in models if not m.get("remote")]   # hosted APIs cost money: only by name
     if args.only:
@@ -837,12 +844,12 @@ def main(argv=None):
 
     signal.signal(signal.SIGINT, stop)
     signal.signal(signal.SIGTERM, stop)
-    side_by_side = min(parallel, len({j.label for j in jobs if j.status == "waiting"}))
+    side_by_side = e.models_at_once({j.label for j in jobs if j.status == "waiting"}, parallel, parallel_hosted)
     if side_by_side > 1:
         printer.side_by_side = True
         say(f"up to {side_by_side} models at a time: one line per answer; each answer notes the models "
             "it ran alongside, since they share this machine's speed")
-    worker = threading.Thread(target=e.run, args=(jobs, parallel))
+    worker = threading.Thread(target=e.run, args=(jobs, parallel, parallel_hosted))
     worker.start()
     while worker.is_alive():
         worker.join(0.5)

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/src/tuieval/tui.py

@@ -975,7 +975,7 @@ class SetupScreen(Screen):
         resumed = [j for j in run if "done earlier" in j.note]
         if resumed:
             text += f"   [yellow]{len(resumed)} pack(s) continue from earlier results[/yellow]"
-        at_once = min(self.parallel() or e.parallel_models(), len({j.label for j in run}))
+        at_once = e.models_at_once({j.label for j in run}, self.parallel())
         if at_once > 1:
             text += (f"\n[cyan]Up to {at_once} models at a time[/cyan][dim] (the time above assumes one at a "
                      "time). Answers note the models they ran alongside, since those share the machine's "
@@ -1550,6 +1550,13 @@ def run_title(tier, models, packs):
             f"{', '.join(packs) if len(packs) <= 2 else f'{len(packs)} packs'}")
 
 
+def short_keys(keys, n=4):
+    """A few "model/pack" pairs as "model · pack", the rest counted."""
+    keys = sorted(keys)
+    shown = ", ".join(k.replace("/", " · ", 1) for k in keys[:n])
+    return rich_escape(shown + (f" and {len(keys) - n} more" if len(keys) > n else ""))
+
+
 class QueuedRun:
     """A run waiting its turn. It's planned again when it starts, so it picks up what the runs before
     it finished; it never sets aside answers that weren't confirmed when it was queued. retest: the
@@ -1557,13 +1564,29 @@ class QueuedRun:
     kind = "Run"
 
     def __init__(self, sel, retest, leave, accepted, eng):
-        self.sel, self.retest, self.leave, self.accepted = sel, retest, leave, accepted
+        self.sel, self.retest, self.leave, self.accepted = sel, retest, set(leave), accepted
         self.queued = time.time()
+        self.replan(eng)
+
+    def replan(self, eng):
         self.jobs, self.problem = self.plan(eng)
         jobs = self.jobs or []
-        self.title = run_title(sel["tier"], {j.label for j in jobs}, {j.pack.label for j in jobs})
+        self.title = run_title(self.sel["tier"], {j.label for j in jobs}, {j.pack.label for j in jobs})
         self.est = eng.estimate_seconds(jobs)[0]
 
+    def keys(self):
+        """The "model/pack" pairs it covers."""
+        return {f"{m}/{p}" for m in self.sel["models"] for p in self.sel["packs"]} - self.leave
+
+    def deliberate(self):
+        """Retest or picked questions: asked again on purpose, never treated as a duplicate."""
+        return bool(self.sel.get("force") or self.sel.get("tests"))
+
+    def drop(self, keys, eng):
+        """Leave these pairs out (they're already running or queued)."""
+        self.leave |= set(keys)
+        self.replan(eng)
+
     def tests(self, eng):
         """The picked tests that are still in their packs (a pack edited while queued drops the rest)."""
         out = {}
@@ -2130,8 +2153,7 @@ class RunScreen(Screen):
     @property
     def side_by_side(self):
         """Several models run at a time in this run."""
-        return min(self.parallel or self.app.engine.parallel_models(),
-                   len({j.label for j in self.jobs if j.status != "skipped"})) > 1
+        return self.app.engine.models_at_once({j.label for j in self.jobs if j.status != "skipped"}, self.parallel) > 1
 
     def active(self):
         """Models being loaded or run now, in queue order."""
@@ -2446,7 +2468,7 @@ class RunScreen(Screen):
                 rough |= j.key in self.rough
             left += (j.total - j.done) * spr
             models.add(j.label)
-        lanes = max(1, min(self.parallel or self.app.engine.parallel_models(), len(models)))
+        lanes = max(1, self.app.engine.models_at_once(models, self.parallel))
         return left / lanes, rough
 
     def update_overall(self):
@@ -3237,8 +3259,32 @@ class EvalsApp(App):
         what = s.live_text() if s else "The last run is still stopping its server"
         return what + (f" (+{len(self.queue)} queued)" if self.queue else "")
 
+    def taken(self, tier):
+        """The "model/pack" pairs of this tier already in the run going (not finished) or in the queue."""
+        out = set()
+        run = self.active_session
+        if isinstance(run, RunScreen) and run.running and run.jobs and run.jobs[0].tier == tier:
+            out |= {j.key for j in run.jobs if j.status in ("waiting", "loading", "running")}
+        for q in self.queue:
+            if q.kind == "Run" and q.sel["tier"] == tier:
+                out |= q.keys()
+        return out
+
     # -- the queue: Start or t while something runs adds to it; each item starts when the one before ends
     def start_or_queue(self, item):
+        skipped = set()
+        if self.is_busy() and item.kind == "Run" and not item.deliberate():
+            mine = item.keys()
+            skipped = mine & self.taken(item.sel["tier"])
+            if skipped == mine:
+                self.notify(f"Already running or queued, so nothing was added: {short_keys(skipped)}.", timeout=10)
+                return
+            if skipped:
+                item.drop(skipped, self.engine)
+                if not item.jobs:
+                    self.notify(f"Nothing new to run: {item.problem}. Already running or queued: "
+                                f"{short_keys(skipped)}.", timeout=10)
+                    return
         if not self.is_busy():
             screen, problem = item.make(self.engine)
             if problem:
@@ -3257,11 +3303,11 @@ class EvalsApp(App):
                     self.add_to_run(run, item)
                 elif choice == "queue":
                     self.enqueue(item)
-            what = f"{', '.join(item.sel['models'][:4])}{' …' if len(item.sel['models']) > 4 else ''} · " \
-                   f"{', '.join(item.sel['packs'][:4])}{' …' if len(item.sel['packs']) > 4 else ''}"
+            what = short_keys(item.keys(), 6)
+            skip = (f"\n\nLeft out, already running or queued: {short_keys(skipped)}." if skipped else "")
             self.push_screen(ChoiceScreen(
                 f"A run is going: {run.live_text()}.\n\nAdd {what} to it? Its models run after the ones "
-                "already in it.\nOr queue it as a separate run that starts when this one ends.",
+                f"already in it.\nOr queue it as a separate run that starts when this one ends.{skip}",
                 [("add", "Add to this run", "success"), ("queue", f"Queue it as #{len(self.queue) + 1}", "primary"),
                  ("keep", "Cancel", "default")]), picked)
             return
@@ -3271,6 +3317,9 @@ class EvalsApp(App):
         jobs, problem = item.plan(self.engine)
         added, why = self.engine.add_jobs(jobs) if jobs else ([], problem)
         if not added:
+            if why and why.startswith("everything picked is already in the run"):
+                self.notify(f"Not added: {why}.", timeout=10)   # queueing it again would only duplicate it
+                return
             self.notify(f"Not added: {why}. Queued it instead.", severity="warning", timeout=10)
             self.enqueue(item)
             return
@@ -3282,6 +3331,8 @@ class EvalsApp(App):
     def enqueue(self, item):
         ahead = self.busy_text()
         self.queue.append(item)
+        if not self.active_session:   # the last run already ended (its server may still be stopping):
+            self.set_timer(0.3, self.start_next)   # nothing else would start the queue
         self.notify(f"Queued #{len(self.queue)}: {item.title}. It starts by itself after: {ahead}. "
                     "w shows the queue.", timeout=10)
         self.refresh_live()

{tuieval-0.2.0.dev79 → tuieval-0.2.0.dev86}/tests/test_tuieval.py

@@ -15,6 +15,7 @@ import threading
 import types
 import time
 import unittest
+import unittest.mock
 import urllib.request
 
 HERE = os.path.dirname(os.path.abspath(__file__))
@@ -283,6 +284,11 @@ class Units(unittest.TestCase):
         job = lambda key, label, total, done=0, status="waiting": types.SimpleNamespace(
             key=key, label=label, total=total, done=done, status=status)
         run = RunScreen.__new__(RunScreen)
+        eng = types.SimpleNamespace(models_at_once=lambda labels, parallel: min(parallel, len(labels)))
+        patch = unittest.mock.patch.object(RunScreen, "app", new_callable=unittest.mock.PropertyMock,
+                                           return_value=types.SimpleNamespace(engine=eng))
+        patch.start()
+        self.addCleanup(patch.stop)
         run.parallel, run.rough = 1, {"local/new"}
         run.jobs = [job("hosted/a", "hosted", 100, 100, "done"), job("local/a", "local", 200, 4, "running"),
                     job("local/new", "local", 50)]
@@ -854,6 +860,66 @@ class MarkupInText(unittest.TestCase):
             self.assertEqual(json.loads(p.stdout.strip().splitlines()[-1]), [["m/br", "done", 2]], p.stderr[-2000:])
 
 
+class HostedBeside(unittest.TestCase):
+    """Hosted models (APIs) don't use this machine: they run beside local ones whatever parallel_models
+    says, and local models can join a hosted-only run."""
+
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.ws = os.path.join(self.tmp.name, "ws")
+        tuieval(self.tmp.name, "init", self.ws)
+        d = os.path.join(self.ws, "packs", "apps")
+        os.makedirs(d)
+        write(os.path.join(d, "pack.toml"), 'label = "apps"\n[certify]\nrepeat = 1\n[gate]\nmin_accuracy = 0.1\n')
+        write(os.path.join(d, "tests.yaml"), "".join(
+            f"- {{id: {t}, input: question {t}, expected: 1, reference: 'ANSWER: 1', difficulty: easy}}\n" for t in "abcd"))
+        local = [sys.executable, os.path.join(HERE, "mock_server.py"), "--port", "{port}", "--packs",
+                 os.path.join(self.ws, "packs"), "--model", "{served_name}", "--delay", "0.5"]
+        self.api = Mock(os.path.join(self.ws, "packs"), "oracle", delay=0.5)
+        with open(os.path.join(self.ws, "models.toml"), "a") as f:
+            f.write(f'\n[servers.mk]\ncmd = {json.dumps(local)}\nport = {free_port()}\n'
+                    f'\n[servers.api]\nurl = "http://127.0.0.1:{self.api.port}"\nhosted = true\n'
+                    '\n[[models]]\nlabel = "loc"\nserver = "mk"\nmodel = "loc"\n')
+            for label in ("h1", "h2"):
+                f.write(f'\n[[models]]\nlabel = "{label}"\nserver = "api"\nmodel = "mock"\n')
+
+    def tearDown(self):
+        self.api.stop()
+        self.tmp.cleanup()
+
+    def results(self, label):
+        return json.loads(read(os.path.join(self.ws, "results", label, "apps.json")))["results"]
+
+    def test_hosted_run_beside_local_with_one_slot(self):
+        from tuieval import engine
+        e = engine.Engine(lambda kind, **d: None, root=self.ws)
+        self.assertEqual([e.hosted(e.model(l)) for l in ("loc", "h1", "h2")], [False, True, True])
+        self.assertEqual(e.models_at_once({"loc", "h1", "h2"}, 1, 4), 3)
+        jobs = e.plan(["loc", "h1", "h2"], ["apps"], None, "certify", False, {})
+        e.run(jobs, parallel=1)
+        self.assertEqual([j.status for j in jobs], ["done"] * 3, [(j.key, j.note) for j in jobs])
+        seen = {l: set().union(*(r.get("ran_alongside", []) for r in self.results(l))) for l in ("loc", "h1", "h2")}
+        self.assertEqual(seen["loc"], {"h1", "h2"})          # neither hosted model waited for the local one
+        self.assertIn("h2", seen["h1"])
+
+    def test_local_model_joins_a_hosted_only_run(self):
+        from tuieval import engine
+        events = []
+        e = engine.Engine(lambda kind, **d: events.append(kind), root=self.ws)
+        jobs = e.plan(["h1"], ["apps"], None, "certify", False, {})
+        t = threading.Thread(target=e.run, args=(jobs,))
+        t.start()
+        deadline = time.time() + 120
+        while "request_done" not in events and time.time() < deadline:
+            time.sleep(0.05)
+        added, why = e.add_jobs(e.plan(["loc"], ["apps"], None, "certify", False, {}))
+        self.assertEqual(([j.key for j in added], why), (["loc/apps"], None))
+        t.join(300)
+        self.assertFalse(t.is_alive())
+        self.assertEqual(len(self.results("loc")), 4)
+        self.assertIsNone(e.machine_holder())                # the lock it took is released again
+
+
 class AddToRun(unittest.TestCase):
     """More evals while a run is going: added to it (they run after its other jobs) or queued after it."""
 
@@ -941,6 +1007,10 @@ class AddToRun(unittest.TestCase):
             import asyncio, json, time
             from tuieval.tui import EvalsApp, RunScreen, SetupScreen, ChoiceScreen
             from textual.widgets import SelectionList
+            async def setup(app, pilot):   # a dialog closing can take a moment on a slow machine
+                deadline = time.time() + 30
+                while not isinstance(app.screen, SetupScreen) and time.time() < deadline:
+                    await pilot.pause(0.1)
             async def go():
                 app = EvalsApp({})
                 async with app.run_test(size=(160, 50)) as pilot:
@@ -967,7 +1037,7 @@ class AddToRun(unittest.TestCase):
                     await pilot.pause()
                     asked = isinstance(app.screen, ChoiceScreen)
                     app.screen.dismiss("add")
-                    await pilot.pause()
+                    await setup(app, pilot)
                     app.screen.query_one("#suites", SelectionList).deselect("apps")
                     app.screen.query_one("#suites", SelectionList).select("other")
                     app.screen.selected_models = {"a"}
@@ -975,7 +1045,7 @@ class AddToRun(unittest.TestCase):
                     app.screen.action_start()
                     await pilot.pause()
                     app.screen.dismiss("queue")
-                    await pilot.pause()
+                    await setup(app, pilot)
                     app.screen.query_one("#tier-smoke").value = True    # another tier can't join: queued
                     await pilot.pause()
                     app.screen.action_start()
@@ -1008,6 +1078,76 @@ class AddToRun(unittest.TestCase):
         self.assertEqual(out["second"], [["a/other", "done"]])                  # queued: its own run after
 
 
+    def test_tui_never_queues_the_same_thing_twice(self):
+        """Start while a run is going never adds or queues a model and pack already running or queued."""
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, SetupScreen, ChoiceScreen
+            from textual.widgets import SelectionList
+            async def pick(app, pilot, models, packs=("apps",)):
+                deadline = time.time() + 30
+                while not isinstance(app.screen, SetupScreen) and time.time() < deadline:
+                    await pilot.pause(0.1)
+                s = app.screen
+                for p in ("apps", "other"):
+                    (s.query_one("#suites", SelectionList).select if p in packs
+                     else s.query_one("#suites", SelectionList).deselect)(p)
+                s.selected_models = set(models)
+                s.refresh_models()
+                await pilot.pause()
+                s.action_start()
+                await pilot.pause()
+                return app.screen
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    app.screen.query_one("#tier-certify").value = True
+                    await pick(app, pilot, ["a"])
+                    deadline = time.time() + 180
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run = app.screen
+                    await pilot.press("n")
+                    out = {}
+                    s = await pick(app, pilot, ["a"])                 # the same again: nothing happens
+                    out["same"] = [isinstance(s, ChoiceScreen), len(app.queue)]
+                    s = await pick(app, pilot, ["a", "b"])            # partly new: only b is offered
+                    out["partly"] = [isinstance(s, ChoiceScreen), "Left out" in str(getattr(s, "question", ""))]
+                    s.dismiss("queue")
+                    await pilot.pause()
+                    out["queued"] = [sorted(q.keys()) for q in app.queue]
+                    s = await pick(app, pilot, ["b"])                 # already queued: nothing happens
+                    out["again"] = [isinstance(s, ChoiceScreen), len(app.queue)]
+                    s = await pick(app, pilot, ["a"], ("other",))     # new: offered, queued or started
+                    out["new"] = (isinstance(s, ChoiceScreen) or any("a/other" in q.keys() for q in app.queue)
+                                  or any(j.key == "a/other" for x in app.sessions for j in x.jobs))
+                    if isinstance(s, ChoiceScreen):
+                        s.dismiss("queue")
+                    await pilot.pause()
+                    deadline = time.time() + 300   # the queued runs start one after another: wait for all three
+                    while time.time() < deadline:
+                        await pilot.pause(0.2)
+                        if len(app.sessions) == 3 and not app.queue and not any(x.running for x in app.sessions):
+                            break
+                    out["sessions"] = sorted((j.key, j.status) for x in app.sessions for j in x.jobs)
+                    print(json.dumps(out))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertEqual(out["same"], [False, 0])
+        self.assertEqual(out["partly"], [True, True])
+        self.assertEqual(out["queued"], [["b/apps"]])
+        self.assertEqual(out["again"], [False, 1])
+        self.assertTrue(out["new"])
+        self.assertEqual(out["sessions"], [["a/apps", "done"], ["a/other", "done"], ["b/apps", "done"]])
+
+
 class PTAIndex(unittest.TestCase):
     """The PTA index: parsimony from tokens and speed from time, each relative to the best on a log
     scale, and accuracy, all over the questions every compared model answered."""