tuieval 0.2.0.dev25__tar.gz → 0.2.0.dev32__tar.gz

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
+- More evals during a run: `n` opens Setup, and Start adds them to the run going (same tier) or queues them after it.
 - A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.
 - Less to take in at first: until something has run, the setup screen leaves out context sizes, tuning state and the Tune key; Presets and Hide are off the footer (`?` lists every key); `tuieval help` groups commands into Start here, Day to day and Advanced.
 - Results lead with a plain summary per model ("ready for Support; not decided yet for Coding → run Certify to finish"), in the TUI, `tuieval verdict` (with the command to run next) and the report; the statistics follow.

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev25
+Version: 0.2.0.dev32
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -137,7 +137,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/README.md

@@ -111,7 +111,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/docs/models.md

@@ -71,7 +71,18 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 - **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
 - **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
-- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
+- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4), `gpu_residency_gb` (see the stall guard) and `parallel_models` (see below).
+
+## Several models at a time
+
+By default a run serves one model at a time. On a machine with room for more, set `parallel_models = 2` (or more) under `[machines.<id>]`, type a number in **Models at a time** on the TUI's setup screen (blank uses the machine's setting), or pass `tuieval run --parallel N`.
+
+- Models start in queue order, each in its own lane. The next one waits while its fit-check memory estimate wouldn't fit next to the ones running (without one, e.g. a server that sizes its own memory, the setting decides). Contexts are never shrunk to make room.
+- A second model on the same server gets a free port, so a server's `cmd` must take `{port}`.
+- A server with `before_start` (which may stop other servers) always runs on its own. Tuning is always one model at a time.
+- Answers are judged exactly as in a run of one model. Each answer records the models served while it ran (`ran_alongside`), and a load time measured next to others is marked (`loaded_alongside`), since models side by side share the machine's GPU and memory bandwidth. The speed table and the answer view show it.
+- In the TUI, one model's answers stream at a time: `v` switches to the next, `k` skips the one streaming. `tuieval run` prints one line per answer instead of streaming.
+- Only one run uses the machine at a time across tuieval windows; models at a time applies within a run.
 
 ## Tuning
 
@@ -255,5 +266,6 @@ Optional sections:
 # [machines.m3max-64gb]
 # memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
 # gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
+# parallel_models = 2          # models a run serves at a time (default 1)
 ```
 

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev25'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev25')
+__version__ = version = '0.2.0.dev32'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev32')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/src/tuieval/compare.py

@@ -37,7 +37,8 @@ def _load_native(data):
             "settings": data.get("settings", {}), "load_s": run.get("load_s"),
             "peak_rss_mb": run.get("peak_rss_mb"), "pack": pack["name"], "pack_fp": pack["fingerprint"],
             # hosted (OpenRouter) models: the provider endpoint and its declared quantization
-            "endpoint": run.get("endpoint"), "quantization": run.get("quantization")}
+            "endpoint": run.get("endpoint"), "quantization": run.get("quantization"),
+            "loaded_alongside": run.get("loaded_alongside") or []}
     rows = []
     for r in data["results"]:
         rows.append({
@@ -54,6 +55,8 @@ def _load_native(data):
             "severity": r.get("severity"), "group": r.get("group"), "repeat": r.get("repeat", 0),
             "difficulty": r.get("difficulty", "unrated"),
             "machine": r.get("machine") or run.get("machine"), "prompt_tokens": r.get("prompt_tokens"),
+            # other models served at the same time (parallel_models): its speed was shared with them
+            "alongside": r.get("ran_alongside") or [],
         })
     return rows, info
 
@@ -210,6 +213,13 @@ def timing(r, machine=None, speeds=None):
     return None, None, None, None, None
 
 
+def alongside_note(rows):
+    """' · n/N ran alongside others' when some answers ran while other models were served (their
+    speed was shared with them), else ''."""
+    shared = sum(bool(r.get("alongside")) for r in rows)
+    return f" · {shared}/{len(rows)} ran alongside others" if shared else ""
+
+
 def speed(rows, infos, machine=None, speeds=None):
     """(header, table, frontier): speed and token use per model.
 
@@ -227,7 +237,9 @@ def speed(rows, infos, machine=None, speeds=None):
     load = {}
     for i in infos:
         cur = load.setdefault(i["label"], {"load_s": None, "peak_rss_mb": None, "tags": i["tags"],
-                                           "think": i["settings"].get("enable_thinking"), "served": set()})
+                                           "think": i["settings"].get("enable_thinking"), "served": set(),
+                                           "loaded_alongside": False})
+        cur["loaded_alongside"] |= bool(i.get("loaded_alongside"))
         if i.get("endpoint"):
             q = i.get("quantization")
             cur["served"].add(f"{'undeclared' if q in (None, 'unknown') else q} via {i['endpoint']}")
@@ -247,6 +259,7 @@ def speed(rows, infos, machine=None, speeds=None):
         here = machine is None or machine in where   # load time and memory only where it ran
         if machine is None:
             kind = "measured on " + (", ".join(where) or "?")
+        kind += alongside_note(v)
         table.append([
             "★" if m in frontier else "", m, "; ".join(sorted(info.get("served") or [])) or "-",
             ",".join(info.get("tags") or []),
@@ -256,7 +269,8 @@ def speed(rows, infos, machine=None, speeds=None):
             _fmt(median(x[3] for x in t), ".0f"), _fmt(median(r["tokens"] for r in v), ".0f"),
             _fmt(median(r["reasoning_tokens"] for r in v), ".0f"), _fmt(median(r["answer_tokens"] for r in v), ".0f"),
             f"{sum(r['tokens'] for r in v) / passes:,.0f}" if passes else "-",
-            _fmt(info.get("load_s") if here else None, ".0f"),
+            _fmt(info.get("load_s") if here else None, ".0f")
+            + (" (alongside others)" if here and info.get("load_s") and info.get("loaded_alongside") else ""),
             _fmt(info["peak_rss_mb"] / 1024 if info.get("peak_rss_mb") and here else None, ".1f"),
         ])
     return header, table, frontier

{tuieval-0.2.0.dev25 → tuieval-0.2.0.dev32}/src/tuieval/engine.py

@@ -229,6 +229,7 @@ class Serving:
     identity: dict             # output-affecting settings, part of the results fingerprint
     profile: dict | None = None
     mtp_layers: int = 0
+    need_gb: float | None = None   # estimated GPU memory at ctx (from the fit check); None when unknown
 
 
 def infer_server(cfg, model):
@@ -664,6 +665,20 @@ def holder_text(holder):
             "One eval or tune per machine: this starts when it ends")
 
 
+class Lane:
+    """One model being run: its own request stream, stall reason and skip switch, so models run
+    side by side (parallel_models) stop independently. Runs and tunes in one lane use the engine's
+    main lane."""
+    def __init__(self, label=None):
+        self.label = label
+        self.stream = None
+        self.stall = None         # set by the stall watcher: why this model's server stalled
+        self.skip = threading.Event()
+        self.procs = []
+        self.port = None
+        self.seen = set()         # other models served during this model's current load or request
+
+
 class Engine:
     def __init__(self, on_event=None, root=None, models_path=None, packs_dir=None,
                  results_dir=None, log_dir=None):
@@ -677,11 +692,14 @@ class Engine:
         self.pack_errors = []
         self.packs = packs_mod.load_packs(self.packs_dir, self.pack_errors)
         self._cancel = threading.Event()
-        self._skip = threading.Event()
-        self._stall = None        # set by the stall watcher: why the current server stalled
+        self._main_lane = Lane()
+        self._here = threading.local()   # .lane: the lane this thread runs
+        self._lanes = {}          # label -> Lane of each model being served now
+        self._lanes_lock = threading.Lock()
         self._procs = []
         self._stop_lock = threading.Lock()
-        self._stream = None
+        self._run_lock = threading.Lock()
+        self._run = None          # the run going: {"jobs", "pending", "open", "machine", "cond"} (add_jobs)
         self._header_cache = {}
         self._serving = {}
         self.tuning_dir = os.path.join(root, "tuning")
@@ -690,6 +708,34 @@ class Engine:
         self._catalogues = {}
         self._add_remote_models()
 
+    # ---- the lane this thread runs (see Lane)
+    def _lane(self):
+        return getattr(self._here, "lane", None) or self._main_lane
+
+    def _all_lanes(self):
+        with self._lanes_lock:
+            return [self._main_lane, *self._lanes.values()]
+
+    @property
+    def _stream(self):
+        return self._lane().stream
+
+    @_stream.setter
+    def _stream(self, value):
+        self._lane().stream = value
+
+    @property
+    def _stall(self):
+        return self._lane().stall
+
+    @_stall.setter
+    def _stall(self, value):
+        self._lane().stall = value
+
+    @property
+    def _skip(self):
+        return self._lane().skip
+
     def reload(self):
         self._serving = {}
         self.cfg = load_config(self.models_path)
@@ -1150,6 +1196,30 @@ class Engine:
         """models.toml [machines.<id>] (optional): memory_headroom_gb, name."""
         return self.cfg.get("machines", {}).get(machine_id or self.machine().id, {})
 
+    def parallel_models(self):
+        """How many models a run serves at a time on this machine: models.toml [machines.<id>]
+        parallel_models (default 1)."""
+        try:
+            return max(1, int(self.machine_settings().get("parallel_models", 1)))
+        except (TypeError, ValueError):
+            return 1
+
+    def memory_need_gb(self, m):
+        """GPU memory a model takes while served here: 0 for one we don't start (a hosted API or a
+        server already running), the fit check's estimate for a local GGUF, else None (unknown)."""
+        if not self.cfg["servers"][m["server"]].get("cmd"):
+            return 0.0
+        return self.serving(m).need_gb
+
+    def memory_available_gb(self):
+        """GPU memory models may take here (what the fit check sizes contexts against)."""
+        mc = self.machine()
+        return mc.gpu_limit_gb - self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
+
+    def runs_alone(self, m):
+        """A server whose before_start step may stop other servers never runs next to another model."""
+        return bool(self.cfg["servers"][m["server"]].get("before_start"))
+
     def gpu_residency_gb(self, machine=None):
         """GPU memory this machine's driver keeps resident without churning (see machines.py)."""
         mc = machine or self.machine()
@@ -1180,7 +1250,7 @@ class Engine:
         kv = m.get("kv_type") or server.get("kv_type")
         want = min(x for x in (server.get("max_ctx"), m.get("ctx"), m.get("max_context"), 10**9) if x)
         want = None if want == 10**9 else want
-        ctx, fits, note, mtp, size = want, True, "", 0, None
+        ctx, fits, note, mtp, size, need = want, True, "", 0, None, None
         path = expand(m["model"])
         if server.get("model_is_path") and os.path.isfile(path):
             try:
@@ -1192,7 +1262,7 @@ class Engine:
                     headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
                     f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
                                      want_ctx=want)
-                    ctx, fits, note = f.max_ctx, f.fits, f.note
+                    ctx, fits, note, need = f.max_ctx, f.fits, f.note, f.need_gb
             except (OSError, ValueError) as e:
                 note = f"couldn't read the GGUF header ({e}); context not checked"
         profile, perf, source = None, [], "n/a"
@@ -1221,7 +1291,7 @@ class Engine:
             # resident), so both are part of what a result is valid for.
             identity["machine"] = mc.id
             identity["perf"] = perf
-        sv = Serving(mc, ctx, kv, fits, note, perf, source, identity, profile, mtp)
+        sv = Serving(mc, ctx, kv, fits, note, perf, source, identity, profile, mtp, need)
         self._serving[key] = sv
         return sv
 
@@ -1289,16 +1359,21 @@ class Engine:
     # ---- control
     def cancel(self):
         self._cancel.set()
-        self._abort()
-
-    def skip_model(self):
-        self._skip.set()
-        self._abort()
-
-    def _abort(self):
-        if self._stream:
-            self._stream.close()
-        for p in list(self._procs):
+        self._abort(self._all_lanes(), list(self._procs))
+
+    def skip_model(self, label=None):
+        """Skip a model being run (label), or every one being run now."""
+        with self._lanes_lock:
+            lanes = [self._lanes[label]] if label in self._lanes else list(self._lanes.values()) or [self._main_lane]
+        for lane in lanes:
+            lane.skip.set()
+        self._abort(lanes, [p for lane in lanes for p in lane.procs] if self._lanes else list(self._procs))
+
+    def _abort(self, lanes, procs):
+        for lane in lanes:
+            if lane.stream:
+                lane.stream.close()
+        for p in procs:
             self._stop(p, grace=10)
 
     def _check(self):
@@ -1322,6 +1397,7 @@ class Engine:
         STALL_WINDOW_S, the GPU driver is churning memory: record why and abort the request."""
         mc = self.machine()
         limit = self.gpu_residency_gb(mc)
+        lane = self._lane()
         samples = []   # (time, user, kernel)
 
         def run():
@@ -1343,15 +1419,15 @@ class Engine:
                     continue
                 share = dk / (du + dk)
                 info["kernel_share_max"] = max(info.get("kernel_share_max") or 0, round(share, 2))
-                if share > self.STALL_KERNEL_SHARE and (du + dk) / dt > self.STALL_MIN_BUSY and not self._stall:
+                if share > self.STALL_KERNEL_SHARE and (du + dk) / dt > self.STALL_MIN_BUSY and not lane.stall:
                     over = f"{alloc:.1f} GB of GPU memory allocated; " if alloc else ""
-                    self._stall = (f"{m['label']} stalled in the GPU driver on {mc.id}: {100 * share:.0f}% of its "
+                    lane.stall = (f"{m['label']} stalled in the GPU driver on {mc.id}: {100 * share:.0f}% of its "
                                    f"CPU time went to the kernel for {self.STALL_WINDOW_S}s while the GPU waited "
                                    f"({over}this machine keeps ~{limit:.0f} GB resident). "
                                    "Lower its GPU memory or retune it.")
-                    self.emit("server_stall", label=m["label"], message=self._stall)
-                    if self._stream:
-                        self._stream.close()
+                    self.emit("server_stall", label=m["label"], message=lane.stall)
+                    if lane.stream:
+                        lane.stream.close()
         threading.Thread(target=run, daemon=True).start()
 
     # ---- processes
@@ -1360,6 +1436,7 @@ class Engine:
         p = subprocess.Popen(cmd, cwd=cwd, stdin=subprocess.DEVNULL, stdout=out,
                              stderr=subprocess.STDOUT, start_new_session=True)
         self._procs.append(p)
+        self._lane().procs.append(p)
         return p
 
     def _stop(self, p, grace=30):
@@ -1381,6 +1458,9 @@ class Engine:
                     p.wait()
             if p in self._procs:
                 self._procs.remove(p)
+            for lane in self._all_lanes():
+                if p in lane.procs:
+                    lane.procs.remove(p)
 
     def _tail(self, path, label, stop):
         def run():
@@ -1472,45 +1552,40 @@ class Engine:
         return any(self.cfg["servers"][j.model["server"]].get("cmd") for j in jobs if j.status == "waiting")
 
     # ---- running
-    def run(self, jobs):
-        """Run the jobs (blocking). Jobs of the same model share one server start."""
+    def run(self, jobs, parallel=None):
+        """Run the jobs (blocking). Jobs of the same model share one server start. parallel: how many
+        models are served at a time (default: parallel_models for this machine). add_jobs() can add
+        more while it runs; they run after the ones before them."""
         self._cancel.clear()
         self.emit("queue_started", jobs=jobs)
         models = sorted({j.label for j in jobs if j.status == "waiting"})
+        machine = bool(jobs) and self.needs_machine(jobs)
         lock = (self.machine_lock(f"{jobs[0].tier} run: {', '.join(models[:3])}"
                                   + (f" +{len(models) - 3}" if len(models) > 3 else ""))
-                if jobs and self.needs_machine(jobs) else contextlib.nullcontext())
+                if machine else contextlib.nullcontext())
+        order = []
+        for j in jobs:
+            if j.label not in order:
+                order.append(j.label)
+        groups = [(label, [j for j in jobs if j.label == label and j.status == "waiting"]) for label in order]
+        with self._run_lock:
+            self._run = {"jobs": jobs, "pending": [g for g in groups if g[1]], "open": True, "machine": machine,
+                         "cond": threading.Condition(self._run_lock)}
         try:
             with lock:
-                order = []
-                for j in jobs:
-                    if j.label not in order:
-                        order.append(j.label)
-                for label in order:
-                    mine = [j for j in jobs if j.label == label and j.status == "waiting"]
-                    if not mine:
-                        continue
-                    self._skip.clear()
-                    try:
-                        try:
-                            self._run_model(mine)
-                        except (Cancelled, ModelFailed):
-                            raise
-                        except Exception as e:  # never let one model's surprise kill the whole queue
-                            raise ModelFailed(f"unexpected error: {e!r}") from e
-                    except ModelFailed as e:
-                        for j in mine:
-                            if j.status in ("waiting", "loading", "running"):
-                                j.status = "skipped" if "by user" in str(e) else "failed"
-                                j.note, j.finished = str(e).splitlines()[0], time.time()
-                                self.emit("job_done", job=j)
-                        self.emit("model_failed", label=label, message=str(e))
+                self._run_lanes(max(1, int(parallel or self.parallel_models())))
+            if self._cancel.is_set():
+                raise Cancelled()
         except Cancelled:
+            with self._run_lock:
+                self._run["open"] = False
             for j in jobs:
                 if j.status in ("waiting", "loading", "running"):
                     j.status, j.note = "skipped", "cancelled (progress kept)" if j.done else "cancelled"
                     self.emit("job_done", job=j)
         finally:
+            with self._run_lock:
+                self._run["open"] = False
             changes = []
             if any(j.status == "done" and j.started and j.tier != "smoke" for j in jobs):
                 try:
@@ -1522,6 +1597,145 @@ class Engine:
                 self.emit("verdicts", changes=changes)
             self.emit("queue_done", jobs=jobs, cancelled=self._cancel.is_set())
 
+    def add_jobs(self, new):
+        """Add planned jobs to the run going on, after its other jobs. Returns (added, why not): a
+        pack already waiting or running in it isn't added twice, and nothing is added once the run is
+        finishing or cancelled, or when it holds no machine lock (hosted only) and these need one.
+        Runs on the caller's thread, so the caller shows what was added (no event)."""
+        with self._run_lock:
+            r = self._run
+            if not r or not r["open"] or self._cancel.is_set():
+                return [], "the run is finishing"
+            if not r["machine"] and self.needs_machine(new):
+                return [], "the run going uses only hosted models, and these need this machine"
+            busy = {j.key for j in r["jobs"] if j.status in ("waiting", "loading", "running")}
+            added = [j for j in new if j.status == "waiting" and j.key not in busy]
+            r["jobs"].extend(added)
+            for j in added:   # a model still waiting takes them on; otherwise it comes again at the end
+                group = next((g for g in r["pending"] if g[0] == j.label), None)
+                if group:
+                    group[1].append(j)
+                else:
+                    r["pending"].append((j.label, [j]))
+            r["cond"].notify_all()
+        return added, None if added else "everything picked is already in the run"
+
+    def _run_lanes(self, parallel):
+        """Run each model's jobs in its own lane, up to `parallel` at a time, in order: the next
+        model starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
+        next to the ones running (memory_need_gb; an unknown need is left to the setting). Lanes
+        wait for jobs added meanwhile (add_jobs) until every model is done."""
+        pending, running, told = self._run["pending"], {}, set()
+        cond = self._run["cond"]
+
+        def blocked(label):
+            """Why the next model can't start now (None: it can)."""
+            if not running:
+                return None
+            if len(running) >= parallel:
+                return ""
+            m = self.model(label)
+            if self.runs_alone(m) or any(self.runs_alone(self.model(l)) for l in running):
+                alone = label if self.runs_alone(m) else next(l for l in running if self.runs_alone(self.model(l)))
+                return (f"{alone}'s server runs a before_start step that may stop other servers, so it runs "
+                        "on its own")
+            need = self.memory_need_gb(m)
+            used = sum(n for n in running.values() if n)
+            room = self.memory_available_gb()
+            if need and used + need > room:
+                return (f"needs ~{need:.0f} GB; {', '.join(running)} take ~{used:.0f} of {room:.0f} GB, so it "
+                        "starts when there's room")
+            return None
+
+        def lane():
+            while True:
+                mine, note = None, None
+                with cond:   # events go out after the lock is released: the UI may be waiting for it (add_jobs)
+                    if self._cancel.is_set() or not pending and not running:
+                        self._run["open"] = False   # nothing more can be added
+                        cond.notify_all()
+                        return
+                    # in order, except that a model added again while it runs waits for itself to end
+                    # without holding up the ones behind it
+                    nxt = next((i for i, g in enumerate(pending) if g[0] not in running), None)
+                    label = pending[nxt][0] if nxt is not None else None
+                    why = blocked(label) if label else ""   # none can start: a model still runs, more may come
+                    if why is None:
+                        label, mine = pending.pop(nxt)
+                        running[label] = self.memory_need_gb(self.model(label))
+                    elif why and label not in told:
+                        told.add(label)
+                        note = f"{label} waits: {why}"
+                    else:
+                        cond.wait(1.0)
+                if note:
+                    self.emit("model_waiting", label=label, message=note)
+                if mine is None:
+                    continue
+                try:
+                    self._run_lane(label, mine)
+                finally:
+                    with cond:
+                        running.pop(label, None)
+                        cond.notify_all()
+
+        if parallel == 1:
+            lane()
+            return
+        threads = [threading.Thread(target=lane, name=f"lane-{i + 1}") for i in range(parallel)]
+        for t in threads:
+            t.start()
+        for t in threads:
+            t.join()
+
+    def _run_lane(self, label, mine):
+        """One model's jobs, in this thread's own lane. Other models served meanwhile are noted on
+        each answer (ran_alongside), since they share the machine's GPU and memory bandwidth."""
+        lane = Lane(label)
+        m = mine[0].model
+        with self._lanes_lock:
+            for other in self._lanes.values():
+                other.seen.add(label)
+            lane.seen = set(self._lanes)
+            if self.cfg["servers"][m["server"]].get("cmd"):
+                lane.port = self._lane_port(m)
+            self._lanes[label] = lane
+        self._here.lane = lane
+        try:
+            try:
+                self._run_model(mine)
+            except (Cancelled, ModelFailed):
+                raise
+            except Exception as e:  # never let one model's surprise kill the whole queue
+                raise ModelFailed(f"unexpected error: {e!r}") from e
+        except ModelFailed as e:
+            for j in mine:
+                if j.status in ("waiting", "loading", "running"):
+                    j.status = "skipped" if "by user" in str(e) else "failed"
+                    j.note, j.finished = str(e).splitlines()[0], time.time()
+                    self.emit("job_done", job=j)
+            self.emit("model_failed", label=label, message=str(e))
+        except Cancelled:
+            pass   # run() marks what's left
+        finally:
+            self._here.lane = None
+            with self._lanes_lock:
+                self._lanes.pop(label, None)
+
+    def _lane_port(self, m):
+        """The server's port, or a free one when another model being served has it (call with
+        _lanes_lock held)."""
+        port = self.cfg["servers"][m["server"]].get("port")
+        if port is None or not any(l.port == port for l in self._lanes.values()):
+            return port
+        taken = {l.port for l in self._lanes.values()}
+        while True:
+            with socket.socket() as s:
+                s.bind(("127.0.0.1", 0))
+                free = s.getsockname()[1]
+            if free not in taken:
+                return free
+
     @contextlib.contextmanager
     def serve(self, m, perf_args=None, port=None, log_name=None):
         """Start a model's server (or use the running one), wait until it serves the expected name,
@@ -1649,11 +1863,16 @@ class Engine:
         for j in jobs:
             j.status = "loading"
             self.emit("job_update", job=j)
-        with self.serve(m) as (base_url, info):
+        lane = self._lane()
+        with self.serve(m, port=lane.port) as (base_url, info):
+            with self._lanes_lock:
+                loaded_alongside = sorted(lane.seen)   # served while this one loaded: its load time is shared
+
             def run_info():
                 return {"load_s": info["load_s"], "peak_rss_mb": info["peak_mb"](),
                         "server_facts": info["facts"] or None, "gpu_peak_gb": info.get("gpu_peak_gb"),
-                        "kernel_share_max": info.get("kernel_share_max"), **serving_info}
+                        "kernel_share_max": info.get("kernel_share_max"),
+                        **({"loaded_alongside": loaded_alongside} if loaded_alongside else {}), **serving_info}
             failed_in_row = 0
             for j in jobs:
                 self._check()
@@ -1736,7 +1955,7 @@ class Engine:
         return tests
 
     # ---- sittings: a pack run over several sessions (stopped and resumed) keeps its total time
-    SITTING_KEYS = ("load_s", "peak_rss_mb", "gpu_peak_gb", "kernel_share_max", "machine")
+    SITTING_KEYS = ("load_s", "peak_rss_mb", "gpu_peak_gb", "kernel_share_max", "machine", "loaded_alongside")
 
     def _sitting(self, job, info):
         now = time.time()
@@ -1811,6 +2030,9 @@ class Engine:
         for k in ("peak_rss_mb", "gpu_peak_gb", "kernel_share_max"):
             if peak(k) is not None:
                 out[k] = peak(k)
+        along = sorted({l for s in sittings for l in s.get("loaded_alongside") or []})
+        if along:
+            out["loaded_alongside"] = along
         return out
 
     def _run_job(self, job, base_url, run_info, alive=lambda: True):
@@ -1862,6 +2084,9 @@ class Engine:
                 if (test["id"], rep) in done_keys:
                     continue
                 self._check()
+                lane = self._lane()
+                with self._lanes_lock:
+                    lane.seen = set(self._lanes) - {job.label}
                 self.emit("request_started", job=job, test=test["description"], test_id=test["id"],
                           has_image=bool(test.get("image")), repeat=rep, difficulty=test.get("difficulty", "unrated"))
                 waits = list(self.SERVER_RETRY_WAITS_S)
@@ -1870,7 +2095,7 @@ class Engine:
                     try:
                         body = self._body(job, test, rep)
                         res = client.stream_chat(base_url, body,
-                                                 lambda kind, text: self.emit("delta", stream=kind, text=text),
+                                                 lambda kind, text: self.emit("delta", stream=kind, text=text, label=job.label),
                                                  timeout=timeout, stream=self._stream, headers=headers)
                     except client.Cancelled:
                         self._check()
@@ -1917,6 +2142,8 @@ class Engine:
                        **({"checks": g["checks"]} if "checks" in g else {}),
                        "machine": self.machine().id,
                        **({"quantization": ep.get("quantization")} if ep else {}),
+                       # other models served while this answer ran: they shared the machine, so its speed did too
+                       **({"ran_alongside": sorted(lane.seen)} if lane.seen else {}),
                        "sent": "+".join(msg["role"] for msg in body["messages"]),  # proof: single turn
                        **({"seed": body["seed"]} if "seed" in body else {}), **meta}
                 # The same long answer twice for one question, with sampling on, means a cached response