tuieval 0.2.0.dev99__tar.gz → 0.2.0.dev103__tar.gz

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## Unreleased
 
+- Run screen: `x` removes the highlighted pack from the run going if it hasn't started (the reverse of adding it); a model left with nothing isn't started.
+- `k` (skip model) asks first, naming the model. Yes/no dialogs start on No; `y` answers yes, `n` or Esc no.
+- Fix: hosted models no longer count as running alongside local ones (they don't share the machine), so a one-model-at-a-time setup no longer gets the "ran alongside other models" note. Older results are read the same way.
 - Results, Per question: the model summary shows every picked model without scrolling, and fits the window: shorter column names, and long model names cut in the middle (both ends kept). The PTA index table fits the same way.
 - PTA index: the accuracy bar shows answers cut off at max_tokens in orange, with `(n% truncated)` beside it and a **truncated** column in the table, so overthinking stands apart from wrong answers.
 - Results → Insights (also `tuieval insights`): charts with what to do next: every model against a reference (95% CI), the best model per use case, fine-tunes against their base (`base` in models.toml), why models fail, accuracy against tokens or time, question health, and the Certify runs that would settle the strongest models' verdicts. `[defaults] reference` sets the default reference.

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev99
+Version: 0.2.0.dev103
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -143,7 +143,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model (after a yes), `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run. `x` takes a pack that hasn't started out of the run; `w` lists queued runs (`del` removes one).
 4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures; click a column name to sort. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/README.md

@@ -117,7 +117,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 
 1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results). To run only some tests of a pack, highlight it and press `e`. Runs of different tests add up, and the pack can PASS once all its tests have run.
 2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
-3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model (after a yes), `c` cancels (finished work is kept and resumes next time). `n` sets up more evals while it runs: add them to this run (same tier) or queue them as the next run. `x` takes a pack that hasn't started out of the run; `w` lists queued runs (`del` removes one).
 4. **Press `r` for results:** the PTA index (parsimony (tokens), time and accuracy as bars), production readiness, per question, and failures; click a column name to sort. `tuieval compare`, `tuieval history` and `tuieval items` have the rest: whether a difference is real, verdict history and test quality.
 
 Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/docs/models.md

@@ -94,7 +94,7 @@ By default a run serves one model at a time. On a machine with room for more, se
 - Models start in queue order, each in its own lane. The next one waits while its fit-check memory estimate wouldn't fit next to the ones running (without one, e.g. a server that sizes its own memory, the setting decides). Contexts are never shrunk to make room.
 - A second model on the same server gets a free port, so a server's `cmd` must take `{port}`.
 - A server with `before_start` (which may stop other servers) always runs on its own. Tuning is always one model at a time.
-- Answers are judged exactly as in a run of one model. Each answer records the models served while it ran (`ran_alongside`), and a load time measured next to others is marked (`loaded_alongside`), since models side by side share the machine's GPU and memory bandwidth. The speed table and the answer view show it.
+- Answers are judged exactly as in a run of one model. Each answer records the models served while it ran (`ran_alongside`), and a load time measured next to others is marked (`loaded_alongside`), since models side by side share the machine's GPU and memory bandwidth. Only local models count: hosted ones share nothing of the machine. The speed table and the answer view show it.
 - In the TUI, one model's answers stream at a time: `v` switches to the next, `k` skips the one streaming. `tuieval run` prints one line per answer instead of streaming.
 - Only one run uses the machine at a time across tuieval windows; models at a time applies within a run.
 

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev99'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev99')
+__version__ = version = '0.2.0.dev103'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev103')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/src/tuieval/compare.py

@@ -41,6 +41,7 @@ def _load_native(data):
             "pack": pack["name"], "pack_fp": pack["fingerprint"],
             # hosted (OpenRouter) models: the provider endpoint and its declared quantization
             "endpoint": run.get("endpoint"), "quantization": run.get("quantization"),
+            "hosted": bool(run.get("hosted") or run.get("endpoint")),
             "loaded_alongside": run.get("loaded_alongside") or []}
     rows = []
     for r in data["results"]:
@@ -78,9 +79,37 @@ def load(paths):
             row.update(raw=rec, path=path, pack_fp=data["pack"]["fingerprint"])
         rows += r
         infos.append(i)
+    # older results also noted hosted models alongside, which share nothing of this machine: a hosted
+    # model's rows note no one, and no one notes it
+    labels, prefixes = configured_hosted(paths)
+    labels |= {i["label"] for i in infos if i["hosted"]}
+
+    def local(along):
+        return [l for l in along if l not in labels and not l.startswith(tuple(prefixes))]
+    for row in rows:
+        row["alongside"] = [] if row["model"] in labels else local(row["alongside"])
+    for i in infos:
+        i["loaded_alongside"] = [] if i["label"] in labels else local(i["loaded_alongside"])
     return rows, infos
 
 
+def configured_hosted(paths):
+    """(labels, label prefixes) models.toml says are hosted, in the workspaces these results/<label>/
+    <pack>.json are from: its hosted models, and the prefix of models named at run time on a hosted
+    server (e.g. "or-"). Covers hosted models whose results aren't loaded."""
+    from . import engine
+    labels, prefixes = set(), set()
+    for root in {os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(p)))) for p in paths}:
+        try:
+            cfg = engine.load_config(os.path.join(root, "models.toml"))
+        except Exception:
+            continue    # no readable models.toml there: the results' own marks still apply
+        labels |= {m["label"] for m in cfg["models"] if engine.hosted(cfg, m)}
+        prefixes |= {engine.remote_label(n, s, "") for n, s in cfg["servers"].items()
+                     if s.get("any_model") and engine.hosted(cfg, {"server": n})}
+    return labels, prefixes
+
+
 def default_paths(results_dir=None):
     """results/<label>/<pack>.json, excluding smoke runs and the archive."""
     results_dir = results_dir or workspace.path("results")

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/src/tuieval/engine.py

@@ -83,6 +83,21 @@ def builtin_config():
         return tomllib.load(f)
 
 
+def hosted(cfg, m):
+    """A model served by an API elsewhere: tuieval doesn't start its server and its url isn't this
+    machine (or its server sets hosted = true). A server you started yourself on localhost still
+    counts as local (it uses the GPU)."""
+    if m.get("remote"):
+        return True
+    server = cfg["servers"][m["server"]]
+    if "hosted" in server:   # say so for a server the url doesn't give away (e.g. a local proxy to an API)
+        return bool(server["hosted"])
+    if server.get("cmd"):
+        return False
+    host = urllib.parse.urlparse(server.get("url") or "").hostname or ""
+    return bool(host) and host not in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+
+
 def load_config(path):
     """Read models.toml, fill in the built-ins it leaves out (builtin.toml) and each model's defaults.
     [defaults] are filled key by key; [sampling] only when models.toml has none (sampling is part of
@@ -442,6 +457,7 @@ class Job:
     earlier: list = dataclasses.field(default_factory=list)  # earlier sittings (see Engine._sittings)
     stop_early: bool = False  # stop once the pack's FAIL is certain (verdict.fail_certain)
     first: list = dataclasses.field(default_factory=list)   # test ids round 1 asks first (most failed)
+    claimed: bool = False   # its model's server took it on: it can no longer be removed from the run
 
     @property
     def label(self):
@@ -675,8 +691,9 @@ class Lane:
     """One model being run: its own request stream, stall reason and skip switch, so models run
     side by side (parallel_models) stop independently. Runs and tunes in one lane use the engine's
     main lane."""
-    def __init__(self, label=None):
+    def __init__(self, label=None, hosted=False):
         self.label = label
+        self.hosted = hosted      # a hosted model shares nothing of this machine, so it's never noted alongside
         self.stream = None
         self.stall = None         # set by the stall watcher: why this model's server stalled
         self.skip = threading.Event()
@@ -1282,18 +1299,7 @@ class Engine:
                 + min(parallel_hosted or self.parallel_hosted(), hosted))
 
     def hosted(self, m):
-        """A model served by an API elsewhere: tuieval doesn't start its server and its url isn't this
-        machine (or its server sets hosted = true). A server you started yourself on localhost still
-        counts as local (it uses the GPU)."""
-        if m.get("remote"):
-            return True
-        server = self.cfg["servers"][m["server"]]
-        if "hosted" in server:   # say so for a server the url doesn't give away (e.g. a local proxy to an API)
-            return bool(server["hosted"])
-        if server.get("cmd"):
-            return False
-        host = urllib.parse.urlparse(server.get("url") or "").hostname or ""
-        return bool(host) and host not in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+        return hosted(self.cfg, m)
 
     def memory_need_gb(self, m):
         """GPU memory a model takes while served here: 0 for one we don't start (a hosted API or a
@@ -1723,6 +1729,33 @@ class Engine:
             r["cond"].notify_all()
         return added, None if added else "everything picked is already in the run"
 
+    def removable(self, job):
+        """Why a job of the run going can't be removed (None: it can): only a pack not yet started."""
+        if job.status in ("done", "failed", "skipped"):
+            return "it has finished"
+        if job.claimed or job.status == "running":
+            return "it's running (k skips the model)"
+        return None
+
+    def remove_jobs(self, keys):
+        """Take jobs not yet started (removable) out of the run going, the reverse of add_jobs: they never
+        run, and a model left with none isn't started. Returns the jobs removed."""
+        removed = []
+        with self._run_lock:
+            r = self._run
+            if not r or not r["open"]:
+                return removed
+            for j in [j for j in r["jobs"] if j.key in keys and self.removable(j) is None]:
+                j.status, j.note = "removed", "removed from the run"
+                r["jobs"].remove(j)
+                for _, group in r["pending"]:
+                    if j in group:
+                        group.remove(j)
+                removed.append(j)
+            r["pending"][:] = [g for g in r["pending"] if g[1]]   # the lanes hold this very list
+            r["cond"].notify_all()
+        return removed
+
     def _run_lanes(self, parallel, parallel_hosted=4):
         """Run each model's jobs in its own lane: up to `parallel` local models at a time, in order (the
         next one starts once a lane is free, nothing running stops it (runs_alone), and its memory fits
@@ -1817,15 +1850,23 @@ class Engine:
         for t in threads:
             t.join()
 
+    def _local_lanes(self, lane, exclude=None):
+        """Labels of the local models served now (call with _lanes_lock held); none for a hosted lane."""
+        if lane.hosted:
+            return set()
+        return {l for l, o in self._lanes.items() if not o.hosted} - {exclude}
+
     def _run_lane(self, label, mine):
         """One model's jobs, in this thread's own lane. Other models served meanwhile are noted on
-        each answer (ran_alongside), since they share the machine's GPU and memory bandwidth."""
-        lane = Lane(label)
+        each answer (ran_alongside), since they share the machine's GPU and memory bandwidth. Hosted
+        models share neither, so they note nothing and nobody notes them."""
         m = mine[0].model
+        lane = Lane(label, self.hosted(m))
         with self._lanes_lock:
-            for other in self._lanes.values():
-                other.seen.add(label)
-            lane.seen = set(self._lanes)
+            if not lane.hosted:
+                for other in self._lanes.values():
+                    other.seen.add(label)
+            lane.seen = self._local_lanes(lane)
             if self.cfg["servers"][m["server"]].get("cmd"):
                 lane.port = self._lane_port(m)
             self._lanes[label] = lane
@@ -1987,6 +2028,8 @@ class Engine:
                         "kv_type": sv.kv_type, "perf_args": sv.perf_args if local else None,
                         "perf_source": sv.perf_source, "server_version": self.server_version(m) if local else None}
         ep = self.endpoint(m)
+        if self.hosted(m):
+            serving_info["hosted"] = True
         if ep:
             serving_info.update(endpoint=ep["tag"], provider=ep.get("provider"), quantization=ep.get("quantization"))
         for j in jobs:
@@ -2006,6 +2049,10 @@ class Engine:
             failed_in_row = 0
             for j in jobs:
                 self._check()
+                with self._run_lock:
+                    if j.status == "removed":   # taken out of the run while its model ran another pack
+                        continue
+                    j.claimed = True
                 try:
                     self._run_job(j, base_url, run_info, alive=info["alive"])
                 except BaseException as ex:
@@ -2244,7 +2291,7 @@ class Engine:
                 self._check()
                 lane = self._lane()
                 with self._lanes_lock:
-                    lane.seen = set(self._lanes) - {job.label}
+                    lane.seen = self._local_lanes(lane, job.label)
                 self.emit("request_started", job=job, test=test["description"], test_id=test["id"],
                           has_image=bool(test.get("image")), repeat=rep, difficulty=test.get("difficulty", "unrated"))
                 waits = list(self.SERVER_RETRY_WAITS_S)

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/src/tuieval/tui.py

@@ -1828,16 +1828,33 @@ class RetestScreen(ModalScreen):
 
 
 class ConfirmScreen(ModalScreen):
-    def __init__(self, question):
+    """Yes or no; No has the focus, so Enter by reflex changes nothing. y: yes; n or Esc: no
+    (and no_keys, e.g. the key that opened it, pressed again)."""
+    BINDINGS = [Binding("y", "answer(True)", "Yes", show=False), Binding("n", "answer(False)", "No", show=False),
+                Binding("escape", "answer(False)", "No", show=False)]
+
+    def __init__(self, question, no_keys=()):
         super().__init__()
         self.question = question
+        self.no_keys = no_keys
 
     def compose(self) -> ComposeResult:
         with Vertical(id="dialog"):
             yield Label(self.question)
             with Horizontal(classes="dialog-buttons"):
-                yield Button("Yes", id="yes", variant="error")
-                yield Button("No", id="no")
+                yield Button("Yes (y)", id="yes", variant="error")
+                yield Button("No (n)", id="no")
+
+    def on_mount(self):
+        self.query_one("#no", Button).focus()
+
+    def on_key(self, event):
+        if event.key in self.no_keys:
+            event.stop()
+            self.dismiss(False)
+
+    def action_answer(self, yes):
+        self.dismiss(yes)
 
     @on(Button.Pressed)
     def answer(self, event):
@@ -2114,6 +2131,7 @@ class RunScreen(Screen):
     # priority: these work whichever pane has focus (the text panes would otherwise swallow keys)
     BINDINGS = [
         Binding("k", "skip_model", "Skip model", priority=True),
+        Binding("x", "remove_job", "Remove from run", priority=True),
         Binding("c", "cancel_all", "Cancel all", priority=True),
         Binding("f", "toggle_follow", "Pause scroll", priority=True),
         Binding("v", "watch_next", "Next model", priority=True),
@@ -2124,7 +2142,7 @@ class RunScreen(Screen):
         Binding("q", "back", "Back", show=False, priority=True),
     ]
     kind = "Run"
-    LIVE_ONLY = {"skip_model", "cancel_all", "toggle_follow"}
+    LIVE_ONLY = {"skip_model", "remove_job", "cancel_all", "toggle_follow"}
     DONE_ONLY = set()
 
     def __init__(self, jobs, parallel=None):
@@ -2180,6 +2198,8 @@ class RunScreen(Screen):
     def check_action(self, action, parameters):
         if action == "watch_next":
             return self.running and self.side_by_side
+        if action == "remove_job":   # only while something is left to remove
+            return self.running and any(self.app.engine.removable(j) is None for j in self.jobs)
         if action in self.LIVE_ONLY:
             return self.running
         if action in self.DONE_ONLY:
@@ -2219,7 +2239,7 @@ class RunScreen(Screen):
         with Vertical(id="top"):
             yield Static(id="overall")
             yield ProgressBar(id="bar", show_eta=False)
-            yield DataTable(id="queue", cursor_type="none", zebra_stripes=True)
+            yield DataTable(id="queue", cursor_type="row", zebra_stripes=True)
         yield Static(id="current")
         with Horizontal(id="done-bar"):
             yield Button("New run  [n]", id="done-new", variant="success")
@@ -2365,10 +2385,12 @@ class RunScreen(Screen):
                 self.total += j.total
             self.query_one("#bar", ProgressBar).update(total=max(self.total, 1))
             self.event(f"added to this run: {', '.join(j.key for j in d['jobs'])}")
+            self.refresh_bindings()
             self.update_overall()
         elif kind in ("job_started", "job_update", "job_done"):
             job = d["job"]
             self.update_job(job)
+            self.refresh_bindings()   # x shows while a pack can still be removed
             if kind == "job_started" and d.get("resumed"):
                 earlier = (f" ({fmt_secs(d['earlier_s'])} over {d['sittings']} earlier sitting(s))"
                            if d.get("sittings") else "")
@@ -2611,13 +2633,63 @@ class RunScreen(Screen):
 
     # -- actions
     def action_skip_model(self):
-        if self.running:
-            if self.side_by_side and self.watch:
-                self.app.engine.skip_model(self.watch)
-                self.notify(f"Skipping {self.watch}; the others keep going…")
+        if not self.running:
+            return
+        one = self.watch if self.side_by_side and self.watch else None
+        names = [one] if one else self.active()
+        if not names:
+            return
+
+        def skip(yes):
+            if not yes or not self.running:
+                return
+            if one:
+                self.app.engine.skip_model(one)
+                self.notify(f"Skipping {one}; the others keep going…")
             else:
                 self.app.engine.skip_model()
                 self.notify("Skipping the current model…")
+        self.app.push_screen(ConfirmScreen(
+            f"Skip {', '.join(names)}? Answers so far are kept (a later run picks up from them); the rest "
+            f"of {'its' if len(names) == 1 else 'their'} questions in this run are skipped.", no_keys=("k",)), skip)
+
+    def action_remove_job(self):
+        """Take the highlighted pack out of the run if it hasn't started (the reverse of adding it)."""
+        if not self.running:
+            return
+        q = self.query_one("#queue", DataTable)
+        try:
+            key = q.coordinate_to_cell_key(q.cursor_coordinate).row_key.value
+        except Exception:
+            key = None
+        job = next((j for j in self.jobs if j.key == key), None)
+        if job is None:
+            self.notify("Highlight a pack in the queue (click it or use ↑/↓ there), then press x.")
+            return
+        why = self.app.engine.removable(job)
+        if why:
+            self.notify(f"{job.label} · {job.pack.label} can't be removed: {why}.")
+            return
+
+        def remove(yes):
+            if not yes:
+                return
+            removed = self.app.engine.remove_jobs({job.key})
+            if not removed:
+                self.notify(f"{job.label} · {job.pack.label} started meanwhile, so it stays (k skips the model).")
+                return
+            for j in removed:
+                q.remove_row(j.key)
+                self.total -= j.total
+            self.query_one("#bar", ProgressBar).update(total=max(self.total, 1))
+            self.event(f"removed from this run: {', '.join(j.key for j in removed)}")
+            self.update_overall()
+            self.refresh_bindings()
+            self.app.refresh_live()
+            self.notify(f"Removed from the run: {job.label} · {job.pack.label}.")
+        self.app.push_screen(ConfirmScreen(
+            f"Remove {job.label} · {job.pack.label} from this run? It hasn't started, so nothing is lost; "
+            "Start can add it again later.", no_keys=("x",)), remove)
 
     def action_watch_next(self):
         """Stream the next model running alongside (from its next answer)."""
@@ -3489,7 +3561,7 @@ class EvalsApp(App):
             return
         run.handle("jobs_added", {"jobs": added})
         models = sorted({j.label for j in added})
-        self.notify(f"Added {len(added)} pack run(s) to the run going: {', '.join(models)}. w shows it.", timeout=10)
+        self.notify(f"Added {len(added)} pack run(s) to the run going: {', '.join(models)}. x removes a pack not started yet.", timeout=10)
         self.refresh_live()
 
     def enqueue(self, item):

{tuieval-0.2.0.dev99 → tuieval-0.2.0.dev103}/tests/test_tuieval.py

@@ -893,15 +893,42 @@ class HostedBeside(unittest.TestCase):
 
     def test_hosted_run_beside_local_with_one_slot(self):
         from tuieval import engine
-        e = engine.Engine(lambda kind, **d: None, root=self.ws)
+        together = set()
+
+        def on_event(kind, **d):
+            if kind == "request_started":
+                together.add(frozenset(e._lanes))
+        e = engine.Engine(on_event, root=self.ws)
         self.assertEqual([e.hosted(e.model(l)) for l in ("loc", "h1", "h2")], [False, True, True])
         self.assertEqual(e.models_at_once({"loc", "h1", "h2"}, 1, 4), 3)
         jobs = e.plan(["loc", "h1", "h2"], ["apps"], None, "certify", False, {})
         e.run(jobs, parallel=1)
         self.assertEqual([j.status for j in jobs], ["done"] * 3, [(j.key, j.note) for j in jobs])
-        seen = {l: set().union(*(r.get("ran_alongside", []) for r in self.results(l))) for l in ("loc", "h1", "h2")}
-        self.assertEqual(seen["loc"], {"h1", "h2"})          # neither hosted model waited for the local one
-        self.assertIn("h2", seen["h1"])
+        self.assertIn(frozenset({"loc", "h1", "h2"}), together)   # neither hosted model waited for the local one
+        # hosted models share nothing of this machine: no one notes them alongside, and they note no one
+        for label in ("loc", "h1", "h2"):
+            self.assertFalse(any("ran_alongside" in r for r in self.results(label)), label)
+        data = json.loads(read(os.path.join(self.ws, "results", "h1", "apps.json")))
+        self.assertTrue(data["run"]["hosted"])
+        self.assertNotIn("loaded_alongside", json.loads(read(os.path.join(self.ws, "results", "loc", "apps.json")))["run"])
+
+    def test_older_results_drop_hosted_models_alongside(self):
+        from tuieval import compare
+        paths = []
+        # h1's results say it's hosted; or-gone (named at run time, results deleted) by its server's prefix
+        for label, run, along in (("loc", {}, ["h1", "loc2", "or-gone"]), ("h1", {"endpoint": "x/fp8"}, ["loc"])):
+            data = {"format": "evals/1", "model": {"label": label}, "pack": {"name": "p", "fingerprint": "f"},
+                    "run": {**run, "loaded_alongside": along}, "settings": {},
+                    "results": [{"test": "t", "description": "d", "pass": True, "score": 1.0, "reason": "",
+                                 "ran_alongside": along}]}
+            path = os.path.join(self.ws, "results", label, "p.json")
+            os.makedirs(os.path.dirname(path), exist_ok=True)
+            with open(path, "w") as f:
+                json.dump(data, f)
+            paths.append(path)
+        rows, infos = compare.load(paths)
+        self.assertEqual({r["model"]: r["alongside"] for r in rows}, {"loc": ["loc2"], "h1": []})
+        self.assertEqual({i["label"]: i["loaded_alongside"] for i in infos}, {"loc": ["loc2"], "h1": []})
 
     def test_local_model_joins_a_hosted_only_run(self):
         from tuieval import engine
@@ -1003,6 +1030,141 @@ class AddToRun(unittest.TestCase):
         loads = [d["label"] for k, d in events if k == "model_loading"]
         self.assertEqual(loads.count("a"), 2)                                 # a's new pack: after it ended
 
+    def test_engine_removes_jobs_not_started(self):
+        from tuieval import engine
+        events = []
+        e = engine.Engine(lambda kind, **d: events.append((kind, d)), root=self.ws)
+        jobs = e.plan(["a"], ["apps", "other"], None, "certify", False, {})
+        t = threading.Thread(target=e.run, args=(jobs, 1))
+        t.start()
+        deadline = time.time() + 180   # CI machines can take long to start servers
+        while not any(k == "request_done" for k, _ in events) and time.time() < deadline:
+            time.sleep(0.05)
+        added, _ = e.add_jobs(e.plan(["b"], ["apps"], None, "certify", False, {}))
+        self.assertEqual([j.key for j in added], ["b/apps"])
+        self.assertEqual(e.remove_jobs({"a/apps"}), [])                      # answering: k skips it instead
+        # a's next pack (its server is up) and b (never started) both go
+        self.assertEqual(sorted(j.key for j in e.remove_jobs({"a/other", "b/apps"})), ["a/other", "b/apps"])
+        self.finish(e, t)
+        self.assertEqual([(j.key, j.status) for j in jobs], [("a/apps", "done")])
+        self.assertNotIn(("model_loading", "b"), [(k, d.get("label")) for k, d in events])
+        self.assertFalse(os.path.exists(os.path.join(self.ws, "results", "a", "other.json")))
+        self.assertEqual(e.remove_jobs({"a/apps"}), [])                      # the run is over
+
+    def test_tui_removes_a_pack_from_the_run(self):
+        path = os.path.join(self.ws, "models.toml")
+        write(path, read(path).replace('"--delay", "0.2"', '"--delay", "3"'))   # still running while it's asked
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, ConfirmScreen
+            from textual.widgets import SelectionList, DataTable
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.query_one("#suites", SelectionList).select("other")
+                    s.selected_models = {"a", "b"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+                    await pilot.pause()
+                    s.action_start()
+                    deadline = time.time() + 180     # CI machines can take long to start servers
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run, out = app.screen, {}
+                    q = run.query_one("#queue", DataTable)
+                    total = run.total
+                    q.move_cursor(row=q.get_row_index("b/apps"))
+                    await pilot.press("x")
+                    await pilot.pause()
+                    out["asked"] = isinstance(app.screen, ConfirmScreen) and app.screen.focused.id == "no"
+                    await pilot.press("n")
+                    await pilot.pause()
+                    out["kept"] = [j.key for j in run.jobs]
+                    q.move_cursor(row=q.get_row_index("b/apps"))
+                    await pilot.press("x")
+                    await pilot.pause()
+                    await pilot.press("y")
+                    await pilot.pause()
+                    out["after"] = [[j.key for j in run.jobs], list(map(str, (k.value for k in q.rows))),
+                                    total - run.total]
+                    q.move_cursor(row=q.get_row_index("a/apps"))      # running: stays, nothing asked
+                    await pilot.press("x")
+                    await pilot.pause()
+                    out["running"] = [isinstance(app.screen, ConfirmScreen), "a/apps" in [j.key for j in run.jobs]]
+                    app.engine.cancel()
+                    deadline = time.time() + 60
+                    while time.time() < deadline and run.running:
+                        await pilot.pause(0.2)
+                    print(json.dumps(out))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertTrue(out["asked"])
+        self.assertIn("b/apps", out["kept"])
+        jobs, rows, fewer = out["after"]
+        self.assertNotIn("b/apps", jobs)
+        self.assertNotIn("b/apps", rows)
+        self.assertEqual(fewer, 4)                     # its 4 questions are no longer counted
+        self.assertEqual(out["running"], [False, True])
+
+    def test_tui_asks_before_skipping_a_model(self):
+        path = os.path.join(self.ws, "models.toml")
+        write(path, read(path).replace('"--delay", "0.2"', '"--delay", "3"'))   # still running while it's asked
+        code = textwrap.dedent("""
+            import asyncio, json, time
+            from tuieval.tui import EvalsApp, RunScreen, ConfirmScreen
+            from textual.widgets import SelectionList
+            async def go():
+                app = EvalsApp({})
+                async with app.run_test(size=(160, 50)) as pilot:
+                    await pilot.pause()
+                    s = app.screen
+                    s.query_one("#suites", SelectionList).select("apps")
+                    s.selected_models = {"a"}
+                    s.refresh_models()
+                    s.query_one("#tier-certify").value = True
+                    await pilot.pause()
+                    s.action_start()
+                    deadline = time.time() + 180     # CI machines can take long to start servers
+                    while time.time() < deadline:
+                        await pilot.pause(0.1)
+                        if isinstance(app.screen, RunScreen) and app.screen.counts()[0]:
+                            break
+                    run, out = app.screen, {}
+                    for keys in (["k", "enter"], ["k", "k"], ["k", "n"], ["k", "escape"]):   # each one: no
+                        await pilot.press(keys[0])
+                        await pilot.pause()
+                        out.setdefault("asked", []).append(isinstance(app.screen, ConfirmScreen)
+                                                           and app.screen.focused.id == "no")
+                        await pilot.press(keys[1])
+                        await pilot.pause()
+                    out["kept"] = [run.running, app.screen is run, [j.status for j in run.jobs]]
+                    await pilot.press("k")
+                    await pilot.pause()
+                    await pilot.press("y")
+                    deadline = time.time() + 120
+                    while time.time() < deadline and run.running:
+                        await pilot.pause(0.2)
+                    out["skipped"] = [j.status for j in run.jobs]
+                    print(json.dumps(out))
+            asyncio.run(go())
+        """)
+        p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, timeout=600,
+                           env=dict(os.environ, TUIEVAL_HOME=self.ws, TUIEVAL_DETECT_PORTS=""))
+        out = json.loads(p.stdout.strip().splitlines()[-1]) if p.stdout.strip() else p.stderr
+        self.assertIsInstance(out, dict, out)
+        self.assertEqual(out["asked"], [True] * 4)
+        self.assertEqual(out["kept"], [True, True, ["running"]])
+        self.assertEqual(out["skipped"], ["skipped"])
+
     def test_tui_adds_or_queues_while_running(self):
         code = textwrap.dedent("""
             import asyncio, json, time