tuieval 0.2.0.dev5__tar.gz → 0.2.0.dev8__tar.gz

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## Unreleased
 
-- `tuieval tune` no longer rejects settings because the model's loading pushed other apps to swap (e.g. a large model locked in RAM with `--load-mode mlock`); only swapping while the loaded server works counts. Swap at load is noted in the output and the profile (`server_load_swapped_mb`).
+- `tuieval tune` scores options on projected full-length answers (`[tune] answer_tokens`, default 1024), so generation speed counts as in real runs.
+- `tuieval tune` reports an option's memory cost instead of rejecting it; it rejects only on critical memory pressure or over `[tune] swap_limit_mb`.
 - Warm-start tuning recognises fine-tunes whose GGUF describes the MTP draft layer differently (e.g. listing KV heads per layer) as the same model shape, so they start from an already-tuned sibling instead of tuning in full.
 - The fit check (context sized from the GGUF header, llama.cpp's memory use) only applies to servers whose command takes `{ctx}`. Servers that size their own memory keep the model's `max_context` instead of an estimate that didn't apply to them; `fit_check = true|false` on a server overrides it.
 - `tuieval export pi` has no default presets path any more: set `[export.pi] presets` to your llama.cpp router's `--models-preset` file.

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev5
+Version: 0.2.0.dev8
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/docs/models.md

@@ -77,8 +77,8 @@ The best server flags differ per model and per machine, so tuieval splits them b
 `tuieval tune <model>` (or `t` on the setup screen, for the ticked models) finds the fastest speed flags for a model on this machine:
 
 1. If `llama-bench` is installed, it sweeps threads, micro-batch and flash attention first (fast, no server starts).
-2. Then it starts the real server with one knob changed at a time and times a fixed **built-in** workload (three short prompts, three medium ones and one ~8k-token prompt), so tuning needs no packs and speeds are comparable between workspaces.
-3. An **output guard** rejects any option that changes greedy answers beyond noise. Candidates under which macOS swaps while the server works are rejected; swapping while the model loads (e.g. other apps making room for a model locked in RAM) is only noted.
+2. Then it starts the real server with one knob changed at a time and times a fixed **built-in** workload (three short prompts, three medium ones and one ~8k-token prompt), so tuning needs no packs and speeds are comparable between workspaces. Options are scored on the **projected** time: each prompt's measured reading time plus a full-length answer (`[tune] answer_tokens`, default 1024) at its measured generation speed. The workload itself generates 128 tokens per prompt, but eval answers run to thousands, so generation speed counts as much as in real runs.
+3. An **output guard** rejects any option that changes greedy answers beyond noise. Memory is a cost, not a reason to reject: an option that makes macOS push other apps' idle memory to swap can still win, and the result warns about it. An option is rejected only if macOS memory pressure turns critical while it runs, or, with `[tune] swap_limit_mb` set, if it swaps more than that beyond the defaults.
 
 Expect 8–15 server starts, about 20–30 minutes for a 27B model, once per model per machine. The result is saved in `tuning/<machine>/<model>.toml` and used by every later run there. Models without a profile run with each knob's first option and show *untuned*. A profile is marked for retuning when the model file or server version changes.
 
@@ -128,7 +128,7 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `request` | fields added to every request (e.g. `{ cache_prompt = false }`) |
 | `perf` | speed-only flags always applied |
 | `tune` | speed-only knobs for `tuieval tune` |
-| `tune_objective` | `total` (default: workload time) or `decode` (tokens/s after a warm-up pass) |
+| `tune_objective` | `projected` (default: workload time with full-length answers), `total` (workload time as run) or `decode` (tokens/s after a warm-up pass) |
 | `version_cmd` | prints the server version, recorded with every result |
 | `health` | readiness path for servers without `/v1/models` (must return JSON with `model`) |
 | `before_start` | a command run before starting the server (e.g. to free memory another process holds) |

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev5'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev5')
+__version__ = version = '0.2.0.dev8'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev8')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/machines.py

@@ -225,6 +225,17 @@ def swapped_out_bytes():
 RESIDENCY_FRACTION = 0.5
 
 
+def memory_pressure_level():
+    """macOS memory pressure: 1 normal, 2 warning, 4 critical (the system starts ending processes).
+    None elsewhere."""
+    try:
+        out = subprocess.run(["sysctl", "-n", "kern.memorystatus_vm_pressure_level"], capture_output=True,
+                             text=True, timeout=5).stdout
+        return int(out.strip())
+    except (OSError, subprocess.TimeoutExpired, ValueError):
+        return None
+
+
 def gpu_residency_gb(machine, override=None):
     """GPU memory the driver keeps resident without churning (models.toml
     [machines.<id>] gpu_residency_gb overrides the half-of-RAM default)."""

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/run_evals.py

@@ -523,13 +523,17 @@ def cmd_tune(argv):
         pr = result["profile"]
         ms = pr["measured"]
         gain = ", " + tune.gain_text(ms) if tune.gain_text(ms) else ""
-        result = f"{ms['tg_tps']} tok/s decode" if ms.get("objective") == "decode" else f"{ms['total_s']}s for the workload"
+        result = f"{ms['tg_tps']} tok/s decode" if ms.get("objective") == "decode" else \
+            f"{ms['projected_s']}s projected for the workload" if ms.get("objective") == "projected" and ms.get("projected_s") \
+            else f"{ms['total_s']}s for the workload"
         say(f"{label}: {' '.join(pr['args']) or '(no knobs)'}  ->  {result}{gain}", GREEN)
         if pr["meta"].get("warm_start"):
             ws = pr["meta"]["warm_start"]
             say(f"{label}: started from {ws['from']}'s flags; re-tried {', '.join(ws['retested']) or 'nothing'}")
         if not pr["meta"].get("answer_guard", True):
             say(f"{label}: its answers depend on these settings; rerun its evals on this machine", "\033[33m")
+        for w in pr["meta"].get("warnings", []):
+            say(f"{label}: warning: {w}", YELLOW)
         if pr["meta"].get("rejected"):
             say(f"{label}: rejected because answers changed: {'; '.join(pr['meta']['rejected'])}")
         if a.export_pi and not export_pi(e, label):

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/templates/models.toml

@@ -101,6 +101,8 @@ headers = { "X-Title" = "tuieval" }
 # request_timeout_s = 600      # a tuning request that takes longer fails that candidate
 # warm_start = true            # start from a tuned model with the same architecture and shapes (a fine-tune)
 # warm_retest = ["spec", "ubatch"]   # the knobs a warm start re-tries; the rest are inherited
+# answer_tokens = 1024         # answer length options are scored for (projected objective)
+# swap_limit_mb = 500          # reject options that swap more than this beyond the defaults (default: report only)
 
 # Optional: llama-bench for the tuner's fast first stage (found on PATH as llama-bench otherwise).
 # [bench]

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/tui.py

@@ -2565,6 +2565,8 @@ class EvalsApp(App):
                             p = tune.tune(e, label, lambda kind, **d: on_event(kind, label=label, **d))
                             ms = p["measured"]
                             gain = f" ({tune.gain_text(ms)})" if tune.gain_text(ms) else ""
+                            if p["meta"].get("warnings"):
+                                gain += " (warning: it pushes other apps' memory to swap; see the log)"
                             if not p["meta"].get("answer_guard", True):
                                 gain += " (answers depend on these settings: rerun its evals here)"
                             done.append(f"{label}{gain}")

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/src/tuieval/tune.py

@@ -18,17 +18,24 @@ retrained or dropped its MTP layers, and its quant mix shifts the best micro-bat
 it tunes in full instead. `tuieval tune --cold` (or [tune] warm_start = false) always tunes in full.
 
 One objective per server (models.toml `tune_objective`):
-  total   (default) total seconds for the workload: short prompts, medium ones and one long
-          (~8k-token) prompt, fixed generation length
-  decode  generated tokens per second, measured on a second pass over a few prompts after a
-          warm-up pass (for servers whose decode speed grows as their caches warm)
+  projected  (default) seconds the workload would take with full-length answers: each prompt's
+             measured reading time plus [tune] answer_tokens (default 1024) at its measured
+             generation speed. The workload generates 128 tokens per prompt, but eval answers run
+             to thousands, so generation speed counts as much as it does in real runs.
+  total      wall-clock seconds for the workload as run (128-token answers)
+  decode     generated tokens per second, measured on a second pass over a few prompts after a
+             warm-up pass (for servers whose decode speed grows as their caches warm)
 An output guard compares greedy answers with the default flags; an option that changes them beyond
 noise is rejected, because speed flags must not change answers. Servers whose answers depend on
 the machine's memory settings (`outputs_depend_on_machine`) skip the guard; their tuned flags
 become part of the results fingerprint instead, so retuning marks that machine's results
-outdated. Any candidate under which macOS swaps while the server works is rejected (swapping
-while the model loads, e.g. to lock it in RAM, is only noted). Servers without tune knobs are only
-measured.
+outdated.
+
+Memory: a candidate is rejected if macOS memory pressure turns critical while it works, or, with
+[tune] swap_limit_mb set, if it swaps more than that beyond the defaults. Otherwise memory is a
+reported cost, not a reason to reject: a flag that pushes other apps' idle memory to swap can still
+win, and the result warns about it. Swapping the defaults already cause is a warning too, and
+swapping while the model loads is only noted. Servers without tune knobs are only measured.
 
 The knobs and their options come from models.toml [servers.<name>.tune]; placeholders {p},
 {p_minus_2} and {all} are this machine's core counts.
@@ -49,7 +56,10 @@ from . import profiles
 
 GEN_TOKENS = 128          # generated per workload prompt
 DECODE_TOKENS = 256      # generated per prompt for the decode objective
-SWAP_LIMIT = 256 * 2**20  # a candidate that swaps out more than this is rejected
+SWAP_NOTABLE = 256 * 2**20  # swap worth a warning (bytes); rejecting for swap needs [tune] swap_limit_mb
+ANSWER_TOKENS = 1024       # answer length the projected objective scores ([tune] answer_tokens)
+PRESSURE_CRITICAL = 4      # macOS memory pressure level that rejects a candidate
+PRESSURE_EVERY_S = 2
 GPU_MARGIN_GB = 0.75      # a candidate whose GPU allocation comes this close to the residency limit
                           # is rejected even before it stalls: servers grow as requests arrive
 REQUEST_LIMIT_S = 600     # a tuning request taking longer fails the candidate ([tune] request_timeout_s)
@@ -70,16 +80,22 @@ class Measure:
     load_s: float | None = None
     error: str = ""
     facts: dict = dataclasses.field(default_factory=dict)   # from the server log, e.g. expert capacity
+    swapped_mb: float | None = None  # macOS swap-outs while the loaded server worked
+    projected_s: float | None = None  # the workload with full-length answers (projected objective)
+    pressure: int | None = None      # highest macOS memory pressure level while it worked
 
     def score(self, objective):
         """Lower is better."""
         if objective == "decode":
             return 1 / self.tg_tps if self.tg_tps else float("inf")
+        if objective == "projected" and self.projected_s is not None:
+            return self.projected_s
         return self.total_s
 
     def summary(self):
         facts = "".join(f" · {k} {v}" for k, v in self.facts.items())
-        return (f"{self.total_s:.1f}s total · gen {self.tg_tps} tok/s · prompt {self.pp_tps} tok/s"
+        projected = f"{self.projected_s:.0f}s projected · " if self.projected_s is not None else ""
+        return (f"{projected}{self.total_s:.1f}s total · gen {self.tg_tps} tok/s · prompt {self.pp_tps} tok/s"
                 f" · ttft {self.ttft_s}s{facts}")
 
 
@@ -192,20 +208,21 @@ def _request(eng, base_url, body, name, emit, limit):
 
 
 def measure(eng, m, base_url, work, gen_tokens=GEN_TOKENS, passes=1, emit=lambda *a, **k: None,
-            limit=REQUEST_LIMIT_S):
+            limit=REQUEST_LIMIT_S, answer_tokens=ANSWER_TOKENS):
     """Time the workload (after a warm-up request). With passes > 1 the earlier passes warm the
-    server's caches and only the last pass is reported."""
+    server's caches and only the last pass is reported. answer_tokens: the answer length the
+    projected time is for."""
     out = Measure()
     for p in range(passes):
         if passes > 1:
             emit("tune_progress", message=f"    pass {p + 1} of {passes}" + (" (warm-up)" if p < passes - 1 else " (timed)"))
-        out = _measure_once(eng, m, base_url, work, gen_tokens, emit, limit)
+        out = _measure_once(eng, m, base_url, work, gen_tokens, emit, limit, answer_tokens)
         if out.error:
             break
     return out
 
 
-def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit):
+def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit, answer_tokens=ANSWER_TOKENS):
     sampling = engine_mod.effective_sampling(eng.cfg, m)
     request = eng.cfg["servers"][m["server"]].get("request", {})
     timeout = eng.cfg["defaults"]["request_timeout_ms"] / 1000
@@ -216,7 +233,7 @@ def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit):
     if warm["error"]:
         out.error = warm["error"]
         return out
-    ttfts, p_tok, p_s, g_tok, g_s = [], 0, 0.0, 0, 0.0
+    ttfts, p_tok, p_s, g_tok, g_s, projected = [], 0, 0.0, 0, 0.0, 0.0
     for name, msgs in work:
         eng._check()
         emit("tune_progress", message=f"    {name} ({gen_tokens} tokens)")
@@ -228,6 +245,13 @@ def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit):
         emit("tune_progress", message=f"      done: {met['completion_tokens']} tokens in {res['total_s']:.1f}s"
                                       + (f", {met['gen_tps']} tok/s" if met["gen_tps"] else ""))
         out.total_s += res["total_s"]
+        # The same request with a full-length answer: its reading time plus answer_tokens at the
+        # generation speed measured here (as run if the server doesn't report that speed).
+        if met["completion_tokens"] and met["gen_tps"]:
+            reading = max(res["total_s"] - met["completion_tokens"] / met["gen_tps"], 0.0)
+            projected += reading + answer_tokens / met["gen_tps"]
+        else:
+            projected += res["total_s"]
         out.texts.append((res["reasoning"] or "") + (res["answer"] or ""))
         if name != "long" and res.get("ttft_s") is not None:
             ttfts.append(res["ttft_s"])
@@ -236,6 +260,7 @@ def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit):
         if met["completion_tokens"] and met["gen_tps"]:
             g_tok, g_s = g_tok + met["completion_tokens"], g_s + met["completion_tokens"] / met["gen_tps"]
     out.total_s = round(out.total_s, 2)
+    out.projected_s = round(projected, 1)
     out.ttft_s = round(sum(ttfts) / len(ttfts), 3) if ttfts else None
     out.pp_tps = round(p_tok / p_s, 1) if p_s else None
     out.tg_tps = round(g_tok / g_s, 2) if g_s else None
@@ -244,12 +269,37 @@ def _measure_once(eng, m, base_url, work, gen_tokens, emit, limit):
 
 def gain_pct(objective, base, best):
     """How much better the chosen settings are than the defaults: % more decode tok/s for the
-    decode objective, % less total time otherwise."""
+    decode objective, % less (projected or total) time otherwise."""
     if base is None:
         return None
     if objective == "decode":
         return round(100 * (best.tg_tps / base.tg_tps - 1)) if base.tg_tps and best.tg_tps else None
-    return round(100 * (1 - best.total_s / base.total_s)) if base.total_s else None
+    b, n = base.score(objective), best.score(objective)
+    return round(100 * (1 - n / b)) if b else None
+
+
+class PressureWatch:
+    """The highest macOS memory pressure level seen while a block runs (None where unknown)."""
+
+    def __enter__(self):
+        self.level, self._stop = machines.memory_pressure_level(), threading.Event()
+        self._thread = threading.Thread(target=self._run, daemon=True)
+        self._thread.start()
+        return self
+
+    def _sample(self):
+        lvl = machines.memory_pressure_level()
+        if lvl is not None:
+            self.level = max(self.level or 0, lvl)
+
+    def _run(self):
+        while not self._stop.wait(PRESSURE_EVERY_S):
+            self._sample()
+
+    def __exit__(self, *exc):
+        self._stop.set()
+        self._thread.join(PRESSURE_EVERY_S + 1)
+        self._sample()
 
 
 def gain_text(measured):
@@ -451,18 +501,21 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
     sv = eng.serving(m)
     if not sv.fits:
         raise engine_mod.ModelFailed(sv.fit_note)
-    objective = server.get("tune_objective", "total")
+    objective = server.get("tune_objective", "projected")
     if objective == "decode":
         work, gen_tokens, passes = decode_workload(eng), DECODE_TOKENS, 2
     else:
         work, gen_tokens, passes = workload(eng, sv.ctx), GEN_TOKENS, 1
     settings = eng.cfg.get("tune", {})
+    answer_tokens = settings.get("answer_tokens", ANSWER_TOKENS)
+    swap_limit = settings.get("swap_limit_mb")   # MB beyond the defaults' swap that rejects; None: report only
     threshold = settings.get("guard_similarity", 0.6)
     guard = not server.get("outputs_depend_on_machine")
     knob_opts = knobs(eng, m, sv) if server.get("cmd") else {}
     fixed = list(server.get("perf", [])) if server.get("cmd") else []
     cache, starts, notes, rejected, bad = {}, [0], [], [], set()   # bad: (knob, option) that failed
-    warned = []
+    warned, warnings = [], []
+    swap_ref = [None]   # MB the defaults swapped while serving: the machine's baseline
 
     def evaluate(choices, why):
         k = tuple(sorted(choices.items()))
@@ -481,18 +534,20 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
                 # Loading may push other apps to swap once (e.g. a model locked in RAM with mlock);
                 # that's recorded, not held against the settings. Swapping while serving is.
                 loaded = machines.swapped_out_bytes()
-                r = measure(eng, m, url, work, gen_tokens, passes, emit,
-                            settings.get("request_timeout_s", REQUEST_LIMIT_S))
+                with PressureWatch() as pressure:
+                    r = measure(eng, m, url, work, gen_tokens, passes, emit,
+                                settings.get("request_timeout_s", REQUEST_LIMIT_S), answer_tokens=answer_tokens)
                 end = machines.swapped_out_bytes()
+                r.pressure = pressure.level
                 if None not in (swap0, loaded, end):
                     swapped = (end - loaded) / 2**20
                     at_load = (loaded - swap0) / 2**20
-                    if at_load > SWAP_LIMIT / 2**20:
+                    if at_load > SWAP_NOTABLE / 2**20:
                         emit("tune_step", message=f"    note: loading pushed {at_load:.0f} MB of other apps to swap "
                                                   "(close apps for more headroom)")
                 r.load_s = round(info["load_s"], 1) if info["load_s"] else None
                 r.facts = dict(info["facts"])
-                if swapped is not None and at_load > SWAP_LIMIT / 2**20:
+                if swapped is not None and at_load > SWAP_NOTABLE / 2**20:
                     r.facts["load_swapped_mb"] = round(at_load)
                 # Settled allocation after the timed pass (brief peaks while processing a prompt are
                 # harmless; sustained allocation over the limit is what makes the driver churn).
@@ -511,17 +566,35 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
                                               "close other apps for faster and fairer results")
         except engine_mod.ModelFailed as e:
             r = Measure(error=str(e).splitlines()[0])
-        if not r.error and swapped is not None and swapped > SWAP_LIMIT / 2**20:
-            r.error = f"macOS swapped {swapped:.0f} MB while the server worked (memory too tight)"
+        r.swapped_mb = swapped
+        # Swapping every candidate shares (model, context, prompt cache, other apps) says nothing about
+        # a flag; only swapping beyond the defaults' does, and it's a cost to report unless it harms:
+        # critical memory pressure, or more than the user's swap_limit_mb.
+        extra = None if swapped is None or swap_ref[0] is None else swapped - swap_ref[0]
+        if extra is not None and extra > SWAP_NOTABLE / 2**20:
+            r.facts["extra_swap_mb"] = round(extra)
+        if not r.error and r.pressure is not None and r.pressure >= PRESSURE_CRITICAL:
+            r.error = "macOS memory pressure turned critical (memory too tight)"
+        elif not r.error and swap_limit is not None and extra is not None and extra > swap_limit:
+            r.error = f"macOS swapped {extra:.0f} MB more than with the defaults (over swap_limit_mb = {swap_limit})"
         cache[k] = r
         emit("tune_result", message=f"    failed: {r.error}" if r.error else "    " + r.summary(), result=r, args=args)
         return r
 
+    def set_baseline(r):
+        """The defaults' swapping is the machine's baseline: warned about, never held against a flag."""
+        swap_ref[0] = r.swapped_mb or 0
+        if swap_ref[0] > SWAP_NOTABLE / 2**20:
+            warnings.append(f"serving this model pushed {swap_ref[0]:.0f} MB of other apps' memory to swap on "
+                            "this machine; close other apps or lower this model's context for more headroom")
+            emit("tune_step", message="    warning: " + warnings[-1])
+
     base = None
     if not server.get("cmd") or not knob_opts:
         r = evaluate({}, "measuring (no speed knobs to tune)")
         if r.error:
             raise engine_mod.ModelFailed(r.error)
+        set_baseline(r)
         best, best_r = {}, r
         method, bench_info = "measured", None
     else:
@@ -530,6 +603,7 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
         if base is None or base.error:
             raise engine_mod.ModelFailed(f"the server doesn't start with the default flags: "
                                          f"{base.error if base else 'no starts left'}")
+        set_baseline(base)
         settled, bench_info, warm_info = {}, None, None
         best, best_r, search = dict(default), base, list(knob_opts)
         sib = find_sibling(eng, m, knob_opts, sv.machine.id) \
@@ -598,6 +672,11 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
         if warm_info:
             notes.insert(0, f"warm start from {warm_info['from']}: inherited "
                             f"{', '.join(warm_info['inherited']) or 'nothing'}")
+    if best_r.facts.get("extra_swap_mb") and best_r is not base:
+        warnings.append(f"the chosen flags use more memory than the defaults: macOS swapped "
+                        f"~{best_r.facts['extra_swap_mb']} MB more of other apps' memory; close apps during runs, "
+                        "or set [tune] swap_limit_mb to rule such flags out")
+        emit("tune_step", message="    warning: " + warnings[-1])
     size = sv.identity.get("model_bytes")
     profile = {
         "args": resolve(knob_opts, best),
@@ -606,15 +685,18 @@ def tune(eng, label, emit=lambda *a, **k: None, max_starts=16, min_gain=0.03, us
                      "tg_tps": best_r.tg_tps, "load_s": best_r.load_s,
                      "default_total_s": base.total_s if knob_opts else None,
                      "default_tg_tps": base.tg_tps if knob_opts else None, "objective": objective,
+                     "projected_s": best_r.projected_s,
+                     "default_projected_s": base.projected_s if knob_opts else None,
                      **{f"server_{k}": v for k, v in best_r.facts.items()},
                      "gain_pct": gain_pct(objective, base, best_r)},
         "meta": {"method": method, "date": profiles.now(), "machine": sv.machine.summary,
                  "server_version": eng.server_version(m), "model_file": os.path.basename(m["model"]),
                  "model_bytes": size, "ctx": sv.ctx, "server_starts": starts[0],
-                 "workload": f"{len(work)} prompts x {gen_tokens} tokens" + (f", {passes} passes" if passes > 1 else ""),
+                 "workload": f"{len(work)} prompts x {gen_tokens} tokens" + (f", {passes} passes" if passes > 1 else "")
+                 + (f", scored for {answer_tokens}-token answers" if objective == "projected" else ""),
                  "llama_bench": bool(bench_info), "answer_guard": guard,
                  **({"warm_start": warm_info} if knob_opts and warm_info else {}),
-                 "rejected": rejected, "notes": notes[:10]},
+                 "rejected": rejected, "notes": notes[:10], "warnings": warnings},
     }
     path = profiles.save(sv.machine.id, label, profile, eng.tuning_dir)
     eng._serving.clear()

{tuieval-0.2.0.dev5 → tuieval-0.2.0.dev8}/tests/test_tuieval.py

@@ -308,12 +308,17 @@ class WarmTune(unittest.TestCase):
             yield "http://x", {"load_s": 1, "facts": {}}
         eng.serve = serve
 
-        def measure(eng_, m, url, work, *a):
+        def measure(eng_, m, url, work, *a, **k):
             args = eng.starts[-1]
             fast = {"6": 1, "256": 1, "draft-mtp": 3}
             return tune.Measure(total_s=10 - sum(v for k, v in fast.items() if k in args), texts=["same"])
         self.orig = tune.measure
         tune.measure = measure
+        from tuieval import machines
+        self.pressure = lambda args: 1   # macOS memory pressure per start's flags: normal
+        orig_pressure = machines.memory_pressure_level
+        machines.memory_pressure_level = lambda: self.pressure(eng.starts[-1] if eng.starts else [])
+        self.addCleanup(setattr, machines, "memory_pressure_level", orig_pressure)
         self.eng, self.tune, self.profiles = eng, tune, profiles
         for label, args in (("sib", ["-t", "6", "-ub", "256", "--spec-type", "draft-mtp"]),
                             ("other", ["-t", "8"])):
@@ -325,14 +330,15 @@ class WarmTune(unittest.TestCase):
         self.tmp.cleanup()
 
     def _swap(self, at_load_mb, while_serving_mb):
-        """Fake macOS swap counter: each server start swaps at_load_mb while loading (between the
-        reading before the start and the one once it serves) and while_serving_mb during the work."""
+        """Fake macOS swap counter: each server start swaps at_load_mb while loading and
+        while_serving_mb(args) during the work (args: that start's flags)."""
         from tuieval import machines
         state = {"n": 0, "total": 0}
 
         def swapped_out_bytes():
             step = state["n"] % 3          # 0: before the start, 1: loaded, 2: after the work
-            state["total"] += {0: 0, 1: at_load_mb, 2: while_serving_mb}[step] * 2**20
+            mb = {0: 0, 1: at_load_mb, 2: while_serving_mb(self.eng.starts[-1]) if step == 2 else 0}[step]
+            state["total"] += mb * 2**20
             state["n"] += 1
             return state["total"]
         orig = machines.swapped_out_bytes
@@ -340,15 +346,52 @@ class WarmTune(unittest.TestCase):
         self.addCleanup(setattr, machines, "swapped_out_bytes", orig)
 
     def test_swapping_while_loading_is_only_noted(self):
-        self._swap(at_load_mb=900, while_serving_mb=0)
+        self._swap(at_load_mb=900, while_serving_mb=lambda args: 0)
         p = self.tune.tune(self.eng, "new", use_bench=False)
         self.assertEqual(p["measured"]["server_load_swapped_mb"], 900)
+        self.assertEqual(p["meta"]["warnings"], [])
+
+    def test_swapping_every_candidate_shares_is_a_warning(self):
+        self._swap(at_load_mb=0, while_serving_mb=lambda args: 800)
+        p = self.tune.tune(self.eng, "new", use_bench=False)
+        self.assertEqual(p["args"], ["-t", "6", "-ub", "256", "--spec-type", "draft-mtp"])   # tuned as usual
+        self.assertIn("800 MB", p["meta"]["warnings"][0])
+
+    def test_a_flag_that_costs_memory_still_wins_with_a_warning(self):
+        # -ub 256 is the fastest micro-batch here, and costs 600 MB of swap the defaults don't
+        self._swap(at_load_mb=0, while_serving_mb=lambda args: 600 if "256" in args else 100)
+        p = self.tune.tune(self.eng, "new", use_bench=False, warm=False)
+        self.assertIn("256", p["args"])
+        self.assertTrue(any("~500 MB more" in w for w in p["meta"]["warnings"]))
+
+    def test_swap_limit_rules_out_flags_that_cost_memory(self):
+        self.eng.cfg["tune"] = {"swap_limit_mb": 256}
+        self._swap(at_load_mb=0, while_serving_mb=lambda args: 600 if "256" in args else 100)
+        p = self.tune.tune(self.eng, "new", use_bench=False, warm=False)
+        self.assertNotIn("256", p["args"])
+        self.assertTrue(any("over swap_limit_mb = 256" in n for n in p["meta"]["notes"]))
 
-    def test_swapping_while_serving_fails_the_settings(self):
-        self._swap(at_load_mb=0, while_serving_mb=500)
-        with self.assertRaises(self.tune.engine_mod.ModelFailed) as cm:
-            self.tune.tune(self.eng, "new", use_bench=False)
-        self.assertIn("swapped 500 MB while the server worked", str(cm.exception))
+    def test_critical_memory_pressure_rejects_a_flag(self):
+        self.pressure = lambda args: 4 if "draft-mtp" in args else 1
+        p = self.tune.tune(self.eng, "new", use_bench=False, warm=False)
+        self.assertNotIn("draft-mtp", p["args"])
+        self.assertTrue(any("memory pressure turned critical" in n for n in p["meta"]["notes"]))
+
+    def test_projected_time_scores_full_length_answers(self):
+        # 10 s for 128 tokens at 16 tok/s: 2 s reading the prompt + 8 s generating
+        res = {"error": None, "total_s": 10.0, "ttft_s": 2.0, "reasoning": "", "answer": "x",
+               "usage": {"completion_tokens": 128, "prompt_tokens": 50},
+               "timings": {"predicted_per_second": 16.0, "prompt_per_second": 25.0}}
+        self.eng.cfg["sampling"] = {"temperature": 0}
+        self.eng._check = lambda: None
+        orig = self.tune._request
+        self.tune._request = lambda *a, **k: dict(res)
+        self.addCleanup(setattr, self.tune, "_request", orig)
+        out = self.tune._measure_once(self.eng, self.eng.model("new"), "http://x", [("p", [])], 128,
+                                      lambda *a, **k: None, 60, answer_tokens=1024)
+        self.assertEqual((out.total_s, out.projected_s), (10.0, 66.0))   # 2 s + 1024 / 16
+        self.assertEqual(out.score("projected"), 66.0)
+        self.assertEqual(out.score("total"), 10.0)
 
     def test_family_ignores_the_mtp_layer(self):
         # one GGUF lists KV heads per layer, none on the MTP (last) layer; the other gives one number
@@ -388,7 +431,7 @@ class WarmTune(unittest.TestCase):
         self.profiles.save("mac", "sib", {"args": ["-t", "8", "-ub", "1024"], "meta": {"method": "tuned",
                                           "server_version": "v1"}}, self.eng.tuning_dir)
         orig = self.tune.measure
-        self.tune.measure = lambda *a: self.tune.Measure(
+        self.tune.measure = lambda *a, **k: self.tune.Measure(
             total_s=99 if "1024" in self.eng.starts[-1] else orig(*a).total_s, texts=["same"])
         p = self.tune.tune(self.eng, "new", use_bench=False)
         self.assertNotIn("warm_start", p["meta"])