tuieval 0.2.0.dev1__tar.gz → 0.2.0.dev3__tar.gz

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/CHANGELOG.md

@@ -2,12 +2,15 @@
 
 ## Unreleased
 
+- The fit check (context sized from the GGUF header, llama.cpp's memory use) only applies to servers whose command takes `{ctx}`. Servers that size their own memory keep the model's `max_context` instead of an estimate that didn't apply to them; `fit_check = true|false` on a server overrides it.
+- `tuieval export pi` has no default presets path any more: set `[export.pi] presets` to your llama.cpp router's `--models-preset` file.
+- A new README screenshot from a demo workspace.
 - Versions come from git tags: a tag `vX.Y.Z` is that release, and every commit on `main` after it is published automatically as a dev build `X.(Y+1).0.devN` (N = commits since the release). `pip install tuieval` keeps installing releases only; `pip install --pre tuieval` gets the latest build.
 
 ## 0.1.7 (replaces 0.1.6, withdrawn)
 
 - `tuieval tune` starts warm for fine-tunes: when a model with the same architecture and tensor shapes is already tuned on this machine, its flags are the starting point and only speculative decoding and micro-batch are re-tried (~4-6 server starts instead of 8-15). It tunes in full if the inherited flags fail, change answers or are slower than the defaults. `--cold` (or `[tune] warm_start = false`) always tunes in full; `[tune] warm_retest` names the re-tried knobs.
-- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama-router preset section and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
+- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama.cpp router presets section and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
 - `tuieval list` lists the models, with hidden ones separately (`--packs` lists the packs).
 - `tuieval remove <model>` / `--hidden` removes models: their models.toml entry, results, tuning profiles and hidden mark, archived in removed/.
 - `tuieval --help` gives every command its own line.

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev1
+Version: 0.2.0.dev3
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -47,7 +47,7 @@ tuieval                        # open the TUI
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
-![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
 ## What you get
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/README.md

@@ -21,7 +21,7 @@ tuieval                        # open the TUI
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
-![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
 ## What you get
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/docs/models.md

@@ -68,8 +68,8 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 ## Machines
 
-- **Machine id** is detected automatically (`m1max-32gb`, `m4pro-24gb`, …; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
-- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers.
+- **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
+- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
 - **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
 
 ## Tuning
@@ -90,18 +90,18 @@ The knobs are `[servers.<name>.tune]` in `models.toml`: each knob is a list of o
 
 `tuieval export pi <model>` makes pi serve a model the way the evals did: the same file, the output-affecting flags, the context from the fit check, the model's sampling, and the speed flags tuned on this machine. `tuieval tune <model> --export-pi` does it right after tuning.
 
-It writes a `[<id>]` section in llama-router's presets file (keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning). It exports models on llama servers.
+It writes a `[<id>]` section in the model presets file of a llama.cpp router (`llama-server --models-preset <file>`; keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning). It exports models on llama servers.
 
-It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards (`llama-router restart`).
+It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards so it reads the new presets.
 
 The id pi sees defaults to the GGUF's name; set `pi_id` and `pi_name` on the model (or pass `--id`/`--name`). Paths and provider names come from `[export.pi]` in `models.toml`:
 
 ```toml
 [export.pi]
-presets = "~/models/presets.ini"
-pi_models = "~/.pi/agent/models.json"
+presets = "/path/to/presets.ini"           # required: the router's --models-preset file
+pi_models = "~/.pi/agent/models.json"       # pi's default
 llama_provider = "llama"     # pi's provider name for the router
-servers = ["llama"]          # models.toml servers that llama-router can serve
+servers = ["llama"]          # models.toml servers the router can serve
 ```
 
 ## Speed verdicts
@@ -110,7 +110,7 @@ Speed is judged separately, per machine: the same model can be production-grade
 
 ## The stall guard (Apple Silicon)
 
-Measured on an M1 Max 32 GB: once the system's GPU allocations pass about half of RAM, the GPU driver evicts and re-maps memory on every GPU job, the server spends 70–95% of its CPU in the kernel while the GPU idles, and servers that submit many small GPU jobs slow to a crawl. For servers with `stall_guard = true`, tuieval samples the server's own vs kernel CPU time and the system's GPU allocation every 5 s during runs and tuning. If more than 70% of its CPU goes to the kernel for a minute, the run stops with an explanation instead of crawling for hours. Finished answers are kept and resume next time. If your machine behaves differently, set `gpu_residency_gb` under `[machines.<id>]`.
+On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM, the GPU driver evicts and re-maps memory on every GPU job, the server spends 70–95% of its CPU in the kernel while the GPU idles, and servers that submit many small GPU jobs slow to a crawl. For servers with `stall_guard = true`, tuieval samples the server's own vs kernel CPU time and the system's GPU allocation every 5 s during runs and tuning. If more than 70% of its CPU goes to the kernel for a minute, the run stops with an explanation instead of crawling for hours. Finished answers are kept and resume next time. If your machine behaves differently, set `gpu_residency_gb` under `[machines.<id>]`.
 
 ## Server options
 
@@ -120,7 +120,8 @@ Measured on an M1 Max 32 GB: once the system's GPU allocations pass about half o
 | `url` | an already-running server's base URL |
 | `port` | the port `cmd` serves on |
 | `cwd` | folder to start `cmd` in |
-| `model_is_path` | `model` is a file or folder: check it exists (and, for GGUFs, that it fits) before starting |
+| `model_is_path` | `model` is a file or folder: check it exists before starting |
+| `fit_check` | size the context from the GGUF header and skip models that don't fit (the fit check; it assumes llama.cpp's memory use). Default: on for servers whose `cmd` takes `{ctx}`, off for others, whose context is the model's `max_context` |
 | `vision_args` | flags appended for models with `mmproj` |
 | `kv_type`, `max_ctx` | llama defaults: KV-cache type, upper bound on context |
 | `ctx_flag` | the server's context flag, so long packs are skipped if a configured context is too small |

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev1'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev1')
+__version__ = version = '0.2.0.dev3'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev3')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/compare.py

@@ -3,7 +3,7 @@
 Usage:
     tuieval compare                          # everything in results/<model>/<pack>.json
     tuieval compare --speed                  # speed & tokens table (TTFT, tok/s, tokens, memory)
-    tuieval compare --machine m4pro-24gb     # speed on another machine (measured there, or projected)
+    tuieval compare --machine m3max-64gb     # speed on another machine (measured there, or projected)
     tuieval compare --pairwise --failures    # is each difference real? which tests failed?
     tuieval compare results/some-model/*.json    # specific files
 """

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/engine.py

@@ -96,6 +96,16 @@ def load_config(path):
     return cfg
 
 
+def fit_check(server):
+    """Whether tuieval sizes the context for this server's models from GGUF headers (machines.fit,
+    which assumes llama.cpp's memory use). By default only servers whose command takes {ctx}, i.e.
+    where tuieval chooses the context; others size their own memory, and their context is the
+    model's max_context. models.toml `fit_check = true|false` on a server overrides it."""
+    if "fit_check" in server:
+        return bool(server["fit_check"])
+    return any("{ctx}" in str(a) for a in server.get("cmd", []))
+
+
 def effective_sampling(cfg, m):
     """The request settings this model runs with: [sampling], then the model's own overrides."""
     s = dict(cfg["sampling"])
@@ -1018,13 +1028,15 @@ class Engine:
         path = expand(m["model"])
         if server.get("model_is_path") and os.path.isfile(path):
             try:
-                mmproj = expand(m.get("mmproj", ""))
-                extra = os.path.getsize(mmproj) if mmproj and os.path.isfile(mmproj) else 0
-                headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
-                f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
-                                 want_ctx=want)
                 info = machines.read_gguf(path)
-                ctx, fits, note, mtp, size = f.max_ctx, f.fits, f.note, info["mtp_layers"], info["bytes"]
+                mtp, size = info["mtp_layers"], info["bytes"]
+                if fit_check(server):
+                    mmproj = expand(m.get("mmproj", ""))
+                    extra = os.path.getsize(mmproj) if mmproj and os.path.isfile(mmproj) else 0
+                    headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
+                    f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
+                                     want_ctx=want)
+                    ctx, fits, note = f.max_ctx, f.fits, f.note
             except (OSError, ValueError) as e:
                 note = f"couldn't read the GGUF header ({e}); context not checked"
         profile, perf, source = None, [], "n/a"

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/export.py

@@ -1,17 +1,18 @@
 """Export a model's serving settings to the pi coding agent, so pi serves it exactly as the evals did:
 the same model file, output-affecting flags and context, plus the speed flags tuned on this machine.
 
-  a section in llama-router's presets (~/models/presets.ini; keys that equal the [*] section are
-  left out) and an entry under pi's `llama` provider modelOverrides. Models on llama servers only.
+  a section in a llama.cpp router's model presets file (llama-server --models-preset; keys that
+  equal its [*] section are left out) and an entry under pi's `llama` provider modelOverrides.
+  Models on llama servers only.
 
 Every file is backed up next to itself (<file>.bak-tuieval-<time>) before it changes, and the diff
 is shown first. Paths come from models.toml [export.pi]:
 
   [export.pi]
-  presets = "~/models/presets.ini"            # llama-router --models-preset
+  presets = "/path/to/presets.ini"            # required: the router's --models-preset file
   pi_models = "~/.pi/agent/models.json"
   llama_provider = "llama"                     # pi's provider name for the router
-  servers = ["llama"]                          # models.toml servers that llama-router can serve
+  servers = ["llama"]                          # models.toml servers the router can serve
 """
 import configparser
 import difflib
@@ -23,7 +24,7 @@ import time
 
 from . import engine as engine_mod
 
-DEFAULTS = {"presets": "~/models/presets.ini", "pi_models": "~/.pi/agent/models.json",
+DEFAULTS = {"presets": None, "pi_models": "~/.pi/agent/models.json",
             "llama_provider": "llama", "servers": ["llama"]}
 # llama.cpp short flags -> the long names presets use as keys
 SHORT = {"-m": "model", "-c": "ctx-size", "-t": "threads", "-tb": "threads-batch", "-ub": "ubatch-size",
@@ -127,7 +128,7 @@ def plan(eng, label, model_id=None, name=None):
     m = eng.model(label)
     s = settings(eng)
     if m["server"] not in s["servers"]:
-        raise ExportError(f"{label} runs on the {m['server']} server; pi export writes llama-router presets, "
+        raise ExportError(f"{label} runs on the {m['server']} server; pi export writes llama.cpp router presets, "
                           "for llama servers ([export.pi] servers lists them)")
     model_id = model_id or m.get("pi_id") or default_id(m)
     name = name or m.get("pi_name") or model_id
@@ -157,6 +158,8 @@ def plan(eng, label, model_id=None, name=None):
     for k, key in SAMPLING_KEYS.items():
         if sampling.get(k) is not None:
             flags[key] = str(sampling[k])
+    if not s.get("presets"):
+        raise ExportError("set [export.pi] presets in models.toml to your llama.cpp router's --models-preset file")
     path = engine_mod.expand(s["presets"])
     old = _read(path)
     if not old:
@@ -183,7 +186,7 @@ def plan(eng, label, model_id=None, name=None):
                   "maxTokens": entry.get("maxTokens", sampling.get("max_tokens")),
                   "reasoning": bool(sampling.get("enable_thinking", True)),
                   "input": ["text", "image"] if vision else ["text"]})
-    notes.append("restart llama-router to load the new presets (llama-router restart)")
+    notes.append("restart the llama.cpp router so it reads the new presets")
     if sampling.get("reasoning_effort"):
         notes.append(f"the evals ran reasoning_effort {sampling['reasoning_effort']}: pick that thinking level in pi")
     pi_new = json.dumps(pi, indent=2, ensure_ascii=False) + "\n"

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/machines.py

@@ -1,6 +1,6 @@
 """Which machine this is, what a model needs, and whether it fits.
 
-    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m1max-32gb")
+    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m3max-64gb")
     read_gguf(path)       -> GGUF header facts (architecture, layers, KV heads, context length, …)
     fit(model_path, …)    -> Fit (largest context that fits this machine, estimated memory)
 
@@ -219,8 +219,7 @@ def swapped_out_bytes():
 
 
 # ---------------------------------------------------------------- GPU residency
-# Measured on an M1 Max 32 GB with macOS 27.0: once the system's GPU allocations exceed about half
-# of RAM, the GPU driver evicts and re-maps memory on every command submission (70-95% of the
+# On Apple Silicon Macs, once the system's GPU allocations exceed about half of RAM, the GPU driver evicts and re-maps memory on every command submission (70-95% of the
 # server's CPU in the kernel, GPU idle), whatever iogpu.wired_limit_mb says. Servers that submit
 # many small GPU jobs collapse; see the stall guard in docs/models.md.
 RESIDENCY_FRACTION = 0.5

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/run_evals.py

@@ -576,7 +576,7 @@ def cmd_export(argv):
                                 description="Write a model's serving settings (model file, output-affecting flags, "
                                             "context and the speed flags tuned on this machine) to another tool. "
                                             "Shows the diff and asks first; every file is backed up.")
-    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama-router presets and pi's "
+    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama.cpp router presets and pi's "
                                                   "models.json)")
     p.add_argument("labels", nargs="+", help="models to export")
     p.add_argument("--id", help="the model id pi sees (default: models.toml pi_id, else the GGUF's name)")

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/src/tuieval/templates/models.toml

@@ -107,7 +107,7 @@ headers = { "X-Title" = "tuieval" }
 # cmd = ["~/llama.cpp/build/bin/llama-bench"]
 
 # Optional per-machine settings, by machine id (tuieval machines shows the ids).
-# [machines.m4pro-24gb]
+# [machines.m3max-64gb]
 # memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
 # gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev3}/tests/test_tuieval.py

@@ -240,6 +240,31 @@ def fake_gguf(path, embedding=5120, size=0):
         f.write(out + b"\0" * max(0, size - len(out)))
 
 
+class FitCheck(unittest.TestCase):
+    def test_default_follows_the_ctx_placeholder(self):
+        from tuieval import engine
+        self.assertTrue(engine.fit_check({"cmd": ["llama-server", "-c", "{ctx}"]}))
+        self.assertFalse(engine.fit_check({"cmd": ["other-server", "--model", "{model}"]}))
+        self.assertFalse(engine.fit_check({"url": "http://x"}))
+        self.assertTrue(engine.fit_check({"cmd": ["other"], "fit_check": True}))
+        self.assertFalse(engine.fit_check({"cmd": ["llama", "-c", "{ctx}"], "fit_check": False}))
+
+    def test_own_memory_servers_keep_their_context(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            fake_gguf(os.path.join(tmp, "m.gguf"), size=1000)
+            with open(os.path.join(ws, "models.toml"), "a") as f:
+                f.write(f'\n[servers.own]\ncmd = ["own-server", "--model", "{{model}}"]\nport = 18500\nmodel_is_path = true\n'
+                        f'\n[[models]]\nlabel = "sized"\nserver = "llama"\nmodel = "{tmp}/m.gguf"\n'
+                        f'\n[[models]]\nlabel = "own"\nserver = "own"\nmodel = "{tmp}/m.gguf"\nmax_context = 131072\n')
+            code = ("from tuieval import engine; e = engine.Engine(); "
+                    "print(e.serving(e.model('own')).ctx, e.serving(e.model('sized')).fit_note != '')")
+            out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                                 env=dict(os.environ, TUIEVAL_HOME=ws)).stdout.split()
+            self.assertEqual(out, ["131072", "True"])   # own server: its max_context; llama: fit-checked
+
+
 class WarmTune(unittest.TestCase):
     """tune.tune starting from a tuned model of the same family, on a fake engine whose speed is a
     function of the flags."""
@@ -398,6 +423,11 @@ class ExportPi(unittest.TestCase):
         pi = json.loads(next(c for c in changes if c.path.endswith("models.json")).new)
         self.assertEqual(pi["providers"]["llama"]["modelOverrides"]["Old"]["contextWindow"], 98304)
 
+    def test_presets_path_is_required(self):
+        del self.eng.cfg["export"]["pi"]["presets"]
+        with self.assertRaises(self.export.ExportError):
+            self.export.plan(self.eng, "l")
+
     def test_only_llama_servers(self):
         with self.assertRaises(self.export.ExportError):
             self.export.plan(self.eng, "s")