tuieval 0.2.0.dev1__tar.gz → 0.2.0.dev2__tar.gz

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/CHANGELOG.md

@@ -2,12 +2,14 @@
 
 ## Unreleased
 
+- `tuieval export pi` has no default presets path any more: set `[export.pi] presets` to your llama.cpp router's `--models-preset` file.
+- A new README screenshot from a demo workspace.
 - Versions come from git tags: a tag `vX.Y.Z` is that release, and every commit on `main` after it is published automatically as a dev build `X.(Y+1).0.devN` (N = commits since the release). `pip install tuieval` keeps installing releases only; `pip install --pre tuieval` gets the latest build.
 
 ## 0.1.7 (replaces 0.1.6, withdrawn)
 
 - `tuieval tune` starts warm for fine-tunes: when a model with the same architecture and tensor shapes is already tuned on this machine, its flags are the starting point and only speculative decoding and micro-batch are re-tried (~4-6 server starts instead of 8-15). It tunes in full if the inherited flags fail, change answers or are slower than the defaults. `--cold` (or `[tune] warm_start = false`) always tunes in full; `[tune] warm_retest` names the re-tried knobs.
-- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama-router preset section and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
+- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama.cpp router presets section and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
 - `tuieval list` lists the models, with hidden ones separately (`--packs` lists the packs).
 - `tuieval remove <model>` / `--hidden` removes models: their models.toml entry, results, tuning profiles and hidden mark, archived in removed/.
 - `tuieval --help` gives every command its own line.

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev1
+Version: 0.2.0.dev2
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -47,7 +47,7 @@ tuieval                        # open the TUI
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
-![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
 ## What you get
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/README.md

@@ -21,7 +21,7 @@ tuieval                        # open the TUI
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
-![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+![tuieval's setup screen with five eval packs and three models with their verdicts](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
 
 ## What you get
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/docs/models.md

@@ -68,7 +68,7 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 ## Machines
 
-- **Machine id** is detected automatically (`m1max-32gb`, `m4pro-24gb`, …; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
+- **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
 - **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers.
 - **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
 
@@ -90,18 +90,18 @@ The knobs are `[servers.<name>.tune]` in `models.toml`: each knob is a list of o
 
 `tuieval export pi <model>` makes pi serve a model the way the evals did: the same file, the output-affecting flags, the context from the fit check, the model's sampling, and the speed flags tuned on this machine. `tuieval tune <model> --export-pi` does it right after tuning.
 
-It writes a `[<id>]` section in llama-router's presets file (keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning). It exports models on llama servers.
+It writes a `[<id>]` section in the model presets file of a llama.cpp router (`llama-server --models-preset <file>`; keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning). It exports models on llama servers.
 
-It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards (`llama-router restart`).
+It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards so it reads the new presets.
 
 The id pi sees defaults to the GGUF's name; set `pi_id` and `pi_name` on the model (or pass `--id`/`--name`). Paths and provider names come from `[export.pi]` in `models.toml`:
 
 ```toml
 [export.pi]
-presets = "~/models/presets.ini"
-pi_models = "~/.pi/agent/models.json"
+presets = "/path/to/presets.ini"           # required: the router's --models-preset file
+pi_models = "~/.pi/agent/models.json"       # pi's default
 llama_provider = "llama"     # pi's provider name for the router
-servers = ["llama"]          # models.toml servers that llama-router can serve
+servers = ["llama"]          # models.toml servers the router can serve
 ```
 
 ## Speed verdicts
@@ -110,7 +110,7 @@ Speed is judged separately, per machine: the same model can be production-grade
 
 ## The stall guard (Apple Silicon)
 
-Measured on an M1 Max 32 GB: once the system's GPU allocations pass about half of RAM, the GPU driver evicts and re-maps memory on every GPU job, the server spends 70–95% of its CPU in the kernel while the GPU idles, and servers that submit many small GPU jobs slow to a crawl. For servers with `stall_guard = true`, tuieval samples the server's own vs kernel CPU time and the system's GPU allocation every 5 s during runs and tuning. If more than 70% of its CPU goes to the kernel for a minute, the run stops with an explanation instead of crawling for hours. Finished answers are kept and resume next time. If your machine behaves differently, set `gpu_residency_gb` under `[machines.<id>]`.
+On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM, the GPU driver evicts and re-maps memory on every GPU job, the server spends 70–95% of its CPU in the kernel while the GPU idles, and servers that submit many small GPU jobs slow to a crawl. For servers with `stall_guard = true`, tuieval samples the server's own vs kernel CPU time and the system's GPU allocation every 5 s during runs and tuning. If more than 70% of its CPU goes to the kernel for a minute, the run stops with an explanation instead of crawling for hours. Finished answers are kept and resume next time. If your machine behaves differently, set `gpu_residency_gb` under `[machines.<id>]`.
 
 ## Server options
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev1'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev1')
+__version__ = version = '0.2.0.dev2'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev2')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/compare.py

@@ -3,7 +3,7 @@
 Usage:
     tuieval compare                          # everything in results/<model>/<pack>.json
     tuieval compare --speed                  # speed & tokens table (TTFT, tok/s, tokens, memory)
-    tuieval compare --machine m4pro-24gb     # speed on another machine (measured there, or projected)
+    tuieval compare --machine m3max-64gb     # speed on another machine (measured there, or projected)
     tuieval compare --pairwise --failures    # is each difference real? which tests failed?
     tuieval compare results/some-model/*.json    # specific files
 """

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/export.py

@@ -1,17 +1,18 @@
 """Export a model's serving settings to the pi coding agent, so pi serves it exactly as the evals did:
 the same model file, output-affecting flags and context, plus the speed flags tuned on this machine.
 
-  a section in llama-router's presets (~/models/presets.ini; keys that equal the [*] section are
-  left out) and an entry under pi's `llama` provider modelOverrides. Models on llama servers only.
+  a section in a llama.cpp router's model presets file (llama-server --models-preset; keys that
+  equal its [*] section are left out) and an entry under pi's `llama` provider modelOverrides.
+  Models on llama servers only.
 
 Every file is backed up next to itself (<file>.bak-tuieval-<time>) before it changes, and the diff
 is shown first. Paths come from models.toml [export.pi]:
 
   [export.pi]
-  presets = "~/models/presets.ini"            # llama-router --models-preset
+  presets = "/path/to/presets.ini"            # required: the router's --models-preset file
   pi_models = "~/.pi/agent/models.json"
   llama_provider = "llama"                     # pi's provider name for the router
-  servers = ["llama"]                          # models.toml servers that llama-router can serve
+  servers = ["llama"]                          # models.toml servers the router can serve
 """
 import configparser
 import difflib
@@ -23,7 +24,7 @@ import time
 
 from . import engine as engine_mod
 
-DEFAULTS = {"presets": "~/models/presets.ini", "pi_models": "~/.pi/agent/models.json",
+DEFAULTS = {"presets": None, "pi_models": "~/.pi/agent/models.json",
             "llama_provider": "llama", "servers": ["llama"]}
 # llama.cpp short flags -> the long names presets use as keys
 SHORT = {"-m": "model", "-c": "ctx-size", "-t": "threads", "-tb": "threads-batch", "-ub": "ubatch-size",
@@ -127,7 +128,7 @@ def plan(eng, label, model_id=None, name=None):
     m = eng.model(label)
     s = settings(eng)
     if m["server"] not in s["servers"]:
-        raise ExportError(f"{label} runs on the {m['server']} server; pi export writes llama-router presets, "
+        raise ExportError(f"{label} runs on the {m['server']} server; pi export writes llama.cpp router presets, "
                           "for llama servers ([export.pi] servers lists them)")
     model_id = model_id or m.get("pi_id") or default_id(m)
     name = name or m.get("pi_name") or model_id
@@ -157,6 +158,8 @@ def plan(eng, label, model_id=None, name=None):
     for k, key in SAMPLING_KEYS.items():
         if sampling.get(k) is not None:
             flags[key] = str(sampling[k])
+    if not s.get("presets"):
+        raise ExportError("set [export.pi] presets in models.toml to your llama.cpp router's --models-preset file")
     path = engine_mod.expand(s["presets"])
     old = _read(path)
     if not old:
@@ -183,7 +186,7 @@ def plan(eng, label, model_id=None, name=None):
                   "maxTokens": entry.get("maxTokens", sampling.get("max_tokens")),
                   "reasoning": bool(sampling.get("enable_thinking", True)),
                   "input": ["text", "image"] if vision else ["text"]})
-    notes.append("restart llama-router to load the new presets (llama-router restart)")
+    notes.append("restart the llama.cpp router so it reads the new presets")
     if sampling.get("reasoning_effort"):
         notes.append(f"the evals ran reasoning_effort {sampling['reasoning_effort']}: pick that thinking level in pi")
     pi_new = json.dumps(pi, indent=2, ensure_ascii=False) + "\n"

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/machines.py

@@ -1,6 +1,6 @@
 """Which machine this is, what a model needs, and whether it fits.
 
-    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m1max-32gb")
+    detect()              -> Machine (chip, cores, RAM, GPU memory limit, id like "m3max-64gb")
     read_gguf(path)       -> GGUF header facts (architecture, layers, KV heads, context length, …)
     fit(model_path, …)    -> Fit (largest context that fits this machine, estimated memory)
 
@@ -219,8 +219,7 @@ def swapped_out_bytes():
 
 
 # ---------------------------------------------------------------- GPU residency
-# Measured on an M1 Max 32 GB with macOS 27.0: once the system's GPU allocations exceed about half
-# of RAM, the GPU driver evicts and re-maps memory on every command submission (70-95% of the
+# On Apple Silicon Macs, once the system's GPU allocations exceed about half of RAM, the GPU driver evicts and re-maps memory on every command submission (70-95% of the
 # server's CPU in the kernel, GPU idle), whatever iogpu.wired_limit_mb says. Servers that submit
 # many small GPU jobs collapse; see the stall guard in docs/models.md.
 RESIDENCY_FRACTION = 0.5

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/run_evals.py

@@ -576,7 +576,7 @@ def cmd_export(argv):
                                 description="Write a model's serving settings (model file, output-affecting flags, "
                                             "context and the speed flags tuned on this machine) to another tool. "
                                             "Shows the diff and asks first; every file is backed up.")
-    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama-router presets and pi's "
+    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama.cpp router presets and pi's "
                                                   "models.json)")
     p.add_argument("labels", nargs="+", help="models to export")
     p.add_argument("--id", help="the model id pi sees (default: models.toml pi_id, else the GGUF's name)")

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/src/tuieval/templates/models.toml

@@ -107,7 +107,7 @@ headers = { "X-Title" = "tuieval" }
 # cmd = ["~/llama.cpp/build/bin/llama-bench"]
 
 # Optional per-machine settings, by machine id (tuieval machines shows the ids).
-# [machines.m4pro-24gb]
+# [machines.m3max-64gb]
 # memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
 # gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
 

{tuieval-0.2.0.dev1 → tuieval-0.2.0.dev2}/tests/test_tuieval.py

@@ -398,6 +398,11 @@ class ExportPi(unittest.TestCase):
         pi = json.loads(next(c for c in changes if c.path.endswith("models.json")).new)
         self.assertEqual(pi["providers"]["llama"]["modelOverrides"]["Old"]["contextWindow"], 98304)
 
+    def test_presets_path_is_required(self):
+        del self.eng.cfg["export"]["pi"]["presets"]
+        with self.assertRaises(self.export.ExportError):
+            self.export.plan(self.eng, "l")
+
     def test_only_llama_servers(self):
         with self.assertRaises(self.export.ExportError):
             self.export.plan(self.eng, "s")