tuieval 0.2.0.dev2__tar.gz → 0.2.0.dev3__tar.gz

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- The fit check (context sized from the GGUF header, llama.cpp's memory use) only applies to servers whose command takes `{ctx}`. Servers that size their own memory keep the model's `max_context` instead of an estimate that didn't apply to them; `fit_check = true|false` on a server overrides it.
 - `tuieval export pi` has no default presets path any more: set `[export.pi] presets` to your llama.cpp router's `--models-preset` file.
 - A new README screenshot from a demo workspace.
 - Versions come from git tags: a tag `vX.Y.Z` is that release, and every commit on `main` after it is published automatically as a dev build `X.(Y+1).0.devN` (N = commits since the release). `pip install tuieval` keeps installing releases only; `pip install --pre tuieval` gets the latest build.

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev2
+Version: 0.2.0.dev3
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/docs/models.md

@@ -69,7 +69,7 @@ The best server flags differ per model and per machine, so tuieval splits them b
 ## Machines
 
 - **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
-- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers.
+- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
 - **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
 
 ## Tuning
@@ -120,7 +120,8 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `url` | an already-running server's base URL |
 | `port` | the port `cmd` serves on |
 | `cwd` | folder to start `cmd` in |
-| `model_is_path` | `model` is a file or folder: check it exists (and, for GGUFs, that it fits) before starting |
+| `model_is_path` | `model` is a file or folder: check it exists before starting |
+| `fit_check` | size the context from the GGUF header and skip models that don't fit (the fit check; it assumes llama.cpp's memory use). Default: on for servers whose `cmd` takes `{ctx}`, off for others, whose context is the model's `max_context` |
 | `vision_args` | flags appended for models with `mmproj` |
 | `kv_type`, `max_ctx` | llama defaults: KV-cache type, upper bound on context |
 | `ctx_flag` | the server's context flag, so long packs are skipped if a configured context is too small |

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev2'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev2')
+__version__ = version = '0.2.0.dev3'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev3')
 
 __commit_id__ = commit_id = None

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/src/tuieval/engine.py

@@ -96,6 +96,16 @@ def load_config(path):
     return cfg
 
 
+def fit_check(server):
+    """Whether tuieval sizes the context for this server's models from GGUF headers (machines.fit,
+    which assumes llama.cpp's memory use). By default only servers whose command takes {ctx}, i.e.
+    where tuieval chooses the context; others size their own memory, and their context is the
+    model's max_context. models.toml `fit_check = true|false` on a server overrides it."""
+    if "fit_check" in server:
+        return bool(server["fit_check"])
+    return any("{ctx}" in str(a) for a in server.get("cmd", []))
+
+
 def effective_sampling(cfg, m):
     """The request settings this model runs with: [sampling], then the model's own overrides."""
     s = dict(cfg["sampling"])
@@ -1018,13 +1028,15 @@ class Engine:
         path = expand(m["model"])
         if server.get("model_is_path") and os.path.isfile(path):
             try:
-                mmproj = expand(m.get("mmproj", ""))
-                extra = os.path.getsize(mmproj) if mmproj and os.path.isfile(mmproj) else 0
-                headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
-                f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
-                                 want_ctx=want)
                 info = machines.read_gguf(path)
-                ctx, fits, note, mtp, size = f.max_ctx, f.fits, f.note, info["mtp_layers"], info["bytes"]
+                mtp, size = info["mtp_layers"], info["bytes"]
+                if fit_check(server):
+                    mmproj = expand(m.get("mmproj", ""))
+                    extra = os.path.getsize(mmproj) if mmproj and os.path.isfile(mmproj) else 0
+                    headroom = self.machine_settings(mc.id).get("memory_headroom_gb", 4.0)
+                    f = machines.fit(path, mc, kv_type=kv or "f16", headroom_gb=headroom, extra_bytes=extra,
+                                     want_ctx=want)
+                    ctx, fits, note = f.max_ctx, f.fits, f.note
             except (OSError, ValueError) as e:
                 note = f"couldn't read the GGUF header ({e}); context not checked"
         profile, perf, source = None, [], "n/a"

{tuieval-0.2.0.dev2 → tuieval-0.2.0.dev3}/tests/test_tuieval.py

@@ -240,6 +240,31 @@ def fake_gguf(path, embedding=5120, size=0):
         f.write(out + b"\0" * max(0, size - len(out)))
 
 
+class FitCheck(unittest.TestCase):
+    def test_default_follows_the_ctx_placeholder(self):
+        from tuieval import engine
+        self.assertTrue(engine.fit_check({"cmd": ["llama-server", "-c", "{ctx}"]}))
+        self.assertFalse(engine.fit_check({"cmd": ["other-server", "--model", "{model}"]}))
+        self.assertFalse(engine.fit_check({"url": "http://x"}))
+        self.assertTrue(engine.fit_check({"cmd": ["other"], "fit_check": True}))
+        self.assertFalse(engine.fit_check({"cmd": ["llama", "-c", "{ctx}"], "fit_check": False}))
+
+    def test_own_memory_servers_keep_their_context(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ws = os.path.join(tmp, "ws")
+            tuieval(tmp, "init", ws)
+            fake_gguf(os.path.join(tmp, "m.gguf"), size=1000)
+            with open(os.path.join(ws, "models.toml"), "a") as f:
+                f.write(f'\n[servers.own]\ncmd = ["own-server", "--model", "{{model}}"]\nport = 18500\nmodel_is_path = true\n'
+                        f'\n[[models]]\nlabel = "sized"\nserver = "llama"\nmodel = "{tmp}/m.gguf"\n'
+                        f'\n[[models]]\nlabel = "own"\nserver = "own"\nmodel = "{tmp}/m.gguf"\nmax_context = 131072\n')
+            code = ("from tuieval import engine; e = engine.Engine(); "
+                    "print(e.serving(e.model('own')).ctx, e.serving(e.model('sized')).fit_note != '')")
+            out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
+                                 env=dict(os.environ, TUIEVAL_HOME=ws)).stdout.split()
+            self.assertEqual(out, ["131072", "True"])   # own server: its max_context; llama: fit-checked
+
+
 class WarmTune(unittest.TestCase):
     """tune.tune starting from a tuned model of the same family, on a fake engine whose speed is a
     function of the flags."""