tuieval 0.1.6__tar.gz → 0.2.0.dev1__tar.gz

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/.gitignore

@@ -11,3 +11,4 @@ dist/
 /logs/
 /tuning/
 /packs/
+src/tuieval/_version.py

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/CHANGELOG.md

@@ -1,9 +1,13 @@
 # Changelog
 
-## 0.1.6
+## Unreleased
+
+- Versions come from git tags: a tag `vX.Y.Z` is that release, and every commit on `main` after it is published automatically as a dev build `X.(Y+1).0.devN` (N = commits since the release). `pip install tuieval` keeps installing releases only; `pip install --pre tuieval` gets the latest build.
+
+## 0.1.7 (replaces 0.1.6, withdrawn)
 
 - `tuieval tune` starts warm for fine-tunes: when a model with the same architecture and tensor shapes is already tuned on this machine, its flags are the starting point and only speculative decoding and micro-batch are re-tried (~4-6 server starts instead of 8-15). It tunes in full if the inherited flags fail, change answers or are slower than the defaults. `--cold` (or `[tune] warm_start = false`) always tunes in full; `[tune] warm_retest` names the re-tried knobs.
-- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama-router preset section or an amalgam router entry, and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
+- `tuieval export pi <model>` (and `tuieval tune --export-pi`) writes a model's serving settings to the pi coding agent: a llama-router preset section and pi's models.json. It shows the diff, asks, and backs every file up. See docs/models.md.
 - `tuieval list` lists the models, with hidden ones separately (`--packs` lists the packs).
 - `tuieval remove <model>` / `--hidden` removes models: their models.toml entry, results, tuning profiles and hidden mark, archived in removed/.
 - `tuieval --help` gives every command its own line.

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.1.6
+Version: 0.2.0.dev1
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -43,6 +43,8 @@ tuieval add ~/models/Some-Model-Q4_K_M.gguf
 tuieval                        # open the TUI
 ```
 
+`pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
+
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
 ![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/README.md

@@ -17,6 +17,8 @@ tuieval add ~/models/Some-Model-Q4_K_M.gguf
 tuieval                        # open the TUI
 ```
 
+`pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
+
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
 
 ![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)

tuieval-0.2.0.dev1/RELEASING.md

@@ -0,0 +1,31 @@
+# Releasing
+
+Versions come from git (hatch-vcs). Nobody edits a version number by hand.
+
+| What | Version | Who gets it |
+|---|---|---|
+| A tag `vX.Y.Z` | `X.Y.Z` | `pip install tuieval` |
+| Every other commit on `main` | `X.(Y+1).0.devN`, N = commits since the last tag | `pip install --pre tuieval` |
+
+CI (`.github/workflows/test.yml`) runs the tests on every push. When they pass, the `publish` job builds the package, checks it with `twine check`, and uploads it to PyPI with trusted publishing (no token is stored anywhere): every push to `main` as a dev build, every `v*` tag as a release.
+
+## Day to day
+
+1. Commit and push to `main`. Add a line under `## Unreleased` in `CHANGELOG.md` for anything users would notice.
+2. CI publishes the dev build. Check it: `pipx install --force --pip-args=--pre tuieval && tuieval --version`.
+
+## A release
+
+Cut one when there's something worth announcing, not for every change. Semantic versioning: PATCH for fixes, MINOR for features (and anything that changes behaviour while the major version is 0), MAJOR for breaking changes after 1.0.
+
+1. In `CHANGELOG.md`, rename `## Unreleased` to `## X.Y.Z` and add a new empty `## Unreleased` above it. Commit and push.
+2. Tag that commit and push the tag: `git tag -a vX.Y.Z -m "X.Y.Z" && git push origin vX.Y.Z`.
+3. CI publishes `X.Y.Z`. Check a fresh install: `pipx install --force tuieval==X.Y.Z && tuieval --version`.
+
+## One-time setup (done once per PyPI project)
+
+On PyPI, under the project's Publishing settings, add a trusted publisher: owner `ashe-wb`, repository `tuieval`, workflow `test.yml`, environment `pypi`.
+
+## Building by hand
+
+`python -m pip install build twine && python -m build && twine check dist/*`. A build needs the git history and tags (a full clone), or it can't tell its version.

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/docs/models.md

@@ -90,10 +90,9 @@ The knobs are `[servers.<name>.tune]` in `models.toml`: each knob is a list of o
 
 `tuieval export pi <model>` makes pi serve a model the way the evals did: the same file, the output-affecting flags, the context from the fit check, the model's sampling, and the speed flags tuned on this machine. `tuieval tune <model> --export-pi` does it right after tuning.
 
-- **llama models:** a `[<id>]` section in llama-router's presets file (keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning).
-- **amalgam models:** a `[models."<id>"]` entry in the amalgam router's `models.toml` and an entry in pi's `amalgam` provider models (copied from the first one there, then name, context and vision set). The router passes the same args to every model, so a model's own `server_args` can't be exported; the export says so.
+It writes a `[<id>]` section in llama-router's presets file (keys that equal its `[*]` section are left out) and an entry under pi's `llama` provider `modelOverrides` (name, context window, vision, reasoning). It exports models on llama servers.
 
-It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards (`llama-router restart` or `amalgam restart`).
+It shows the diff and the notes first (untuned or outdated tuning, other preset sections that name missing files, a reasoning effort to pick in pi), asks, backs every file up as `<file>.bak-tuieval-<time>`, and refuses to write a file that changed in the meantime. `--dry-run` only shows; `--yes` doesn't ask. Restart the router afterwards (`llama-router restart`).
 
 The id pi sees defaults to the GGUF's name; set `pi_id` and `pi_name` on the model (or pass `--id`/`--name`). Paths and provider names come from `[export.pi]` in `models.toml`:
 
@@ -101,10 +100,8 @@ The id pi sees defaults to the GGUF's name; set `pi_id` and `pi_name` on the mod
 [export.pi]
 presets = "~/models/presets.ini"
 pi_models = "~/.pi/agent/models.json"
-amalgam_models = "~/amalgam/local/models.toml"
-llama_provider = "llama"
-amalgam_provider = "amalgam"
-servers = { llama = "llama", amalgam = "amalgam" }   # models.toml server -> which export it gets
+llama_provider = "llama"     # pi's provider name for the router
+servers = ["llama"]          # models.toml servers that llama-router can serve
 ```
 
 ## Speed verdicts

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/pyproject.toml

@@ -1,5 +1,5 @@
 [build-system]
-requires = ["hatchling>=1.24"]
+requires = ["hatchling>=1.24", "hatch-vcs>=0.4"]
 build-backend = "hatchling.build"
 
 [project]
@@ -37,8 +37,17 @@ tuieval = "tuieval.cli:main"
 Homepage = "https://github.com/ashe-wb/tuieval"
 Issues = "https://github.com/ashe-wb/tuieval/issues"
 
+# The version comes from git: a tag vX.Y.Z is that release; every commit after it on main is
+# X.(Y+1).0.devN, N = commits since the tag (see RELEASING.md).
 [tool.hatch.version]
-path = "src/tuieval/__init__.py"
+source = "vcs"
+
+[tool.hatch.version.raw-options]
+version_scheme = "release-branch-semver"
+local_scheme = "no-local-version"
+
+[tool.hatch.build.hooks.vcs]
+version-file = "src/tuieval/_version.py"
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/tuieval"]

tuieval-0.2.0.dev1/src/tuieval/__init__.py

@@ -0,0 +1,5 @@
+"""tuieval: evaluate local and frontier models on your own eval packs, in the terminal."""
+try:
+    from ._version import __version__   # written by the build from the git tag (RELEASING.md)
+except ImportError:                      # a source tree that was never built or installed
+    __version__ = "0+unknown"

tuieval-0.2.0.dev1/src/tuieval/_version.py

@@ -0,0 +1,24 @@
+# file generated by vcs-versioning
+# don't change, don't track in version control
+from __future__ import annotations
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+version: str
+__version__: str
+__version_tuple__: tuple[int | str, ...]
+version_tuple: tuple[int | str, ...]
+commit_id: str | None
+__commit_id__: str | None
+
+__version__ = version = '0.2.0.dev1'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev1')
+
+__commit_id__ = commit_id = None

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/src/tuieval/export.py

@@ -1,10 +1,8 @@
 """Export a model's serving settings to the pi coding agent, so pi serves it exactly as the evals did:
 the same model file, output-affecting flags and context, plus the speed flags tuned on this machine.
 
-  llama models    a section in llama-router's presets (~/models/presets.ini; keys that equal the
-                  [*] section are left out) and an entry under pi's `llama` provider modelOverrides
-  amalgam models  a [models."<id>"] entry in the amalgam router's models.toml and an entry in pi's
-                  `amalgam` provider models
+  a section in llama-router's presets (~/models/presets.ini; keys that equal the [*] section are
+  left out) and an entry under pi's `llama` provider modelOverrides. Models on llama servers only.
 
 Every file is backed up next to itself (<file>.bak-tuieval-<time>) before it changes, and the diff
 is shown first. Paths come from models.toml [export.pi]:
@@ -12,10 +10,8 @@ is shown first. Paths come from models.toml [export.pi]:
   [export.pi]
   presets = "~/models/presets.ini"            # llama-router --models-preset
   pi_models = "~/.pi/agent/models.json"
-  amalgam_models = "~/amalgam/local/models.toml"
-  llama_provider = "llama"                     # pi provider names
-  amalgam_provider = "amalgam"
-  servers = { llama = "llama", amalgam = "amalgam" }   # models.toml server -> export kind
+  llama_provider = "llama"                     # pi's provider name for the router
+  servers = ["llama"]                          # models.toml servers that llama-router can serve
 """
 import configparser
 import difflib
@@ -28,8 +24,7 @@ import time
 from . import engine as engine_mod
 
 DEFAULTS = {"presets": "~/models/presets.ini", "pi_models": "~/.pi/agent/models.json",
-            "amalgam_models": "~/amalgam/local/models.toml", "llama_provider": "llama",
-            "amalgam_provider": "amalgam", "servers": {"llama": "llama", "amalgam": "amalgam"}}
+            "llama_provider": "llama", "servers": ["llama"]}
 # llama.cpp short flags -> the long names presets use as keys
 SHORT = {"-m": "model", "-c": "ctx-size", "-t": "threads", "-tb": "threads-batch", "-ub": "ubatch-size",
          "-b": "batch-size", "-fa": "flash-attn", "-ngl": "n-gpu-layers", "-np": "parallel",
@@ -42,9 +37,7 @@ class ExportError(Exception):
 
 
 def settings(eng):
-    s = {**DEFAULTS, **eng.cfg.get("export", {}).get("pi", {})}
-    s["servers"] = {**DEFAULTS["servers"], **s.get("servers", {})}
-    return s
+    return {**DEFAULTS, **eng.cfg.get("export", {}).get("pi", {})}
 
 
 def default_id(m):
@@ -133,12 +126,11 @@ def plan(eng, label, model_id=None, name=None):
     """(changes, notes) that export one model to pi. Nothing is written."""
     m = eng.model(label)
     s = settings(eng)
-    kind = s["servers"].get(m["server"])
-    if kind not in ("llama", "amalgam"):
-        raise ExportError(f"{label} runs on the {m['server']} server; pi export knows llama and amalgam "
-                          "servers ([export.pi] servers maps others)")
+    if m["server"] not in s["servers"]:
+        raise ExportError(f"{label} runs on the {m['server']} server; pi export writes llama-router presets, "
+                          "for llama servers ([export.pi] servers lists them)")
     model_id = model_id or m.get("pi_id") or default_id(m)
-    name = name or m.get("pi_name") or f"{model_id} ({kind})"
+    name = name or m.get("pi_name") or model_id
     sv = eng.serving(m)
     notes = []
     sampling = engine_mod.effective_sampling(eng.cfg, m)
@@ -150,85 +142,48 @@ def plan(eng, label, model_id=None, name=None):
     pi = json.loads(pi_old)
     providers = pi.setdefault("providers", {})
     changes = []
-    if kind == "llama":
-        if sv.perf_source == "untuned":
-            notes.append(f"{label} isn't tuned on this machine: exporting the untuned defaults (tuieval tune {label})")
-        elif sv.perf_source.startswith("outdated"):
-            notes.append(f"{label}'s tuning is {sv.perf_source}; consider tuieval tune {label}")
-        cmd, _, _ = eng.server_command(m)
-        if cmd[0] == "env":
-            raise ExportError("the server command sets environment variables, which presets can't hold")
-        first = next(i for i, a in enumerate(cmd) if a.startswith("-"))
-        flags = {}
-        for k, v in flags_to_keys(cmd[first:]):
-            if k not in ROUTER_OWNED:
-                flags[k] = v   # a repeated flag: the last wins, as on the command line
-        for k, key in SAMPLING_KEYS.items():
-            if sampling.get(k) is not None:
-                flags[key] = str(sampling[k])
-        path = engine_mod.expand(s["presets"])
-        old = _read(path)
-        if not old:
-            raise ExportError(f"presets file {path} not found ([export.pi] presets)")
-        sections = _ini_sections(old)
-        common = sections.get("*", {})
-        order = ["model", "mmproj"] + [k for k in flags if k not in ("model", "mmproj")]
-        lines = [f"{k:<16} = {flags[k]}" for k in order if k in flags and common.get(k) != flags[k]]
-        tuned = {k for opts in eng.cfg["servers"][m["server"]].get("tune", {}).values() for o in opts
-                 for k, _ in flags_to_keys(o)}
-        for k in common:
-            if k in tuned and k not in flags:
-                notes.append(f"presets [*] sets {k} = {common[k]}, which {label}'s tuned flags leave off; "
-                             "the router will still apply it")
-        broken = [sec for sec, kv in sections.items() if sec != model_id
-                  and any(kv.get(k) and not os.path.exists(os.path.expanduser(kv[k])) for k in ("model", "mmproj"))]
-        if broken:
-            notes.append(f"{path}: {len(broken)} other section(s) name files that don't exist, so pi can't load "
-                         f"them: {', '.join(broken)}")
-        changes.append(Change(path, old, set_ini_section(old, model_id, lines)))
-        prov = providers.setdefault(s["llama_provider"], {})
-        entry = prov.setdefault("modelOverrides", {}).setdefault(model_id, {})
-        entry.update({"name": name, "contextWindow": sv.ctx,
-                      "maxTokens": entry.get("maxTokens", sampling.get("max_tokens")),
-                      "reasoning": bool(sampling.get("enable_thinking", True)),
-                      "input": ["text", "image"] if vision else ["text"]})
-        notes.append("restart llama-router to load the new presets (llama-router restart)")
-    else:
-        path = engine_mod.expand(s["amalgam_models"])
-        old = _read(path)
-        if not old:
-            raise ExportError(f"amalgam router config {path} not found ([export.pi] amalgam_models)")
-        import tomllib
-        router = tomllib.loads(old)
-        want = m["model"]
-        have = router.get("models", {}).get(model_id, {}).get("path")
-        if have is None:
-            new = old.rstrip("\n") + f'\n\n[models."{model_id}"]\npath = "{want}"\n'
-        elif engine_mod.expand(have) != engine_mod.expand(want):
-            new = re.sub(r'(\[models\."' + re.escape(model_id) + r'"\][^\[]*?path\s*=\s*)"[^"]*"',
-                         lambda h: h.group(1) + f'"{want}"', old, count=1)
-        else:
-            new = old
-        if new != old:
-            changes.append(Change(path, old, new))
-        args = [str(a) for a in router.get("args", [])]
-        if m.get("server_args"):
-            notes.append(f"{label} has server_args {m['server_args']}, but the amalgam router passes the same "
-                         f"args to every model ({' '.join(args)}): pi won't get them")
-        ctx = m.get("max_context") or sv.ctx
-        if "--max-context" in args[:-1] and ctx and str(ctx) != args[args.index("--max-context") + 1]:
-            notes.append(f"the router serves --max-context {args[args.index('--max-context') + 1]}; the evals ran {ctx}")
-        prov = providers.setdefault(s["amalgam_provider"], {})
-        models = prov.setdefault("models", [])
-        entry = next((x for x in models if x.get("id") == model_id), None)
-        if entry is None:
-            template = {k: v for k, v in (models[0] if models else {}).items() if k not in ("id", "name")}
-            entry = {"id": model_id, "name": name, **template}
-            models.append(entry)
-        entry.update({"name": name, "reasoning": bool(sampling.get("enable_thinking", True)),
-                      "input": ["text", "image"] if vision else ["text"], "contextWindow": ctx})
-        entry.setdefault("maxTokens", sampling.get("max_tokens"))
-        notes.append("restart the amalgam router to serve it (amalgam restart)")
+    if sv.perf_source == "untuned":
+        notes.append(f"{label} isn't tuned on this machine: exporting the untuned defaults (tuieval tune {label})")
+    elif sv.perf_source.startswith("outdated"):
+        notes.append(f"{label}'s tuning is {sv.perf_source}; consider tuieval tune {label}")
+    cmd, _, _ = eng.server_command(m)
+    if cmd[0] == "env":
+        raise ExportError("the server command sets environment variables, which presets can't hold")
+    first = next(i for i, a in enumerate(cmd) if a.startswith("-"))
+    flags = {}
+    for k, v in flags_to_keys(cmd[first:]):
+        if k not in ROUTER_OWNED:
+            flags[k] = v   # a repeated flag: the last wins, as on the command line
+    for k, key in SAMPLING_KEYS.items():
+        if sampling.get(k) is not None:
+            flags[key] = str(sampling[k])
+    path = engine_mod.expand(s["presets"])
+    old = _read(path)
+    if not old:
+        raise ExportError(f"presets file {path} not found ([export.pi] presets)")
+    sections = _ini_sections(old)
+    common = sections.get("*", {})
+    order = ["model", "mmproj"] + [k for k in flags if k not in ("model", "mmproj")]
+    lines = [f"{k:<16} = {flags[k]}" for k in order if k in flags and common.get(k) != flags[k]]
+    tuned = {k for opts in eng.cfg["servers"][m["server"]].get("tune", {}).values() for o in opts
+             for k, _ in flags_to_keys(o)}
+    for k in common:
+        if k in tuned and k not in flags:
+            notes.append(f"presets [*] sets {k} = {common[k]}, which {label}'s tuned flags leave off; "
+                         "the router will still apply it")
+    broken = [sec for sec, kv in sections.items() if sec != model_id
+              and any(kv.get(k) and not os.path.exists(os.path.expanduser(kv[k])) for k in ("model", "mmproj"))]
+    if broken:
+        notes.append(f"{path}: {len(broken)} other section(s) name files that don't exist, so pi can't load "
+                     f"them: {', '.join(broken)}")
+    changes.append(Change(path, old, set_ini_section(old, model_id, lines)))
+    prov = providers.setdefault(s["llama_provider"], {})
+    entry = prov.setdefault("modelOverrides", {}).setdefault(model_id, {})
+    entry.update({"name": name, "contextWindow": sv.ctx,
+                  "maxTokens": entry.get("maxTokens", sampling.get("max_tokens")),
+                  "reasoning": bool(sampling.get("enable_thinking", True)),
+                  "input": ["text", "image"] if vision else ["text"]})
+    notes.append("restart llama-router to load the new presets (llama-router restart)")
     if sampling.get("reasoning_effort"):
         notes.append(f"the evals ran reasoning_effort {sampling['reasoning_effort']}: pick that thinking level in pi")
     pi_new = json.dumps(pi, indent=2, ensure_ascii=False) + "\n"

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/src/tuieval/run_evals.py

@@ -576,11 +576,11 @@ def cmd_export(argv):
                                 description="Write a model's serving settings (model file, output-affecting flags, "
                                             "context and the speed flags tuned on this machine) to another tool. "
                                             "Shows the diff and asks first; every file is backed up.")
-    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama-router presets or the amalgam "
-                                                  "router, and pi's models.json)")
+    p.add_argument("target", choices=["pi"], help="pi: the pi coding agent (llama-router presets and pi's "
+                                                  "models.json)")
     p.add_argument("labels", nargs="+", help="models to export")
     p.add_argument("--id", help="the model id pi sees (default: models.toml pi_id, else the GGUF's name)")
-    p.add_argument("--name", help="the name pi shows (default: models.toml pi_name, else '<id> (<server>)')")
+    p.add_argument("--name", help="the name pi shows (default: models.toml pi_name, else the id)")
     p.add_argument("--dry-run", action="store_true", help="show the changes only")
     p.add_argument("--yes", action="store_true", help="write without asking")
     p.add_argument("--models", default=None, help="default: the workspace's models.toml")

{tuieval-0.1.6 → tuieval-0.2.0.dev1}/tests/test_tuieval.py

@@ -329,7 +329,7 @@ class WarmTune(unittest.TestCase):
 
 
 class ExportPi(unittest.TestCase):
-    """export.plan/apply against temporary presets, pi models.json and amalgam router files."""
+    """export.plan/apply against a temporary presets file and pi models.json."""
     PRESETS = textwrap.dedent("""\
         version = 1
 
@@ -356,19 +356,14 @@ class ExportPi(unittest.TestCase):
         fake_gguf(f"{d}/Old.gguf")
         write(f"{d}/presets.ini", self.PRESETS)
         write(f"{d}/models.json", json.dumps({"providers": {
-            "amalgam": {"baseUrl": "http://x", "apiKey": "SECRET", "models": [
-                {"id": "A", "name": "a", "maxTokens": 32768, "thinkingLevelMap": {"off": "none"}}]},
-            "llama": {"modelOverrides": {}}}}, indent=2) + "\n")
-        write(f"{d}/router.toml", 'args = ["--max-context", "131072"]\n\n[models."A"]\npath = "~/a.gguf"\n')
+            "llama": {"apiKey": "SECRET", "modelOverrides": {}}}}, indent=2) + "\n")
         cfg = {"sampling": {"temperature": 1.0, "max_tokens": 16384, "enable_thinking": True},
                "servers": {"llama": {"cmd": ["llama"], "tune": {"cache_reuse": [["--cache-reuse", "256"], []]}},
-                           "amalgam": {"cmd": ["splash"]}},
+                           "mlx": {"cmd": ["mlx"]}},
                "models": [{"label": "l", "server": "llama", "model": f"{d}/Old.gguf",
                            "sampling": {"presence_penalty": 1.5}},
-                          {"label": "s", "server": "amalgam", "model": "~/b.gguf", "vision": True,
-                           "max_context": 65536, "server_args": ["--max-context", "64K"]}],
-               "export": {"pi": {"presets": f"{d}/presets.ini", "pi_models": f"{d}/models.json",
-                                 "amalgam_models": f"{d}/router.toml"}}}
+                          {"label": "s", "server": "mlx", "model": "~/b"}],
+               "export": {"pi": {"presets": f"{d}/presets.ini", "pi_models": f"{d}/models.json"}}}
         eng = types.SimpleNamespace(cfg=cfg)
         eng.model = lambda label: next(m for m in cfg["models"] if m["label"] == label)
         eng.serving = lambda m: types.SimpleNamespace(perf_source="tuned", ctx=98304)
@@ -403,30 +398,28 @@ class ExportPi(unittest.TestCase):
         pi = json.loads(next(c for c in changes if c.path.endswith("models.json")).new)
         self.assertEqual(pi["providers"]["llama"]["modelOverrides"]["Old"]["contextWindow"], 98304)
 
-    def test_amalgam_adds_router_entry_and_pi_model(self):
-        changes, notes = self.export.plan(self.eng, "s", model_id="B", name="B!")
-        router = next(c for c in changes if c.path.endswith("router.toml")).new
-        self.assertTrue(router.endswith('[models."B"]\npath = "~/b.gguf"\n'))
-        pi_change = next(c for c in changes if c.path.endswith("models.json"))
-        entry = json.loads(pi_change.new)["providers"]["amalgam"]["models"][1]
-        self.assertEqual((entry["id"], entry["name"], entry["contextWindow"], entry["input"], entry["maxTokens"]),
-                         ("B", "B!", 65536, ["text", "image"], 32768))
-        self.assertTrue(any("server_args" in n for n in notes))
-        self.assertNotIn("SECRET", pi_change.diff())
+    def test_only_llama_servers(self):
+        with self.assertRaises(self.export.ExportError):
+            self.export.plan(self.eng, "s")
+
+    def test_diff_hides_credentials(self):
+        changes, _ = self.export.plan(self.eng, "l", model_id="New")
+        diff = next(c for c in changes if c.path.endswith("models.json")).diff()
+        self.assertIn('"New"', diff)
+        self.assertNotIn("SECRET", diff)
 
     def test_apply_backs_up_and_refuses_a_file_changed_since(self):
-        changes, _ = self.export.plan(self.eng, "s", model_id="B")
+        changes, _ = self.export.plan(self.eng, "l", model_id="New")
         backups = self.export.apply(changes)
         self.assertEqual(len(backups), 2)
-        self.assertIn('"B"', read(f"{self.d}/router.toml"))
-        self.assertNotIn('"B"', read(next(b for b in backups if "router.toml" in b)))
-        self.assertEqual(self.export.plan(self.eng, "s", model_id="B")[0], [])   # already exported
-        changes, _ = self.export.plan(self.eng, "s", model_id="C")
-        write(f"{self.d}/router.toml", "edited\n")
+        self.assertIn("[New]", read(f"{self.d}/presets.ini"))
+        self.assertNotIn("[New]", read(next(b for b in backups if "presets.ini" in b)))
+        self.assertEqual(self.export.plan(self.eng, "l", model_id="New")[0], [])   # already exported
+        changes, _ = self.export.plan(self.eng, "l", model_id="Other")
+        write(f"{self.d}/presets.ini", "edited\n")
         with self.assertRaises(self.export.ExportError):
             self.export.apply(changes)
 
-
 class Remove(unittest.TestCase):
     TOML = textwrap.dedent("""\
         [servers.llama]

tuieval-0.1.6/RELEASING.md

@@ -1,7 +0,0 @@
-# Releasing
-
-1. Bump `__version__` in `src/tuieval/__init__.py` and add a `CHANGELOG.md` entry.
-2. Commit, push, and wait for CI to pass.
-3. Build from a clean checkout and check: `python -m pip install build twine && python -m build && twine check dist/*`
-4. Publish to PyPI: `twine upload dist/*` (username `__token__`, password: a PyPI API token).
-5. Check a fresh install: `pipx install --force tuieval==<version> && tuieval --version`

tuieval-0.1.6/src/tuieval/__init__.py

@@ -1,2 +0,0 @@
-"""tuieval: evaluate local and frontier models on your own eval packs, in the terminal."""
-__version__ = "0.1.6"