tuieval 0.2.0.dev14__tar.gz → 0.2.0.dev28__tar.gz

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## Unreleased
 
+- Run several models at a time: `parallel_models` per machine, **Models at a time** in the TUI, or `tuieval run --parallel N` (default 1). Models wait for memory to fit, and answers note what ran alongside them.
+- A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.
+- Less to take in at first: until something has run, the setup screen leaves out context sizes, tuning state and the Tune key; Presets and Hide are off the footer (`?` lists every key); `tuieval help` groups commands into Start here, Day to day and Advanced.
+- Results lead with a plain summary per model ("ready for Support; not decided yet for Coding → run Certify to finish"), in the TUI, `tuieval verdict` (with the command to run next) and the report; the statistics follow.
+- First packs without hand-written YAML: `tuieval new-pack <name> --from questions.csv` (question and answer columns) and `--about "<topic>"` (writes a prompt for a strong model to draft the tests). A test can take its prompt from a file (`input_file`). `tuieval selftest` errors say how to fix them.
+- `tuieval init` in a terminal is a guided setup: it finds models on servers already running (LM Studio, Ollama, llama-server, vLLM), adds the ones you pick, creates a starter pack and runs a Smoke check (`--yes` for no questions).
+- `tuieval add` without a model lists the models on running servers to pick from; `tuieval add <id>` finds the server that has it and adds that server to models.toml. The TUI's add dialog lists them too, with rarely needed options folded away.
+- `?` on any TUI screen explains what it's for, its keys, and the words tuieval uses (tiers, verdicts, critical failures, gates).
+- The setup screen is ready on first open: the only pack and model are ticked, the tier is Smoke until something has run, and ticked boxes are clearly marked (unticked ones are empty).
+- First runs fail fast with the fix instead of hanging: a server that isn't running fails in seconds (it used to wait up to 20 minutes), a missing `llama-server` says how to install it, and `tuieval run` with no models says how to add one.
+- `tuieval doctor` checks servers, models, packs and API keys, and says what to fix.
 - Run only some tests of a pack: `e` in the TUI, or `tuieval run --tests`. Runs add up; `--force` reruns just the picked tests. Presets keep the pick.
 - `tuieval tune` scores options on projected full-length answers (`[tune] answer_tokens`, default 1024), so generation speed counts as in real runs.
 - `tuieval tune` reports an option's memory cost instead of rejecting it; it rejects only on critical memory pressure or over `[tune] swap_limit_mb`.

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev14
+Version: 0.2.0.dev28
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -43,12 +43,21 @@ Then you get an answer to the questions that matter:
 
 ```bash
 pipx install tuieval          # or: uv tool install tuieval, or pip install tuieval   (Python 3.11+)
-tuieval init my-evals && cd my-evals
-tuieval new-pack my-first-pack # a pack of example questions to edit
-tuieval add ~/models/Some-Model-Q4_K_M.gguf
-tuieval                        # open the TUI
+tuieval init my-evals          # guided: finds models already running (LM Studio, Ollama, vLLM…),
+                               # adds a starter pack and runs a 3-question check
+cd my-evals && tuieval         # open the TUI (? explains any screen)
 ```
 
+Whatever runs your models, the first steps are short:
+
+| You have | Do this |
+|---|---|
+| **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
+
+Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
+
 `pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
@@ -117,7 +126,8 @@ tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it
 ```bash
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
-tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval add                                                         # pick from the models on running servers
+tuieval add qwen3:8b                                                # finds the running server that has it
 tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
 ```
 
@@ -130,7 +140,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
-Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
 ## Command line
 
@@ -139,6 +149,7 @@ tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
 tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
+tuieval doctor                                # check servers, models, packs and keys, and what to fix
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/README.md

@@ -17,12 +17,21 @@ Then you get an answer to the questions that matter:
 
 ```bash
 pipx install tuieval          # or: uv tool install tuieval, or pip install tuieval   (Python 3.11+)
-tuieval init my-evals && cd my-evals
-tuieval new-pack my-first-pack # a pack of example questions to edit
-tuieval add ~/models/Some-Model-Q4_K_M.gguf
-tuieval                        # open the TUI
+tuieval init my-evals          # guided: finds models already running (LM Studio, Ollama, vLLM…),
+                               # adds a starter pack and runs a 3-question check
+cd my-evals && tuieval         # open the TUI (? explains any screen)
 ```
 
+Whatever runs your models, the first steps are short:
+
+| You have | Do this |
+|---|---|
+| **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
+
+Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
+
 `pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
@@ -91,7 +100,8 @@ tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it
 ```bash
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
-tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval add                                                         # pick from the models on running servers
+tuieval add qwen3:8b                                                # finds the running server that has it
 tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
 ```
 
@@ -104,7 +114,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
-Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
 ## Command line
 
@@ -113,6 +123,7 @@ tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
 tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
+tuieval doctor                                # check servers, models, packs and keys, and what to fix
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/docs/models.md

@@ -1,6 +1,6 @@
 # Models, servers and machines
 
-Everything about models lives in your workspace's `models.toml`. `tuieval init` writes one with three servers ready to use:
+Everything about models lives in your workspace's `models.toml`. `tuieval init` writes a short one; three servers are built in and ready to use (their full definitions are at the end of this page, under [Built-in servers and defaults](#built-in-servers-and-defaults)):
 
 | server | what it is |
 |---|---|
@@ -14,7 +14,8 @@ Everything about models lives in your workspace's `models.toml`. `tuieval init`
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4      # GGUF -> llama
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --no-think              # same model, thinking off (label …-nothink)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf    # vision model
-tuieval add qwen3:8b --server local                                  # a model your running server serves
+tuieval add                                                          # pick from the models on servers already running
+tuieval add qwen3:8b                                                 # finds the running server that has it
 tuieval scan --add                                                   # every new GGUF under model_dirs
 tuieval list                                                         # models; hidden ones listed separately
 tuieval remove --hidden                                              # clean up every hidden model
@@ -70,7 +71,18 @@ The best server flags differ per model and per machine, so tuieval splits them b
 
 - **Machine id** is detected automatically (e.g. `m3max-64gb`, `m2-16gb`; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
 - **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers. Servers that size their own memory (their `cmd` doesn't take `{ctx}`) are left alone: their context is the model's `max_context`; `fit_check` on the server changes that.
-- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
+- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4), `gpu_residency_gb` (see the stall guard) and `parallel_models` (see below).
+
+## Several models at a time
+
+By default a run serves one model at a time. On a machine with room for more, set `parallel_models = 2` (or more) under `[machines.<id>]`, type a number in **Models at a time** on the TUI's setup screen (blank uses the machine's setting), or pass `tuieval run --parallel N`.
+
+- Models start in queue order, each in its own lane. The next one waits while its fit-check memory estimate wouldn't fit next to the ones running (without one, e.g. a server that sizes its own memory, the setting decides). Contexts are never shrunk to make room.
+- A second model on the same server gets a free port, so a server's `cmd` must take `{port}`.
+- A server with `before_start` (which may stop other servers) always runs on its own. Tuning is always one model at a time.
+- Answers are judged exactly as in a run of one model. Each answer records the models served while it ran (`ran_alongside`), and a load time measured next to others is marked (`loaded_alongside`), since models side by side share the machine's GPU and memory bandwidth. The speed table and the answer view show it.
+- In the TUI, one model's answers stream at a time: `v` switches to the next, `k` skips the one streaming. `tuieval run` prints one line per answer instead of streaming.
+- Only one run uses the machine at a time across tuieval windows; models at a time applies within a run.
 
 ## Tuning
 
@@ -142,3 +154,118 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `api_key_env` | environment variable holding the API key |
 | `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
 | `headers` | extra HTTP headers |
+
+## Built-in servers and defaults
+
+A workspace's `models.toml` only needs what differs from these. `[defaults]` fill in key by key. `[sampling]` applies only when `models.toml` has no `[sampling]` section at all, because sampling is part of every result's fingerprint. A server you define replaces the built-in one with the same name; to change one setting, copy the whole block into `models.toml` and edit it. `tuieval add` adds a `[servers.<name>]` block with just a `url` when it finds a model on a server that's already running.
+
+```toml
+[defaults] fill in key by key; [sampling] only when models.toml has no [sampling] at all (it's part of
+# every result's fingerprint); a server here is replaced whole by one of the same name in models.toml.
+
+[defaults]
+repeat = 3
+request_timeout_ms = 1800000   # per request; long thinking runs need it
+ready_timeout_s = 1200         # how long a model may take to load before giving up
+model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
+
+# Sent with every request. A model may override it (see `thinking` / `sampling` below); results
+# record the settings each model actually ran with, and the scorecard shows any difference.
+[sampling]
+temperature = 0.7
+top_p = 0.95
+top_k = 20
+min_p = 0.0
+max_tokens = 16384
+enable_thinking = true         # test every model in the SAME mode
+
+# llama.cpp's server (https://github.com/ggml-org/llama.cpp), started for each GGUF model.
+[servers.llama]
+# Output-affecting flags only. Anything here must be identical wherever results are compared,
+# so it's part of each result's fingerprint. Speed-only flags live in `perf` and `tune` below.
+cmd = [
+  "llama-server",
+  "-m", "{model}",
+  "--alias", "{served_name}",
+  "--host", "127.0.0.1", "--port", "{port}",
+  "--jinja",
+  "--metrics",
+  "-np", "1",
+  "-ctk", "{kv_type}", "-ctv", "{kv_type}",
+  "-c", "{ctx}",
+  "--reasoning-format", "auto",
+]
+port = 8080
+model_is_path = true           # check the GGUF exists and fits before starting
+vision_args = ["--mmproj", "{mmproj}"]   # appended only for models that set mmproj
+kv_type = "f16"                # a model's own `kv_type` (e.g. "q8_0") makes it a different model entry
+max_ctx = 65536                # upper bound; the fit check lowers it per machine (see machines.py)
+version_cmd = ["llama-server", "--version"]   # recorded with every result
+# Added to every request. cache_prompt = false: each answer is computed from scratch, so a repeat of
+# the same question reuses nothing from the last time it was asked (and its TTFT is honest).
+# Answers record any prompt tokens the server reused.
+request = { cache_prompt = false }
+
+# Speed-only flags, always applied (not tuned).
+perf = ["-ngl", "99"]
+
+# Speed-only knobs `tuieval tune` tries, one at a time from the best so far. The first option of
+# each knob is the default for models not tuned on this machine yet. Placeholders: {p} P-cores,
+# {p_minus_2}, {all} all cores. Options that stop the server from starting are skipped.
+# Check the flags against `llama-server --help` for your build.
+[servers.llama.tune]
+threads = [["-t", "{p}", "-tb", "{p}"], ["-t", "{p_minus_2}", "-tb", "{p}"], ["-t", "{all}", "-tb", "{all}"]]
+ubatch = [["-ub", "512"], ["-ub", "256"], ["-ub", "1024"]]
+batch = [["-b", "2048"], ["-b", "4096"]]
+flash_attn = [["-fa", "on"], ["-fa", "off"]]
+
+# A server that's already running (LM Studio, Ollama, vLLM, a remote box): give its URL and no cmd.
+# Its models are added with `tuieval add <model id> --server local`.
+[servers.local]
+url = "http://127.0.0.1:1234"   # LM Studio's default; Ollama is http://127.0.0.1:11434, vLLM :8000
+
+# OpenRouter: ANY model it serves, with no [[models]] entry. Name it when you run:
+#     tuieval run --tier smoke --only openrouter:qwen/qwen3-235b-a22b-2507
+#     TUI: tick "+ openrouter: any model" in the model list, or press "a" and type openrouter:<model id>
+# Results go to results/or-<id>/ and the model stays listed once it has results. A plain
+# `tuieval run` never includes OpenRouter models (they cost money); name them.
+# Needs OPENROUTER_API_KEY in the environment of the shell that starts tuieval.
+# pin_endpoint: every answer of a model comes from one provider endpoint (the closest to the released
+# weights), with no fallback, since providers run different quantizations.
+# To keep providers that store or train on prompts (and could learn your tests) out entirely, add
+#   request = { provider = { data_collection = "deny" } }
+[servers.openrouter]
+url = "https://openrouter.ai/api/v1"
+any_model = true
+label_prefix = "or"
+pin_endpoint = true
+api_key_env = "OPENROUTER_API_KEY"
+thinking_param = "reasoning"     # [sampling] enable_thinking -> OpenRouter's reasoning.enabled
+headers = { "X-Title" = "tuieval" }
+```
+
+`[defaults] detect_ports` (default `[1234, 11434, 8080, 8000]`, the usual LM Studio, Ollama, llama-server and vLLM ports) is where `tuieval add`, `tuieval init` and `tuieval doctor` look for servers that are already running. `TUIEVAL_DETECT_PORTS` (comma-separated; empty means none) overrides it.
+
+Optional sections:
+
+```toml
+# Optional tuner settings.
+# [tune]
+# guard_similarity = 0.6       # greedy answers under a candidate flag must stay this similar to the defaults'
+# request_timeout_s = 600      # a tuning request that takes longer fails that candidate
+# warm_start = true            # start from a tuned model with the same architecture and shapes (a fine-tune)
+# warm_retest = ["spec", "ubatch"]   # the knobs a warm start re-tries; the rest are inherited
+# answer_tokens = 1024         # answer length options are scored for (projected objective)
+# swap_limit_mb = 500          # reject options that swap more than this beyond the defaults (default: report only)
+
+# Optional: llama-bench for the tuner's fast first stage (found on PATH as llama-bench otherwise).
+# [bench]
+# cmd = ["~/llama.cpp/build/bin/llama-bench"]
+
+# Optional per-machine settings, by machine id (tuieval machines shows the ids).
+# [machines.m3max-64gb]
+# memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
+# gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
+# parallel_models = 2          # models a run serves at a time (default 1)
+```
+

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/docs/writing-packs.md

@@ -10,6 +10,11 @@ tuieval new-pack support-bot --grader reply      # answer | rag | reply | tool_c
 
 Edit `packs/support-bot/tests.yaml`, replace the examples with your own questions, and run `tuieval selftest support-bot` after every change.
 
+Two shortcuts to a first set of real questions:
+
+- **From a spreadsheet:** `tuieval new-pack quiz --from questions.csv` makes one test per row from a `question` and an `answer` column (optional: `id`, `category`, `difficulty`, `tolerance`, and `wrong` with known-wrong answers separated by `|`). Numbers are checked with the tolerance, words case-insensitively, and `NOT_AVAILABLE` means the right answer is "the question doesn't say".
+- **Drafted by a model:** `tuieval new-pack support-bot --grader reply --about "refund questions for a shoe store"` also writes `DRAFT-PROMPT.md`. Paste it into the strongest model you have, save the YAML it writes as `tests.yaml`, and run `tuieval selftest`: it catches drafted tests whose expected answer is wrong or whose grader can't tell a mistake from the right answer. Read the questions anyway; you know your domain better than the model.
+
 To try a different set of questions, copy a pack (`cp -r packs/support-bot packs/support-bot-v2`), edit the copy, and pick whichever you want in the TUI.
 
 ## Fingerprints
@@ -55,6 +60,7 @@ min_consistency = 0.95           # share of variant groups where every variant p
   input: |                         # the user message
     A customer bought shoes 20 days ago …
   image: images/receipt.png        # optional: sent as an image (needs vision)
+  input_file: prompts/refund.txt   # optional, instead of input: the user message from a file in the pack
   grader: answer                   # optional: override the pack's grader
   group: refund-window-2           # optional: variants of one case share a group (consistency gate)
   critical: true                   # optional: any failure of this test is critical (disqualifying)

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev14'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev14')
+__version__ = version = '0.2.0.dev28'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev28')
 
 __commit_id__ = commit_id = None

tuieval-0.2.0.dev28/src/tuieval/builtin.toml

@@ -0,0 +1,83 @@
+# Built-in defaults and servers, used when a workspace's models.toml doesn't define them.
+# [defaults] fill in key by key; [sampling] only when models.toml has no [sampling] at all (it's part of
+# every result's fingerprint); a server here is replaced whole by one of the same name in models.toml.
+
+[defaults]
+repeat = 3
+request_timeout_ms = 1800000   # per request; long thinking runs need it
+ready_timeout_s = 1200         # how long a model may take to load before giving up
+model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
+
+# Sent with every request. A model may override it (see `thinking` / `sampling` below); results
+# record the settings each model actually ran with, and the scorecard shows any difference.
+[sampling]
+temperature = 0.7
+top_p = 0.95
+top_k = 20
+min_p = 0.0
+max_tokens = 16384
+enable_thinking = true         # test every model in the SAME mode
+
+# llama.cpp's server (https://github.com/ggml-org/llama.cpp), started for each GGUF model.
+[servers.llama]
+# Output-affecting flags only. Anything here must be identical wherever results are compared,
+# so it's part of each result's fingerprint. Speed-only flags live in `perf` and `tune` below.
+cmd = [
+  "llama-server",
+  "-m", "{model}",
+  "--alias", "{served_name}",
+  "--host", "127.0.0.1", "--port", "{port}",
+  "--jinja",
+  "--metrics",
+  "-np", "1",
+  "-ctk", "{kv_type}", "-ctv", "{kv_type}",
+  "-c", "{ctx}",
+  "--reasoning-format", "auto",
+]
+port = 8080
+model_is_path = true           # check the GGUF exists and fits before starting
+vision_args = ["--mmproj", "{mmproj}"]   # appended only for models that set mmproj
+kv_type = "f16"                # a model's own `kv_type` (e.g. "q8_0") makes it a different model entry
+max_ctx = 65536                # upper bound; the fit check lowers it per machine (see machines.py)
+version_cmd = ["llama-server", "--version"]   # recorded with every result
+# Added to every request. cache_prompt = false: each answer is computed from scratch, so a repeat of
+# the same question reuses nothing from the last time it was asked (and its TTFT is honest).
+# Answers record any prompt tokens the server reused.
+request = { cache_prompt = false }
+
+# Speed-only flags, always applied (not tuned).
+perf = ["-ngl", "99"]
+
+# Speed-only knobs `tuieval tune` tries, one at a time from the best so far. The first option of
+# each knob is the default for models not tuned on this machine yet. Placeholders: {p} P-cores,
+# {p_minus_2}, {all} all cores. Options that stop the server from starting are skipped.
+# Check the flags against `llama-server --help` for your build.
+[servers.llama.tune]
+threads = [["-t", "{p}", "-tb", "{p}"], ["-t", "{p_minus_2}", "-tb", "{p}"], ["-t", "{all}", "-tb", "{all}"]]
+ubatch = [["-ub", "512"], ["-ub", "256"], ["-ub", "1024"]]
+batch = [["-b", "2048"], ["-b", "4096"]]
+flash_attn = [["-fa", "on"], ["-fa", "off"]]
+
+# A server that's already running (LM Studio, Ollama, vLLM, a remote box): give its URL and no cmd.
+# Its models are added with `tuieval add <model id> --server local`.
+[servers.local]
+url = "http://127.0.0.1:1234"   # LM Studio's default; Ollama is http://127.0.0.1:11434, vLLM :8000
+
+# OpenRouter: ANY model it serves, with no [[models]] entry. Name it when you run:
+#     tuieval run --tier smoke --only openrouter:qwen/qwen3-235b-a22b-2507
+#     TUI: tick "+ openrouter: any model" in the model list, or press "a" and type openrouter:<model id>
+# Results go to results/or-<id>/ and the model stays listed once it has results. A plain
+# `tuieval run` never includes OpenRouter models (they cost money); name them.
+# Needs OPENROUTER_API_KEY in the environment of the shell that starts tuieval.
+# pin_endpoint: every answer of a model comes from one provider endpoint (the closest to the released
+# weights), with no fallback, since providers run different quantizations.
+# To keep providers that store or train on prompts (and could learn your tests) out entirely, add
+#   request = { provider = { data_collection = "deny" } }
+[servers.openrouter]
+url = "https://openrouter.ai/api/v1"
+any_model = true
+label_prefix = "or"
+pin_endpoint = true
+api_key_env = "OPENROUTER_API_KEY"
+thinking_param = "reasoning"     # [sampling] enable_thinking -> OpenRouter's reasoning.enabled
+headers = { "X-Title" = "tuieval" }

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/src/tuieval/cli.py

@@ -2,28 +2,33 @@
 
 Usage: tuieval [--workspace DIR] [command] [options]
 
-  tuieval                     open the TUI
-  tuieval init [DIR]          create a workspace (models.toml, packs/, graders/)
-  tuieval new-pack NAME       a new pack from a starter template (--grader answer|rag|reply|tool_call|code)
-  tuieval add PATH|ID         register a model in models.toml (a GGUF, or a model id)
-  tuieval scan                list GGUFs in model_dirs that aren't in models.toml yet
-  tuieval list                the models tuieval knows, hidden ones separately (--packs: the packs)
-  tuieval remove MODEL        remove models, or every hidden one (--hidden); archived in removed/
+Start here
+  tuieval init [DIR]          create a workspace; in a terminal, a guided setup to a first result
+  tuieval                     open the TUI (? explains any screen)
+  tuieval doctor              check servers, models, packs and API keys, and say what to fix
+  tuieval add [PATH|ID]       add a model: a GGUF, a model id, or none to pick from running servers
+  tuieval new-pack NAME       a new pack of example questions (--from CSV, --about TOPIC, --grader …)
+  tuieval selftest            check every test's reference and wrong answers against its grader
   tuieval run [...]           run evals from the command line (tuieval run --help)
+  tuieval verdict             which model is ready for what, and what to run next
+
+Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
-  tuieval regrade             re-score stored answers after changing a grader
-  tuieval verdict             production readiness (PASS/FAIL per model and use case)
-  tuieval report              the readiness verdicts with their evidence, as a markdown file
+  tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
-  tuieval selftest            check every test's reference and wrong answers against its grader
   tuieval items               tests that don't separate models or look broken
   tuieval capture LOG --pack  turn a real failure (logs/live/*.txt) into a new test
+  tuieval regrade             re-score stored answers after changing a grader
+  tuieval list                the models tuieval knows, hidden ones separately (--packs: the packs)
+  tuieval remove MODEL        remove models, or every hidden one (--hidden); archived in removed/
+
+Advanced
+  tuieval scan                list GGUFs in model_dirs that aren't in models.toml yet
   tuieval machines            this machine, other recorded machines, and fit and tuning per model
   tuieval tune MODEL          find the fastest speed-only server flags for a model on this machine
   tuieval export pi MODEL     serve a model in the pi coding agent with its tuned flags
   tuieval watch               a proxy that shows the reasoning of any app using your server
-  tuieval help                this help
-  tuieval --version
+  tuieval help · --version
 
 The workspace is --workspace DIR, else $TUIEVAL_HOME, else the current folder.
 """
@@ -67,6 +72,9 @@ def main(argv=None):
         if not workspace.is_workspace():
             sys.exit(f"{workspace.root()} isn't a tuieval workspace; run `tuieval init` first")
         return scaffold.cmd_new_pack(rest)
+    if cmd == "doctor":
+        from . import doctor
+        return doctor.cmd_doctor(rest)
     if cmd == "run":
         from . import run_evals
         return run_evals.main(rest)

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev28}/src/tuieval/compare.py

@@ -37,7 +37,8 @@ def _load_native(data):
             "settings": data.get("settings", {}), "load_s": run.get("load_s"),
             "peak_rss_mb": run.get("peak_rss_mb"), "pack": pack["name"], "pack_fp": pack["fingerprint"],
             # hosted (OpenRouter) models: the provider endpoint and its declared quantization
-            "endpoint": run.get("endpoint"), "quantization": run.get("quantization")}
+            "endpoint": run.get("endpoint"), "quantization": run.get("quantization"),
+            "loaded_alongside": run.get("loaded_alongside") or []}
     rows = []
     for r in data["results"]:
         rows.append({
@@ -54,6 +55,8 @@ def _load_native(data):
             "severity": r.get("severity"), "group": r.get("group"), "repeat": r.get("repeat", 0),
             "difficulty": r.get("difficulty", "unrated"),
             "machine": r.get("machine") or run.get("machine"), "prompt_tokens": r.get("prompt_tokens"),
+            # other models served at the same time (parallel_models): its speed was shared with them
+            "alongside": r.get("ran_alongside") or [],
         })
     return rows, info
 
@@ -210,6 +213,13 @@ def timing(r, machine=None, speeds=None):
     return None, None, None, None, None
 
 
+def alongside_note(rows):
+    """' · n/N ran alongside others' when some answers ran while other models were served (their
+    speed was shared with them), else ''."""
+    shared = sum(bool(r.get("alongside")) for r in rows)
+    return f" · {shared}/{len(rows)} ran alongside others" if shared else ""
+
+
 def speed(rows, infos, machine=None, speeds=None):
     """(header, table, frontier): speed and token use per model.
 
@@ -227,7 +237,9 @@ def speed(rows, infos, machine=None, speeds=None):
     load = {}
     for i in infos:
         cur = load.setdefault(i["label"], {"load_s": None, "peak_rss_mb": None, "tags": i["tags"],
-                                           "think": i["settings"].get("enable_thinking"), "served": set()})
+                                           "think": i["settings"].get("enable_thinking"), "served": set(),
+                                           "loaded_alongside": False})
+        cur["loaded_alongside"] |= bool(i.get("loaded_alongside"))
         if i.get("endpoint"):
             q = i.get("quantization")
             cur["served"].add(f"{'undeclared' if q in (None, 'unknown') else q} via {i['endpoint']}")
@@ -247,6 +259,7 @@ def speed(rows, infos, machine=None, speeds=None):
         here = machine is None or machine in where   # load time and memory only where it ran
         if machine is None:
             kind = "measured on " + (", ".join(where) or "?")
+        kind += alongside_note(v)
         table.append([
             "★" if m in frontier else "", m, "; ".join(sorted(info.get("served") or [])) or "-",
             ",".join(info.get("tags") or []),
@@ -256,7 +269,8 @@ def speed(rows, infos, machine=None, speeds=None):
             _fmt(median(x[3] for x in t), ".0f"), _fmt(median(r["tokens"] for r in v), ".0f"),
             _fmt(median(r["reasoning_tokens"] for r in v), ".0f"), _fmt(median(r["answer_tokens"] for r in v), ".0f"),
             f"{sum(r['tokens'] for r in v) / passes:,.0f}" if passes else "-",
-            _fmt(info.get("load_s") if here else None, ".0f"),
+            _fmt(info.get("load_s") if here else None, ".0f")
+            + (" (alongside others)" if here and info.get("load_s") and info.get("loaded_alongside") else ""),
             _fmt(info["peak_rss_mb"] / 1024 if info.get("peak_rss_mb") and here else None, ".1f"),
         ])
     return header, table, frontier