tuieval 0.2.0.dev14__tar.gz → 0.2.0.dev23__tar.gz

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+- A new workspace's `models.toml` is 23 lines: the llama, local and OpenRouter servers and the defaults are built in (docs/models.md lists them in full; a server defined in `models.toml` replaces the built-in one with that name). Existing workspaces keep their settings; built-in servers they don't define become available, and adding a model still prefers the workspace's own servers.
+- Less to take in at first: until something has run, the setup screen leaves out context sizes, tuning state and the Tune key; Presets and Hide are off the footer (`?` lists every key); `tuieval help` groups commands into Start here, Day to day and Advanced.
+- Results lead with a plain summary per model ("ready for Support; not decided yet for Coding → run Certify to finish"), in the TUI, `tuieval verdict` (with the command to run next) and the report; the statistics follow.
+- First packs without hand-written YAML: `tuieval new-pack <name> --from questions.csv` (question and answer columns) and `--about "<topic>"` (writes a prompt for a strong model to draft the tests). A test can take its prompt from a file (`input_file`). `tuieval selftest` errors say how to fix them.
+- `tuieval init` in a terminal is a guided setup: it finds models on servers already running (LM Studio, Ollama, llama-server, vLLM), adds the ones you pick, creates a starter pack and runs a Smoke check (`--yes` for no questions).
+- `tuieval add` without a model lists the models on running servers to pick from; `tuieval add <id>` finds the server that has it and adds that server to models.toml. The TUI's add dialog lists them too, with rarely needed options folded away.
+- `?` on any TUI screen explains what it's for, its keys, and the words tuieval uses (tiers, verdicts, critical failures, gates).
+- The setup screen is ready on first open: the only pack and model are ticked, the tier is Smoke until something has run, and ticked boxes are clearly marked (unticked ones are empty).
+- First runs fail fast with the fix instead of hanging: a server that isn't running fails in seconds (it used to wait up to 20 minutes), a missing `llama-server` says how to install it, and `tuieval run` with no models says how to add one.
+- `tuieval doctor` checks servers, models, packs and API keys, and says what to fix.
 - Run only some tests of a pack: `e` in the TUI, or `tuieval run --tests`. Runs add up; `--force` reruns just the picked tests. Presets keep the pick.
 - `tuieval tune` scores options on projected full-length answers (`[tune] answer_tokens`, default 1024), so generation speed counts as in real runs.
 - `tuieval tune` reports an option's memory cost instead of rejecting it; it rejects only on critical memory pressure or over `[tune] swap_limit_mb`.

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tuieval
-Version: 0.2.0.dev14
+Version: 0.2.0.dev23
 Summary: Evaluate local and frontier LLMs on your own questions: accuracy, speed, tokens and PASS/FAIL verdicts, in the terminal.
 Project-URL: Homepage, https://github.com/ashe-wb/tuieval
 Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
@@ -43,12 +43,21 @@ Then you get an answer to the questions that matter:
 
 ```bash
 pipx install tuieval          # or: uv tool install tuieval, or pip install tuieval   (Python 3.11+)
-tuieval init my-evals && cd my-evals
-tuieval new-pack my-first-pack # a pack of example questions to edit
-tuieval add ~/models/Some-Model-Q4_K_M.gguf
-tuieval                        # open the TUI
+tuieval init my-evals          # guided: finds models already running (LM Studio, Ollama, vLLM…),
+                               # adds a starter pack and runs a 3-question check
+cd my-evals && tuieval         # open the TUI (? explains any screen)
 ```
 
+Whatever runs your models, the first steps are short:
+
+| You have | Do this |
+|---|---|
+| **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
+
+Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
+
 `pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
@@ -117,7 +126,8 @@ tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it
 ```bash
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
-tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval add                                                         # pick from the models on running servers
+tuieval add qwen3:8b                                                # finds the running server that has it
 tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
 ```
 
@@ -130,7 +140,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
-Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
 ## Command line
 
@@ -139,6 +149,7 @@ tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
 tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
+tuieval doctor                                # check servers, models, packs and keys, and what to fix
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/README.md

@@ -17,12 +17,21 @@ Then you get an answer to the questions that matter:
 
 ```bash
 pipx install tuieval          # or: uv tool install tuieval, or pip install tuieval   (Python 3.11+)
-tuieval init my-evals && cd my-evals
-tuieval new-pack my-first-pack # a pack of example questions to edit
-tuieval add ~/models/Some-Model-Q4_K_M.gguf
-tuieval                        # open the TUI
+tuieval init my-evals          # guided: finds models already running (LM Studio, Ollama, vLLM…),
+                               # adds a starter pack and runs a 3-question check
+cd my-evals && tuieval         # open the TUI (? explains any screen)
 ```
 
+Whatever runs your models, the first steps are short:
+
+| You have | Do this |
+|---|---|
+| **LM Studio, Ollama or vLLM** | Start its server and load a model, then `tuieval init my-evals`: it finds the model. Later: `tuieval add` lists new ones. |
+| **GGUF files** | Install llama.cpp (`brew install llama.cpp`), then `tuieval add ~/models/Some-Model-Q4_K_M.gguf`. tuieval starts `llama-server` for it, sized to your machine. |
+| **An OpenRouter key** | `export OPENROUTER_API_KEY=…`, then `tuieval run --tier smoke --only openrouter:<model id>`. |
+
+Stuck? `tuieval doctor` checks servers, models, packs and keys, and says what to fix.
+
 `pip install tuieval` gets the latest release. Every change on `main` is also published as a dev build (`X.Y.0.devN`); get it with `pipx install --pip-args=--pre tuieval` or `pip install --pre tuieval`.
 
 Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
@@ -91,7 +100,8 @@ tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it
 ```bash
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
-tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval add                                                         # pick from the models on running servers
+tuieval add qwen3:8b                                                # finds the running server that has it
 tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
 ```
 
@@ -104,7 +114,7 @@ See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinnin
 3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
 4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
 
-Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model. `?` explains any screen.
 
 ## Command line
 
@@ -113,6 +123,7 @@ tuieval run                                   # Screen every model on every pack
 tuieval run --tier certify --only a,b --packs support-bot,coding
 tuieval run --packs coding --tests parse-dates,fix-bug   # only these tests (or --tests coding:parse-dates)
 tuieval run --dry-run                         # the plan and server commands
+tuieval doctor                                # check servers, models, packs and keys, and what to fix
 tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
 tuieval report                                # the same with evidence, as a markdown file
 tuieval history                               # every verdict change over time

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/docs/models.md

@@ -1,6 +1,6 @@
 # Models, servers and machines
 
-Everything about models lives in your workspace's `models.toml`. `tuieval init` writes one with three servers ready to use:
+Everything about models lives in your workspace's `models.toml`. `tuieval init` writes a short one; three servers are built in and ready to use (their full definitions are at the end of this page, under [Built-in servers and defaults](#built-in-servers-and-defaults)):
 
 | server | what it is |
 |---|---|
@@ -14,7 +14,8 @@ Everything about models lives in your workspace's `models.toml`. `tuieval init`
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4      # GGUF -> llama
 tuieval add ~/models/Some-Model-Q4_K_M.gguf --no-think              # same model, thinking off (label …-nothink)
 tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf    # vision model
-tuieval add qwen3:8b --server local                                  # a model your running server serves
+tuieval add                                                          # pick from the models on servers already running
+tuieval add qwen3:8b                                                 # finds the running server that has it
 tuieval scan --add                                                   # every new GGUF under model_dirs
 tuieval list                                                         # models; hidden ones listed separately
 tuieval remove --hidden                                              # clean up every hidden model
@@ -142,3 +143,117 @@ On Apple Silicon Macs, once the system's GPU allocations pass about half of RAM,
 | `api_key_env` | environment variable holding the API key |
 | `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
 | `headers` | extra HTTP headers |
+
+## Built-in servers and defaults
+
+A workspace's `models.toml` only needs what differs from these. `[defaults]` fill in key by key. `[sampling]` applies only when `models.toml` has no `[sampling]` section at all, because sampling is part of every result's fingerprint. A server you define replaces the built-in one with the same name; to change one setting, copy the whole block into `models.toml` and edit it. `tuieval add` adds a `[servers.<name>]` block with just a `url` when it finds a model on a server that's already running.
+
+```toml
+[defaults] fill in key by key; [sampling] only when models.toml has no [sampling] at all (it's part of
+# every result's fingerprint); a server here is replaced whole by one of the same name in models.toml.
+
+[defaults]
+repeat = 3
+request_timeout_ms = 1800000   # per request; long thinking runs need it
+ready_timeout_s = 1200         # how long a model may take to load before giving up
+model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
+
+# Sent with every request. A model may override it (see `thinking` / `sampling` below); results
+# record the settings each model actually ran with, and the scorecard shows any difference.
+[sampling]
+temperature = 0.7
+top_p = 0.95
+top_k = 20
+min_p = 0.0
+max_tokens = 16384
+enable_thinking = true         # test every model in the SAME mode
+
+# llama.cpp's server (https://github.com/ggml-org/llama.cpp), started for each GGUF model.
+[servers.llama]
+# Output-affecting flags only. Anything here must be identical wherever results are compared,
+# so it's part of each result's fingerprint. Speed-only flags live in `perf` and `tune` below.
+cmd = [
+  "llama-server",
+  "-m", "{model}",
+  "--alias", "{served_name}",
+  "--host", "127.0.0.1", "--port", "{port}",
+  "--jinja",
+  "--metrics",
+  "-np", "1",
+  "-ctk", "{kv_type}", "-ctv", "{kv_type}",
+  "-c", "{ctx}",
+  "--reasoning-format", "auto",
+]
+port = 8080
+model_is_path = true           # check the GGUF exists and fits before starting
+vision_args = ["--mmproj", "{mmproj}"]   # appended only for models that set mmproj
+kv_type = "f16"                # a model's own `kv_type` (e.g. "q8_0") makes it a different model entry
+max_ctx = 65536                # upper bound; the fit check lowers it per machine (see machines.py)
+version_cmd = ["llama-server", "--version"]   # recorded with every result
+# Added to every request. cache_prompt = false: each answer is computed from scratch, so a repeat of
+# the same question reuses nothing from the last time it was asked (and its TTFT is honest).
+# Answers record any prompt tokens the server reused.
+request = { cache_prompt = false }
+
+# Speed-only flags, always applied (not tuned).
+perf = ["-ngl", "99"]
+
+# Speed-only knobs `tuieval tune` tries, one at a time from the best so far. The first option of
+# each knob is the default for models not tuned on this machine yet. Placeholders: {p} P-cores,
+# {p_minus_2}, {all} all cores. Options that stop the server from starting are skipped.
+# Check the flags against `llama-server --help` for your build.
+[servers.llama.tune]
+threads = [["-t", "{p}", "-tb", "{p}"], ["-t", "{p_minus_2}", "-tb", "{p}"], ["-t", "{all}", "-tb", "{all}"]]
+ubatch = [["-ub", "512"], ["-ub", "256"], ["-ub", "1024"]]
+batch = [["-b", "2048"], ["-b", "4096"]]
+flash_attn = [["-fa", "on"], ["-fa", "off"]]
+
+# A server that's already running (LM Studio, Ollama, vLLM, a remote box): give its URL and no cmd.
+# Its models are added with `tuieval add <model id> --server local`.
+[servers.local]
+url = "http://127.0.0.1:1234"   # LM Studio's default; Ollama is http://127.0.0.1:11434, vLLM :8000
+
+# OpenRouter: ANY model it serves, with no [[models]] entry. Name it when you run:
+#     tuieval run --tier smoke --only openrouter:qwen/qwen3-235b-a22b-2507
+#     TUI: tick "+ openrouter: any model" in the model list, or press "a" and type openrouter:<model id>
+# Results go to results/or-<id>/ and the model stays listed once it has results. A plain
+# `tuieval run` never includes OpenRouter models (they cost money); name them.
+# Needs OPENROUTER_API_KEY in the environment of the shell that starts tuieval.
+# pin_endpoint: every answer of a model comes from one provider endpoint (the closest to the released
+# weights), with no fallback, since providers run different quantizations.
+# To keep providers that store or train on prompts (and could learn your tests) out entirely, add
+#   request = { provider = { data_collection = "deny" } }
+[servers.openrouter]
+url = "https://openrouter.ai/api/v1"
+any_model = true
+label_prefix = "or"
+pin_endpoint = true
+api_key_env = "OPENROUTER_API_KEY"
+thinking_param = "reasoning"     # [sampling] enable_thinking -> OpenRouter's reasoning.enabled
+headers = { "X-Title" = "tuieval" }
+```
+
+`[defaults] detect_ports` (default `[1234, 11434, 8080, 8000]`, the usual LM Studio, Ollama, llama-server and vLLM ports) is where `tuieval add`, `tuieval init` and `tuieval doctor` look for servers that are already running. `TUIEVAL_DETECT_PORTS` (comma-separated; empty means none) overrides it.
+
+Optional sections:
+
+```toml
+# Optional tuner settings.
+# [tune]
+# guard_similarity = 0.6       # greedy answers under a candidate flag must stay this similar to the defaults'
+# request_timeout_s = 600      # a tuning request that takes longer fails that candidate
+# warm_start = true            # start from a tuned model with the same architecture and shapes (a fine-tune)
+# warm_retest = ["spec", "ubatch"]   # the knobs a warm start re-tries; the rest are inherited
+# answer_tokens = 1024         # answer length options are scored for (projected objective)
+# swap_limit_mb = 500          # reject options that swap more than this beyond the defaults (default: report only)
+
+# Optional: llama-bench for the tuner's fast first stage (found on PATH as llama-bench otherwise).
+# [bench]
+# cmd = ["~/llama.cpp/build/bin/llama-bench"]
+
+# Optional per-machine settings, by machine id (tuieval machines shows the ids).
+# [machines.m3max-64gb]
+# memory_headroom_gb = 3       # GPU memory kept free for macOS and other apps (default 4)
+# gpu_residency_gb = 12        # GPU memory the driver keeps resident before churning (default: half of RAM)
+```
+

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/docs/writing-packs.md

@@ -10,6 +10,11 @@ tuieval new-pack support-bot --grader reply      # answer | rag | reply | tool_c
 
 Edit `packs/support-bot/tests.yaml`, replace the examples with your own questions, and run `tuieval selftest support-bot` after every change.
 
+Two shortcuts to a first set of real questions:
+
+- **From a spreadsheet:** `tuieval new-pack quiz --from questions.csv` makes one test per row from a `question` and an `answer` column (optional: `id`, `category`, `difficulty`, `tolerance`, and `wrong` with known-wrong answers separated by `|`). Numbers are checked with the tolerance, words case-insensitively, and `NOT_AVAILABLE` means the right answer is "the question doesn't say".
+- **Drafted by a model:** `tuieval new-pack support-bot --grader reply --about "refund questions for a shoe store"` also writes `DRAFT-PROMPT.md`. Paste it into the strongest model you have, save the YAML it writes as `tests.yaml`, and run `tuieval selftest`: it catches drafted tests whose expected answer is wrong or whose grader can't tell a mistake from the right answer. Read the questions anyway; you know your domain better than the model.
+
 To try a different set of questions, copy a pack (`cp -r packs/support-bot packs/support-bot-v2`), edit the copy, and pick whichever you want in the TUI.
 
 ## Fingerprints
@@ -55,6 +60,7 @@ min_consistency = 0.95           # share of variant groups where every variant p
   input: |                         # the user message
     A customer bought shoes 20 days ago …
   image: images/receipt.png        # optional: sent as an image (needs vision)
+  input_file: prompts/refund.txt   # optional, instead of input: the user message from a file in the pack
   grader: answer                   # optional: override the pack's grader
   group: refund-window-2           # optional: variants of one case share a group (consistency gate)
   critical: true                   # optional: any failure of this test is critical (disqualifying)

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/src/tuieval/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0.dev14'
-__version_tuple__ = version_tuple = (0, 2, 0, 'dev14')
+__version__ = version = '0.2.0.dev23'
+__version_tuple__ = version_tuple = (0, 2, 0, 'dev23')
 
 __commit_id__ = commit_id = None

tuieval-0.2.0.dev23/src/tuieval/builtin.toml

@@ -0,0 +1,83 @@
+# Built-in defaults and servers, used when a workspace's models.toml doesn't define them.
+# [defaults] fill in key by key; [sampling] only when models.toml has no [sampling] at all (it's part of
+# every result's fingerprint); a server here is replaced whole by one of the same name in models.toml.
+
+[defaults]
+repeat = 3
+request_timeout_ms = 1800000   # per request; long thinking runs need it
+ready_timeout_s = 1200         # how long a model may take to load before giving up
+model_dirs = ["~/models"]      # where "Scan for models" looks for new GGUFs
+
+# Sent with every request. A model may override it (see `thinking` / `sampling` below); results
+# record the settings each model actually ran with, and the scorecard shows any difference.
+[sampling]
+temperature = 0.7
+top_p = 0.95
+top_k = 20
+min_p = 0.0
+max_tokens = 16384
+enable_thinking = true         # test every model in the SAME mode
+
+# llama.cpp's server (https://github.com/ggml-org/llama.cpp), started for each GGUF model.
+[servers.llama]
+# Output-affecting flags only. Anything here must be identical wherever results are compared,
+# so it's part of each result's fingerprint. Speed-only flags live in `perf` and `tune` below.
+cmd = [
+  "llama-server",
+  "-m", "{model}",
+  "--alias", "{served_name}",
+  "--host", "127.0.0.1", "--port", "{port}",
+  "--jinja",
+  "--metrics",
+  "-np", "1",
+  "-ctk", "{kv_type}", "-ctv", "{kv_type}",
+  "-c", "{ctx}",
+  "--reasoning-format", "auto",
+]
+port = 8080
+model_is_path = true           # check the GGUF exists and fits before starting
+vision_args = ["--mmproj", "{mmproj}"]   # appended only for models that set mmproj
+kv_type = "f16"                # a model's own `kv_type` (e.g. "q8_0") makes it a different model entry
+max_ctx = 65536                # upper bound; the fit check lowers it per machine (see machines.py)
+version_cmd = ["llama-server", "--version"]   # recorded with every result
+# Added to every request. cache_prompt = false: each answer is computed from scratch, so a repeat of
+# the same question reuses nothing from the last time it was asked (and its TTFT is honest).
+# Answers record any prompt tokens the server reused.
+request = { cache_prompt = false }
+
+# Speed-only flags, always applied (not tuned).
+perf = ["-ngl", "99"]
+
+# Speed-only knobs `tuieval tune` tries, one at a time from the best so far. The first option of
+# each knob is the default for models not tuned on this machine yet. Placeholders: {p} P-cores,
+# {p_minus_2}, {all} all cores. Options that stop the server from starting are skipped.
+# Check the flags against `llama-server --help` for your build.
+[servers.llama.tune]
+threads = [["-t", "{p}", "-tb", "{p}"], ["-t", "{p_minus_2}", "-tb", "{p}"], ["-t", "{all}", "-tb", "{all}"]]
+ubatch = [["-ub", "512"], ["-ub", "256"], ["-ub", "1024"]]
+batch = [["-b", "2048"], ["-b", "4096"]]
+flash_attn = [["-fa", "on"], ["-fa", "off"]]
+
+# A server that's already running (LM Studio, Ollama, vLLM, a remote box): give its URL and no cmd.
+# Its models are added with `tuieval add <model id> --server local`.
+[servers.local]
+url = "http://127.0.0.1:1234"   # LM Studio's default; Ollama is http://127.0.0.1:11434, vLLM :8000
+
+# OpenRouter: ANY model it serves, with no [[models]] entry. Name it when you run:
+#     tuieval run --tier smoke --only openrouter:qwen/qwen3-235b-a22b-2507
+#     TUI: tick "+ openrouter: any model" in the model list, or press "a" and type openrouter:<model id>
+# Results go to results/or-<id>/ and the model stays listed once it has results. A plain
+# `tuieval run` never includes OpenRouter models (they cost money); name them.
+# Needs OPENROUTER_API_KEY in the environment of the shell that starts tuieval.
+# pin_endpoint: every answer of a model comes from one provider endpoint (the closest to the released
+# weights), with no fallback, since providers run different quantizations.
+# To keep providers that store or train on prompts (and could learn your tests) out entirely, add
+#   request = { provider = { data_collection = "deny" } }
+[servers.openrouter]
+url = "https://openrouter.ai/api/v1"
+any_model = true
+label_prefix = "or"
+pin_endpoint = true
+api_key_env = "OPENROUTER_API_KEY"
+thinking_param = "reasoning"     # [sampling] enable_thinking -> OpenRouter's reasoning.enabled
+headers = { "X-Title" = "tuieval" }

{tuieval-0.2.0.dev14 → tuieval-0.2.0.dev23}/src/tuieval/cli.py

@@ -2,28 +2,33 @@
 
 Usage: tuieval [--workspace DIR] [command] [options]
 
-  tuieval                     open the TUI
-  tuieval init [DIR]          create a workspace (models.toml, packs/, graders/)
-  tuieval new-pack NAME       a new pack from a starter template (--grader answer|rag|reply|tool_call|code)
-  tuieval add PATH|ID         register a model in models.toml (a GGUF, or a model id)
-  tuieval scan                list GGUFs in model_dirs that aren't in models.toml yet
-  tuieval list                the models tuieval knows, hidden ones separately (--packs: the packs)
-  tuieval remove MODEL        remove models, or every hidden one (--hidden); archived in removed/
+Start here
+  tuieval init [DIR]          create a workspace; in a terminal, a guided setup to a first result
+  tuieval                     open the TUI (? explains any screen)
+  tuieval doctor              check servers, models, packs and API keys, and say what to fix
+  tuieval add [PATH|ID]       add a model: a GGUF, a model id, or none to pick from running servers
+  tuieval new-pack NAME       a new pack of example questions (--from CSV, --about TOPIC, --grader …)
+  tuieval selftest            check every test's reference and wrong answers against its grader
   tuieval run [...]           run evals from the command line (tuieval run --help)
+  tuieval verdict             which model is ready for what, and what to run next
+
+Day to day
   tuieval compare [...]       scorecards (--speed, --pairwise, --failures)
-  tuieval regrade             re-score stored answers after changing a grader
-  tuieval verdict             production readiness (PASS/FAIL per model and use case)
-  tuieval report              the readiness verdicts with their evidence, as a markdown file
+  tuieval report              the verdicts with their evidence, as a markdown file
   tuieval history             every verdict change, newest first
-  tuieval selftest            check every test's reference and wrong answers against its grader
   tuieval items               tests that don't separate models or look broken
   tuieval capture LOG --pack  turn a real failure (logs/live/*.txt) into a new test
+  tuieval regrade             re-score stored answers after changing a grader
+  tuieval list                the models tuieval knows, hidden ones separately (--packs: the packs)
+  tuieval remove MODEL        remove models, or every hidden one (--hidden); archived in removed/
+
+Advanced
+  tuieval scan                list GGUFs in model_dirs that aren't in models.toml yet
   tuieval machines            this machine, other recorded machines, and fit and tuning per model
   tuieval tune MODEL          find the fastest speed-only server flags for a model on this machine
   tuieval export pi MODEL     serve a model in the pi coding agent with its tuned flags
   tuieval watch               a proxy that shows the reasoning of any app using your server
-  tuieval help                this help
-  tuieval --version
+  tuieval help · --version
 
 The workspace is --workspace DIR, else $TUIEVAL_HOME, else the current folder.
 """
@@ -67,6 +72,9 @@ def main(argv=None):
         if not workspace.is_workspace():
             sys.exit(f"{workspace.root()} isn't a tuieval workspace; run `tuieval init` first")
         return scaffold.cmd_new_pack(rest)
+    if cmd == "doctor":
+        from . import doctor
+        return doctor.cmd_doctor(rest)
     if cmd == "run":
         from . import run_evals
         return run_evals.main(rest)

tuieval-0.2.0.dev23/src/tuieval/doctor.py

@@ -0,0 +1,131 @@
+"""tuieval doctor: checks the setup and says what to fix (servers, models, packs, API keys).
+
+    tuieval doctor
+
+Each line is ✓ (fine), ! (worth knowing) or ✗ (a run would fail), with the fix. Exits 1 if any ✗.
+"""
+import argparse
+import os
+import platform
+import sys
+
+from . import engine
+from . import workspace
+
+OK, WARN, BAD = "✓", "!", "✗"
+COLOR = {OK: "\033[32m", WARN: "\033[33m", BAD: "\033[31m"}
+
+
+class Report:
+    def __init__(self):
+        self.bad = 0
+        self.color = sys.stdout.isatty() and not os.environ.get("NO_COLOR")
+
+    def line(self, mark, text, fix=""):
+        self.bad += mark == BAD
+        m = f"{COLOR[mark]}{mark}\033[0m" if self.color else mark
+        print(f"  {m} {text}" + (f"\n      → {fix}" if fix else ""))
+
+    def head(self, text):
+        print(f"\n{text}")
+
+
+def check(models_path=None):
+    """Print the report; returns the number of problems that would make a run fail."""
+    r = Report()
+    r.head("tuieval")
+    py = platform.python_version()
+    r.line(OK if sys.version_info >= (3, 11) else BAD, f"Python {py}", "" if sys.version_info >= (3, 11)
+           else "tuieval needs Python 3.11 or newer")
+    try:
+        e = engine.Engine(models_path=models_path)
+    except Exception as ex:   # a broken models.toml: nothing else can be checked
+        r.line(BAD, f"models.toml in {workspace.root()}: {ex}", "fix the file (tuieval init makes a fresh one elsewhere)")
+        return r.bad
+    r.line(OK, f"workspace {e.root}")
+
+    r.head("Packs")
+    for err in e.pack_errors:
+        r.line(BAD, err, "fix the pack, then: tuieval selftest")
+    if e.packs:
+        tests = sum(len(p.tests) for p in e.packs.values())
+        r.line(OK, f"{len(e.packs)} pack(s), {tests} tests  (tuieval selftest checks their answers)")
+    elif not e.pack_errors:
+        r.line(BAD, "no packs yet", "tuieval new-pack my-first-pack   (a pack of example questions to edit)")
+
+    r.head("Servers")
+    found = engine.detect_servers(engine.detect_ports(e.cfg))
+    used = {m["server"] for m in e.cfg["models"]}
+    answering = {}
+    for name, server in e.cfg["servers"].items():
+        if server.get("any_model") or server.get("api_key_env"):
+            key = server.get("api_key_env")
+            if key and os.environ.get(key):
+                r.line(OK, f"{name}: {key} is set")
+            elif key:
+                r.line(WARN, f"{name}: {key} isn't set (needed only for {name} models)",
+                       f"export {key}=… in the shell you start tuieval from")
+            continue
+        if server.get("cmd"):
+            missing = engine.missing_program([str(a) for a in server["cmd"]],
+                                             engine.expand(server["cwd"]) if server.get("cwd") else None)
+            if missing:
+                r.line(BAD if name in used else WARN,
+                       f"{name}: {missing} isn't installed" + ("" if name in used else " (no model uses it yet)"),
+                       engine.install_hint(missing))
+            else:
+                r.line(OK, f"{name}: {server['cmd'][0]} is installed; tuieval starts it for each model")
+            continue
+        url = server.get("url", "").rstrip("/").removesuffix("/v1")
+        ids = engine.probe_models(url) if url else None
+        answering[name] = ids
+        if ids is not None:
+            r.line(OK, f"{name}: {url} answers, serving {len(ids)} model(s)" + (f": {', '.join(ids[:5])}" if ids else ""))
+        else:
+            elsewhere = [f for f in found if f["url"] != url]
+            fix = (f"start the server at {url}, or "
+                   + ("run `tuieval add` to add the models of the one running at " + ", ".join(
+                       f"{f['url']}" + (f" ({f['usual']})" if f["usual"] else "") for f in elsewhere)
+                      if elsewhere else engine.url_fix(e.cfg, name)))
+            r.line(BAD if name in used else WARN,
+                   f"{name}: nothing answers at {url}" + ("" if name in used else " (no model uses it yet)"), fix)
+    known = {s.get("url", "").rstrip("/").removesuffix("/v1") for s in e.cfg["servers"].values()}
+    for f in found:
+        if f["url"] not in known:
+            r.line(WARN, f"found a server at {f['url']}" + (f" ({f['usual']}'s usual port)" if f["usual"] else "")
+                   + f" serving {len(f['models'])} model(s), not in models.toml",
+                   "add its models with: tuieval add")
+
+    r.head("Models")
+    if not e.cfg["models"]:
+        r.line(BAD, "no models yet", "tuieval add <GGUF path or model id>, or press a in the TUI")
+    for m in e.cfg["models"]:
+        server = e.cfg["servers"][m["server"]]
+        label = f"{m['label']} ({m['server']})"
+        if server.get("model_is_path"):
+            path = engine.expand(m["model"])
+            r.line(OK if os.path.exists(path) else BAD, f"{label}: {path}",
+                   "" if os.path.exists(path) else "the file is gone; fix `model` in models.toml or tuieval remove it")
+        elif m["server"] in answering:
+            ids = answering[m["server"]]
+            if ids is None:
+                r.line(BAD, f"{label}: its server isn't running", "see Servers above")
+            elif engine.serves(ids, m["served_name"]):
+                r.line(OK, f"{label}: served")
+            else:
+                r.line(BAD, f"{label}: the server doesn't list {m['served_name']!r}",
+                       f"load it in the server, or use one of: {', '.join(ids[:8]) or 'none listed'}")
+        else:
+            r.line(OK, label)
+
+    print("\n" + ("Ready: open the TUI with `tuieval`, or run `tuieval run --tier smoke`." if not r.bad
+                  else f"{r.bad} problem(s) to fix before a run (✗ above)."))
+    return r.bad
+
+
+def cmd_doctor(argv):
+    p = argparse.ArgumentParser(prog="tuieval doctor", description=__doc__,
+                                formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--models", default=None, help="default: the workspace's models.toml")
+    a = p.parse_args(argv)
+    sys.exit(1 if check(a.models) else 0)