tuieval 0.1.0__tar.gz

tuieval-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+build/
+dist/
+.DS_Store
+# a workspace used while developing; never commit your own packs or results here
+/my-evals/
+/results/
+/logs/
+/tuning/
+/packs/

tuieval-0.1.0/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0
+
+First public release.
+
+- TUI and CLI for evaluating local models on your own eval packs: accuracy, speed and token use, with PASS / FAIL / INCONCLUSIVE verdicts per use case.
+- Workspaces (`tuieval init`, `--workspace`, `TUIEVAL_HOME`) keep your packs, models and results apart from the tool.
+- `tuieval new-pack` starter templates for the built-in graders: answer, rag, reply, tool_call, code.
+- Custom graders from the workspace's `graders/` folder or a pack's own `grader.py`.
+- Servers: llama.cpp (with per-machine fit check and speed tuning), any OpenAI-compatible server, OpenRouter (pinned to one provider endpoint).
+- Speed tuning uses a built-in workload, so it needs no packs.

tuieval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 tuieval contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tuieval-0.1.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.5
+Name: tuieval
+Version: 0.1.0
+Summary: Evaluate local LLMs on your own eval packs, in the terminal: accuracy, speed and tokens, with PASS/FAIL verdicts.
+Project-URL: Homepage, https://github.com/ashe-wb/tuieval
+Project-URL: Issues, https://github.com/ashe-wb/tuieval/issues
+Author: tuieval contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmark,evals,evaluation,llama.cpp,llm,local-llm,tui
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console :: Curses
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: pyyaml>=6
+Requires-Dist: textual<9,>=8
+Description-Content-Type: text/markdown
+
+# tuieval
+
+**Evaluate local LLMs on your own questions, in the terminal.** Compare models (tiny, dense, MoE; llama.cpp GGUFs, LM Studio, Ollama, vLLM, OpenRouter) on what *you* use them for, measuring **accuracy, speed and token use** together, and get a **PASS / FAIL / INCONCLUSIVE** verdict per use case. Grading is automatic; there's no LLM judge.
+
+tuieval ships with **no built-in benchmark**. Public benchmarks leak into training data and rarely match your work. You write *eval packs* (folders of your own questions with checkable answers), and tuieval runs them, grades them, and tells you which model is ready for that job on this machine.
+
+```bash
+pipx install tuieval          # or: pip install tuieval   (Python 3.11+)
+tuieval init my-evals && cd my-evals
+tuieval new-pack my-first-pack # a pack of example questions to edit
+tuieval add ~/models/Some-Model-Q4_K_M.gguf
+tuieval                        # open the TUI
+```
+
+Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
+
+![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+
+## What you get
+
+- **A TUI** to pick packs and models, watch reasoning and answers stream live with TTFT, tokens/s, memory and a running score, and browse results.
+- **Verdicts you can act on.** A pack passes only with zero critical failures over enough trials, and an accuracy whose 95% lower bound clears your bar. *INCONCLUSIVE* says what evidence is missing.
+- **Two tiers:** *Screen* a sample of every pack to drop weak models fast, then *Certify* finalists on every test with repeats. Certification reuses the screening answers.
+- **Honest numbers:** every request is a fresh single-turn conversation, with prompt caching off and repeat rounds in different orders. Each answer records what was sent, and the run warns about reused prompts or identical repeats.
+- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
+- **Tests that check themselves.** Every test carries a reference answer and known-wrong answers, and `tuieval selftest` checks the grader accepts the first and rejects the second, and that each gate is reachable at all.
+- **History.** Verdict changes are appended to `results/verdicts.jsonl`, and replaced results are moved to `history/`, never overwritten.
+
+## Eval packs
+
+A pack is a folder in your workspace's `packs/`:
+
+```
+packs/support-bot/
+  pack.toml       label, use case, grader, gate (what PASS means)
+  system.txt      the system prompt
+  tests.yaml      the questions, with expected answers, references and known-wrong answers
+```
+
+```yaml
+- id: refund-window
+  difficulty: medium
+  input: A customer bought shoes 20 days ago and wants a refund. Our policy allows 30 days. Can they get one?
+  max_words: 40
+  must_include: ["yes|can"]
+  reference: "Yes, they're within the 30-day window, so they can get a refund."
+  wrong: ["No, the refund window has passed."]
+```
+
+Built-in graders: `answer` (a number or word on an `ANSWER:` line, or a correct refusal when the data isn't there), `rag` (grounded answers with citations), `reply` (free-form replies against rules), `tool_call` (the right tool with the right arguments, or rightly none) and `code` (Python run against hidden tests). Your own grader is one Python file in the workspace's `graders/` folder.
+
+`tuieval new-pack <name> --grader <grader>` creates a pack with working examples for any grader. **See [docs/writing-packs.md](docs/writing-packs.md)** for every field, gates, critical failures, difficulty labels, custom graders and how to write tests that separate good models from weak ones.
+
+## The workspace
+
+Your packs, models and results live in a **workspace** folder, apart from the tool:
+
+```
+my-evals/
+  models.toml      your servers and models
+  packs/           your eval packs
+  graders/         your own graders (optional)
+  results/ logs/ tuning/ reports/ presets.toml     written by tuieval
+```
+
+tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it in git or a synced folder: results from one machine count on another when the answer-changing settings match, so you can Certify on your fastest machine.
+
+## Models
+
+```bash
+tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
+tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
+tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
+```
+
+See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinning, machines, tuning and every option.
+
+## The TUI
+
+1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results).
+2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+
+## Command line
+
+```bash
+tuieval run                                   # Screen every model on every pack
+tuieval run --tier certify --only a,b --packs support-bot,coding
+tuieval run --dry-run                         # the plan and server commands
+tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
+tuieval report                                # the same with evidence, as a markdown file
+tuieval history                               # every verdict change over time
+tuieval compare --speed --pairwise --failures # scorecards
+tuieval selftest                              # check every test's reference and wrong answers
+tuieval items                                 # tests that don't separate models or look broken
+tuieval capture logs/live/<file> --pack X     # turn a real failure into a new test
+tuieval regrade                               # re-score stored answers after changing a grader
+tuieval machines                              # this machine and others, fit and tuning per model
+tuieval tune <model>                          # fastest speed flags for a model on this machine
+tuieval watch --upstream http://localhost:8080   # show the reasoning of any app using your server
+tuieval help
+```
+
+## Reading results
+
+- **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
+- **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
+- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
+- **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
+
+## Safety
+
+⚠️ The `code` grader executes model-written code on your machine. Run code packs inside a container or VM with no credentials in the environment.
+
+## Development
+
+```bash
+git clone https://github.com/ashe-wb/tuieval && cd tuieval
+python -m venv .venv && .venv/bin/pip install -e .
+.venv/bin/python -m unittest discover -s tests    # uses a mock server; no model needed
+```
+
+## License
+
+MIT. See [LICENSE](LICENSE).

tuieval-0.1.0/README.md

@@ -0,0 +1,130 @@
+# tuieval
+
+**Evaluate local LLMs on your own questions, in the terminal.** Compare models (tiny, dense, MoE; llama.cpp GGUFs, LM Studio, Ollama, vLLM, OpenRouter) on what *you* use them for, measuring **accuracy, speed and token use** together, and get a **PASS / FAIL / INCONCLUSIVE** verdict per use case. Grading is automatic; there's no LLM judge.
+
+tuieval ships with **no built-in benchmark**. Public benchmarks leak into training data and rarely match your work. You write *eval packs* (folders of your own questions with checkable answers), and tuieval runs them, grades them, and tells you which model is ready for that job on this machine.
+
+```bash
+pipx install tuieval          # or: pip install tuieval   (Python 3.11+)
+tuieval init my-evals && cd my-evals
+tuieval new-pack my-first-pack # a pack of example questions to edit
+tuieval add ~/models/Some-Model-Q4_K_M.gguf
+tuieval                        # open the TUI
+```
+
+Once you've added your own packs and models, the setup screen looks like this (packs on the left, models with a verdict code per use case on the right, the highlighted model's details below):
+
+![tuieval's setup screen with five eval packs and a dozen local models](https://raw.githubusercontent.com/ashe-wb/tuieval/main/docs/images/tui-setup.png)
+
+## What you get
+
+- **A TUI** to pick packs and models, watch reasoning and answers stream live with TTFT, tokens/s, memory and a running score, and browse results.
+- **Verdicts you can act on.** A pack passes only with zero critical failures over enough trials, and an accuracy whose 95% lower bound clears your bar. *INCONCLUSIVE* says what evidence is missing.
+- **Two tiers:** *Screen* a sample of every pack to drop weak models fast, then *Certify* finalists on every test with repeats. Certification reuses the screening answers.
+- **Honest numbers:** every request is a fresh single-turn conversation, with prompt caching off and repeat rounds in different orders. Each answer records what was sent, and the run warns about reused prompts or identical repeats.
+- **Per-machine speed.** A fit check picks the largest context that fits your Mac's GPU memory, `tuieval tune` finds the fastest speed-only server flags (with a guard that rejects flags that change answers), and readiness includes a *Fast enough?* table per machine, measured or projected.
+- **Tests that check themselves.** Every test carries a reference answer and known-wrong answers, and `tuieval selftest` checks the grader accepts the first and rejects the second, and that each gate is reachable at all.
+- **History.** Verdict changes are appended to `results/verdicts.jsonl`, and replaced results are moved to `history/`, never overwritten.
+
+## Eval packs
+
+A pack is a folder in your workspace's `packs/`:
+
+```
+packs/support-bot/
+  pack.toml       label, use case, grader, gate (what PASS means)
+  system.txt      the system prompt
+  tests.yaml      the questions, with expected answers, references and known-wrong answers
+```
+
+```yaml
+- id: refund-window
+  difficulty: medium
+  input: A customer bought shoes 20 days ago and wants a refund. Our policy allows 30 days. Can they get one?
+  max_words: 40
+  must_include: ["yes|can"]
+  reference: "Yes, they're within the 30-day window, so they can get a refund."
+  wrong: ["No, the refund window has passed."]
+```
+
+Built-in graders: `answer` (a number or word on an `ANSWER:` line, or a correct refusal when the data isn't there), `rag` (grounded answers with citations), `reply` (free-form replies against rules), `tool_call` (the right tool with the right arguments, or rightly none) and `code` (Python run against hidden tests). Your own grader is one Python file in the workspace's `graders/` folder.
+
+`tuieval new-pack <name> --grader <grader>` creates a pack with working examples for any grader. **See [docs/writing-packs.md](docs/writing-packs.md)** for every field, gates, critical failures, difficulty labels, custom graders and how to write tests that separate good models from weak ones.
+
+## The workspace
+
+Your packs, models and results live in a **workspace** folder, apart from the tool:
+
+```
+my-evals/
+  models.toml      your servers and models
+  packs/           your eval packs
+  graders/         your own graders (optional)
+  results/ logs/ tuning/ reports/ presets.toml     written by tuieval
+```
+
+tuieval uses the current folder, or `--workspace DIR` / `TUIEVAL_HOME`. Keep it in git or a synced folder: results from one machine count on another when the answer-changing settings match, so you can Certify on your fastest machine.
+
+## Models
+
+```bash
+tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4     # llama.cpp (llama-server on PATH)
+tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf   # vision
+tuieval add qwen3:8b --server local                                 # an already-running server (set its url)
+tuieval run --tier smoke --only openrouter:qwen/qwen3-32b           # any OpenRouter model, no config needed
+```
+
+See **[docs/models.md](docs/models.md)** for servers, OpenRouter endpoint pinning, machines, tuning and every option.
+
+## The TUI
+
+1. **Pick packs and models** (space ticks; type to filter models by label or tag). Each model shows a short code per use case, e.g. `C✓ S?` (✓ pass, ✗ fail, ? inconclusive; grey = from earlier results).
+2. **Pick a tier** (Smoke to check setup, Screen, Certify) and **press `s`**. The line above the buttons shows how many answers that is and roughly how long it will take. For each model, tuieval starts its server (or uses a running one), checks the right model is loaded, runs every selected pack, then stops it.
+3. **Watch the run:** progress with ETA, live score, tok/s, TTFT and memory, the current test with reasoning and answer side by side, recent results with the grader's reason. `k` skips a model, `c` cancels (finished work is kept and resumes next time). Runs started while one is going are queued.
+4. **Press `r` for results:** production readiness, verdict history, scorecard, speed & tokens (★ marks models nothing beats on both accuracy and time), per question, is the difference real?, tests that separate models, by difficulty, failures, and test quality.
+
+Other keys: `t` tunes the ticked models' speed flags, `a` adds a model, `m` scans for GGUFs, `p` saves or loads a preset, `x` hides a model.
+
+## Command line
+
+```bash
+tuieval run                                   # Screen every model on every pack
+tuieval run --tier certify --only a,b --packs support-bot,coding
+tuieval run --dry-run                         # the plan and server commands
+tuieval verdict                               # PASS / FAIL / INCONCLUSIVE per model and use case
+tuieval report                                # the same with evidence, as a markdown file
+tuieval history                               # every verdict change over time
+tuieval compare --speed --pairwise --failures # scorecards
+tuieval selftest                              # check every test's reference and wrong answers
+tuieval items                                 # tests that don't separate models or look broken
+tuieval capture logs/live/<file> --pack X     # turn a real failure into a new test
+tuieval regrade                               # re-score stored answers after changing a grader
+tuieval machines                              # this machine and others, fit and tuning per model
+tuieval tune <model>                          # fastest speed flags for a model on this machine
+tuieval watch --upstream http://localhost:8080   # show the reasoning of any app using your server
+tuieval help
+```
+
+## Reading results
+
+- **Start with Production readiness.** A single critical failure already means FAIL, whatever the accuracy: a model that breaks a hard rule once in 300 answers will do it in production. The Failures tab and `tuieval report` list exactly which answers failed.
+- **INCONCLUSIVE is not "nearly passed".** It says what's missing (usually a Certify run, or more trials).
+- **Trust "Is the difference real?" over raw percentages.** Differences of one or two tests are usually noise.
+- **Read `trunc` before accuracy.** A model that runs out of tokens isn't wrong; it's thinking too long for the budget.
+- **Watch TTFT for interactive use.** A model that is 5% more accurate but takes 3 s longer to start answering may be the worse choice.
+
+## Safety
+
+⚠️ The `code` grader executes model-written code on your machine. Run code packs inside a container or VM with no credentials in the environment.
+
+## Development
+
+```bash
+git clone https://github.com/ashe-wb/tuieval && cd tuieval
+python -m venv .venv && .venv/bin/pip install -e .
+.venv/bin/python -m unittest discover -s tests    # uses a mock server; no model needed
+```
+
+## License
+
+MIT. See [LICENSE](LICENSE).

tuieval-0.1.0/docs/models.md

@@ -0,0 +1,120 @@
+# Models, servers and machines
+
+Everything about models lives in your workspace's `models.toml`. `tuieval init` writes one with three servers ready to use:
+
+| server | what it is |
+|---|---|
+| `llama` | [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server`, started by tuieval for each GGUF model, with a per-machine fit check and speed tuning |
+| `local` | any OpenAI-compatible server that's already running (LM Studio, Ollama, vLLM, a remote box): just a `url` |
+| `openrouter` | any model on [OpenRouter](https://openrouter.ai/models), named at run time, pinned to one provider endpoint |
+
+## Adding models
+
+```bash
+tuieval add ~/models/Some-Model-Q4_K_M.gguf --tags 9b,dense,q4      # GGUF -> llama
+tuieval add ~/models/Some-Model-Q4_K_M.gguf --no-think              # same model, thinking off (label …-nothink)
+tuieval add ~/models/VL-Q4.gguf --mmproj ~/models/VL-mmproj.gguf    # vision model
+tuieval add qwen3:8b --server local                                  # a model your running server serves
+tuieval scan --add                                                   # every new GGUF under model_dirs
+tuieval list                                                         # models, packs, result status
+```
+
+In the TUI, `a` adds a model and `m` scans your model folders. Each model is one `[[models]]` block:
+
+| field | meaning |
+|---|---|
+| `server` | which `[servers.*]` block serves it |
+| `model` | GGUF path (llama) or model id (other servers) |
+| `label` | names `results/<label>/`; default: the file or repo name, lowercased. Never rename a label that has results. |
+| `served_name` | the model name sent with each request (default: the label for llama, the model id elsewhere) |
+| `vision`, `mmproj` | the model reads images; `mmproj` is llama's vision projector and implies `vision` |
+| `thinking = false` | run with thinking off |
+| `tags = [...]` | filter by them in the TUI and with `--tags` |
+| `sampling = {...}` | override `[sampling]` for this model |
+| `server_args = [...]` | extra server flags (treated as answer-changing) |
+| `ctx` (llama) | cap the context; the fit check may lower it further per machine |
+| `max_context` (other servers) | the context the server provides; packs whose prompts need more are skipped |
+| `kv_type` (llama) | e.g. `"q8_0"`: a quantized KV cache, which makes it a different model (give it its own label) |
+| `tools = false` | packs that need tool calling are skipped |
+
+Results record the settings each model actually ran with, and the results screen notes any differences between models.
+
+## OpenRouter
+
+OpenRouter models need no block. Name any model from openrouter.ai/models when you run, e.g. `tuieval run --tier smoke --only openrouter:qwen/qwen3-235b-a22b-2507`. In the TUI, tick **+ openrouter: any model** at the bottom of the model list and pick from the live list (filter by typing; it shows context and price), or press `a` and type `openrouter:<model id>`. A wrong id is refused with the closest matches.
+
+- Needs `OPENROUTER_API_KEY` exported in the shell that starts tuieval. A plain `tuieval run` never includes OpenRouter models, since they cost money.
+- Vision, tool calling and context size come from OpenRouter's model list; packs the model can't do are skipped.
+- **One provider per model.** Left alone, OpenRouter spreads requests over providers running different quantizations. `pin_endpoint = true` picks one endpoint per model (closest to the released weights first: bf16 > fp8 > undeclared > lower; then tool and sampling support, uptime and price), sends every request only there with no fallback, keeps it while it's offered, and records it with the results. Answers from any other provider aren't counted.
+- Hosted providers cache shared prompt openings and that can't be switched off. It doesn't change answers, but those answers (⟲ in the run) are left out of TTFT and prompt-speed numbers.
+- To keep providers that store or train on prompts (and could learn your tests) out entirely, add `request = { provider = { data_collection = "deny" } }` under `[servers.openrouter]`.
+
+## Every answer is independent
+
+Each request is a fresh single-turn conversation: the system prompt and one question, never an earlier answer. Repeats go round by round, each round asking the tests in its own fixed shuffled order, so a question never follows itself. The llama server is told not to reuse earlier prompts (`request = { cache_prompt = false }`). Each answer records the messages sent and any prompt tokens the server reports reusing, and the run warns if that's ever not 0. If a question gets the exact same long answer twice with sampling on, the run flags it as a probable cached response. `tuieval selftest` checks all of this.
+
+## Answer-changing vs speed-only flags
+
+The best server flags differ per model and per machine, so tuieval splits them by what they change:
+
+| Changes **answers**: same everywhere, part of each result's fingerprint | Changes **only speed**: tuned per model per machine |
+|---|---|
+| model file and quant, KV-cache type, `--jinja`, `--reasoning-format`, sampling, mmproj (`cmd` in `models.toml`) | threads, batch sizes, flash attention, cache reuse (`perf` and `[servers.llama.tune]`) |
+
+**Quality results travel.** Tuning never invalidates results, and a result from one machine counts on another as long as the answer-changing settings match. So you can Certify on your fastest machine and only tune (and optionally Screen) on the others: sync the workspace folder between them.
+
+## Machines
+
+- **Machine id** is detected automatically (`m1max-32gb`, `m4pro-24gb`, …; set `EVALS_MACHINE` to rename it). Every result records the machine, the server version and the exact speed flags used. `tuieval machines` lists this machine and every other machine that has run the evals (they record themselves in `tuning/`).
+- **Fit check (Apple Silicon, llama):** before starting a GGUF, tuieval reads its header (layers, KV heads, hybrid attention layers) and picks the largest context that fits this machine's GPU memory, capped at `max_ctx`. A model that can't fit at 8k context is skipped with *doesn't fit on <machine>* instead of swapping. Packs that need more context than fits are skipped. It never quantizes the KV cache on its own, since that changes answers.
+- **Per-machine settings** go under `[machines.<id>]`: `memory_headroom_gb` (GPU memory kept free for macOS, default 4) and `gpu_residency_gb` (see the stall guard).
+
+## Tuning
+
+`tuieval tune <model>` (or `t` on the setup screen, for the ticked models) finds the fastest speed flags for a model on this machine:
+
+1. If `llama-bench` is installed, it sweeps threads, micro-batch and flash attention first (fast, no server starts).
+2. Then it starts the real server with one knob changed at a time and times a fixed **built-in** workload (three short prompts, three medium ones and one ~8k-token prompt), so tuning needs no packs and speeds are comparable between workspaces.
+3. An **output guard** rejects any option that changes greedy answers beyond noise. Candidates that make macOS swap are rejected.
+
+Expect 8–15 server starts, about 20–30 minutes for a 27B model, once per model per machine. The result is saved in `tuning/<machine>/<model>.toml` and used by every later run there. Models without a profile run with each knob's first option and show *untuned*. A profile is marked for retuning when the model file or server version changes.
+
+The knobs are `[servers.<name>.tune]` in `models.toml`: each knob is a list of options, each option a list of flags. Placeholders: `{p}` P-cores, `{p_minus_2}`, `{all}` all cores, `{gpu_safe_gb}` (the GPU residency limit minus 2 GB; also `_minus_1`, `_minus_2`, `_plus_1`).
+
+## Speed verdicts
+
+Speed is judged separately, per machine: the same model can be production-grade in quality and still too slow on a smaller machine. Results → Production readiness has a **Fast enough?** table: p90 seconds per answer against each pack's `max_p90_s`, for every machine. It's *measured* where the model ran, and otherwise *projected* from each answer's token counts and that machine's tuned speeds. `tuieval compare --machine <id>` shows the same on the command line.
+
+## The stall guard (Apple Silicon)
+
+Measured on an M1 Max 32 GB: once the system's GPU allocations pass about half of RAM, the GPU driver evicts and re-maps memory on every GPU job, the server spends 70–95% of its CPU in the kernel while the GPU idles, and servers that submit many small GPU jobs slow to a crawl. For servers with `stall_guard = true`, tuieval samples the server's own vs kernel CPU time and the system's GPU allocation every 5 s during runs and tuning. If more than 70% of its CPU goes to the kernel for a minute, the run stops with an explanation instead of crawling for hours. Finished answers are kept and resume next time. If your machine behaves differently, set `gpu_residency_gb` under `[machines.<id>]`.
+
+## Server options
+
+| option | meaning |
+|---|---|
+| `cmd` | command that starts the server (no `cmd` = an already-running server at `url`). Placeholders: `{model}`, `{served_name}`, `{port}`, `{mmproj}`, `{ctx}`, `{kv_type}`, `{root}` (the workspace), and the tune placeholders. An argument `env:NAME=value` sets an environment variable instead. |
+| `url` | an already-running server's base URL |
+| `port` | the port `cmd` serves on |
+| `cwd` | folder to start `cmd` in |
+| `model_is_path` | `model` is a file or folder: check it exists (and, for GGUFs, that it fits) before starting |
+| `vision_args` | flags appended for models with `mmproj` |
+| `kv_type`, `max_ctx` | llama defaults: KV-cache type, upper bound on context |
+| `ctx_flag` | the server's context flag, so long packs are skipped if a configured context is too small |
+| `request` | fields added to every request (e.g. `{ cache_prompt = false }`) |
+| `perf` | speed-only flags always applied |
+| `tune` | speed-only knobs for `tuieval tune` |
+| `tune_objective` | `total` (default: workload time) or `decode` (tokens/s after a warm-up pass) |
+| `version_cmd` | prints the server version, recorded with every result |
+| `health` | readiness path for servers without `/v1/models` (must return JSON with `model`) |
+| `before_start` | a command run before starting the server (e.g. to free memory another process holds) |
+| `env` | environment variables for the server |
+| `log_facts` | `{name = "regex"}` read from the server's startup log and recorded with every run |
+| `require_facts` | `{name = "regex"}` the startup log must show, or the run refuses to start (e.g. a setting that must be off for fair evals) |
+| `stall_guard` | watch for GPU-driver stalls (see above) |
+| `outputs_depend_on_machine` | answers depend on this machine's memory settings: results only count on the machine that produced them, and tuned flags are fingerprinted |
+| `any_model`, `label_prefix` | any model the server lists can be named at run time as `<server>:<id>` (OpenRouter) |
+| `pin_endpoint` | pin each model to one provider endpoint (OpenRouter) |
+| `api_key_env` | environment variable holding the API key |
+| `thinking_param` | `reasoning` to send `[sampling] enable_thinking` as OpenRouter's `reasoning.enabled` |
+| `headers` | extra HTTP headers |

tuieval-0.1.0/docs/writing-packs.md

@@ -0,0 +1,160 @@
+# Writing eval packs
+
+tuieval ships with no packs: you write the questions that matter for what you use models for. A pack is a folder in your workspace's `packs/` with a `pack.toml` and one or more test files. Every folder there with a `pack.toml` shows up in the TUI. Delete or rename a folder (prefix it with `_` to hide it) and it's gone.
+
+The quickest start is a starter template with example tests that already pass `tuieval selftest`:
+
+```bash
+tuieval new-pack support-bot --grader reply      # answer | rag | reply | tool_call | code
+```
+
+Edit `packs/support-bot/tests.yaml`, replace the examples with your own questions, and run `tuieval selftest support-bot` after every change.
+
+To try a different set of questions, copy a pack (`cp -r packs/support-bot packs/support-bot-v2`), edit the copy, and pick whichever you want in the TUI.
+
+## Fingerprints
+
+Every pack has a fingerprint of what the model sees and what decides pass/fail (questions, images, system prompt, tools, expected answers, grader). Results record it, so after you edit a question the TUI marks older results `~ outdated` and reruns them instead of comparing different question sets. Editing a description, category, difficulty, reference answer, wrong answers or gate doesn't make results outdated.
+
+## pack.toml
+
+```toml
+label = "Support bot"            # shown in the TUI
+group = "Customer support"       # the use case it counts toward (see below)
+description = "…"
+grader = "reply"                 # default grader for its tests (see Graders)
+system = "system.txt"            # system prompt file (optional)
+tools = "tools.yaml"             # OpenAI-style tool definitions sent with every request (optional)
+needs = ["vision"]               # vision | tools | long_context | <python module>: skipped where unavailable
+order = 30                       # position in the list
+tests = ["tests.yaml"]           # optional; default: tests.* first, then every other .yaml/.csv file
+screen = 20                      # tests in a Screen run (spread across categories, one variant per group)
+
+[certify]
+repeat = 3                       # repeats in a Certify run (default: models.toml [defaults] repeat)
+
+[gate]                           # release criteria; see "Gates" below
+min_accuracy = 0.90              # the 95% lower bound of the pass rate must reach this to PASS
+max_critical_rate = 0.01         # needs 3/rate failure-free critical trials (rule of three): 300 here
+max_p90_s = 30                   # 90th-percentile seconds per answer, judged per machine
+max_truncation = 0.01            # share of answers cut off by max_tokens
+min_consistency = 0.95           # share of variant groups where every variant passes
+```
+
+`group` doubles as the **use case** in the readiness verdicts: all packs in a group must PASS for the use case to PASS. For example, a "Coding" use case could be a `coding` pack plus a `coding-sql` pack.
+
+`needs` entries other than `vision`, `tools` and `long_context` name Python modules the pack's grading needs (for example `pandas`, when your hidden tests use it). The pack is skipped, with a note, where that module isn't installed in tuieval's Python environment.
+
+## Tests (YAML)
+
+```yaml
+- id: refund-window               # optional, must be unique in the pack (default: from description)
+  description: refund window       # shown while running and in results
+  category: policy                 # optional grouping (Screen runs spread across categories)
+  difficulty: medium               # easy | medium | hard: your label, shown in the TUI, never sent to the model
+  input: |                         # the user message
+    A customer bought shoes 20 days ago …
+  image: images/receipt.png        # optional: sent as an image (needs vision)
+  grader: answer                   # optional: override the pack's grader
+  group: refund-window-2           # optional: variants of one case share a group (consistency gate)
+  critical: true                   # optional: any failure of this test is critical (disqualifying)
+  expected: 30                     # grader-specific fields from here on
+  tolerance: 0
+  reference: "…\nANSWER: 30"       # a correct model answer (never sent to the model)
+  wrong: ["ANSWER: 14"]            # known-bad answers the grader must reject
+```
+
+**`reference` and `wrong` make a test check itself.** `tuieval selftest` grades every reference (it must pass with full score) and every wrong answer (it must fail). A test whose expected answer is wrong, or whose grader can't tell a classic mistake from the right answer, is caught before it fails a good model. Give every new test a reference, and a `wrong` entry for the mistake you most expect.
+
+`selftest` also checks that no two tests send the same prompt, that tool tests expect tools the pack defines, that no `TODO` placeholders are left, that each request is one fresh single-turn conversation, and that the gate is reachable (see below).
+
+## Tests (CSV)
+
+For question-and-answer packs, a spreadsheet is often easier. Columns: `id, input, expected, tolerance, expected_text, category` (and any other test field). Empty cells are ignored.
+
+```csv
+id,input,expected,tolerance,expected_text,category
+q1,"What is 15% of 80? End with ANSWER: <number>",12,0,,math
+q2,"Capital of Peru? End with ANSWER: <city>",,,Lima,geo
+```
+
+## Graders
+
+| grader | checks | test fields |
+|---|---|---|
+| `answer` | the last `ANSWER: …` line | `expected` (number or `NOT_AVAILABLE`) + `tolerance`, or `expected_text` (word, case-insensitive; a list means any of them) |
+| `code` | last ```` ```python ```` block against hidden tests | `hidden_tests` with `# SETUP` / `# CHECK: name` blocks; score = fraction of checks passed |
+| `rag` | answer grounded in passages, with citations | as `answer`, plus `expected_sources: [P3]`; half credit if the answer is right but the citation isn't |
+| `tool_call` | the tool the model called | `expect_tool: {name, arguments}`, or `expect_no_tool: true` (+ `clarify: true`) |
+| `reply` | a free-form reply | `max_words`, `no_markdown`, `must_include` (`"a\|b"` = either), `must_not_include`, `must_match`, `ends_with_question` |
+
+⚠️ The `code` grader executes model-written code on your machine. Run code packs inside a container or VM with no credentials in the environment.
+
+### Your own graders
+
+A grader is a Python function registered by name. Put it in your workspace's `graders/` folder (one `.py` file each), or in a pack's own `grader.py` so the pack carries its grading with it:
+
+```python
+# graders/ticket_route.py
+import json
+
+from tuieval.graders import grader, result
+
+
+@grader("ticket_route", template={"expected_queue": "TODO"})
+def ticket_route(answer, test, meta):
+    """The model replies with JSON like {"queue": "billing"}. Security-report tests are marked
+    `critical: true` in tests.yaml, so they count as critical trials."""
+    try:
+        queue = json.loads(answer).get("queue")
+    except (ValueError, AttributeError):
+        return result(False, "not a JSON object")
+    if queue != test["expected_queue"]:
+        # sending a security report anywhere but the security queue is disqualifying
+        severe = test["expected_queue"] == "security"
+        return result(False, f"routed to {queue!r}", severity="critical" if severe else None)
+    return result(True, f"routed to {queue!r}")
+```
+
+- `answer` is the model's final answer (reasoning already removed); `test` is the test's fields; `meta` has `finish` and `tool_calls`.
+- `result(ok, reason, score=None, severity=None)`: `score` defaults to 1.0 or 0.0; `severity="critical"` marks a disqualifying failure.
+- `critical=True` makes every test of this grader a critical trial (any of its failures *can* be critical). Otherwise only tests with `critical: true` or `expected: NOT_AVAILABLE` are.
+- `template` is the skeleton `tuieval capture` writes for a new test of this grader.
+
+Truncated and empty answers are failed before your grader runs. After changing a grader, `tuieval regrade` re-scores the stored answers without rerunning any model.
+
+## Gates
+
+A pack **PASSes** only on a full Certify run, when all of these hold:
+
+- **zero critical failures**, over enough critical trials to show the critical rate is below `max_critical_rate` (rule of three: 300 clean trials for 1%, 60 for 5%)
+- the **95% lower bound** of accuracy clears `min_accuracy`. Because it's the lower bound, the observed score must be higher, more so for small packs: with 30 trials, clearing an 80% bar takes about 29 correct.
+- truncation, consistency and (per machine) p90 latency are within their budgets
+
+`tuieval selftest` rejects a gate that a flawless certification run couldn't pass, and says what to add (tests, repeats, or a looser rate).
+
+## What makes a failure critical
+
+Graders mark failures that should disqualify a model whatever its accuracy:
+
+- `answer`, `rag`: inventing an answer where the right one is `NOT_AVAILABLE`
+- `tool_call`: an action nobody asked for (wrong tool, an unwanted or extra call)
+- any test with `critical: true`
+- whatever your own graders return with `severity="critical"`
+
+## Difficulty labels
+
+Every test has a `difficulty` of easy, medium or hard. It's for you, not the model: requests are built only from `input`, `image`, the system prompt and tools, so the label never reaches the model. The TUI shows it in the pack list (e.g. `12E 20M 8H`), next to the current question, in Recent results and Failures, and in Results → By difficulty.
+
+A useful convention: **easy** = one rule or a direct read; **medium** = arithmetic, one trap, or two conditions; **hard** = rules in conflict, multi-step reasoning, or misleading data. Once several models have run, `tuieval items` flags labels the results contradict ("easier than labelled": a hard test every model passes; "harder than labelled": an easy test most models fail). `tuieval selftest` warns about tests without a label.
+
+## Writing tests that separate production-ready models
+
+- **Spec every behaviour you check.** A correct answer written only from the prompt must pass.
+- **Test the boundaries** (exactly at a limit, one past it) and the conflicts between rules, not only typical cases.
+- **Write each case twice** in a different surface form (layout, wording, field order) with the same `group`: a production model must not flip its answer.
+- **Put traps where real use has them:** missing data, injected instructions, misleading visuals.
+- **Use invented names, products and numbers**, so a model can't score from memory.
+- **Generate volume with a script.** Critical gates need hundreds of trials; a small generator with a fixed seed that writes `generated.yaml` (references and wrong answers included) scales far better than hand-writing. Rerun it rather than editing its output.
+- **After a few models have run, `tuieval items`** lists tests nobody fails (no signal), everybody fails (check the test), weak models beat strong ones on (inverted), or one model flips on (flaky).
+- **Turn real failures into tests:** `tuieval capture logs/live/<file> --pack <name>` writes a skeleton to `captured.yaml` with the model's bad answer under `wrong`; fill in the TODOs.