llm-evals 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2ae773e535e412da4e212478e21e0d8cbb828e41393328a2e76344511e55694c
-  data.tar.gz: be95ec246ee851a62230279e2892541415baab88b8220a3ed2f7f507c41654fa
+  metadata.gz: 1df9a8ec28e61b18c3b929fbf1aff863f01b4083e762dcc8dfa24b0a4c107da7
+  data.tar.gz: ae89677e193f2c6369663df56d92276de19b2d4be3de5c83502612840f79d96e
 SHA512:
-  metadata.gz: 7d880476a22a64188d8266071248f2e2cc9fe06b2d57680f0d52045bf5d32bd22336e1664f5d79186721a1aa9ef44b9c266e67090d5f0deadbe873cbaff7f572
-  data.tar.gz: f68267cbca0e553ff8c524e407de65a6e448167f06b53ddb8506e3b333b1a2a6663f2e05ef5e1db19b3684ea7d177ab8cbf52b22d83f024d7001ca892312d959
+  metadata.gz: 719e4775de99d347928c359df4a92ae253c5686a153e9970acb02e0baf23a8dd17c0811e2d6476993f1b563b1bef28d02efc721a0b44440a47db25999fc90bec
+  data.tar.gz: 1cd81a2ca9d699ca1b1f15ce4ac3fca1af31288ada47e4fbff395255ad997e04d2a4e9b37e7303696c4b0fe9b2c1bb7dce22b2b133ca93a9aa7f5d8c54bfc8a3

data/CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## Unreleased
+
+- Breaking: `results.display` now defaults to `both`: every run prints the terminal summary, builds the HTML report from its JSON and opens it in the browser. Use `--display tui` or `--no-open` to turn that off.
+
+- Group the `run` and `compare` flags in `--help` under General, TUI and Browser Report.
+
+- Breaking: move every class out of the top of `LLM::Evals` into a module of its own. There are no aliases; rename as follows:
+
+  | Module                 | Constants                                                                                      |
+  | ---------------------- | ---------------------------------------------------------------------------------------------- |
+  | `LLM::Evals::Runtime`  | `Configuration`, `Helper`; `Runtime` is now `Runtime::Environment`                             |
+  | `LLM::Evals::Datasets` | `Case`, `Discovery`                                                                            |
+  | `LLM::Evals::Scoring`  | `Scorer`, `Metric`, `WeightedScore`, `SchemaValidator`                                         |
+  | `LLM::Evals::Models`   | `Model`, `Contender`, `Throttle`                                                               |
+  | `LLM::Evals::Actions`  | `Experiment`, `Adapter`, `Verify`, `ErrorBudget`                                               |
+  | `LLM::Evals::Results`  | `Reply`, `Score`, `ProviderResponse` (was `Completion`), `RunResult`, `RunOutcome`, `Warnings` |
+  | `LLM::Evals::Stats`    | `CallStats`, `Usage` (`Stats` itself stays)                                                    |
+
+  `LLM::Evals::Tracker` is now `LLM::Evals::Trackers::Base`, and `LLM::Evals::ConfigUpgrade` is `LLM::Evals::Config::Upgrade`. `LLM::Evals.configure` and `.configuration` are unchanged.
+
+- Add `Model#ask` and `Providers::TypeSafe#ask`: typed questions to TypeSafe's API, through the same limits and retries.
+
+- Add `evals upgrade-config`, which rewrites a lineup to the current format, keeping its comments and leaving `<file>.bak`.
+
+- Breaking: the lineup lists models under a provider section (`openrouter: { vendors: ... }`, `typesafe: { models: ... }`). Top-level `vendors:` and `direct:` are errors.
+
+- Add `rate_limit: { burst, per_minute }` per provider, vendor and model, through `ruby-limiter`. Waiting is left out of `time`.
+
+- Turn RubyLLM's retries off and retry here, honoring `Retry-After` across a vendor's calls, up to 5 attempts.
+
+- Add `errors.abort_after` to the lineup: a run stops once it has `total` errors and, if set, `percent` of finished calls have failed. Partial results are kept and the command exits non-zero.
+
+- Add `evals weights`, which prints each weighted column's direction, weight and real share of the score, and names in red any scorer metric the weights leave out.
+
+- Report calls per provider, vendor and model: requests, errors, requests per second, average and 95th percentile latency, in the terminal and as `call_stats` in the JSON. Add `latency` and `p95` as columns `weighted_score` can weight.
+
+- Draw failed cases in red on the progress bars of `run` and `compare`, so a failing model or account shows while the run is still going.
+
 ## 0.4.0 (2026-10-30)
 
 Renamed from `agent-evals`: gem `llm-evals`, namespace `LLM::Evals`, executable `evals` (with `llm-evals` as an identical spare, for a system where `evals` is taken; shell completion targets `evals`), lineup file `.llm-evals.config.yml`. The old names are gone. The version continues from `agent-evals` 0.3.1.

data/README.md

@@ -8,15 +8,20 @@ A flexible CLI for running multi-case evaluation suites and comparing LLMs and p
 
 > [!IMPORTANT]
 > This gem is not affiliated with the excellent [RubyLLM](https://rubyllm.com/) gem, though it uses RubyLLM under the hood.
-
-> [!NOTE]
-> The `llm-evals` gem installs two commands: `evals` and `llm-evals`. Use `llm-evals` if another installed package already provides the `evals` command. The CLI and runner use [Dry.rb](https://dry-rb.org/), and the default model connection uses OpenRouter.
 >
-> The gem also supports OpenTelemetry and can send evaluation data to services such as [Braintrust](https://braintrust.ai) and [W&B Weave](https://wandb.ai/site/weave/). Some design ideas came from [Pydantic Evals](https://pydantic.dev/logfire/evals).
+> The `llm-evals` gem installs two commands: `evals` and `llm-evals`. Use `llm-evals` if another installed package already provides the `evals` command. The CLI and the runner uses [dry-rb](https://dry-rb.org/) and my extensions [dry-cli.tools](https://dry-cli.tools).
+>
+> In order run evals against pretty much any model, you need an account, with credits, on an [OpenRouter](https://openrouter.ai), and set the `OPENROUTER_API_KEY` to the key generated in OpenRouter.
 >
-> I originally developed the gem for [Qualified.At](https://qualified.at), a Ruby on Rails SaaS product.
+> If you also want to include [Typesafe's Jev model](https://typesafe.ai/blog/introducing-system-one-models-and-jev), this gem has a direct connector for Jev, so you'll need to set `TYPESAFE_API_KEY` in your environment as well.
+
+______________________________________________________________________
+
+The gem also supports OpenTelemetry and can send evaluation data to services such as [Braintrust](https://braintrust.ai) and [W&B Weave](https://wandb.ai/site/weave/). Some design ideas came from [Pydantic Evals](https://pydantic.dev/logfire/evals).
+
+> _I originally developed this gem for my fremium product [Qualified.At](https://qualified.at), a Ruby on Rails SaaS product._
 
-## Evals are unit tests for probabilistic software
+## Evals are unit tests for probabilistic software made by AI
 
 An eval checks whether a model meets the behavior you expect on a set of representative cases. Repeating the same cases lets you measure how prompts, parameters, and models affect the results. Your scorer decides whether those results are good; the eval makes the comparison consistent.
 
@@ -29,6 +34,8 @@ An eval checks whether a model meets the behavior you expect on a set of represe
 - [Independent Eval Research](https://www.braintrust.dev/evals) (Braintrust)
 - [How to Build an LLM Eval Pipeline in GitHub Actions](https://www.braintrust.dev/articles/llm-eval-pipeline-github-actions) (Braintrust)
 
+______________________________________________________________________
+
 ### A real-world example: Qualified.At
 
 > [!NOTE]
@@ -49,16 +56,50 @@ A tax customer might write: *"I'm married with three children, filing jointly, h
 When we built Qualified.At, it wasn't clear which provider or model would extract answers most accurately from unstructured text and map them to a JSON schema. Accuracy matters, but so do latency and cost. At 1,000 leads, a cost of $0.0002 per call adds up to $0.20; a cost of $0.50 per call adds up to $500.
 
 > [!IMPORTANT]
-> Which model captures the most answers, assigns them correctly, avoids hallucinations, and doesn't miss details? We want accuracy with low cost and low latency. If a user waits 30 seconds to get to the next question, that's a problem.
+> So, **which model captures the most answers**, **assigns them correctly**, **avoids hallucinations**, and **doesn't miss details**? We want accuracy with low cost and low latency. If a user waits 30 seconds to get to the next question, that's a problem. **That very question is why I built this gem.**
+
+______________________________________________________________________
+
+## Weighted Scoring
+
+Quite importantly, and often skipped over in the Eval design, is a concept of a **single weighted score**, that attaches fixed weights to various metrics.
+
+Evals are naturally a **multi-variate optimization problem**, something that has been solved in statistics and operations research many decades ago.
 
-That question is why I built this gem. The report below compares models on accuracy, completeness, latency, cost, and other metrics for this extraction task.
+> [!NOTE]
+> So if the **cost** is the most important factor, we give the cost the largest weight. If the **accuracy** is the most important, then it gets the highest weight. The weights configuration handles metrics that improve when they increase, as well as those that improve when they decrease.
+
+Here is a weight section definitions from the YAML configuration file `evals/.llm-evals.config.yml` (see below) that generated the HTML report which follows:
+
+![weights](docs/img/yaml-weights.avif)
 
-![LLM evaluation HTML report](docs/img/llm-evals-html-report.avif)
+## Running Evals
+
+Considering your `evals` folder is setup correctly, and the config file lives right under the `evals`, as `.llm-evals.config.yml`, you should just be able to run:
+
+```bash
+export OPENROUTER_API_KEY=....
+export TYPESAFE_API_KEY=...
+
+evals run
+```
 
 > [!NOTE]
-> *The screenshot is a little out of date. I later reversed the model order so the best scores appear at the top. In this run, Anthropic's `claude-haiku-5.5` was the best fit for my priorities: speed, cost, and accuracy.*
+> You need just those two because the evals go through ***OpenRouter***, and the only exception at the moment is Jev, which has a direct driver.
+
+The following is a screenshot of an HTML report from running evals in the [Qualified.At](https://qualified.at) Rails application, which compares various models on accuracy, completeness, latency, cost, and other metrics for the extraction task of the first question the lead answers.
+
+I set the *temperature* to 0.1 to get the least "creative" and most "precise" results, and for most models I am trying two sets of effort: "*high*" and "*low*":
+
+![HTML Report](docs/img/qualified-at-evals.avif)
 
-I had used evals before and wanted to understand the ideas better and build something reusable for the Ruby community. Since this is a CLI tool, I use Dry.rb utilities such as `dry-cli`. I also maintain three extensions at [dry-cli.tools](https://dry-cli.tools).
+### A Specific Experiment
+
+While the above shows the summarized results across 10 DSL examples, each with 4 prompts (so 40 total), the following screenshot from the same HTML report shows just one DSL use-case (tax collection) and how different the results are.
+
+I've highlighted in green the top models, their low pricing, and the overall winner for tax preparation — and not that in this case it's not the same model as the global winner. This is important.
+
+![taxes](docs/img/one-dataset-results.avif)
 
 ## Datasets
 
@@ -142,14 +183,20 @@ The recipes change into each suite's folder and set `RUBYLIB` for the config-onl
 
 ### Where to run from
 
-`llm-evals` finds a suite by searching **upward from the current directory** for `evals/eval_helper.rb` or `evals/.llm-evals.config.yml`. The repository root has neither, so running there fails:
+`llm-evals` finds a suite by searching **upward from the current directory** for `evals/eval_helper.rb` or `evals/.llm-evals.config.yml`. If that finds nothing, it searches **up to four directories below** the current directory for `.llm-evals.config.yml`, skipping hidden directories and `node_modules`, `vendor`, `tmp`, `log`, `pkg` and `coverage`. The shallowest file wins, then the first by name. Run from the repository root, it finds a fixture:
 
 ```text
 $ bundle exec exe/evals cases
-error: no suite is configured: register an adapter and a scorer in evals/eval_helper.rb, ...
+
+┌ Info ────────────────────────────────────────────────────────────────────────┐
+│ Evals configuration: config_only_suite                                       │
+│ File: spec/fixtures/config_only_suite/evals/.llm-evals.config.yml            │
+└──────────────────────────────────────────────────────────────────────────────┘
 ```
 
-Run the command from a suite's folder (the parent of its `evals/` directory), or pass the helper path with `--helper`.
+#### What is this?
+
+A file the search above ran, and found the eval suite four directories deep. Your shell stays where it was. That green box is the first thing a run prints, and it follows the help. When there is no file, the box is a yellow warning. `--no-search` turns the search off, and `-C`/`--config` skips it.
 
 ### The toy suite (`eval_helper.rb` style)
 
@@ -188,7 +235,7 @@ Files to look at:
 | --------------------------------------- | ---------------------------------------------------------- |
 | `evals/eval_helper.rb`                  | `LLM::Evals.configure`: suite, cases glob, adapter, scorer |
 | `lib/toy_suite/evals.rb`                | the adapter and scorer classes                             |
-| `evals/.llm-evals.config.yml`           | the lineup, including a `direct:` entry the suite refuses  |
+| `evals/.llm-evals.config.yml`           | the lineup, including a TypeSafe entry the suite refuses   |
 | `evals/datasets/`                       | the case layout on disk                                    |
 
 The warning `typesafe/jev-latest refused: needs typesafe; this suite produces prompt` is intentional: it shows a contender being left out rather than scored 0.
@@ -211,7 +258,7 @@ Use the toy suite as a template:
 
 1. Copy its `evals/` folder into your project.
 1. Replace the adapter and scorer with your own (see [Writing a suite](#writing-a-suite)).
-1. List real models under `vendors:` in `.llm-evals.config.yml` and drop the `fake:` entries.
+1. List real models under `openrouter:` in `.llm-evals.config.yml` and drop the `fake:` section.
 1. Add `gem "ruby_llm"` to your Gemfile and set `OPENROUTER_API_KEY`, in your shell or the suite's `.env` (see [Environment](#environment)).
 
 ## Install
@@ -257,43 +304,51 @@ require "bundler/setup"
 require "llm/evals"
 require "my_gem"
 
-class MyAdapter < LLM::Evals::Adapter
+class MyAdapter < LLM::Evals::Actions::Adapter
   # Once per dataset, outside the threads.
   def prepare(dataset) = @schema = MyGem.schema_for(dataset)
 
   def call(kase, model:)
-    completion = model.complete(
-      system: MyGem.system_prompt(@schema), user: kase.input, json: true)
-    
+    provider_response = model.complete(
+      system: MyGem.system_prompt(@schema), user: kase.input, json: true
+    )
+
     # The production parser, so scores reflect production.
-    output = MyGem.parse(completion.text)
-    LLM::Evals::Reply.new(output: output, raw: completion.text, usage: completion.usage)
+    output = MyGem.parse(provider_response.text)
+    LLM::Evals::Results::Reply.new(
+      output: output,
+      raw:    provider_response.text,
+      usage:  provider_response.usage
+    )
   rescue MyGem::NotJSON => e
     # The model's failure: scored 0.
-    LLM::Evals::Reply.new(error: e.message, raw: completion&.text)
+    LLM::Evals::Results::Reply.new(
+      error: e.message,
+      raw:   provider_response&.text
+    )
   end
 
   # What the scorer needs beyond the case.
-  def context(kase) = { schema: @schema }
+  def context(_kase) = { schema: @schema }
 
   # Offline, for `llm-evals verify`.
   def verify(dataset) = MyGem.schema_problems(dataset)
 end
 
-class MyScorer < LLM::Evals::Scorer
+class MyScorer < LLM::Evals::Scoring::Scorer
   def metrics
     [
-      LLM::Evals::Metric.new(
-        name: "field_accuracy",
+      LLM::Evals::Scoring::Metric.new(
+        name:        "field_accuracy",
         description: "Mean per-field score.",
-        primary: true
+        primary:     true
       ),
-      LLM::Evals::Metric.new(
-        name: "exact_match",
+      LLM::Evals::Scoring::Metric.new(
+        name:        "exact_match",
         description: "1 when every field is right."
       ),
-      LLM::Evals::Metric.new(
-        name: "restraint",
+      LLM::Evals::Scoring::Metric.new(
+        name:        "restraint",
         description: "Of the fields the text never answers, the share left null."
       )
     ]
@@ -304,22 +359,22 @@ class MyScorer < LLM::Evals::Scorer
     mean = scores.values.sum / scores.size
     misses = scores.reject { |_, v| v == 1.0 }.keys
 
-    LLM::Evals::Score.new(
-      value: mean,
-      metrics: {
+    LLM::Evals::Results::Score.new(
+      value:       mean,
+      metrics:     {
         "field_accuracy" => mean,
-        "exact_match" => scores.values.all?(1.0) ? 1.0 : 0.0,
+        "exact_match"    => scores.values.all?(1.0) ? 1.0 : 0.0,
         # nil: this case does not apply; it is skipped, not zeroed.
-        "restraint" => nil
+        "restraint"      => nil
       },
-      diagnostics: misses.to_h do |f| 
-        [f, { want: kase.expected[f], got: reply.output[f] }] 
+      diagnostics: misses.to_h do |f|
+        [f, { want: kase.expected[f], got: reply.output[f] }]
       end
     )
   end
 
   def verify(kase, context:)
-    LLM::Evals::SchemaValidator.new(context[:schema]).problems(kase.expected)
+    LLM::Evals::Scoring::SchemaValidator.new(context[:schema]).problems(kase.expected)
   end
 
   def summary(kase) = "#{kase.expected.count { |_, v| !v.nil? }}/#{kase.expected.size} answered"
@@ -333,6 +388,7 @@ LLM::Evals.configure do |c|
   # After the env files load, before any model is called.
   c.before_run { MyGem.reset_configuration! }
 end
+
 ```
 
 A complete, runnable suite lives in [`spec/fixtures/toy_suite/evals/`](spec/fixtures/toy_suite/evals/): two datasets, two variants, the gem's own `Testing::EchoAdapter` and a fake provider. It runs without a network.
@@ -399,42 +455,45 @@ results:
   autosave: true             # save every completed run as JSON (default)
   destination: ./evals/results/
   temp_dir: ./tmp            # temporary JSON when autosave is false
-  display: browser           # none, tui or browser
-
-                             # This section requires OPENROUTER_API_KEY
-vendors:                     # the OpenRouter list; every entry takes a prompt
-  anthropic:
-    - model: claude-fable-5
-    - model: claude-opus-5
-    - model: claude-sonnet-5.5
-    - model: claude-haiku-5.5
-    	# a list expands into one contender per value
-      effort: [default, low, high] 
-  openai:
-    - model: gpt-4.1-mini
-    - model: gpt-4o-mini
-    - model: gpt-6.1-sol
-  google:
-    - model: gemini-2.5-flash
-    - model: gemini-4-argon
-  meta-llama:
-    - model: llama-3.3-70b-instruct
-  typesafe:
-    - model: jev-router      # Jev on OpenRouter, a chat model
-  moonshotai:
-    - model: kimi-k3
-
-direct:                      # a vendor's own API, one request type each
-  typesafe:                  # request type "typesafe": noul, choice, score
-    - model: jev-latest      # and this requires TYPESAFE_API_KEY
+  display: both              # none, tui, browser or both (default)
+
+                             # A section named after a provider states the key it needs.
+openrouter:                  # requires OPENROUTER_API_KEY; every entry takes a prompt
+  rate_limit: { burst: 1000, per_minute: 500 }
+  vendors:
+    anthropic:
+      rate_limit: { burst: 10, per_minute: 50 }
+      models:
+        - model: claude-opus-5                  # the vendor's limit only
+        - model: claude-fable-5                 # both limits
+          rate_limit: { burst: 5, per_minute: 20 }
+        - model: claude-haiku-5.5
+          # a list expands into one contender per value
+          effort: [default, low, high]              
+    openai:                                     # no limit of its own
+      models:
+        - model: gpt-6.1-sol
+    typesafe:
+      models:
+        - model: jev-router                     # Jev on OpenRouter, a chat model
+
+typesafe:  # requires TYPESAFE_API_KEY; request type "typesafe": noul, choice, score
+  models:
+    - jev-latest  # a bare name is the same as `- model: jev-latest`
 ```
 
-**`vendors:`** Each entry becomes `<vendor>/<model>`, the OpenRouter id. `effort` and `temperature` take one value or a list, and lists multiply: `effort: [low, high]` with `temperature: [0.0, 0.7]` runs four experiments for that model. Anything left out falls back to `defaults`. Flags (`-e`, `-t`, `--max-tokens`, `-p`) beat both.
+**Provider sections.** Each top-level key with `vendors:` or `models:` names a provider. An entry under `vendors:` becomes `<vendor>/<model>`, the OpenRouter id. An entry under `models:` becomes the provider's prefix and the name, such as `typesafe/jev-latest`. The contender carries its provider, so a direct `typesafe/jev-latest` and the OpenRouter chat model `typesafe/jev-router` never get confused. An unknown provider or key is a configuration error naming the file. The old top-level `vendors:` and `direct:` are errors too; `evals upgrade-config` rewrites them, keeping your comments and leaving `<file>.bak`, and `-n` prints the result instead.
+
+**Settings.** `effort` and `temperature` take one value or a list, and lists multiply: `effort: [low, high]` with `temperature: [0.0, 0.7]` runs four experiments for that model. Anything left out falls back to `defaults`. Flags (`-e`, `-t`, `--max-tokens`, `-p`) beat both.
 
-**`direct:`** Each key names a provider the gem knows (today only `typesafe`). Entries take the same `{model:, effort:, temperature:, max_tokens:}` format, expand the same way, and render as `typesafe/jev-latest`. The contender carries its provider, so a direct `typesafe/jev-latest` and the OpenRouter chat model `typesafe/jev-router` never get confused. An unknown key is a configuration error naming the file and the known providers.
+**`rate_limit`.** It goes on a provider, a vendor or a model, and needs both numbers: the first `burst` calls of the run go at once, then `per_minute`, evenly spaced. A burst is spent once per run and does not refill. A model listed twice shares one limit, so setting it on both entries is an error. A call waits on every limit that matches it. Limits hold for one process across all `-j` and `-p`, and models given with `-m` obey them too. Time spent waiting is left out of `time`, `latency` and `p95`; `time` subtracts the wait averaged across the experiment's workers.
+
+**Retries.** RubyLLM's retries are off. Rate limits, overloads, server errors and dropped connections are retried up to 5 attempts in all, and a retry shuts that vendor for every thread, for as long as `Retry-After` says or for 1, 2, 4 then 8 seconds when it says nothing. A provider asking for more than 5 minutes is not retried; the case counts as an error.
 
 #### Weighted Score
 
+We already chatted about the weghted score above, but this is a deeper dive.
+
 > [!IMPORTANT]
 > This is one of those features you never think you'd need, until you realize your results are sortable by a single column and it does not incorporate all of the others! What to do!? The answer is right here. Create a new score, weighted because it gives more or less weight to various resulting scores, and supports both results that are better when they are smaller (latency) and the ones that are better when they are larger (match %, accuracy %), etc.s
 
@@ -452,16 +511,33 @@ weighted_score:
 
 `wscore = Σ(|weight| × goodness) / Σ|weight|`, from 0% to 100%, higher is better. Metrics count as they are, or as `1 - value` under a negative weight. When `errors` is weighted, errors come off the total: 20 points for one, 25 for two, 30 for three, and so on, down to 0%. A failed case has no metric scores, so the other weighted metrics do not reflect that failure. A contender with no errors earns the full positive contribution from the `errors` weight. `cost` and `time` go on a log scale between the lowest and highest in the table, so each doubling costs the same, however expensive the priciest model is. That makes a weighted score comparable only within one table.
 
-You can weight `score`, `errors`, `time`, `cost` and your scorer's metrics. Anything else, tokens included, is refused before a model is called. A contender missing a weighted value shows `-`, and one with errors cannot be named best. Without `weighted_score`, nothing changes.
+`evals weights` prints each column's direction and real share of the score: `↓` in bold blue where low is good, `↑` in bold yellow where high is good. A scorer metric the weights leave out is listed in red with no values. Weights need not add up to 1.
+
+You can weight `score`, `errors`, `time`, `cost`, `latency`, `p95` and your scorer's metrics. `latency` and `p95` are the average and 95th percentile seconds of the calls that worked (across datasets, a contender's `p95` is the average of each dataset's, an approximation good enough to rank by), while `time` is how long the experiment took, waiting included. Anything else, tokens included, is refused before a model is called. A contender missing a weighted value shows `-`, and one with errors cannot be named best. Without `weighted_score`, nothing changes.
+
+#### Aborting a failing run
+
+```yaml
+errors:
+  abort_after:
+    total: 1000   # required
+    percent: 60   # optional: also needs 60% of finished calls to have failed
+```
+
+Both limits must be met. A case counts as an error when the call raises or the model's reply is an error, such as one that is not JSON. Calls in flight finish, nothing new starts, the report is written from what finished, and the command exits non-zero. Without `errors`, a run never aborts.
+
+#### When every call fails
+
+With no configuration, 100 failed calls in a row pause the run: no new calls start, and it prints the error messages and which models, datasets and cases are affected, or that all of them are. It then asks whether to keep running. If you carry on, another 100 failures in a row ask again. If you stop, the partial results are reported and the command exits 1. `-y` / `--yes` prints the same details and keeps running. When stdin is not a terminal and `--yes` is absent, the run stops, since there is nobody to ask.
 
 #### Request types
 
 `request_types` is required. It identifies the kind of request the suite's adapter sends. The types belong to the providers and show in `llm-evals --help`; a suite only names them:
 
-| Type       | Provider                                 | Meaning                                              |
-| ---------- | ---------------------------------------- | ---------------------------------------------------- |
-| `prompt`   | OpenRouter (everything under `vendors:`) | a chat completion: system and user text in, text out |
-| `typesafe` | `direct: typesafe:`                      | typed questions: noul, choice, score                 |
+| Type       | Provider                                    | Meaning                                              |
+| ---------- | ------------------------------------------- | ---------------------------------------------------- |
+| `prompt`   | OpenRouter (everything under `openrouter:`) | a chat completion: system and user text in, text out |
+| `typesafe` | `typesafe:`                                 | typed questions: noul, choice, score                 |
 
 A contender runs only when its provider's type is in the suite's list. Otherwise it is **refused**:
 
@@ -503,7 +579,7 @@ evals completion zsh                     # shell completion (bash too)
 
 Flags shared by `run` and `compare`: `-m/--models`, `-d/--dataset` (every dataset when omitted), `-c/--case`, `-e/--effort`, `-t/--temperature`, `--max-tokens`, `-p/--parallelism`, `-j/--concurrency`, `-C/--config FILE`, `--project NAME`, `--tracker NAME`, `--local`, `--no-open`, `-n/--dry-run`, `-o/--output [FILE]`, `--format tui|html|json`, `--totals overall|variant|both`, `--display none|tui|browser`, `--autosave/--no-autosave`, `--destination DIR` and `--temp-dir DIR`. `--totals` controls the TUI totals tables: the default `overall` compares contenders across all cases, `variant` compares each case folder (such as `complete` or `vague`) across datasets, and `both` prints both views. Completed runs are saved as timestamped JSON by default, independently of the selected display. Browser and TUI reports are rendered from that JSON; `--no-autosave` uses a temporary JSON snapshot under `temp_dir`. Use `llm-evals render results.json` to regenerate the HTML report without making model calls again. `-l` and `-L` are reserved for logging and not yet implemented.
 
-**Reading the output.** `results.display` selects no report, a terminal leaderboard, or a browser report. The TUI has one column for each metric declared by the scorer and ranks by the primary metric. It also shows a glossary built from each metric's `description`. Browser reports show global and dataset standings, sortable tables, case inputs, expected answers, scores, and diagnostics. Provider names and model IDs use separate colors. Headline standings show effort and temperature alongside each model. Every JSON snapshot records both parsed CLI options and effective run settings, including expanded model settings and selected cases. `render` turns a saved snapshot into HTML later. Progress bars and warnings go to STDERR. Output from libraries is saved to `log/<command>.warnings.<stamp>.log` so it doesn't garble the terminal display.
+**Reading the output.** `results.display` selects no report, a terminal leaderboard, a browser report, or, by default, both: the HTML report is built from the run's JSON and opened in the browser. `--display tui` skips the page, and `--no-open` writes it without opening it. The TUI has one column for each metric declared by the scorer and ranks by the primary metric. It also shows a glossary built from each metric's `description`. Browser reports show global and dataset standings, sortable tables, case inputs, expected answers, scores, and diagnostics. Provider names and model IDs use separate colors. Headline standings show effort and temperature alongside each model. Every JSON snapshot records both parsed CLI options and effective run settings, including expanded model settings and selected cases. `render` turns a saved snapshot into HTML later. Progress bars and warnings go to STDERR. Output from libraries is saved to `log/<command>.warnings.<stamp>.log` so it doesn't garble the terminal display.
 
 ## Trackers
 
@@ -537,7 +613,7 @@ Logfire has no Ruby gem of its own. It is an OTLP backend for the standard OpenT
 A tracker is a class with three callbacks. Register it with `c.tracker` in the helper (or in a plugin gem loaded through `require:`), then choose it by name:
 
 ```ruby
-class Weave < LLM::Evals::Tracker
+class Weave < LLM::Evals::Trackers::Base
   # Selects it when set and nothing else is chosen.
   def self.env_key = "WANDB_API_KEY"
 
@@ -582,8 +658,8 @@ LLM::Evals.configure { |c| c.tracker Weave }
 
 | Variable                      | Used for                                                  |
 | ----------------------------- | --------------------------------------------------------- |
-| `OPENROUTER_API_KEY`          | required for any `vendors:` model (OpenRouter)            |
-| `TYPESAFE_API_KEY`            | required for a `direct: typesafe:` model                  |
+| `OPENROUTER_API_KEY`          | required for any `openrouter:` model                      |
+| `TYPESAFE_API_KEY`            | required for a `typesafe:` model                          |
 | `BRAINTRUST_API_KEY`          | the Braintrust tracker; selects it when nothing else does |
 | `BRAINTRUST_PROJECT_ID`       | Braintrust project UUID; beats the lineup's `project`     |
 | `LOGFIRE_TOKEN`               | the Logfire tracker (a write token); selects it likewise  |
@@ -598,9 +674,9 @@ The suite loads files listed in `env_files`, without overriding variables alread
 
 A provider wraps an API. It declares the request types it supports, required keys, available models, and a `complete` method.
 
-- A provider that serves `vendors:` ids declares which ones with `prefix`.
-- A `direct?` provider is reached only through a `direct:` key matching its `name`.
-- Register it with `c.provider`. Providers are checked in reverse registration order for a `vendors:` ID. OpenRouter claims every ID, so it is always checked last.
+- A provider declares the ids it serves with `prefix`, and its section in the lineup carries its `name`.
+- A `direct?` provider is reached only through its own section.
+- Register it with `c.provider`. Providers are checked in reverse registration order for an ID outside a direct provider's section. OpenRouter claims every ID, so it is always checked last.
 
 ```ruby
 class Ollama < LLM::Evals::Providers::Base
@@ -640,7 +716,7 @@ class Ollama < LLM::Evals::Providers::Base
       Net::HTTP.post(URI("#{HOST}/api/generate"), JSON.generate(body)).body
     )
 
-    LLM::Evals::Completion.new(
+    LLM::Evals::Results::ProviderResponse.new(
       text: response.fetch("response"),
       usage: {
         input_tokens: response["prompt_eval_count"],
@@ -657,7 +733,7 @@ LLM::Evals.configure { |c| c.provider Ollama.new }
 
 `model.effort`, `model.temperature` and `model.max_tokens` carry the contender's settings; nil means "send nothing". A transport failure should raise: the run counts it as an error, not as a wrong answer.
 
-`Providers::TypeSafe` is a direct provider (`direct: typesafe:`, request type `typesafe`). It ships with TypeSafe's model list, key and pricing, which is enough for `verify` and the cost column. Its `complete` is not implemented yet and raises "no typesafe consumer yet": the typed request shape will be defined by the first suite that declares `request_types: [typesafe]`.
+`Providers::TypeSafe` is a direct provider (`typesafe:`, request type `typesafe`). It ships with TypeSafe's model list, key and pricing, which is enough for `verify` and the cost column. Its `complete` is not implemented yet and raises "no typesafe consumer yet": the typed request shape will be defined by the first suite that declares `request_types: [typesafe]`.
 
 ## Public API
 
@@ -667,7 +743,7 @@ module LLM::Evals
     :id, :dataset, :variant, :input, :expected, :path, :metadata)
   Usage      = Data.define(
     :input_tokens, :output_tokens, :cost_usd)
-  Completion = Data.define(
+  ProviderResponse = Data.define(
     :text, :usage, :raw)
   # error: the model failed, scored 0. A raise: transport failure.
   Reply      = Data.define(
@@ -680,13 +756,13 @@ module LLM::Evals
     :value, :metrics, :outcomes, :diagnostics)
   RunResult  = Data.define(
     :experiment, :case, :reply, :score, :duration, :error)
-  # provider: nil for vendors:, a name for direct:.
+  # provider: nil, or a direct provider's name.
   Contender  = Data.define(
     :model, :effort, :temperature, :max_tokens, :provider)
 
   class Model
     attr_reader :id, :provider, :settings
-    def complete(system: nil, user:, json: true) = Completion
+    def complete(system: nil, user:, json: true) = ProviderResponse
   end
 
   class Adapter

data/exe/llm-evals

@@ -4,11 +4,66 @@
 # No Bundler here: a Gemfile in the user's directory must not decide which
 # gems load. The suite's own eval_helper.rb may `require "bundler/setup"`.
 $LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
-require "llm/evals"
 
-puts "\e[1;31mERROR: \e[0;33mThis gem comes with an executable named 'evals'"
-puts "       You invoked 'llm-evals'. Please use 'evals' to use the gems's CLI.\n\e[0m"
+# warn prints nothing under -W0, and the welcome is only ever shown once.
+# rubocop:disable Style/StderrPuts
+require "fileutils"
+require "yaml"
+require "date"
+
+module LLM
+  module Evals
+    # The key under which the semaphore file records the first run.
+    WARNING_HASH_KEY = "warning_shown_on"
+    # Where the semaphore file lives.
+    SEMAPHORE_DIR    = "#{Dir.home}/.local/share/llm-evals".freeze
+    # Remembers that the welcome was shown.
+    SEMAPHORE_FILE   = "#{SEMAPHORE_DIR}/warning-semaphore.yml".freeze
+
+    # What the semaphore file held.
+    SEMAPHORE_CONFIG = if File.size?(SEMAPHORE_FILE)
+                         YAML.safe_load_file(SEMAPHORE_FILE, aliases: true)
+                       else
+                         {}
+                       end
+
+    unless SEMAPHORE_CONFIG.is_a?(Hash) && SEMAPHORE_CONFIG[WARNING_HASH_KEY]
+      FileUtils.mkdir_p(SEMAPHORE_DIR)
+      File.write(SEMAPHORE_FILE, { WARNING_HASH_KEY => DateTime.now.to_s }.to_yaml)
+
+      $stderr.puts "\e[0;33mWelcome to the llm-evals CLI!\n\n"
 
-puts "Thanks for using llm-evals!"
+      $stderr.puts "\e[1;31mNOTE:  \e[0;33mThis gem comes with an executable named \e[1;32mevals\e[0;33m.\n"
+      $stderr.puts "       You invoked 'llm-evals'. While we prefer that you use 'evals' for the CLI,"
+      $stderr.puts "       we understand that sometimes commands such as 'evals' may clash with others."
+      $stderr.puts "       For this reason you can use \e[1;32mllm-evals\e[0;33m if you prefer moving forward.\e[0m"
+      $stderr.puts
+      $stderr.puts "This message won't be shown again, and we thank you for using llm-evals!"
 
-exit 0
+      require "io/console"
+
+      $stdin.tty? && $stdin.raw do
+        $stderr.print "      Press any key to continue..."
+        waited = 0
+        delta = 0.1
+        loop do
+          if $stdin.wait_readable(0)
+            $stdin.read_nonblock(1)
+            break
+          end
+          sleep delta
+          waited += delta
+          if waited > 20
+            $stderr.puts "\n      No key pressed after 20 seconds, continuing..."
+            break
+          end
+        end
+      end
+    end
+  end
+end
+
+# rubocop:enable Style/StderrPuts
+
+require "llm/evals"
+LLM::Evals::Launcher.new(ARGV).execute!

data/lib/llm/evals/actions/adapter.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module LLM
+  module Evals
+    module Actions
+      # What a suite implements to run one case: everything the model sees, and
+      # everything done to its answer before it is scored. An adapter never
+      # scores; that is the {Scoring::Scorer}'s, and keeping them apart is what lets one
+      # suite compare prompts and another compare models with the same runner.
+      #
+      # @abstract Subclass and override {#call}.
+      class Adapter
+        # Called once per dataset before its cases run; load what the dataset
+        # shares (a schema, a flow definition) here, outside the threads.
+        #
+        # @param _dataset [String] the dataset's name
+        # @return [void]
+        def prepare(_dataset) = nil
+
+        # Runs one case against one model.
+        #
+        # @param kase [Datasets::Case] the case to run
+        # @param model [Models::Model] the model at the setting under test
+        # @return [Results::Reply] with `error:` set when the model failed to answer; raise for a transport failure
+        # @raise [NotImplementedError] until a subclass overrides it
+        def call(kase, model:)
+          raise NotImplementedError, "#{self.class}#call(kase, model:) must return an LLM::Evals::Results::Reply"
+        end
+
+        # What the scorer needs to know about a case beyond the case itself: a
+        # schema, field types, the prompt. Passed to {Scoring::Scorer#call} as `context:`.
+        #
+        # @param _kase [Datasets::Case] the case about to be scored
+        # @return [Hash]
+        def context(_kase) = {}
+
+        # Offline checks of a dataset's shared files. `llm-evals verify` prints them.
+        #
+        # @param _dataset [String] the dataset's name
+        # @return [Array<String>] problems; empty when the dataset is sound
+        def verify(_dataset) = []
+
+        # Recorded with each Braintrust experiment, e.g. which prompt version ran.
+        #
+        # @return [Hash]
+        def metadata = {}
+      end
+    end
+  end
+end