rtr-bench 0.1.3 → 0.1.5

package/README.md

@@ -1,14 +1,14 @@
 # RTR-bench
 
-**RTR-bench** is a benchmark that scores language models on cultural fluency, built from real human data. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed and the y-axis is Uniting vs Dividing. Across 174 closed windows there have been almost 40k individual submissions. Each one is a unique, real human data point.
+**RTR-bench** tests whether language models can predict how a crowd interprets an ambiguous prompt. It investigates one aspect of cultural fluency: anticipating how people assign meaning to words, people, institutions, and ideas. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project where participants respond to a daily prompt by placing a coordinate on a 2×2 grid: Honest vs Performed on one axis, Uniting vs Dividing on the other. The dataset used here contains 39,828 submissions across 165 closed windows.
 
-Most labeled datasets are made by asking annotators to apply a rubric. This one was not. Nobody was given a definition of honest or performed, nobody was trained, and there was no correct answer to converge on. People arrived, interpreted a prompt, placed a submission, and left. What makes the data unusual is that the crowd's answer is a distribution, rather than a label. Not "people say X", but "31% of people said this, 44% said that, and the rest said something else."
+These submissions were collected through participation in a generative art project. Participants were not trained to apply an annotation rubric or converge on a correct answer. Each placement recorded an interpretation, and together those placements formed a distribution. That distribution preserves both agreement and disagreement: how strongly one reading prevailed, which alternatives attracted people, and where the crowd split.
 
-That distribution is what makes it useful for evaluating a model. Every window has a dominant quadrant, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize: the axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret.
+Commonly cited benchmarks such as [HLE](https://lastexam.ai/) and [SWE-bench](https://www.swebench.com/) evaluate expert answers or software changes against defined criteria for success. RTR-bench evaluates a model against an observed social response. A crowd’s interpretation can be contradictory, unfamiliar, or difficult to anticipate, and accurately predicting it requires distinguishing what a prompt means to that audience from the reading the model would otherwise produce.
 
-Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), measure whether a model can produce a correct answer: multiple choice, math, and code. RTR-bench measures a different subject. There is no answer key in this data, only how people actually interpreted an ambiguous prompt, so a model is scored against the crowd rather than against a key. That is what makes it a test of whether a model can read the room.
+For each prompt, the model answers two separate questions: which of the four quadrants received the most human submissions, and which quadrant represents its own interpretation. Both responses are compared with the recorded crowd response. The headline score measures how often the model correctly predicts the crowd’s dominant quadrant. This gives a straightforward measure of agreement, although it captures only part of the underlying distribution.
 
-**RTR-bench** is not a knowledge test, and that is deliberate. It is a prediction task. Given a prompt, a model has to decide how the crowd interpreted it, and then decide how it would interpret it itself. Both answers are scored against the real submissions.
+The score is one way to compare models; their patterns of agreement and error are another. Which interpretations do models consistently anticipate? Where do different models make the same mistake? How does asking a model to predict the crowd change its answer? RTR-bench uses these questions to examine what it means for a model to read the room - with the room here defined by the participants in Mixed Messages.
 
 ## How it works
 
@@ -29,48 +29,33 @@ Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [H
 
 ## Quickstart
 
-Scoring needs no database access, just an OpenRouter API key.
+You need Node.js 18 or newer and an OpenRouter API key with available credits. The published package includes the benchmark dataset; no repository setup is required.
 
-### Test a model
-
-Node 18 or newer. No clone, no install.
+Create a folder for your results, add your API key, and run a model:
 
 ```bash
 mkdir rtr-run && cd rtr-run
 echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
-
-# see ids and prices. costs nothing
-npx --yes rtr-bench catalog --limit 20
-
-# run one model, then score it
-npx --yes rtr-bench z-ai/glm-5.3-flash
+npx --yes rtr-bench@latest z-ai/glm-5.3-flash
 ```
 
-`--yes` skips npx's install prompt. A run makes 1,320 calls, takes about four minutes on a cheap model, costs a few cents, and writes `results/` in the folder you created. Add `--concurrency 8` to go faster.
+Replace `sk-or-your-key` with your OpenRouter API key. To evaluate another model, replace `z-ai/glm-5.3-flash` with its OpenRouter model ID. `npx` downloads and caches the package automatically; `--yes` skips its install prompt.
 
-By default OpenRouter may route each call to any host that serves the model, and hosts differ in quantisation. List them, with price and precision:
+The command runs the benchmark, prints the score, and saves responses, scores, and a report in `results/`. A full run schedules 1,320 calls: two questions, each asked in four option orders, across 165 prompts. Retries can add calls. Time and cost depend on the model and provider.
+
+To list available model IDs and token prices:
 
 ```bash
-npx --yes rtr-bench providers z-ai/glm-5.3-flash
+npx --yes rtr-bench@latest catalog --limit 20
 ```
 
-Pin one with `--provider Wafer`. A pinned run refuses to fall back, and the summary names the host that served the calls.
-
-### Work on the code
-
-The repository runs the TypeScript sources directly, so this path needs Node 22.18 or newer.
+To try the command on just five prompts before running the full benchmark:
 
 ```bash
-git clone https://github.com/1999labs/RTR-bench
-cd RTR-bench
-npm install
-echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
-node src/cli.ts catalog --limit 20
-node src/cli.ts run --models z-ai/glm-5.3-flash --concurrency 8
-node src/cli.ts report
+npx --yes rtr-bench@latest z-ai/glm-5.3-flash --limit 5
 ```
 
-Rebuilding the dataset needs the project's database connection, so `npm run export` is included for inspection only.
+This smaller run checks your setup; its score is not comparable with the full benchmark results. Rerunning without `--limit 5` reuses completed responses and runs the remaining prompts.
 
 ## Results
 
@@ -90,11 +75,49 @@ Rebuilding the dataset needs the project's database connection, so `npm run expo
 
 ![RTR-bench: score against cost](results/leaderboard-scatter.png)
 
-Eleven models were run against the same 165 windows, and all of them landed between 32.1 and 46.1 out of 100. The floors sit low: guessing among four readings scores 25, and always naming the crowd's most common quadrant scores 31.5. Nearly every model clears both, and none of them comes close to reading the room. The order at the top is not a ranking either. At 165 windows the confidence intervals are about seven points wide, so gemini-3.8-flash at 46.1 and claude-haiku-5.5 at 43.0 are the same result measured twice, once for 78 cents and once for 17.
+Eleven models were tested against the same 165 windows, representing 39,828 human submissions. Scores ranged from 32.1 to 46.1, compared with a 31.5 baseline for always choosing the most common quadrant and 33.9 for a simple lexical heuristic. Gemini scored highest, correctly predicting 76 of the 165 windows. Its bootstrapped 95% confidence interval was 38.2–53.3%, however, so small differences between models should be treated cautiously.
+
+Perhaps the value of the benchmark lies more in exposing model behavior than in actually ranking models. As described above, benchmarks such as HLE and SWE-bench evaluate whether a model can answer an expert question or resolve a software issue against defined criteria. RTR-bench asks whether a model can anticipate how a particular crowd interpreted something ambiguous. An incorrect prediction does not establish that the model’s interpretation is unreasonable. It shows where that interpretation departs from the room.
+
+The clearest shared departure was Honest × Dividing. This was the crowd’s dominant quadrant in 39 of 165 windows, or 23.6%. Across the models’ 1,591 final predictions without a tie between option orders, however, they selected it only 160 times, or 10.1%. Including tied predictions as misses, models correctly identified just 15.9% of the windows dominated by Honest × Dividing, compared with 60.7% for Honest × Uniting. The models were much better at anticipating authentic unity than anticipating division that participants interpreted as honest.
+
+Donald Trump and Elon Musk illustrate both the pattern and the limits of the current score. None of the eleven models correctly predicted the dominant quadrant for either Trump’s window or either of Musk’s two windows. For Trump, 634 of 738 submissions—85.9%—landed in Honest × Dividing. In the first Musk window, the same quadrant won with only 278 of 720 submissions, or 38.6%, narrowly ahead of Performed × Dividing at 34.7%. Both count as an incorrect prediction, although one misses an overwhelming consensus and the other misses a closely divided crowd. At the other end, all eleven models correctly predicted Handwritten Notes, The CIA, and Altruism.
+
+These results return to what makes the source data unusual: the crowd’s answer is a distribution. The current score reduces that distribution to its largest quadrant, making comparison straightforward while leaving much of the disagreement unmeasured. My main takeaway is that these models share a difficulty anticipating when this audience reads division as honest. Establishing how broadly that pattern holds will require more audiences, prompt variants, and future windows. Next, I would like to test more frontier models, report performance separately where the crowd was decisive, and evaluate predictions of the full distribution alongside the dominant-quadrant score. Conviction and polarization could then help describe the kinds of disagreement a model understands, and the kinds it consistently misreads.
+
+## Optional usage
+
+OpenRouter may route calls to different providers serving the same model. To list providers, token prices, and numerical precision:
+
+```bash
+npx --yes rtr-bench@latest providers z-ai/glm-5.3-flash
+```
+
+To use a specific provider, add `--provider` with a name from that list:
+
+```bash
+npx --yes rtr-bench@latest z-ai/glm-5.3-flash --provider Wafer
+```
+
+A pinned run refuses to fall back to another provider, and the summary names the providers that served the calls. Provider availability can change.
+
+The default is six concurrent calls. To allow up to eight calls at once, add `--concurrency 8`. This may shorten the run if the provider's rate limits allow it.
+
+## Development
+
+To modify the benchmark or run the repository's source code, use the setup below. These commands run TypeScript directly and require Node.js 22.18 or newer.
+
+```bash
+git clone https://github.com/1999labs/RTR-bench
+cd RTR-bench
+npm install
+echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
+node src/cli.ts z-ai/glm-5.3-flash
+```
 
-The models are more alike in how they fail than in how they score. Across 1,591 answers they name honest+uniting 67.7 percent of the time and honest+dividing 18.2 percent of the time, while the crowd chose honest+dividing in 39 of the 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candour. Six of the seven windows that no model got right are honest+dividing, among them Donald Trump and Elon Musk, where the crowd agreed with itself 86 and 70 percent. The 78-cent model and the two-cent one miss the same quadrant.
+Replace `sk-or-your-key` with your OpenRouter API key. The run automatically generates scores and a report in `results/`.
 
-Cost barely enters into it. Two cents buys 32.7, seventeen cents buys 43.0, seventy-eight cents buys 46.1, and the whole table cost $2.56 to produce. What the numbers describe is a set of models that can tell whether a word feels warm or hostile, and cannot tell what this particular audience would do with it. That gap is the thing the benchmark was built to measure, and eleven models later it is still open.
+Rebuilding the dataset requires the project's database connection and the configuration in `scripts/export.mjs`. The exported dataset is already included, so running the benchmark does not require database access.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rtr-bench",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "Can a language model predict how a crowd reads a word? A cultural fluency benchmark built from Mixed Messages.",
   "type": "module",
   "license": "MIT",