rtr-bench 0.1.2 → 0.1.4

package/README.md

@@ -1,14 +1,14 @@
 # RTR-bench
 
-[Mixed Messages](https://mixedmessages.fyi) is a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed, and the y-axis is Uniting vs Dividing. Across 175 windows, there have been almost 40k individual submissions. Each is a unique human data point. 
+**RTR-bench** is a benchmark that scores language models on cultural fluency, built from real human data. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed and the y-axis is Uniting vs Dividing. Across 174 closed windows there have been almost 40k individual submissions. Each one is a unique, real human data point.
 
 Most labeled datasets are made by asking annotators to apply a rubric. This one was not. Nobody was given a definition of honest or performed, nobody was trained, and there was no correct answer to converge on. People arrived, interpreted a prompt, placed a submission, and left. What makes the data unusual is that the crowd's answer is a distribution, rather than a label. Not "people say X", but "31% of people said this, 44% said that, and the rest said something else."
 
-That distribution is what makes it useful for evaluating a model. Every window has a majority reading and a real minority, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize. The axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret. 
+That distribution is what makes it useful for evaluating a model. Every window has a dominant quadrant, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize: the axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret.
 
-Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), are exam-like: multiple choice, math, and code. Models have been optimized for their shape. But they are very mechanical, and their results are not indicative of a model's ability to "read the room." If the industry ambition is a model that surpasses human intelligence, the comparison has to include the things humans are good at, like having cultural fluency, which models generally lack. That's why Mixed Messages provides a unique and differentiated dataset to benchmark against.
+Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), measure whether a model can produce a correct answer: multiple choice, math, and code. RTR-bench measures a different subject. There is no answer key in this data, only how people actually interpreted an ambiguous prompt, so a model is scored against the crowd rather than against a key. That is what makes it a test of whether a model can read the room.
 
-**RTR-bench**, or read-the-room, is not a knowledge test. That is deliberate. It is an attempt at a different kind of benchmark, built from a creative act rather than an exam structure. Cultural fluency resists the traditional benchmark shape, which is why this one is a prediction task that measures whether a model can anticipate how people would interpret a prompt, and whether it would have interpreted that prompt the same way, both against real human data.
+**RTR-bench** is not a knowledge test, and that is deliberate. It is a prediction task. Given a prompt, a model has to decide how the crowd interpreted it, and then decide how it would interpret it itself. Both answers are scored against the real submissions.
 
 ## How it works
 
@@ -29,28 +29,45 @@ Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [H
 
 ## Quickstart
 
-Scoring needs no database access.
+Scoring needs no database access, just an OpenRouter API key.
 
-```bash
-git clone https://github.com/1999labs/RTR-bench
-cd RTR-bench
-npm install
+### Test a model
 
-# keep the key in a file, not in shell history
+Node 18 or newer. No clone, no install.
+
+```bash
+mkdir rtr-run && cd rtr-run
 echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
 
-# list ids and prices
-node src/cli.ts catalog --limit 20
+# see ids and prices. costs nothing
+npx --yes rtr-bench catalog --limit 20
 
-# run a model, then score everything on disk
-node src/cli.ts run --models z-ai/glm-5.3-flash --concurrency 8
-node src/cli.ts report
+# run one model, then score it
+npx --yes rtr-bench z-ai/glm-5.3-flash
 ```
 
-Published on npm, the same run is one command with no clone:
+`--yes` skips npx's install prompt. A run makes 1,320 calls, takes about four minutes on a cheap model, costs a few cents, and writes `results/` in the folder you created. Add `--concurrency 8` to go faster.
+
+By default OpenRouter may route each call to any host that serves the model, and hosts differ in quantisation. List them, with price and precision:
 
 ```bash
-npx rtr-bench z-ai/glm-5.3-flash
+npx --yes rtr-bench providers z-ai/glm-5.3-flash
+```
+
+Pin one with `--provider Wafer`. A pinned run refuses to fall back, and the summary names the host that served the calls.
+
+### Work on the code
+
+The repository runs the TypeScript sources directly, so this path needs Node 22.18 or newer.
+
+```bash
+git clone https://github.com/1999labs/RTR-bench
+cd RTR-bench
+npm install
+echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
+node src/cli.ts catalog --limit 20
+node src/cli.ts run --models z-ai/glm-5.3-flash --concurrency 8
+node src/cli.ts report
 ```
 
 Rebuilding the dataset needs the project's database connection, so `npm run export` is included for inspection only.
@@ -59,16 +76,25 @@ Rebuilding the dataset needs the project's database connection, so `npm run expo
 
 | Model | Provider | Score | Cost |
 |---|---|---|---|
-| z-ai/glm-5.3-flash | TBD | TBD | TBD |
-| z-ai/glm-5.3 | TBD | TBD | TBD |
-| minimax/minimax-m3.1-flash | TBD | TBD | TBD |
-| deepseek/deepseek-v4.1-flash-1017 | TBD | TBD | TBD |
-| tencent/hy4 | TBD | TBD | TBD |
-| google/gemini-3.8-flash | TBD | TBD | TBD |
-| google/gemini-4-carbon | TBD | TBD | TBD |
-| moonshotai/kimi-k3.1-pro | TBD | TBD | TBD |
-| x-ai/grok-4.7 | TBD | TBD | TBD |
-| qwen/qwen3.8-max-0902 | TBD | TBD | TBD |
+| z-ai/glm-5.3-flash | Tenstorrent (fp8) | 41.2 | $0.0281 |
+| z-ai/glm-5.3 | Tenstorrent (fp8) | 40.0 | $0.2828 |
+| minimax/minimax-m3.1-flash | GMICloud (fp8) | 32.1 | $0.2444 |
+| xiaomi/mimo-v2.6-flash | GMICloud (bf16) | 40.6 | $0.1216 |
+| deepseek/deepseek-v4.1-flash-1017 | Morph (fp8) | 40.0 | $0.3414 |
+| tencent/hy3 | Tencent (fp8) | 40.0 | $0.2665 |
+| google/gemini-3.8-flash | Google AI Studio | 46.1 | $0.7790 |
+| openai/gpt-6-luna | OpenAI | 37.0 | $0.1329 |
+| anthropic/claude-haiku-5.5 | Claude Platform on AWS | 43.0 | $0.1696 |
+| thinkingmachines/inkling-small | DeepInfra (fp8) | 37.0 | $0.1733 |
+| openai/gpt-oss-120b | DekaLLM (bf16) | 32.7 | $0.0205 |
+
+![RTR-bench: score against cost](results/leaderboard-scatter.png)
+
+Eleven models were tested against the same 165 windows. Scores concentrated in the high 30s and low 40s. Models were more alike in how they fail than in how they score. Across 1,591 answers, they decided on Honest x Uniting 67.7% of the time and Honest x Dividing 18.2% of the time, while the dominant quadrant was Honest x Dividing in 39 of those 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candor. 
+
+There were seven prompts that no model got right. They were Donald Trump, Elon Musk, Las Vegas, Barack Obama, Reputation, Dreams, and Honor. The dominant quadrant for each of them besides Dreams was Honest x Dividing. There were three prompts that every model got right. They were Handwritten Notes, The CIA, and Altruism. 
+
+My main takeaway is that the same blind spot shows up in every model, and it is not a vocabulary problem. Ask it to pick the dominant quadrant for Trump or Musk and the model can easily choose dividing instead of uniting, but has a hard time determining honest or performed. That is why Honest X Dividing is the quadrant the models got wrong the most. Next, I would like to score more closed frontier models, score only the windows where the crowd was very decisive, and somehow factor into the benchmark the conviction and polarization scores from each window. 
 
 ## License
 

package/dist/cli.js

@@ -120,7 +120,8 @@ function readModel(dir) {
         order: rec.order,
         task,
         parsed: rec.parsed ?? null,
-        cost: rec.usage?.cost ?? rec.usage?.total_cost ?? 0
+        cost: rec.usage?.cost ?? rec.usage?.total_cost ?? 0,
+        provider: rec.provider ?? null
       });
     }
   }
@@ -236,6 +237,10 @@ function scoreModel(model, windows) {
   const mean = (xs) => xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : NaN;
   const predictAcc = mean(predHits);
   const readAgree = mean(readHits);
+  const providers = {};
+  for (const c of calls) {
+    if (c.provider) providers[c.provider] = (providers[c.provider] ?? 0) + 1;
+  }
   return {
     model: display,
     windows: predHits.length,
@@ -247,7 +252,8 @@ function scoreModel(model, windows) {
     gap: readAgree - predictAcc,
     instability: pred.length ? pred.filter((a) => a.unstable).length / pred.length : NaN,
     parseFailures: calls.filter((c) => !c.parsed).length,
-    costUsd: calls.reduce((a, c) => a + c.cost, 0)
+    costUsd: calls.reduce((a, c) => a + c.cost, 0),
+    providers
   };
 }
 function majorityBaseline(windows) {
@@ -313,19 +319,30 @@ import fs3 from "node:fs";
 import path2 from "node:path";
 var API = "https://openrouter.ai/api/v1/chat/completions";
 var apiKey = () => process.env.OPENROUTER_API_KEY ?? "";
+var nextSlot = 0;
+async function pace(rpm) {
+  if (!rpm) return;
+  const gap = 6e4 / rpm;
+  const now = Date.now();
+  const start = Math.max(now, nextSlot);
+  nextSlot = start + gap;
+  const wait = start - now;
+  if (wait > 0) await new Promise((r) => setTimeout(r, wait));
+}
 function slug(model) {
   return model.replace(/[^a-zA-Z0-9]+/g, "-").toLowerCase();
 }
-async function call(model, task, word, order) {
+async function call(model, task, word, order, provider) {
   const messages = buildMessages(task, word, order);
   const body = {
     model,
     messages,
     temperature: 0,
     reasoning: { effort: "low" },
-    max_tokens: 400,
+    max_tokens: 2e3,
     usage: { include: true }
   };
+  if (provider) body.provider = { order: [provider], allow_fallbacks: false };
   const post = async () => fetch(API, {
     method: "POST",
     headers: {
@@ -360,12 +377,32 @@ async function call(model, task, word, order) {
     const text = await res.text();
     throw new Error(`HTTP ${status}: ${text.slice(0, 300)}`);
   }
-  const j = await res.json();
+  const j0 = await res.json();
+  let j = j0;
+  const noContent = !(j0?.choices?.[0]?.message?.content ?? "") && j0?.choices?.[0]?.finish_reason === "length";
+  if (noContent) {
+    const bigger = { ...body, max_tokens: 4e3 };
+    const r2 = await fetch(API, {
+      method: "POST",
+      headers: {
+        Authorization: `Bearer ${apiKey()}`,
+        "Content-Type": "application/json",
+        "HTTP-Referer": "https://mixedmessages.fyi",
+        "X-Title": "RTR-bench"
+      },
+      body: JSON.stringify(bigger)
+    });
+    if (r2.ok) {
+      j = await r2.json();
+      body.max_tokens = 4e3;
+    }
+  }
   return {
     json: j,
     body,
     temperature: "temperature" in body ? 0 : null,
-    reasoningEffort: body.reasoning?.effort ?? null
+    reasoningEffort: body.reasoning?.effort ?? null,
+    maxTokens: body.max_tokens
   };
 }
 async function withRetry(fn, tries = 4) {
@@ -388,11 +425,31 @@ async function runModel(windows, opts) {
     throw new Error(
       "OPENROUTER_API_KEY is not set. Put it in .env in this directory, or export it in your shell."
     );
-  const dir = path2.join("results", "raw", slug(opts.model));
+  try {
+    const res = await fetch(`${API.replace("/chat/completions", "")}/credits`, {
+      headers: { Authorization: `Bearer ${apiKey()}` }
+    });
+    if (res.ok) {
+      const d = (await res.json()).data;
+      const remaining = d.total_credits - d.total_usage;
+      if (remaining <= 0)
+        throw new Error(
+          `OpenRouter balance is $${remaining.toFixed(4)}. Add credit at https://openrouter.ai/settings/credits, then rerun.`
+        );
+    }
+  } catch (e) {
+    if (String(e.message).startsWith("OpenRouter balance")) throw e;
+  }
+  const dir = path2.join(
+    "results",
+    "raw",
+    slug(opts.model) + (opts.provider ? `@${slug(opts.provider)}` : "")
+  );
   let done = 0;
   let failed = 0;
   let tokens = 0;
   let usd = 0;
+  const served = {};
   const list = opts.limit ? windows.slice(0, opts.limit) : windows;
   const jobs = [];
   for (const w of list) {
@@ -403,7 +460,17 @@ async function runModel(windows, opts) {
     }
   }
   const file2 = (j) => path2.join(dir, j.task, `${j.day}-${j.order}.json`);
-  const queue = jobs.filter((j) => !fs3.existsSync(file2(j)));
+  const want = opts.provider ?? null;
+  const isDone = (f) => {
+    if (!fs3.existsSync(f)) return false;
+    try {
+      const rec = JSON.parse(fs3.readFileSync(f, "utf8"));
+      return !rec.error && rec.parsed && (rec.providerRequested ?? null) === want;
+    } catch {
+      return false;
+    }
+  };
+  const queue = jobs.filter((j) => !isDone(file2(j)));
   console.log(
     `${opts.model}: ${jobs.length} calls, ${jobs.length - queue.length} already on disk, ${queue.length} to go`
   );
@@ -415,13 +482,15 @@ async function runModel(windows, opts) {
       fs3.mkdirSync(path2.dirname(target), { recursive: true });
       let rec;
       try {
-        const { json, body, temperature, reasoningEffort } = await withRetry(
-          () => call(opts.model, j.task, j.word, j.order)
+        await pace(opts.rpm);
+        const { json, body, temperature, reasoningEffort, maxTokens } = await withRetry(
+          () => call(opts.model, j.task, j.word, j.order, opts.provider)
         );
         const raw = json?.choices?.[0]?.message?.content ?? "";
         const usage = json?.usage ?? {};
         tokens += usage.total_tokens ?? 0;
-        usd += Number(json?.usage?.total_cost ?? 0);
+        usd += Number(usage.cost ?? usage.total_cost ?? 0);
+        if (json?.provider) served[json.provider] = (served[json.provider] ?? 0) + 1;
         rec = {
           model: opts.model,
           day: j.day,
@@ -431,6 +500,8 @@ async function runModel(windows, opts) {
           promptHash: PROMPT_HASH,
           temperature,
           reasoningEffort,
+          providerRequested: opts.provider ?? null,
+          maxTokens,
           request: body,
           raw,
           parsed: parseAnswer(raw),
@@ -450,6 +521,8 @@ async function runModel(windows, opts) {
           promptHash: PROMPT_HASH,
           temperature: 0,
           reasoningEffort: null,
+          providerRequested: opts.provider ?? null,
+          maxTokens: null,
           request: null,
           raw: "",
           parsed: null,
@@ -470,8 +543,29 @@ async function runModel(windows, opts) {
     }
   };
   await Promise.all(Array.from({ length: opts.concurrency }, worker));
+  const names = Object.keys(served);
+  const mix = names.length ? names.map((n) => `${n} ${served[n]}`).join(", ") : "unknown";
   console.log(`${opts.model}: wrote ${done}, failed ${failed}, tokens ${tokens.toLocaleString()}`);
-  return { done, failed, tokens, usd };
+  console.log(`${opts.model}: providers ${mix}${opts.provider ? " (pinned)" : ""}`);
+  return { done, failed, tokens, usd, providers: served };
+}
+async function listEndpoints(model) {
+  const res = await fetch(`${API.replace("/chat/completions", "")}/models/${model}/endpoints`, {
+    headers: { "User-Agent": "RTR-bench", Accept: "application/json" }
+  });
+  if (!res.ok) throw new Error(`endpoints HTTP ${res.status}`);
+  const j = await res.json();
+  const eps = j.data?.endpoints ?? [];
+  return eps.map((e) => ({
+    provider: e.provider ?? e.provider_name ?? "?",
+    quant: e.quantization ?? "?",
+    inUsd: +(Number(e.pricing?.prompt) * 1e6).toFixed(3),
+    outUsd: +(Number(e.pricing?.completion) * 1e6).toFixed(3),
+    ctx: e.context_length ?? null,
+    status: e.status ?? null
+  })).sort(
+    (a, b) => (a.status === 0 ? 0 : 1) - (b.status === 0 ? 0 : 1) || a.inUsd - b.inUsd
+  );
 }
 async function listModels() {
   const res = await fetch("https://openrouter.ai/api/v1/models", {
@@ -533,12 +627,12 @@ function writeReport() {
   lines.push("## Leaderboard");
   lines.push("");
   lines.push(
-    "| model | score /100 | n | cost | 95% CI | axis x | axis y | Brier | own reading agrees | crowd distance | gap | unstable |"
+    "| model | score /100 | n | cost | provider | 95% CI | axis x | axis y | Brier | own reading agrees | crowd distance | gap | unstable |"
   );
-  lines.push("|---|---|---|---|---|---|---|---|---|---|---|---|");
+  lines.push("|---|---|---|---|---|---|---|---|---|---|---|---|---|");
   for (const s of ranked) {
     lines.push(
-      `| \`${s.model}\` | **${(100 * s.predict.acc).toFixed(1)}** | ${s.windows} | $${s.costUsd.toFixed(4)} | ${ci(s.predict.ci)} | ${pct(s.axes.x)} | ${pct(s.axes.y)} | ${round(s.brier)} | ${pct(s.read.agree)} | ${round(s.crowdDistance)} | ${s.gap >= 0 ? "+" : ""}${pct(s.gap)} | ${pct(s.instability)} |`
+      `| \`${s.model}\` | **${(100 * s.predict.acc).toFixed(1)}** | ${s.windows} | $${s.costUsd.toFixed(4)} | ${providerLabel(s)} | ${ci(s.predict.ci)} | ${pct(s.axes.x)} | ${pct(s.axes.y)} | ${round(s.brier)} | ${pct(s.read.agree)} | ${round(s.crowdDistance)} | ${s.gap >= 0 ? "+" : ""}${pct(s.gap)} | ${pct(s.instability)} |`
     );
   }
   lines.push("");
@@ -611,6 +705,12 @@ function writeReport() {
   console.log("wrote results/leaderboard.md and results/leaderboard.svg");
   return data;
 }
+function providerLabel(s) {
+  const names = Object.keys(s.providers);
+  if (names.length === 0) return "?";
+  if (names.length === 1) return names[0];
+  return `${names.length} providers`;
+}
 function scoreLine(s) {
   const [lo, hi] = s.predict.ci;
   return [
@@ -618,7 +718,8 @@ function scoreLine(s) {
     `${(100 * s.predict.acc).toFixed(1)} / 100`,
     `${s.windows} windows`,
     `95% CI ${(100 * lo).toFixed(1)} to ${(100 * hi).toFixed(1)}`,
-    `cost $${s.costUsd.toFixed(4)}`
+    `cost $${s.costUsd.toFixed(4)}`,
+    `provider ${providerLabel(s)}`
   ].join("   ");
 }
 function svg(scores, baselines, windowCount) {
@@ -719,8 +820,10 @@ if (cmd === "catalog") {
   const windows = loadWindows().map((w) => ({ day: w.day, word: w.word }));
   const limit = flags.limit ? Number(flags.limit) : void 0;
   const concurrency = Number(flags.concurrency ?? 6);
+  const rpm = flags.rpm ? Number(flags.rpm) : void 0;
+  const provider = flags.provider ? String(flags.provider) : void 0;
   for (const model of models) {
-    await runModel(windows, { model, tasks: [...tasks], concurrency, limit });
+    await runModel(windows, { model, tasks: [...tasks], concurrency, limit, provider, rpm });
   }
   const data = writeReport();
   for (const s of data.scores) {
@@ -744,11 +847,16 @@ if (cmd === "catalog") {
     model: cmd,
     tasks: ["predict", "read"],
     concurrency: Number(flags.concurrency ?? 6),
-    limit: flags.limit ? Number(flags.limit) : void 0
+    limit: flags.limit ? Number(flags.limit) : void 0,
+    provider: flags.provider ? String(flags.provider) : void 0
   });
   const data = writeReport();
   const s = data.scores.find((x) => x.model === cmd);
   if (s) console.log(scoreLine(s));
+} else if (cmd === "providers") {
+  const id = rest.find((a) => !a.startsWith("--"));
+  if (!id) throw new Error("usage: rtr-bench providers <model-id>");
+  console.table(await listEndpoints(id));
 } else {
   const here = path4.dirname(fileURLToPath2(import.meta.url));
   console.log(fs5.readFileSync(path4.join(here, "..", "README.md"), "utf8").slice(0, 1200));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rtr-bench",
-  "version": "0.1.2",
+  "version": "0.1.4",
   "description": "Can a language model predict how a crowd reads a word? A cultural fluency benchmark built from Mixed Messages.",
   "type": "module",
   "license": "MIT",
@@ -25,6 +25,7 @@
   "files": [
     "dist/",
     "data/",
+    "results/leaderboard-scatter.png",
     "README.md",
     "LICENSE"
   ],