rtr-bench 0.1.1 → 0.1.3

package/README.md

@@ -1,14 +1,14 @@
 # RTR-bench
 
-[Mixed Messages](https://mixedmessages.fyi) is a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed, and the y-axis is Uniting vs Dividing. Across 175 windows, there have been almost 40k individual submissions. Each is a unique human data point. 
+**RTR-bench** is a benchmark that scores language models on cultural fluency, built from real human data. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed and the y-axis is Uniting vs Dividing. Across 174 closed windows there have been almost 40k individual submissions. Each one is a unique, real human data point.
 
 Most labeled datasets are made by asking annotators to apply a rubric. This one was not. Nobody was given a definition of honest or performed, nobody was trained, and there was no correct answer to converge on. People arrived, interpreted a prompt, placed a submission, and left. What makes the data unusual is that the crowd's answer is a distribution, rather than a label. Not "people say X", but "31% of people said this, 44% said that, and the rest said something else."
 
-That distribution is what makes it useful for evaluating a model. Every window has a majority reading and a real minority, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize. The axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret. 
+That distribution is what makes it useful for evaluating a model. Every window has a dominant quadrant, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize: the axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret.
 
-Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), are exam-like: multiple choice, math, and code. Models have been optimized for their shape. But they are very mechanical, and their results are not indicative of a model's ability to "read the room." If the industry ambition is a model that surpasses human intelligence, the comparison has to include the things humans are good at, like having cultural fluency, which models generally lack. That's why Mixed Messages provides a unique and differentiated dataset to benchmark against.
+Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), measure whether a model can produce a correct answer: multiple choice, math, and code. RTR-bench measures a different subject. There is no answer key in this data, only how people actually interpreted an ambiguous prompt, so a model is scored against the crowd rather than against a key. That is what makes it a test of whether a model can read the room.
 
-**RTR-bench**, or read-the-room, is not a knowledge test. That is deliberate. It is an attempt at a different kind of benchmark, built from a creative act rather than an exam structure. Cultural fluency resists the traditional benchmark shape, which is why this one is a prediction task that measures whether a model can anticipate how people would interpret a prompt, and whether it would have interpreted that prompt the same way, both against real human data.
+**RTR-bench** is not a knowledge test, and that is deliberate. It is a prediction task. Given a prompt, a model has to decide how the crowd interpreted it, and then decide how it would interpret it itself. Both answers are scored against the real submissions.
 
 ## How it works
 
@@ -29,28 +29,45 @@ Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [H
 
 ## Quickstart
 
-Scoring needs no database access.
+Scoring needs no database access, just an OpenRouter API key.
 
-```bash
-git clone https://github.com/1999labs/RTR-bench
-cd RTR-bench
-npm install
+### Test a model
 
-# keep the key in a file, not in shell history
+Node 18 or newer. No clone, no install.
+
+```bash
+mkdir rtr-run && cd rtr-run
 echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
 
-# list ids and prices
-node src/cli.ts catalog --limit 20
+# see ids and prices. costs nothing
+npx --yes rtr-bench catalog --limit 20
 
-# run a model, then score everything on disk
-node src/cli.ts run --models <model-id> --concurrency 8
-node src/cli.ts report
+# run one model, then score it
+npx --yes rtr-bench z-ai/glm-5.3-flash
 ```
 
-Published on npm, the same run is one command with no clone:
+`--yes` skips npx's install prompt. A run makes 1,320 calls, takes about four minutes on a cheap model, costs a few cents, and writes `results/` in the folder you created. Add `--concurrency 8` to go faster.
+
+By default OpenRouter may route each call to any host that serves the model, and hosts differ in quantisation. List them, with price and precision:
 
 ```bash
-npx rtr-bench <model-id>
+npx --yes rtr-bench providers z-ai/glm-5.3-flash
+```
+
+Pin one with `--provider Wafer`. A pinned run refuses to fall back, and the summary names the host that served the calls.
+
+### Work on the code
+
+The repository runs the TypeScript sources directly, so this path needs Node 22.18 or newer.
+
+```bash
+git clone https://github.com/1999labs/RTR-bench
+cd RTR-bench
+npm install
+echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
+node src/cli.ts catalog --limit 20
+node src/cli.ts run --models z-ai/glm-5.3-flash --concurrency 8
+node src/cli.ts report
 ```
 
 Rebuilding the dataset needs the project's database connection, so `npm run export` is included for inspection only.
@@ -59,16 +76,25 @@ Rebuilding the dataset needs the project's database connection, so `npm run expo
 
 | Model | Provider | Score | Cost |
 |---|---|---|---|
-| z-ai/glm-5.3-flash | TBD | TBD | TBD |
-| z-ai/glm-5.3 | TBD | TBD | TBD |
-| minimax/minimax-m3.1-flash | TBD | TBD | TBD |
-| deepseek/deepseek-v4.1-flash-1017 | TBD | TBD | TBD |
-| tencent/hy4 | TBD | TBD | TBD |
-| google/gemini-3.8-flash | TBD | TBD | TBD |
-| google/gemini-4-carbon | TBD | TBD | TBD |
-| moonshotai/kimi-k3.1-pro | TBD | TBD | TBD |
-| x-ai/grok-4.7 | TBD | TBD | TBD |
-| qwen/qwen3.8-max-0902 | TBD | TBD | TBD |
+| z-ai/glm-5.3-flash | Tenstorrent (fp8) | 41.2 | $0.0281 |
+| z-ai/glm-5.3 | Tenstorrent (fp8) | 40.0 | $0.2828 |
+| minimax/minimax-m3.1-flash | GMICloud (fp8) | 32.1 | $0.2444 |
+| xiaomi/mimo-v2.6-flash | GMICloud (bf16) | 40.6 | $0.1216 |
+| deepseek/deepseek-v4.1-flash-1017 | Morph (fp8) | 40.0 | $0.3414 |
+| tencent/hy3 | Tencent (fp8) | 40.0 | $0.2665 |
+| google/gemini-3.8-flash | Google AI Studio | 46.1 | $0.7790 |
+| openai/gpt-6-luna | OpenAI | 37.0 | $0.1329 |
+| anthropic/claude-haiku-5.5 | Claude Platform on AWS | 43.0 | $0.1696 |
+| thinkingmachines/inkling-small | DeepInfra (fp8) | 37.0 | $0.1733 |
+| openai/gpt-oss-120b | DekaLLM (bf16) | 32.7 | $0.0205 |
+
+![RTR-bench: score against cost](results/leaderboard-scatter.png)
+
+Eleven models were run against the same 165 windows, and all of them landed between 32.1 and 46.1 out of 100. The floors sit low: guessing among four readings scores 25, and always naming the crowd's most common quadrant scores 31.5. Nearly every model clears both, and none of them comes close to reading the room. The order at the top is not a ranking either. At 165 windows the confidence intervals are about seven points wide, so gemini-3.8-flash at 46.1 and claude-haiku-5.5 at 43.0 are the same result measured twice, once for 78 cents and once for 17.
+
+The models are more alike in how they fail than in how they score. Across 1,591 answers they name honest+uniting 67.7 percent of the time and honest+dividing 18.2 percent of the time, while the crowd chose honest+dividing in 39 of the 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candour. Six of the seven windows that no model got right are honest+dividing, among them Donald Trump and Elon Musk, where the crowd agreed with itself 86 and 70 percent. The 78-cent model and the two-cent one miss the same quadrant.
+
+Cost barely enters into it. Two cents buys 32.7, seventeen cents buys 43.0, seventy-eight cents buys 46.1, and the whole table cost $2.56 to produce. What the numbers describe is a set of models that can tell whether a word feels warm or hostile, and cannot tell what this particular audience would do with it. That gap is the thing the benchmark was built to measure, and eleven models later it is still open.
 
 ## License
 

package/dist/cli.js

@@ -120,7 +120,8 @@ function readModel(dir) {
         order: rec.order,
         task,
         parsed: rec.parsed ?? null,
-        cost: rec.usage?.cost ?? rec.usage?.total_cost ?? 0
+        cost: rec.usage?.cost ?? rec.usage?.total_cost ?? 0,
+        provider: rec.provider ?? null
       });
     }
   }
@@ -236,6 +237,10 @@ function scoreModel(model, windows) {
   const mean = (xs) => xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : NaN;
   const predictAcc = mean(predHits);
   const readAgree = mean(readHits);
+  const providers = {};
+  for (const c of calls) {
+    if (c.provider) providers[c.provider] = (providers[c.provider] ?? 0) + 1;
+  }
   return {
     model: display,
     windows: predHits.length,
@@ -247,7 +252,8 @@ function scoreModel(model, windows) {
     gap: readAgree - predictAcc,
     instability: pred.length ? pred.filter((a) => a.unstable).length / pred.length : NaN,
     parseFailures: calls.filter((c) => !c.parsed).length,
-    costUsd: calls.reduce((a, c) => a + c.cost, 0)
+    costUsd: calls.reduce((a, c) => a + c.cost, 0),
+    providers
   };
 }
 function majorityBaseline(windows) {
@@ -313,19 +319,30 @@ import fs3 from "node:fs";
 import path2 from "node:path";
 var API = "https://openrouter.ai/api/v1/chat/completions";
 var apiKey = () => process.env.OPENROUTER_API_KEY ?? "";
+var nextSlot = 0;
+async function pace(rpm) {
+  if (!rpm) return;
+  const gap = 6e4 / rpm;
+  const now = Date.now();
+  const start = Math.max(now, nextSlot);
+  nextSlot = start + gap;
+  const wait = start - now;
+  if (wait > 0) await new Promise((r) => setTimeout(r, wait));
+}
 function slug(model) {
   return model.replace(/[^a-zA-Z0-9]+/g, "-").toLowerCase();
 }
-async function call(model, task, word, order) {
+async function call(model, task, word, order, provider) {
   const messages = buildMessages(task, word, order);
   const body = {
     model,
     messages,
     temperature: 0,
     reasoning: { effort: "low" },
-    max_tokens: 400,
+    max_tokens: 2e3,
     usage: { include: true }
   };
+  if (provider) body.provider = { order: [provider], allow_fallbacks: false };
   const post = async () => fetch(API, {
     method: "POST",
     headers: {
@@ -360,12 +377,32 @@ async function call(model, task, word, order) {
     const text = await res.text();
     throw new Error(`HTTP ${status}: ${text.slice(0, 300)}`);
   }
-  const j = await res.json();
+  const j0 = await res.json();
+  let j = j0;
+  const noContent = !(j0?.choices?.[0]?.message?.content ?? "") && j0?.choices?.[0]?.finish_reason === "length";
+  if (noContent) {
+    const bigger = { ...body, max_tokens: 4e3 };
+    const r2 = await fetch(API, {
+      method: "POST",
+      headers: {
+        Authorization: `Bearer ${apiKey()}`,
+        "Content-Type": "application/json",
+        "HTTP-Referer": "https://mixedmessages.fyi",
+        "X-Title": "RTR-bench"
+      },
+      body: JSON.stringify(bigger)
+    });
+    if (r2.ok) {
+      j = await r2.json();
+      body.max_tokens = 4e3;
+    }
+  }
   return {
     json: j,
     body,
     temperature: "temperature" in body ? 0 : null,
-    reasoningEffort: body.reasoning?.effort ?? null
+    reasoningEffort: body.reasoning?.effort ?? null,
+    maxTokens: body.max_tokens
   };
 }
 async function withRetry(fn, tries = 4) {
@@ -388,11 +425,31 @@ async function runModel(windows, opts) {
     throw new Error(
       "OPENROUTER_API_KEY is not set. Put it in .env in this directory, or export it in your shell."
     );
-  const dir = path2.join("results", "raw", slug(opts.model));
+  try {
+    const res = await fetch(`${API.replace("/chat/completions", "")}/credits`, {
+      headers: { Authorization: `Bearer ${apiKey()}` }
+    });
+    if (res.ok) {
+      const d = (await res.json()).data;
+      const remaining = d.total_credits - d.total_usage;
+      if (remaining <= 0)
+        throw new Error(
+          `OpenRouter balance is $${remaining.toFixed(4)}. Add credit at https://openrouter.ai/settings/credits, then rerun.`
+        );
+    }
+  } catch (e) {
+    if (String(e.message).startsWith("OpenRouter balance")) throw e;
+  }
+  const dir = path2.join(
+    "results",
+    "raw",
+    slug(opts.model) + (opts.provider ? `@${slug(opts.provider)}` : "")
+  );
   let done = 0;
   let failed = 0;
   let tokens = 0;
   let usd = 0;
+  const served = {};
   const list = opts.limit ? windows.slice(0, opts.limit) : windows;
   const jobs = [];
   for (const w of list) {
@@ -403,7 +460,17 @@ async function runModel(windows, opts) {
     }
   }
   const file2 = (j) => path2.join(dir, j.task, `${j.day}-${j.order}.json`);
-  const queue = jobs.filter((j) => !fs3.existsSync(file2(j)));
+  const want = opts.provider ?? null;
+  const isDone = (f) => {
+    if (!fs3.existsSync(f)) return false;
+    try {
+      const rec = JSON.parse(fs3.readFileSync(f, "utf8"));
+      return !rec.error && rec.parsed && (rec.providerRequested ?? null) === want;
+    } catch {
+      return false;
+    }
+  };
+  const queue = jobs.filter((j) => !isDone(file2(j)));
   console.log(
     `${opts.model}: ${jobs.length} calls, ${jobs.length - queue.length} already on disk, ${queue.length} to go`
   );
@@ -415,13 +482,15 @@ async function runModel(windows, opts) {
       fs3.mkdirSync(path2.dirname(target), { recursive: true });
       let rec;
       try {
-        const { json, body, temperature, reasoningEffort } = await withRetry(
-          () => call(opts.model, j.task, j.word, j.order)
+        await pace(opts.rpm);
+        const { json, body, temperature, reasoningEffort, maxTokens } = await withRetry(
+          () => call(opts.model, j.task, j.word, j.order, opts.provider)
         );
         const raw = json?.choices?.[0]?.message?.content ?? "";
         const usage = json?.usage ?? {};
         tokens += usage.total_tokens ?? 0;
-        usd += Number(json?.usage?.total_cost ?? 0);
+        usd += Number(usage.cost ?? usage.total_cost ?? 0);
+        if (json?.provider) served[json.provider] = (served[json.provider] ?? 0) + 1;
         rec = {
           model: opts.model,
           day: j.day,
@@ -431,6 +500,8 @@ async function runModel(windows, opts) {
           promptHash: PROMPT_HASH,
           temperature,
           reasoningEffort,
+          providerRequested: opts.provider ?? null,
+          maxTokens,
           request: body,
           raw,
           parsed: parseAnswer(raw),
@@ -450,6 +521,8 @@ async function runModel(windows, opts) {
           promptHash: PROMPT_HASH,
           temperature: 0,
           reasoningEffort: null,
+          providerRequested: opts.provider ?? null,
+          maxTokens: null,
           request: null,
           raw: "",
           parsed: null,
@@ -470,8 +543,29 @@ async function runModel(windows, opts) {
     }
   };
   await Promise.all(Array.from({ length: opts.concurrency }, worker));
+  const names = Object.keys(served);
+  const mix = names.length ? names.map((n) => `${n} ${served[n]}`).join(", ") : "unknown";
   console.log(`${opts.model}: wrote ${done}, failed ${failed}, tokens ${tokens.toLocaleString()}`);
-  return { done, failed, tokens, usd };
+  console.log(`${opts.model}: providers ${mix}${opts.provider ? " (pinned)" : ""}`);
+  return { done, failed, tokens, usd, providers: served };
+}
+async function listEndpoints(model) {
+  const res = await fetch(`${API.replace("/chat/completions", "")}/models/${model}/endpoints`, {
+    headers: { "User-Agent": "RTR-bench", Accept: "application/json" }
+  });
+  if (!res.ok) throw new Error(`endpoints HTTP ${res.status}`);
+  const j = await res.json();
+  const eps = j.data?.endpoints ?? [];
+  return eps.map((e) => ({
+    provider: e.provider ?? e.provider_name ?? "?",
+    quant: e.quantization ?? "?",
+    inUsd: +(Number(e.pricing?.prompt) * 1e6).toFixed(3),
+    outUsd: +(Number(e.pricing?.completion) * 1e6).toFixed(3),
+    ctx: e.context_length ?? null,
+    status: e.status ?? null
+  })).sort(
+    (a, b) => (a.status === 0 ? 0 : 1) - (b.status === 0 ? 0 : 1) || a.inUsd - b.inUsd
+  );
 }
 async function listModels() {
   const res = await fetch("https://openrouter.ai/api/v1/models", {
@@ -533,12 +627,12 @@ function writeReport() {
   lines.push("## Leaderboard");
   lines.push("");
   lines.push(
-    "| model | score /100 | n | cost | 95% CI | axis x | axis y | Brier | own reading agrees | crowd distance | gap | unstable |"
+    "| model | score /100 | n | cost | provider | 95% CI | axis x | axis y | Brier | own reading agrees | crowd distance | gap | unstable |"
   );
-  lines.push("|---|---|---|---|---|---|---|---|---|---|---|---|");
+  lines.push("|---|---|---|---|---|---|---|---|---|---|---|---|---|");
   for (const s of ranked) {
     lines.push(
-      `| \`${s.model}\` | **${(100 * s.predict.acc).toFixed(1)}** | ${s.windows} | $${s.costUsd.toFixed(4)} | ${ci(s.predict.ci)} | ${pct(s.axes.x)} | ${pct(s.axes.y)} | ${round(s.brier)} | ${pct(s.read.agree)} | ${round(s.crowdDistance)} | ${s.gap >= 0 ? "+" : ""}${pct(s.gap)} | ${pct(s.instability)} |`
+      `| \`${s.model}\` | **${(100 * s.predict.acc).toFixed(1)}** | ${s.windows} | $${s.costUsd.toFixed(4)} | ${providerLabel(s)} | ${ci(s.predict.ci)} | ${pct(s.axes.x)} | ${pct(s.axes.y)} | ${round(s.brier)} | ${pct(s.read.agree)} | ${round(s.crowdDistance)} | ${s.gap >= 0 ? "+" : ""}${pct(s.gap)} | ${pct(s.instability)} |`
     );
   }
   lines.push("");
@@ -611,6 +705,12 @@ function writeReport() {
   console.log("wrote results/leaderboard.md and results/leaderboard.svg");
   return data;
 }
+function providerLabel(s) {
+  const names = Object.keys(s.providers);
+  if (names.length === 0) return "?";
+  if (names.length === 1) return names[0];
+  return `${names.length} providers`;
+}
 function scoreLine(s) {
   const [lo, hi] = s.predict.ci;
   return [
@@ -618,7 +718,8 @@ function scoreLine(s) {
     `${(100 * s.predict.acc).toFixed(1)} / 100`,
     `${s.windows} windows`,
     `95% CI ${(100 * lo).toFixed(1)} to ${(100 * hi).toFixed(1)}`,
-    `cost $${s.costUsd.toFixed(4)}`
+    `cost $${s.costUsd.toFixed(4)}`,
+    `provider ${providerLabel(s)}`
   ].join("   ");
 }
 function svg(scores, baselines, windowCount) {
@@ -719,8 +820,10 @@ if (cmd === "catalog") {
   const windows = loadWindows().map((w) => ({ day: w.day, word: w.word }));
   const limit = flags.limit ? Number(flags.limit) : void 0;
   const concurrency = Number(flags.concurrency ?? 6);
+  const rpm = flags.rpm ? Number(flags.rpm) : void 0;
+  const provider = flags.provider ? String(flags.provider) : void 0;
   for (const model of models) {
-    await runModel(windows, { model, tasks: [...tasks], concurrency, limit });
+    await runModel(windows, { model, tasks: [...tasks], concurrency, limit, provider, rpm });
   }
   const data = writeReport();
   for (const s of data.scores) {
@@ -744,11 +847,16 @@ if (cmd === "catalog") {
     model: cmd,
     tasks: ["predict", "read"],
     concurrency: Number(flags.concurrency ?? 6),
-    limit: flags.limit ? Number(flags.limit) : void 0
+    limit: flags.limit ? Number(flags.limit) : void 0,
+    provider: flags.provider ? String(flags.provider) : void 0
   });
   const data = writeReport();
   const s = data.scores.find((x) => x.model === cmd);
   if (s) console.log(scoreLine(s));
+} else if (cmd === "providers") {
+  const id = rest.find((a) => !a.startsWith("--"));
+  if (!id) throw new Error("usage: rtr-bench providers <model-id>");
+  console.table(await listEndpoints(id));
 } else {
   const here = path4.dirname(fileURLToPath2(import.meta.url));
   console.log(fs5.readFileSync(path4.join(here, "..", "README.md"), "utf8").slice(0, 1200));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rtr-bench",
-  "version": "0.1.1",
+  "version": "0.1.3",
   "description": "Can a language model predict how a crowd reads a word? A cultural fluency benchmark built from Mixed Messages.",
   "type": "module",
   "license": "MIT",
@@ -25,6 +25,7 @@
   "files": [
     "dist/",
     "data/",
+    "results/leaderboard-scatter.png",
     "README.md",
     "LICENSE"
   ],