rtr-bench 0.1.3 → 0.1.4

package/README.md

@@ -90,11 +90,11 @@ Rebuilding the dataset needs the project's database connection, so `npm run expo
 
 ![RTR-bench: score against cost](results/leaderboard-scatter.png)
 
-Eleven models were run against the same 165 windows, and all of them landed between 32.1 and 46.1 out of 100. The floors sit low: guessing among four readings scores 25, and always naming the crowd's most common quadrant scores 31.5. Nearly every model clears both, and none of them comes close to reading the room. The order at the top is not a ranking either. At 165 windows the confidence intervals are about seven points wide, so gemini-3.8-flash at 46.1 and claude-haiku-5.5 at 43.0 are the same result measured twice, once for 78 cents and once for 17.
+Eleven models were tested against the same 165 windows. Scores concentrated in the high 30s and low 40s. Models were more alike in how they fail than in how they score. Across 1,591 answers, they decided on Honest x Uniting 67.7% of the time and Honest x Dividing 18.2% of the time, while the dominant quadrant was Honest x Dividing in 39 of those 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candor. 
 
-The models are more alike in how they fail than in how they score. Across 1,591 answers they name honest+uniting 67.7 percent of the time and honest+dividing 18.2 percent of the time, while the crowd chose honest+dividing in 39 of the 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candour. Six of the seven windows that no model got right are honest+dividing, among them Donald Trump and Elon Musk, where the crowd agreed with itself 86 and 70 percent. The 78-cent model and the two-cent one miss the same quadrant.
+There were seven prompts that no model got right. They were Donald Trump, Elon Musk, Las Vegas, Barack Obama, Reputation, Dreams, and Honor. The dominant quadrant for each of them besides Dreams was Honest x Dividing. There were three prompts that every model got right. They were Handwritten Notes, The CIA, and Altruism. 
 
-Cost barely enters into it. Two cents buys 32.7, seventeen cents buys 43.0, seventy-eight cents buys 46.1, and the whole table cost $2.56 to produce. What the numbers describe is a set of models that can tell whether a word feels warm or hostile, and cannot tell what this particular audience would do with it. That gap is the thing the benchmark was built to measure, and eleven models later it is still open.
+My main takeaway is that the same blind spot shows up in every model, and it is not a vocabulary problem. Ask it to pick the dominant quadrant for Trump or Musk and the model can easily choose dividing instead of uniting, but has a hard time determining honest or performed. That is why Honest X Dividing is the quadrant the models got wrong the most. Next, I would like to score more closed frontier models, score only the windows where the crowd was very decisive, and somehow factor into the benchmark the conviction and polarization scores from each window. 
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rtr-bench",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "Can a language model predict how a crowd reads a word? A cultural fluency benchmark built from Mixed Messages.",
   "type": "module",
   "license": "MIT",