rtr-bench 0.1.5 → 0.2.0

package/README.md

@@ -1,124 +1,230 @@
 # RTR-bench
 
-**RTR-bench** tests whether language models can predict how a crowd interprets an ambiguous prompt. It investigates one aspect of cultural fluency: anticipating how people assign meaning to words, people, institutions, and ideas. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project where participants respond to a daily prompt by placing a coordinate on a 2×2 grid: Honest vs Performed on one axis, Uniting vs Dividing on the other. The dataset used here contains 39,828 submissions across 165 closed windows.
+**RTR-bench** is an exploratory benchmark that tests whether language models can predict how a crowd interprets an ambiguous prompt, and whether examples from that crowd help them improve. It investigates one aspect of cultural fluency: anticipating how people assign meaning to words, people, institutions, and ideas. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project where participants respond to a daily prompt by placing a coordinate on a 2×2 grid: Honest vs Performed on one axis, Uniting vs Dividing on the other. The dataset used here contains 39,828 submissions across 165 closed windows.
 
 These submissions were collected through participation in a generative art project. Participants were not trained to apply an annotation rubric or converge on a correct answer. Each placement recorded an interpretation, and together those placements formed a distribution. That distribution preserves both agreement and disagreement: how strongly one reading prevailed, which alternatives attracted people, and where the crowd split.
 
 Commonly cited benchmarks such as [HLE](https://lastexam.ai/) and [SWE-bench](https://www.swebench.com/) evaluate expert answers or software changes against defined criteria for success. RTR-bench evaluates a model against an observed social response. A crowd’s interpretation can be contradictory, unfamiliar, or difficult to anticipate, and accurately predicting it requires distinguishing what a prompt means to that audience from the reading the model would otherwise produce.
 
-For each prompt, the model answers two separate questions: which of the four quadrants received the most human submissions, and which quadrant represents its own interpretation. Both responses are compared with the recorded crowd response. The headline score measures how often the model correctly predicts the crowd’s dominant quadrant. This gives a straightforward measure of agreement, although it captures only part of the underlying distribution.
+For each prompt, the model estimates the share of human submissions in all four quadrants. **`Predict`** asks for this distribution without historical examples. **`Learn`** lets the model choose and inspect up to three historical examples before submitting its prediction. Both tracks score the prediction against the recorded crowd distribution, out of 100. Learn also reports its gain or loss against Predict, showing whether gathering evidence improved the answer.
 
-The score is one way to compare models; their patterns of agreement and error are another. Which interpretations do models consistently anticipate? Where do different models make the same mistake? How does asking a model to predict the crowd change its answer? RTR-bench uses these questions to examine what it means for a model to read the room - with the room here defined by the participants in Mixed Messages.
+The scores are one way to compare models; their patterns of agreement and error are another. Which interpretations do models consistently anticipate? Where do different models make the same mistake? Which examples do they seek out, and when does that evidence help or mislead them? RTR-bench uses these questions to examine what it means for a model to read the room—with the room here defined by the participants in Mixed Messages.
+
+RTR-bench is intended to surface patterns of agreement, disagreement, and adaptation that warrant further investigation. Its results describe performance on this audience and prompt set; broader claims about cultural fluency require validation across other audiences and settings. Recorded submissions provide an observable prediction target, but do not establish participants’ underlying beliefs or subjective experiences.
 
 ## How it works
 
-[Mixed Messages](https://mixedmessages.fyi) publishes one prompt, every day. Over a 24-hour submission period, participants interpret that prompt by placing a coordinate on the grid. When the window closes, a rendering algorithm uses the (x, y) coordinates to create an artwork which depicts the day's collective interpretation. Each window has a dominant quadrant, as well as conviction and polarizations cores. The 2x2 grid gives us four possible options for the dominant quadrant, with the following implied readings and coordinates: 
+[Mixed Messages](https://mixedmessages.fyi) publishes one prompt, every day. Over a 24-hour submission period, participants interpret that prompt by placing a coordinate on the grid. When the window closes, a rendering algorithm uses the (x, y) coordinates to create an artwork which depicts the day’s collective interpretation. Each window has a distribution across four quadrants, along with conviction and polarization scores. The 2×2 grid gives us the following readings and coordinates:
 
 | Quadrant | Reading | Coordinates | Color |
 |---|---|---|---|
-| Honest x Uniting | Bringing us together, authentically | x ≥ 5,000, y < 5,000 | Green |
-| Honest x Dividing | Fracturing us, truthfully | x ≥ 5,000, y ≥ 5,000 | Purple |
-| Performed x Uniting | Creating false unity among us | x < 5,000, y < 5,000 | Cyan |
-| Performed x Dividing | Exploiting our divisions, deliberately | x < 5,000, y ≥ 5,000 | Red |
+| Honest × Uniting | Bringing us together, authentically | x ≥ 5,000, y < 5,000 | Green |
+| Honest × Dividing | Fracturing us, truthfully | x ≥ 5,000, y ≥ 5,000 | Purple |
+| Performed × Uniting | Creating false unity among us | x < 5,000, y < 5,000 | Cyan |
+| Performed × Dividing | Exploiting our divisions, deliberately | x < 5,000, y ≥ 5,000 | Red |
+
+**The dataset.** 39,828 submissions across 165 of the 174 closed windows, from the November 12, 2026 export. Only windows with at least 25 submissions are included. `data/windows.json` holds everything: the prompt, submission counts by quadrant, center of mass, and conviction and polarization scores. The latter measures are not used in scoring.
+
+The dataset is divided chronologically into 90 reference windows, 30 development windows, and 45 evaluation windows. Every reference window predates every target. Development outcomes are never added to the reference pool. The same data and frozen splits are available on [Hugging Face](https://huggingface.co/datasets/noahmclaughlin/RTR-bench).
+
+**The tasks.** In **`Predict`**, the model receives the target prompt, audience description, and quadrant definitions, then returns four probabilities totaling one. A prediction of 0.10, 0.50, 0.25, and 0.15 means it expects 10%, 50%, 25%, and 15% of the crowd’s submissions in the respective quadrants. It sees no historical examples or target outcomes. In **`Learn`**, the model can first request a catalog containing the names and IDs of the 90 reference prompts. It chooses an example to inspect and receives that example’s actual distribution and submission count. It can then choose another example or submit its prediction, up to three distinct inspections. The choices are sequential: evidence from one example can influence what it inspects next.
 
-**The dataset.** 39,828 submissions across 165 of the 174 closed windows, as of November 12th. Eight windows had fewer than 25 marks, which is the floor. `data/windows.json` holds everything: the prompt, the submission counts, the dominant quadrant, the center of mass, and the conviction and polarization scores, which are not used in scoring but may be included at a later date. 
+Note that each **`Learn`** episode starts a fresh conversation. The model receives no target score or correctness feedback during the episode, and it can submit before exhausting its inspection budget. The environment allows eight actions and eight model turns. Learning happens through the conversation; the model’s weights are not updated.
+
+**Scoring.** For each target, sum the squared differences between the predicted and observed shares across all four quadrants. Average those errors equally across the declared targets, then calculate:
+
+```text
+Score = 100 × (1 − mean error / 2)
+Learn gain = Learn score − Predict score
+```
 
-**The task.** For each window with N≥25 submissions, the model is asked to determine, given the prompt, which quadrant received the most submissions. Submission counts and dominant quadrants are never shown to the model.
+An exact match scores 100. Invalid or missing terminal predictions receive zero and remain in the denominator. Pending targets keep the run incomplete, without an aggregate headline score. Probabilities must be finite values between zero and one; sums within 0.001 of one are normalized.
 
-**Scoring.** A right answer is one point. A wrong answer is zero points. The final score is calculated as the number of points divided by the number of prompts interpreted (165 as of 11/12/26), times 100. A score of 50 means that the model determined the correct dominant quadrant for half of the prompts. The four dominant quadrant options are rotated against four model calls so that no option gains from position. 3/4 matching answers is the model's final answer. A 2/2 split scores zero. 
+These scores are not percentage accuracy. Reports include uniform predictions and a constant distribution calculated from the reference windows, so a model’s score can be interpreted against simple baselines. Learn’s final score and its gain should be read together: improvement from a weak starting point does not necessarily produce a strong final prediction.
+
+**Consistency.** Repeated runs record fresh predictions under the same declared settings. Their observed ranges describe variation between runs. Reports also include confidence intervals obtained by resampling target entity groups, preserving matched targets when calculating Learn’s gain. These intervals describe variation across targets; they do not measure repeated-run variability or uncertainty about independent people.
+
+The full comparison includes randomly selected examples and examples selected by a fixed text-similarity method. These are controls for Learn, not additional tracks. Predict supplies the no-example control without duplicate calls. See the [Predict protocol](docs/distribution.md) and [Learn protocol](docs/learn-room.md) for the exact prompts, limits, failure handling, and reproducibility rules.
 
 ## Quickstart
 
-You need Node.js 18 or newer and an OpenRouter API key with available credits. The published package includes the benchmark dataset; no repository setup is required.
+Run RTR-bench from Terminal using one of these three commands:
+
+| Command | What it runs |
+|---|---|
+| `npx rtr-bench read-the-room <model>` | **Predict and Learn**, plus Learn's gain or loss against Predict |
+| `npx rtr-bench predict <model>` | **Predict only**: predict the crowd's distribution without examples |
+| `npx rtr-bench learn <model>` | **Learn only**: inspect historical examples, then predict; does not calculate Learn gain |
 
-Create a folder for your results, add your API key, and run a model:
+`<model>` is a placeholder for the model's OpenRouter ID. Each command also needs the provider and run settings. **After the one-time setup below**, this complete example runs both tracks on **five development prompts**, with a **$0.25 spending cap**:
 
 ```bash
-mkdir rtr-run && cd rtr-run
-echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
-npx --yes rtr-bench@latest z-ai/glm-5.3-flash
+npx rtr-bench read-the-room z-ai/glm-5.3-flash \
+  --provider Relace \
+  --split development \
+  --limit 5 \
+  --max-usd 0.25
+```
+
+To run only one track, replace `read-the-room` with `predict` or `learn` and keep the rest of the command. `--limit 5` means five target prompts per selected track. The backslashes let one command continue across several lines; copy the whole block into Terminal.
+
+### Before your first run
+
+1. Install [Node.js](https://nodejs.org/) **22.18 or newer**, which includes `npx`. `npx` downloads and starts RTR-bench for you. You do not need to clone GitHub or run `npm install` or `npm ci` for this path. The dataset is included; no GPU or database setup is needed.
+2. Open Terminal in a folder where you want to keep your results. Any folder you choose is fine. Creating a new folder is optional and does not start an evaluation.
+3. Get an [OpenRouter API key](https://openrouter.ai/settings/keys) and add credits to that account. In your chosen folder, create a plain-text file named `.env` containing the line below, replacing `sk-or-your-key` with your key. Keep this file private.
+
+```text
+OPENROUTER_API_KEY=sk-or-your-key
 ```
 
-Replace `sk-or-your-key` with your OpenRouter API key. To evaluate another model, replace `z-ai/glm-5.3-flash` with its OpenRouter model ID. `npx` downloads and caches the package automatically; `--yes` skips its install prompt.
+Keep Terminal in that same folder and run the complete command above. The benchmark reads `.env`, sends requests through OpenRouter, and charges your OpenRouter account for model usage. `--max-usd` is a spending limit, not a fixed price; the run stops incomplete if it cannot continue within that limit.
+
+These commands require RTR-bench **0.2.0 or newer** on npm. To select that exact release, replace `npx rtr-bench` with `npx rtr-bench@0.2.0`. GitHub updates and npm releases are published separately.
+
+### Read the results
+
+When the run finishes, Terminal prints each selected track's score and the path to `report.html`. Open that file in a browser to inspect the results. A combined `read-the-room` run also prints Learn's gain or loss against Predict.
+
+The benchmark automatically creates `results/learn-room/` inside your current folder and saves reports, requests, responses, action traces, and spending records there. You do not need to create the results folders yourself.
+
+Running the identical command again resumes its saved run rather than starting a fresh evaluation. Changing the model, selected tracks, target set, or settings creates a separate run and spending ledger. A small development run's predictions and budget are not carried into a larger run.
 
-The command runs the benchmark, prints the score, and saves responses, scores, and a report in `results/`. A full run schedules 1,320 calls: two questions, each asked in four option orders, across 165 prompts. Retries can add calls. Time and cost depend on the model and provider.
+### Run the final evaluation
 
-To list available model IDs and token prices:
+Use development prompts to check the setup. Once the settings are fixed, the following command runs **Predict, Learn, and both comparison controls on all 45 final evaluation prompts**, matching the conditions in the results table:
 
 ```bash
-npx --yes rtr-bench@latest catalog --limit 20
+npx rtr-bench read-the-room z-ai/glm-5.3-flash \
+  --provider Relace \
+  --controls \
+  --split evaluation \
+  --limit 45 \
+  --max-usd 0.25
 ```
 
-To try the command on just five prompts before running the full benchmark:
+`--controls` adds two comparisons: one supplies randomly selected historical examples, and the other supplies examples selected by text similarity. Without this flag, `read-the-room` runs only Predict and Learn. A development run is not comparable with the final evaluation results.
+
+To evaluate another model, replace `z-ai/glm-5.3-flash` with its OpenRouter model ID and select a provider that serves it. You can list models and inspect a model's providers with:
+
+```bash
+npx rtr-bench catalog --limit 20
+npx rtr-bench providers z-ai/glm-5.3-flash
+```
+
+Provider availability and pricing can change. Choose a spending cap appropriate for the selected model and provider; $0.25 is the cap for the GLM example, not a recommended budget for every model.
+
+### Use the GitHub source instead (optional)
+
+If you want to inspect or change the code, install [Git](https://git-scm.com/) and run:
 
 ```bash
-npx --yes rtr-bench@latest z-ai/glm-5.3-flash --limit 5
+git clone https://github.com/1999labs/RTR-bench
+cd RTR-bench
+npm ci
 ```
 
-This smaller run checks your setup; its score is not comparable with the full benchmark results. Rerunning without `--limit 5` reuses completed responses and runs the remaining prompts.
+These commands download the project, enter its folder, and install its supporting software. If you already have a checkout, use that folder. Create `.env` there as described above, then replace `npx rtr-bench` with `node src/cli.ts` in the evaluation commands. `src/cli.ts` is the source file that starts the benchmark. This is an alternative to the npm path, not an additional setup step.
 
 ## Results
 
-| Model | Provider | Score | Cost |
-|---|---|---|---|
-| z-ai/glm-5.3-flash | Tenstorrent (fp8) | 41.2 | $0.0281 |
-| z-ai/glm-5.3 | Tenstorrent (fp8) | 40.0 | $0.2828 |
-| minimax/minimax-m3.1-flash | GMICloud (fp8) | 32.1 | $0.2444 |
-| xiaomi/mimo-v2.6-flash | GMICloud (bf16) | 40.6 | $0.1216 |
-| deepseek/deepseek-v4.1-flash-1017 | Morph (fp8) | 40.0 | $0.3414 |
-| tencent/hy3 | Tencent (fp8) | 40.0 | $0.2665 |
-| google/gemini-3.8-flash | Google AI Studio | 46.1 | $0.7790 |
-| openai/gpt-6-luna | OpenAI | 37.0 | $0.1329 |
-| anthropic/claude-haiku-5.5 | Claude Platform on AWS | 43.0 | $0.1696 |
-| thinkingmachines/inkling-small | DeepInfra (fp8) | 37.0 | $0.1733 |
-| openai/gpt-oss-120b | DekaLLM (bf16) | 32.7 | $0.0205 |
+| Model | Provider | Predict | Learn | Learn gain | Full run cost (including controls) |
+|---|---|---:|---:|---:|---:|
+| z-ai/glm-5.3-flash | Relace (fp4) | 87.93 | 85.57 | −2.37 | $0.03150 |
+| z-ai/glm-5.3 | TBD | TBD | TBD | TBD | TBD |
+| minimax/minimax-m3.1-flash | TBD | TBD | TBD | TBD | TBD |
+| xiaomi/mimo-v2.6-flash | TBD | TBD | TBD | TBD | TBD |
+| deepseek/deepseek-v4.1-flash-1017 | TBD | TBD | TBD | TBD | TBD |
+| tencent/hy3 | TBD | TBD | TBD | TBD | TBD |
+| google/gemini-3.8-flash | TBD | TBD | TBD | TBD | TBD |
+| openai/gpt-6-luna | TBD | TBD | TBD | TBD | TBD |
+| anthropic/claude-haiku-5.5 | TBD | TBD | TBD | TBD | TBD |
+| thinkingmachines/inkling-small | TBD | TBD | TBD | TBD | TBD |
+| openai/gpt-oss-120b | TBD | TBD | TBD | TBD | TBD |
 
-![RTR-bench: score against cost](results/leaderboard-scatter.png)
+## Optional usage
 
-Eleven models were tested against the same 165 windows, representing 39,828 human submissions. Scores ranged from 32.1 to 46.1, compared with a 31.5 baseline for always choosing the most common quadrant and 33.9 for a simple lexical heuristic. Gemini scored highest, correctly predicting 76 of the 165 windows. Its bootstrapped 95% confidence interval was 38.2–53.3%, however, so small differences between models should be treated cautiously.
+The examples below use the GitHub source checkout. For model evaluations through the npm package, replace `node src/cli.ts` with `npx rtr-bench`. The synthetic examples use fixture paths from the source checkout.
 
-Perhaps the value of the benchmark lies more in exposing model behavior than in actually ranking models. As described above, benchmarks such as HLE and SWE-bench evaluate whether a model can answer an expert question or resolve a software issue against defined criteria. RTR-bench asks whether a model can anticipate how a particular crowd interpreted something ambiguous. An incorrect prediction does not establish that the model’s interpretation is unreasonable. It shows where that interpretation departs from the room.
+### Run one track
 
-The clearest shared departure was Honest × Dividing. This was the crowd’s dominant quadrant in 39 of 165 windows, or 23.6%. Across the models’ 1,591 final predictions without a tie between option orders, however, they selected it only 160 times, or 10.1%. Including tied predictions as misses, models correctly identified just 15.9% of the windows dominated by Honest × Dividing, compared with 60.7% for Honest × Uniting. The models were much better at anticipating authentic unity than anticipating division that participants interpreted as honest.
+To run Predict alone:
 
-Donald Trump and Elon Musk illustrate both the pattern and the limits of the current score. None of the eleven models correctly predicted the dominant quadrant for either Trump’s window or either of Musk’s two windows. For Trump, 634 of 738 submissions—85.9%—landed in Honest × Dividing. In the first Musk window, the same quadrant won with only 278 of 720 submissions, or 38.6%, narrowly ahead of Performed × Dividing at 34.7%. Both count as an incorrect prediction, although one misses an overwhelming consensus and the other misses a closely divided crowd. At the other end, all eleven models correctly predicted Handwritten Notes, The CIA, and Altruism.
+```bash
+node src/cli.ts predict z-ai/glm-5.3-flash \
+  --provider Relace \
+  --split development \
+  --limit 5 \
+  --max-usd 0.25
+```
 
-These results return to what makes the source data unusual: the crowd’s answer is a distribution. The current score reduces that distribution to its largest quadrant, making comparison straightforward while leaving much of the disagreement unmeasured. My main takeaway is that these models share a difficulty anticipating when this audience reads division as honest. Establishing how broadly that pattern holds will require more audiences, prompt variants, and future windows. Next, I would like to test more frontier models, report performance separately where the crowd was decisive, and evaluate predictions of the full distribution alongside the dominant-quadrant score. Conviction and polarization could then help describe the kinds of disagreement a model understands, and the kinds it consistently misreads.
+Replace `predict` with `learn` to run Learn alone. Each reports its own score; Learn’s gain requires the matching Predict result, which `read-the-room` calculates automatically. The standalone commands use the same prompts and scorer as their conditions in the combined run.
 
-## Optional usage
+Separate commands and changes to `--controls` create separate runs and spending ledgers; saved predictions are not automatically shared between them. Use `read-the-room --controls` from the outset when you want both tracks and the controls.
 
-OpenRouter may route calls to different providers serving the same model. To list providers, token prices, and numerical precision:
+The legacy `learn-room` and `distribution` commands remain available with their original defaults and saved-run formats. In particular, legacy `learn-room` without `--strategy` runs the no-example and random conditions. New commands use the existing `results/learn-room/` storage and report format; internal strategy labels are `none` (Predict), `agent` (Learn), `random`, and `similar`.
+
+### Synthetic runs
+
+Try Predict on the five-target fixture without an API key or any API spending:
+
+```bash
+node src/cli.ts predict \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --limit 5
+```
+
+To exercise Learn and all comparison conditions:
 
 ```bash
-npx --yes rtr-bench@latest providers z-ai/glm-5.3-flash
+node src/cli.ts read-the-room \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --controls \
+  --limit 5
 ```
 
-To use a specific provider, add `--provider` with a name from that list:
+These commands write explicitly labeled synthetic reports with saved outputs and browsable traces. They demonstrate the pipeline, not model performance.
+
+Synthetic run directories are ignored by Git. Fixture inputs and tests stay tracked; regenerate the reports locally with the commands above. Published benchmark reports remain tracked and are marked as generated in `.gitattributes`.
+
+### Providers and concurrency
+
+OpenRouter may route calls to different providers serving the same model. To list providers, token prices, and numerical precision:
 
 ```bash
-npx --yes rtr-bench@latest z-ai/glm-5.3-flash --provider Wafer
+node src/cli.ts providers z-ai/glm-5.3-flash
 ```
 
-A pinned run refuses to fall back to another provider, and the summary names the providers that served the calls. Provider availability can change.
+Use `--provider` to select a provider from that list. A pinned run refuses to fall back to another provider, and reports name the providers that served the calls. Provider availability can change.
+
+Predict and Learn currently execute requests sequentially. Each Learn episode contains a sequence of decisions, with the next request depending on the previous observation. Time and cost depend on the model, provider, token usage, and number of turns.
 
-The default is six concurrent calls. To allow up to eight calls at once, add `--concurrency 8`. This may shorten the run if the provider's rate limits allow it.
+On 45 targets, Predict and Learn together typically use about 270 model calls. Adding `--controls` brings that to about 360: 45 for Predict, 45 for each automatic control, and approximately five calls per Learn episode. Early stopping can reduce that number; additional agent turns and transport retries can increase it. The first final GLM run used 347 calls.
+
+The public commands default to 4,000 output tokens per response. Learn uses the published `transfer-check-v1` policy, up to three inspected examples, and eight model turns. Advanced runs can override `--max-tokens` and, for Learn, `--agent-policy`.
+
+The current client uses OpenRouter. Direct connections to other hosted or local endpoints are not yet implemented.
 
 ## Development
 
-To modify the benchmark or run the repository's source code, use the setup below. These commands run TypeScript directly and require Node.js 22.18 or newer.
+After the GitHub source setup in [Quickstart](#quickstart), use these commands from the project folder to check changes and build the distributable program. Source commands run TypeScript directly and require Node.js 22.18 or newer.
 
 ```bash
-git clone https://github.com/1999labs/RTR-bench
-cd RTR-bench
-npm install
-echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
-node src/cli.ts z-ai/glm-5.3-flash
+npm test
+npm run build
+
+node src/cli.ts read-the-room \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --controls \
+  --limit 5
 ```
 
-Replace `sk-or-your-key` with your OpenRouter API key. The run automatically generates scores and a report in `results/`.
+`npm test` checks the implementation. `npm run build` bundles the source into `dist/cli.js`, the program included in the npm package. The fixture command generates a synthetic report without making model API calls. For paid runs, use the Quickstart commands. See the [npm release instructions](docs/npm-release.md) for packaging and publishing.
+
+Rebuilding the original dataset requires the project’s database connection and the configuration in `scripts/export.mjs`. The exported dataset is already included, so running the benchmark does not require database access. The Hugging Face export and validator preserve the frozen source data and split assignments.
 
-Rebuilding the dataset requires the project's database connection and the configuration in `scripts/export.mjs`. The exported dataset is already included, so running the benchmark does not require database access.
+RTR-bench previously evaluated dominant-quadrant predictions. Those scores use a different protocol and are not directly comparable with Predict and Learn. See the [original benchmark README and results](results/history/original-benchmark/README.md).
 
 ## License
 
-MIT for the code. CC0 for `data/windows.json`.
+MIT for the code. CC0 for `data/windows.json` and its derived dataset exports.

package/data/splits/v1.json

@@ -0,0 +1,184 @@
+{
+  "version": "chronological-v1",
+  "datasetSha256": "ec9d7582b5610a24f4e14dbbb05d13c636b692ec9e133b9381db0fdf483ffb2b",
+  "rule": "Reference: day <= 99; development: days 100..129; evaluation: day >= 130. All reference windows precede all targets. No labels used to choose boundaries.",
+  "grouping": {
+    "method": "Trim, lowercase, collapse whitespace; manually reviewed prompt list for entity aliases.",
+    "aliases": {},
+    "decisions": [
+      "ELON MUSK (days 30 and 99) stays entirely in reference.",
+      "Related but distinct prompts (such as CONTENT CREATORS and INFLUENCERS) remain distinct; grouping prevents same-entity retrieval, not thematic overlap."
+    ]
+  },
+  "reference": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    28,
+    29,
+    30,
+    31,
+    35,
+    36,
+    37,
+    38,
+    39,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    54,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    70,
+    71,
+    72,
+    73,
+    74,
+    77,
+    78,
+    79,
+    80,
+    81,
+    82,
+    83,
+    84,
+    85,
+    86,
+    87,
+    88,
+    89,
+    90,
+    91,
+    92,
+    93,
+    94,
+    95,
+    96,
+    97,
+    98,
+    99
+  ],
+  "development": [
+    100,
+    101,
+    102,
+    103,
+    104,
+    105,
+    106,
+    107,
+    108,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    116,
+    117,
+    118,
+    119,
+    120,
+    121,
+    122,
+    123,
+    124,
+    125,
+    126,
+    127,
+    128,
+    129
+  ],
+  "evaluation": [
+    130,
+    131,
+    132,
+    133,
+    134,
+    135,
+    136,
+    137,
+    138,
+    139,
+    140,
+    141,
+    142,
+    143,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    150,
+    151,
+    152,
+    153,
+    154,
+    155,
+    156,
+    157,
+    158,
+    159,
+    160,
+    161,
+    162,
+    163,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    170,
+    171,
+    172,
+    173,
+    174
+  ]
+}