rtr-bench 0.1.4 → 0.2.0

package/README.md

@@ -1,101 +1,230 @@
 # RTR-bench
 
-**RTR-bench** is a benchmark that scores language models on cultural fluency, built from real human data. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project that asks participants to interpret a daily prompt by placing a coordinate on a 2x2 grid, where the x-axis is Honest vs Performed and the y-axis is Uniting vs Dividing. Across 174 closed windows there have been almost 40k individual submissions. Each one is a unique, real human data point.
+**RTR-bench** is an exploratory benchmark that tests whether language models can predict how a crowd interprets an ambiguous prompt, and whether examples from that crowd help them improve. It investigates one aspect of cultural fluency: anticipating how people assign meaning to words, people, institutions, and ideas. Its source is [Mixed Messages](https://mixedmessages.fyi), a generative art project where participants respond to a daily prompt by placing a coordinate on a 2×2 grid: Honest vs Performed on one axis, Uniting vs Dividing on the other. The dataset used here contains 39,828 submissions across 165 closed windows.
 
-Most labeled datasets are made by asking annotators to apply a rubric. This one was not. Nobody was given a definition of honest or performed, nobody was trained, and there was no correct answer to converge on. People arrived, interpreted a prompt, placed a submission, and left. What makes the data unusual is that the crowd's answer is a distribution, rather than a label. Not "people say X", but "31% of people said this, 44% said that, and the rest said something else."
+These submissions were collected through participation in a generative art project. Participants were not trained to apply an annotation rubric or converge on a correct answer. Each placement recorded an interpretation, and together those placements formed a distribution. That distribution preserves both agreement and disagreement: how strongly one reading prevailed, which alternatives attracted people, and where the crowd split.
 
-That distribution is what makes it useful for evaluating a model. Every window has a dominant quadrant, so a model can be scored on whether it lands with the crowd or against it, and on which axis it misreads when it does. There is also nothing here to memorize: the axes are cultural rather than factual, so no amount of training data tells a model how a crowd interpreted a prompt. The only way to get it right is for the model to understand how people would think and interpret.
+Commonly cited benchmarks such as [HLE](https://lastexam.ai/) and [SWE-bench](https://www.swebench.com/) evaluate expert answers or software changes against defined criteria for success. RTR-bench evaluates a model against an observed social response. A crowd’s interpretation can be contradictory, unfamiliar, or difficult to anticipate, and accurately predicting it requires distinguishing what a prompt means to that audience from the reading the model would otherwise produce.
 
-Commonly-cited benchmarks, such as [SWE-bench](https://www.swebench.com/) and [HLE](https://lastexam.ai/), measure whether a model can produce a correct answer: multiple choice, math, and code. RTR-bench measures a different subject. There is no answer key in this data, only how people actually interpreted an ambiguous prompt, so a model is scored against the crowd rather than against a key. That is what makes it a test of whether a model can read the room.
+For each prompt, the model estimates the share of human submissions in all four quadrants. **`Predict`** asks for this distribution without historical examples. **`Learn`** lets the model choose and inspect up to three historical examples before submitting its prediction. Both tracks score the prediction against the recorded crowd distribution, out of 100. Learn also reports its gain or loss against Predict, showing whether gathering evidence improved the answer.
 
-**RTR-bench** is not a knowledge test, and that is deliberate. It is a prediction task. Given a prompt, a model has to decide how the crowd interpreted it, and then decide how it would interpret it itself. Both answers are scored against the real submissions.
+The scores are one way to compare models; their patterns of agreement and error are another. Which interpretations do models consistently anticipate? Where do different models make the same mistake? Which examples do they seek out, and when does that evidence help or mislead them? RTR-bench uses these questions to examine what it means for a model to read the room—with the room here defined by the participants in Mixed Messages.
+
+RTR-bench is intended to surface patterns of agreement, disagreement, and adaptation that warrant further investigation. Its results describe performance on this audience and prompt set; broader claims about cultural fluency require validation across other audiences and settings. Recorded submissions provide an observable prediction target, but do not establish participants’ underlying beliefs or subjective experiences.
 
 ## How it works
 
-[Mixed Messages](https://mixedmessages.fyi) publishes one prompt, every day. Over a 24-hour submission period, participants interpret that prompt by placing a coordinate on the grid. When the window closes, a rendering algorithm uses the (x, y) coordinates to create an artwork which depicts the day's collective interpretation. Each window has a dominant quadrant, as well as conviction and polarizations cores. The 2x2 grid gives us four possible options for the dominant quadrant, with the following implied readings and coordinates: 
+[Mixed Messages](https://mixedmessages.fyi) publishes one prompt, every day. Over a 24-hour submission period, participants interpret that prompt by placing a coordinate on the grid. When the window closes, a rendering algorithm uses the (x, y) coordinates to create an artwork which depicts the day’s collective interpretation. Each window has a distribution across four quadrants, along with conviction and polarization scores. The 2×2 grid gives us the following readings and coordinates:
 
 | Quadrant | Reading | Coordinates | Color |
 |---|---|---|---|
-| Honest x Uniting | Bringing us together, authentically | x ≥ 5,000, y < 5,000 | Green |
-| Honest x Dividing | Fracturing us, truthfully | x ≥ 5,000, y ≥ 5,000 | Purple |
-| Performed x Uniting | Creating false unity among us | x < 5,000, y < 5,000 | Cyan |
-| Performed x Dividing | Exploiting our divisions, deliberately | x < 5,000, y ≥ 5,000 | Red |
+| Honest × Uniting | Bringing us together, authentically | x ≥ 5,000, y < 5,000 | Green |
+| Honest × Dividing | Fracturing us, truthfully | x ≥ 5,000, y ≥ 5,000 | Purple |
+| Performed × Uniting | Creating false unity among us | x < 5,000, y < 5,000 | Cyan |
+| Performed × Dividing | Exploiting our divisions, deliberately | x < 5,000, y ≥ 5,000 | Red |
+
+**The dataset.** 39,828 submissions across 165 of the 174 closed windows, from the November 12, 2026 export. Only windows with at least 25 submissions are included. `data/windows.json` holds everything: the prompt, submission counts by quadrant, center of mass, and conviction and polarization scores. The latter measures are not used in scoring.
+
+The dataset is divided chronologically into 90 reference windows, 30 development windows, and 45 evaluation windows. Every reference window predates every target. Development outcomes are never added to the reference pool. The same data and frozen splits are available on [Hugging Face](https://huggingface.co/datasets/noahmclaughlin/RTR-bench).
+
+**The tasks.** In **`Predict`**, the model receives the target prompt, audience description, and quadrant definitions, then returns four probabilities totaling one. A prediction of 0.10, 0.50, 0.25, and 0.15 means it expects 10%, 50%, 25%, and 15% of the crowd’s submissions in the respective quadrants. It sees no historical examples or target outcomes. In **`Learn`**, the model can first request a catalog containing the names and IDs of the 90 reference prompts. It chooses an example to inspect and receives that example’s actual distribution and submission count. It can then choose another example or submit its prediction, up to three distinct inspections. The choices are sequential: evidence from one example can influence what it inspects next.
+
+Note that each **`Learn`** episode starts a fresh conversation. The model receives no target score or correctness feedback during the episode, and it can submit before exhausting its inspection budget. The environment allows eight actions and eight model turns. Learning happens through the conversation; the model’s weights are not updated.
+
+**Scoring.** For each target, sum the squared differences between the predicted and observed shares across all four quadrants. Average those errors equally across the declared targets, then calculate:
+
+```text
+Score = 100 × (1 − mean error / 2)
+Learn gain = Learn score − Predict score
+```
 
-**The dataset.** 39,828 submissions across 165 of the 174 closed windows, as of November 12th. Eight windows had fewer than 25 marks, which is the floor. `data/windows.json` holds everything: the prompt, the submission counts, the dominant quadrant, the center of mass, and the conviction and polarization scores, which are not used in scoring but may be included at a later date. 
+An exact match scores 100. Invalid or missing terminal predictions receive zero and remain in the denominator. Pending targets keep the run incomplete, without an aggregate headline score. Probabilities must be finite values between zero and one; sums within 0.001 of one are normalized.
 
-**The task.** For each window with N≥25 submissions, the model is asked to determine, given the prompt, which quadrant received the most submissions. Submission counts and dominant quadrants are never shown to the model.
+These scores are not percentage accuracy. Reports include uniform predictions and a constant distribution calculated from the reference windows, so a model’s score can be interpreted against simple baselines. Learn’s final score and its gain should be read together: improvement from a weak starting point does not necessarily produce a strong final prediction.
 
-**Scoring.** A right answer is one point. A wrong answer is zero points. The final score is calculated as the number of points divided by the number of prompts interpreted (165 as of 11/12/26), times 100. A score of 50 means that the model determined the correct dominant quadrant for half of the prompts. The four dominant quadrant options are rotated against four model calls so that no option gains from position. 3/4 matching answers is the model's final answer. A 2/2 split scores zero. 
+**Consistency.** Repeated runs record fresh predictions under the same declared settings. Their observed ranges describe variation between runs. Reports also include confidence intervals obtained by resampling target entity groups, preserving matched targets when calculating Learn’s gain. These intervals describe variation across targets; they do not measure repeated-run variability or uncertainty about independent people.
+
+The full comparison includes randomly selected examples and examples selected by a fixed text-similarity method. These are controls for Learn, not additional tracks. Predict supplies the no-example control without duplicate calls. See the [Predict protocol](docs/distribution.md) and [Learn protocol](docs/learn-room.md) for the exact prompts, limits, failure handling, and reproducibility rules.
 
 ## Quickstart
 
-Scoring needs no database access, just an OpenRouter API key.
+Run RTR-bench from Terminal using one of these three commands:
 
-### Test a model
+| Command | What it runs |
+|---|---|
+| `npx rtr-bench read-the-room <model>` | **Predict and Learn**, plus Learn's gain or loss against Predict |
+| `npx rtr-bench predict <model>` | **Predict only**: predict the crowd's distribution without examples |
+| `npx rtr-bench learn <model>` | **Learn only**: inspect historical examples, then predict; does not calculate Learn gain |
 
-Node 18 or newer. No clone, no install.
+`<model>` is a placeholder for the model's OpenRouter ID. Each command also needs the provider and run settings. **After the one-time setup below**, this complete example runs both tracks on **five development prompts**, with a **$0.25 spending cap**:
 
 ```bash
-mkdir rtr-run && cd rtr-run
-echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
+npx rtr-bench read-the-room z-ai/glm-5.3-flash \
+  --provider Relace \
+  --split development \
+  --limit 5 \
+  --max-usd 0.25
+```
+
+To run only one track, replace `read-the-room` with `predict` or `learn` and keep the rest of the command. `--limit 5` means five target prompts per selected track. The backslashes let one command continue across several lines; copy the whole block into Terminal.
+
+### Before your first run
+
+1. Install [Node.js](https://nodejs.org/) **22.18 or newer**, which includes `npx`. `npx` downloads and starts RTR-bench for you. You do not need to clone GitHub or run `npm install` or `npm ci` for this path. The dataset is included; no GPU or database setup is needed.
+2. Open Terminal in a folder where you want to keep your results. Any folder you choose is fine. Creating a new folder is optional and does not start an evaluation.
+3. Get an [OpenRouter API key](https://openrouter.ai/settings/keys) and add credits to that account. In your chosen folder, create a plain-text file named `.env` containing the line below, replacing `sk-or-your-key` with your key. Keep this file private.
+
+```text
+OPENROUTER_API_KEY=sk-or-your-key
+```
 
-# see ids and prices. costs nothing
-npx --yes rtr-bench catalog --limit 20
+Keep Terminal in that same folder and run the complete command above. The benchmark reads `.env`, sends requests through OpenRouter, and charges your OpenRouter account for model usage. `--max-usd` is a spending limit, not a fixed price; the run stops incomplete if it cannot continue within that limit.
 
-# run one model, then score it
-npx --yes rtr-bench z-ai/glm-5.3-flash
+These commands require RTR-bench **0.2.0 or newer** on npm. To select that exact release, replace `npx rtr-bench` with `npx rtr-bench@0.2.0`. GitHub updates and npm releases are published separately.
+
+### Read the results
+
+When the run finishes, Terminal prints each selected track's score and the path to `report.html`. Open that file in a browser to inspect the results. A combined `read-the-room` run also prints Learn's gain or loss against Predict.
+
+The benchmark automatically creates `results/learn-room/` inside your current folder and saves reports, requests, responses, action traces, and spending records there. You do not need to create the results folders yourself.
+
+Running the identical command again resumes its saved run rather than starting a fresh evaluation. Changing the model, selected tracks, target set, or settings creates a separate run and spending ledger. A small development run's predictions and budget are not carried into a larger run.
+
+### Run the final evaluation
+
+Use development prompts to check the setup. Once the settings are fixed, the following command runs **Predict, Learn, and both comparison controls on all 45 final evaluation prompts**, matching the conditions in the results table:
+
+```bash
+npx rtr-bench read-the-room z-ai/glm-5.3-flash \
+  --provider Relace \
+  --controls \
+  --split evaluation \
+  --limit 45 \
+  --max-usd 0.25
 ```
 
-`--yes` skips npx's install prompt. A run makes 1,320 calls, takes about four minutes on a cheap model, costs a few cents, and writes `results/` in the folder you created. Add `--concurrency 8` to go faster.
+`--controls` adds two comparisons: one supplies randomly selected historical examples, and the other supplies examples selected by text similarity. Without this flag, `read-the-room` runs only Predict and Learn. A development run is not comparable with the final evaluation results.
 
-By default OpenRouter may route each call to any host that serves the model, and hosts differ in quantisation. List them, with price and precision:
+To evaluate another model, replace `z-ai/glm-5.3-flash` with its OpenRouter model ID and select a provider that serves it. You can list models and inspect a model's providers with:
 
 ```bash
-npx --yes rtr-bench providers z-ai/glm-5.3-flash
+npx rtr-bench catalog --limit 20
+npx rtr-bench providers z-ai/glm-5.3-flash
 ```
 
-Pin one with `--provider Wafer`. A pinned run refuses to fall back, and the summary names the host that served the calls.
+Provider availability and pricing can change. Choose a spending cap appropriate for the selected model and provider; $0.25 is the cap for the GLM example, not a recommended budget for every model.
 
-### Work on the code
+### Use the GitHub source instead (optional)
 
-The repository runs the TypeScript sources directly, so this path needs Node 22.18 or newer.
+If you want to inspect or change the code, install [Git](https://git-scm.com/) and run:
 
 ```bash
 git clone https://github.com/1999labs/RTR-bench
 cd RTR-bench
-npm install
-echo 'OPENROUTER_API_KEY=sk-or-your-key' > .env
-node src/cli.ts catalog --limit 20
-node src/cli.ts run --models z-ai/glm-5.3-flash --concurrency 8
-node src/cli.ts report
+npm ci
 ```
 
-Rebuilding the dataset needs the project's database connection, so `npm run export` is included for inspection only.
+These commands download the project, enter its folder, and install its supporting software. If you already have a checkout, use that folder. Create `.env` there as described above, then replace `npx rtr-bench` with `node src/cli.ts` in the evaluation commands. `src/cli.ts` is the source file that starts the benchmark. This is an alternative to the npm path, not an additional setup step.
 
 ## Results
 
-| Model | Provider | Score | Cost |
-|---|---|---|---|
-| z-ai/glm-5.3-flash | Tenstorrent (fp8) | 41.2 | $0.0281 |
-| z-ai/glm-5.3 | Tenstorrent (fp8) | 40.0 | $0.2828 |
-| minimax/minimax-m3.1-flash | GMICloud (fp8) | 32.1 | $0.2444 |
-| xiaomi/mimo-v2.6-flash | GMICloud (bf16) | 40.6 | $0.1216 |
-| deepseek/deepseek-v4.1-flash-1017 | Morph (fp8) | 40.0 | $0.3414 |
-| tencent/hy3 | Tencent (fp8) | 40.0 | $0.2665 |
-| google/gemini-3.8-flash | Google AI Studio | 46.1 | $0.7790 |
-| openai/gpt-6-luna | OpenAI | 37.0 | $0.1329 |
-| anthropic/claude-haiku-5.5 | Claude Platform on AWS | 43.0 | $0.1696 |
-| thinkingmachines/inkling-small | DeepInfra (fp8) | 37.0 | $0.1733 |
-| openai/gpt-oss-120b | DekaLLM (bf16) | 32.7 | $0.0205 |
+| Model | Provider | Predict | Learn | Learn gain | Full run cost (including controls) |
+|---|---|---:|---:|---:|---:|
+| z-ai/glm-5.3-flash | Relace (fp4) | 87.93 | 85.57 | −2.37 | $0.03150 |
+| z-ai/glm-5.3 | TBD | TBD | TBD | TBD | TBD |
+| minimax/minimax-m3.1-flash | TBD | TBD | TBD | TBD | TBD |
+| xiaomi/mimo-v2.6-flash | TBD | TBD | TBD | TBD | TBD |
+| deepseek/deepseek-v4.1-flash-1017 | TBD | TBD | TBD | TBD | TBD |
+| tencent/hy3 | TBD | TBD | TBD | TBD | TBD |
+| google/gemini-3.8-flash | TBD | TBD | TBD | TBD | TBD |
+| openai/gpt-6-luna | TBD | TBD | TBD | TBD | TBD |
+| anthropic/claude-haiku-5.5 | TBD | TBD | TBD | TBD | TBD |
+| thinkingmachines/inkling-small | TBD | TBD | TBD | TBD | TBD |
+| openai/gpt-oss-120b | TBD | TBD | TBD | TBD | TBD |
+
+## Optional usage
+
+The examples below use the GitHub source checkout. For model evaluations through the npm package, replace `node src/cli.ts` with `npx rtr-bench`. The synthetic examples use fixture paths from the source checkout.
+
+### Run one track
+
+To run Predict alone:
+
+```bash
+node src/cli.ts predict z-ai/glm-5.3-flash \
+  --provider Relace \
+  --split development \
+  --limit 5 \
+  --max-usd 0.25
+```
+
+Replace `predict` with `learn` to run Learn alone. Each reports its own score; Learn’s gain requires the matching Predict result, which `read-the-room` calculates automatically. The standalone commands use the same prompts and scorer as their conditions in the combined run.
+
+Separate commands and changes to `--controls` create separate runs and spending ledgers; saved predictions are not automatically shared between them. Use `read-the-room --controls` from the outset when you want both tracks and the controls.
+
+The legacy `learn-room` and `distribution` commands remain available with their original defaults and saved-run formats. In particular, legacy `learn-room` without `--strategy` runs the no-example and random conditions. New commands use the existing `results/learn-room/` storage and report format; internal strategy labels are `none` (Predict), `agent` (Learn), `random`, and `similar`.
+
+### Synthetic runs
+
+Try Predict on the five-target fixture without an API key or any API spending:
+
+```bash
+node src/cli.ts predict \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --limit 5
+```
+
+To exercise Learn and all comparison conditions:
+
+```bash
+node src/cli.ts read-the-room \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --controls \
+  --limit 5
+```
+
+These commands write explicitly labeled synthetic reports with saved outputs and browsable traces. They demonstrate the pipeline, not model performance.
+
+Synthetic run directories are ignored by Git. Fixture inputs and tests stay tracked; regenerate the reports locally with the commands above. Published benchmark reports remain tracked and are marked as generated in `.gitattributes`.
 
-![RTR-bench: score against cost](results/leaderboard-scatter.png)
+### Providers and concurrency
+
+OpenRouter may route calls to different providers serving the same model. To list providers, token prices, and numerical precision:
+
+```bash
+node src/cli.ts providers z-ai/glm-5.3-flash
+```
+
+Use `--provider` to select a provider from that list. A pinned run refuses to fall back to another provider, and reports name the providers that served the calls. Provider availability can change.
+
+Predict and Learn currently execute requests sequentially. Each Learn episode contains a sequence of decisions, with the next request depending on the previous observation. Time and cost depend on the model, provider, token usage, and number of turns.
+
+On 45 targets, Predict and Learn together typically use about 270 model calls. Adding `--controls` brings that to about 360: 45 for Predict, 45 for each automatic control, and approximately five calls per Learn episode. Early stopping can reduce that number; additional agent turns and transport retries can increase it. The first final GLM run used 347 calls.
+
+The public commands default to 4,000 output tokens per response. Learn uses the published `transfer-check-v1` policy, up to three inspected examples, and eight model turns. Advanced runs can override `--max-tokens` and, for Learn, `--agent-policy`.
+
+The current client uses OpenRouter. Direct connections to other hosted or local endpoints are not yet implemented.
+
+## Development
+
+After the GitHub source setup in [Quickstart](#quickstart), use these commands from the project folder to check changes and build the distributable program. Source commands run TypeScript directly and require Node.js 22.18 or newer.
+
+```bash
+npm test
+npm run build
+
+node src/cli.ts read-the-room \
+  --fixture tests/fixtures/learn-room-four-strategies.json \
+  --controls \
+  --limit 5
+```
 
-Eleven models were tested against the same 165 windows. Scores concentrated in the high 30s and low 40s. Models were more alike in how they fail than in how they score. Across 1,591 answers, they decided on Honest x Uniting 67.7% of the time and Honest x Dividing 18.2% of the time, while the dominant quadrant was Honest x Dividing in 39 of those 165 windows. Division, when a model registers it at all, reads as spin or as bad faith, and almost never as candor. 
+`npm test` checks the implementation. `npm run build` bundles the source into `dist/cli.js`, the program included in the npm package. The fixture command generates a synthetic report without making model API calls. For paid runs, use the Quickstart commands. See the [npm release instructions](docs/npm-release.md) for packaging and publishing.
 
-There were seven prompts that no model got right. They were Donald Trump, Elon Musk, Las Vegas, Barack Obama, Reputation, Dreams, and Honor. The dominant quadrant for each of them besides Dreams was Honest x Dividing. There were three prompts that every model got right. They were Handwritten Notes, The CIA, and Altruism. 
+Rebuilding the original dataset requires the project’s database connection and the configuration in `scripts/export.mjs`. The exported dataset is already included, so running the benchmark does not require database access. The Hugging Face export and validator preserve the frozen source data and split assignments.
 
-My main takeaway is that the same blind spot shows up in every model, and it is not a vocabulary problem. Ask it to pick the dominant quadrant for Trump or Musk and the model can easily choose dividing instead of uniting, but has a hard time determining honest or performed. That is why Honest X Dividing is the quadrant the models got wrong the most. Next, I would like to score more closed frontier models, score only the windows where the crowd was very decisive, and somehow factor into the benchmark the conviction and polarization scores from each window. 
+RTR-bench previously evaluated dominant-quadrant predictions. Those scores use a different protocol and are not directly comparable with Predict and Learn. See the [original benchmark README and results](results/history/original-benchmark/README.md).
 
 ## License
 
-MIT for the code. CC0 for `data/windows.json`.
+MIT for the code. CC0 for `data/windows.json` and its derived dataset exports.

package/data/splits/v1.json

@@ -0,0 +1,184 @@
+{
+  "version": "chronological-v1",
+  "datasetSha256": "ec9d7582b5610a24f4e14dbbb05d13c636b692ec9e133b9381db0fdf483ffb2b",
+  "rule": "Reference: day <= 99; development: days 100..129; evaluation: day >= 130. All reference windows precede all targets. No labels used to choose boundaries.",
+  "grouping": {
+    "method": "Trim, lowercase, collapse whitespace; manually reviewed prompt list for entity aliases.",
+    "aliases": {},
+    "decisions": [
+      "ELON MUSK (days 30 and 99) stays entirely in reference.",
+      "Related but distinct prompts (such as CONTENT CREATORS and INFLUENCERS) remain distinct; grouping prevents same-entity retrieval, not thematic overlap."
+    ]
+  },
+  "reference": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    28,
+    29,
+    30,
+    31,
+    35,
+    36,
+    37,
+    38,
+    39,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    54,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    70,
+    71,
+    72,
+    73,
+    74,
+    77,
+    78,
+    79,
+    80,
+    81,
+    82,
+    83,
+    84,
+    85,
+    86,
+    87,
+    88,
+    89,
+    90,
+    91,
+    92,
+    93,
+    94,
+    95,
+    96,
+    97,
+    98,
+    99
+  ],
+  "development": [
+    100,
+    101,
+    102,
+    103,
+    104,
+    105,
+    106,
+    107,
+    108,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    116,
+    117,
+    118,
+    119,
+    120,
+    121,
+    122,
+    123,
+    124,
+    125,
+    126,
+    127,
+    128,
+    129
+  ],
+  "evaluation": [
+    130,
+    131,
+    132,
+    133,
+    134,
+    135,
+    136,
+    137,
+    138,
+    139,
+    140,
+    141,
+    142,
+    143,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    150,
+    151,
+    152,
+    153,
+    154,
+    155,
+    156,
+    157,
+    158,
+    159,
+    160,
+    161,
+    162,
+    163,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    170,
+    171,
+    172,
+    173,
+    174
+  ]
+}