rtr-bench 0.1.4 → 0.2.0

package/docs/distribution.md

@@ -0,0 +1,102 @@
+# Distribution prediction protocol v1
+
+For the public Predict track, use `node src/cli.ts predict <model>`; see the [Quickstart](../README.md#quickstart). This uses the shared Learn runner’s no-example condition with a 4,000-token default and support for `--max-usd`. `read-the-room <model>` runs Predict and Learn together. The legacy `distribution` command and its fixture format remain available unchanged.
+
+This is a separate RTR-bench track. The original winning-quadrant benchmark retains its prompts, scoring, and result paths. [Learn the Room](learn-room.md) reuses this protocol's probability validation and scorer for its interactive environment and initial baseline comparisons.
+
+## Dataset and frozen split
+
+`data/splits/v1.json` pins the exact `data/windows.json` bytes with SHA-256. The loader checks counts, complete coverage, disjoint groups, and chronology before a run or report.
+
+| Partition | Days | Windows | Purpose |
+|---|---|---:|---|
+| Reference | 1–99, included dataset windows only | 90 | Estimate the constant baseline; future evidence pool |
+| Development | 100–129 | 30 | Prompt and implementation development |
+| Evaluation | 130–174 | 45 | Final comparison after settings are fixed |
+
+The cutoffs use chronology, not crowd labels. Both Elon Musk windows (days 30 and 99) remain in reference. Prompt grouping trims, lowercases, and collapses whitespace, with an explicit alias map for reviewed same-entity variants. The current alias map is empty after reviewing the prompt list. Related concepts are not collapsed merely because they are thematically similar.
+
+Every reference window predates every target. Development outcomes never enter the reference baseline or later evidence pool. This public dataset has already been used for the original benchmark: these partitions control evidence access and tuning in the new experiment, but do not establish unseen training data. The chronological split may also expose changes in the audience over time.
+
+No separate training partition or RL experiment is defined in v1. A limited pilot selects the first N windows of its declared partition and records those IDs before execution.
+
+## Output and scoring
+
+Return a JSON object containing exactly four numeric probabilities:
+
+```json
+{
+  "performed+uniting": 0.20,
+  "honest+uniting": 0.30,
+  "performed+dividing": 0.40,
+  "honest+dividing": 0.10
+}
+```
+
+Values must be finite numbers in [0, 1]. No string coercion, prose, code fences, extra keys, or percentage values are accepted. A sum within 0.001 of one, inclusive, is normalized to one; the original sum and raw response are retained. Larger deviations fail validation.
+
+For each target, derive the observed shares from integer quadrant counts divided by submissions. Sum the four squared differences between predicted and observed shares. Average errors equally across the declared targets, then report `100 * (1 - mean_error / 2)`. A score of 100 is an exact match; 0 is the global worst case. This is not percentage accuracy, and 50 is not a chance baseline.
+
+Final invalid, missing, or API-failed predictions receive penalty 2 (zero points) and stay in the denominator. The report distinguishes an assigned failure penalty from a measured distribution error. Pending targets make a run incomplete; an incomplete run has no aggregate headline score. Reports separately show valid-response error, invalid-output rate, overall failure rate, and valid/invalid/missing/pending counts.
+
+Secondary winning-quadrant accuracy accepts every crowd co-winner. When a prediction has K exactly tied maxima, its credit is the fraction of those K quadrants that are crowd co-winners. This is the expected accuracy of a uniform tie-break, without introducing randomness. A uniform prediction against one crowd winner earns 0.25 credit.
+
+Always report both baselines on the exact same targets:
+
+- Uniform: 0.25 per quadrant.
+- Constant reference: the arithmetic mean of the 90 reference-window distributions. Each reference window has equal weight, irrespective of submission count.
+
+Report gains over baselines in score points. Confidence intervals use 2,000 bootstrap samples of entity groups with seed 7. Paired gains resample matching target differences together. These intervals describe variation across targets, not uncertainty about independent people in this self-selected audience. Five-target pilots are setup checks, not model rankings.
+
+## Calls, failures, and reproducibility
+
+The v1 prompt is frozen in `src/tracks/distribution/prompt.ts`. Only the target word, audience description, quadrant definitions, and output instructions enter the API messages. The runner sends neither reference labels nor target labels for direct distribution prediction.
+
+API runs use temperature 0, a 2,000-token output cap, and one prediction per window. There are no option rotations or automatic prompt-repair attempts. A requested provider is pinned with fallback disabled. Unpinned runs record actual serving providers. Unsupported settings fail explicitly rather than silently changing the request.
+
+Each target has at most three attempts. Only transport failures and HTTP 429/500/502/503/504 are retried, with 1- and 2-second backoffs and a 60-second timeout per attempt. Successful API responses with malformed or absent model content are terminal failures. An interrupted attempt consumes an attempt slot, since its API outcome and charge may be unknown.
+
+Each request and every attempt are saved, including raw responses, errors, provider, timestamps, and available usage. Reports sum known cost and tokens over all attempts and count attempts with unknown cost or tokens separately. Fixture runs make zero API calls and are visibly labeled synthetic.
+
+A deterministic run ID hashes dataset and split fingerprints, target/reference IDs, model and requested provider, prompt and scorer versions, generation/retry settings, and fixture contents when applicable. Identical commands resume terminal records, including failures, without rerolling them. Changing a target set or configuration produces a separate run. Increment protocol/runner versions when changing behavior.
+
+Writes use a temporary file and rename. An exclusive `run.lock` prevents simultaneous writers. After an unclean process exit, first confirm the recorded PID is no longer running, then remove that run's stale lock before resuming. Saved API responses can be recovered without another call. These controls reproduce configuration and scoring, not necessarily identical outputs from nondeterministic model services.
+
+## Commands
+
+Source commands require Node.js 22.18 or newer. The bundled CLI can also run these commands after `npm run build`.
+
+Run the five-target, zero-cost fixture pilot:
+
+```bash
+node src/cli.ts distribution --fixture tests/fixtures/distribution-pilot.json --limit 5
+```
+
+The fixture intentionally includes three valid outputs, one invalid sum, and one empty response. It is a pipeline demonstration, not a model result.
+
+For a future paid API pilot, after selecting a model/provider:
+
+```bash
+node src/cli.ts distribution <model-id> --split development --limit 5 --provider <provider-name>
+```
+
+Omit `--limit` to cover the full partition. Use `--split evaluation` only after development decisions are frozen. `--output <directory>` changes the output root, which defaults to `results/distribution/`. Each run directory contains `manifest.json`, per-target `records/`, and JSON/Markdown/HTML reports. The HTML report supports prompt filtering and inspection of every request and attempt.
+
+Regenerate a report without network access:
+
+```bash
+node src/cli.ts distribution-report <run-directory>
+```
+
+Verify locally without API spending:
+
+```bash
+npm test
+npm run build
+```
+
+## Evaluation status
+
+Both tracks have completed development validation and a [first final evaluation](../results/learn-room/z-ai-glm-5-3-flash-e9d0327447ef1856/README.md) on all 45 frozen evaluation targets. GLM 5.3 Flash through Relace scored **87.93/100** for direct distribution prediction, using the identical no-example condition in Learn the Room without duplicate calls. All 45 direct predictions were valid. Uniform and constant reference-distribution baselines scored 85.39 and 85.56, respectively; these scores are not percentage accuracy.
+
+The same run evaluated all three evidence conditions, for 180 valid predictions total. The transfer-check agent scored 85.57; see the linked report for paired comparisons, uncertainty, usage, and the frozen protocol. The original RTR score remains separate. Future models should use the same declared settings; any policy development informed by these final outcomes must disclose that this holdout has been used.

package/docs/learn-room.md

@@ -0,0 +1,186 @@
+# Learn the Room protocol v2
+
+Learn the Room is a separate evaluation track built on the [distribution protocol](distribution.md). It uses the same chronological split, probability validation, failure penalties, equal-window weighting, and score out of 100. This implementation provides a controlled interactive environment, four evidence strategies, resumable runs, and comparison reports. Strategies are version 2; the runner is version 3 with provider Retry-After handling and optional spending enforcement. The environment and distribution scorer remain version 1. Saved earlier reports remain replayable.
+
+## Public commands
+
+Use `node src/cli.ts read-the-room <model>` for Predict (`none`) and Learn (`agent`), `predict <model>` for Predict alone, or `learn <model>` for Learn alone. Add the boolean `--controls` flag to `read-the-room` for all four strategies. These commands default to 4,000 output tokens per response and, when running Learn, `transfer-check-v1`. They share the existing runner, spending enforcement, reports, and `results/learn-room/` storage. Use `--max-usd` and `--provider` for a capped paid run; see the [Quickstart](../README.md#quickstart).
+
+The legacy `learn-room` command retains its defaults: 2,000 tokens, the original agent policy, and `--strategy both` meaning `none` plus `random`. Its new `--strategy tracks` selects `none` plus `agent`; `--strategy all` still selects all four conditions. New commands do not accept `--strategy`. A combined run reports Learn’s paired gain; Learn alone does not calculate a gain without a Predict baseline. Separate strategy sets have separate run identities and do not share saved predictions or budgets.
+
+## Environment contract
+
+`createEpisode(target, reference, options)` returns separate `agent` and `evaluator` capabilities. A policy receives only `agent`. The evaluator owns the hidden target distribution, scoring, and trace access. Each episode copies its inputs, has independent state, and rejects references that repeat the target entity, are not earlier than the target, or duplicate an ID.
+
+`agent.observe()` supplies the target ID and prompt, quadrant names, inspection budget, and action limit. `agent.act(action)` accepts these JSON actions:
+
+```json
+{"type":"list_examples"}
+{"type":"inspect_example","id":1}
+{"type":"submit_prediction","prediction":{"performed+uniting":0.2,"honest+uniting":0.3,"performed+dividing":0.4,"honest+dividing":0.1}}
+```
+
+| Operation | Observation | Budget behavior |
+|---|---|---|
+| `list_examples` | Reference IDs and prompt text only | Uses one action |
+| `inspect_example` | Selected reference distribution and submission count | Uses one action and, on first successful inspection, one inspection |
+| `submit_prediction` | Validation acknowledgement and terminal status | Uses one action and ends the episode |
+
+Default limits are three distinct inspections and eight total actions. The CLI permits inspection budgets from zero to three. Invalid actions, unknown IDs, repeated catalog requests, and denied inspections all consume an action. A duplicate successful inspection returns cached evidence, consumes another action, and does not consume another inspection. An over-budget request reveals no additional evidence.
+
+A valid submit action with invalid probabilities ends the episode with zero points. Reaching the action limit without a prediction also ends it with zero points. A valid prediction on the final allowed action is accepted. A policy may submit before using its inspection budget. Once ended, further actions are rejected without changing the outcome or trace.
+
+Action observations contain validation status and remaining budgets, but never the target distribution, score, reward, or correctness. The host reads `evaluator.outcome()` only for reporting. `evaluator.abort(reason)` records host termination when a prediction cannot be obtained; it is logged separately from policy actions. Every accepted in-episode action attempt and its observation is recorded. Returned values are copies, so changing an observation cannot alter environment state.
+
+This is an API boundary for model evaluation, not a sandbox for arbitrary untrusted local code. A model adapter must receive only the agent-facing observations and must not have filesystem access, the full dataset, evaluator methods, or reports in its context.
+
+## Four strategies
+
+- **None:** no catalog or examples are supplied to the model. Its prediction messages exactly match direct distribution prediction, and the controller submits the resulting probabilities to the environment.
+- **Random:** the controller lists the catalog, selects the configured number of distinct examples without replacement, inspects them through the environment, and supplies only those observations with the prediction instructions. The model makes one final prediction.
+- **Similar:** the controller ranks reference prompt text by the fixed TF–IDF cosine method below, inspects the highest-ranked examples, and supplies them with the same final prediction instructions as Random.
+- **Agent:** the model chooses one environment action per turn, sees the observation, and chooses its next action. It may stop early or submit without inspecting any examples. Each episode starts with a fresh conversation, and only permitted observations enter that conversation.
+
+The random selector uses SHA-256 of `seed:targetDay` to initialize a 32-bit linear congruential generator, followed by a Fisher–Yates shuffle of sorted reference IDs. The default seed is 7. Selection is independent of execution order, run limits, and other episodes. The random algorithm is unchanged from `learn-room-strategies-v1`; the expanded strategy set is versioned as `learn-room-strategies-v2`.
+
+None, Random, and Similar are automatic baseline controllers. Agent selects evidence interactively. None of the selectors receives target outcomes.
+
+### Similarity method: word-char3-tfidf-v1
+
+Normalize prompt text with Unicode NFKC and lowercase it. Tokenize contiguous Unicode letters/numbers. Features are whole tokens plus length-three character sequences from each token padded with `^` and `$`. Word and character features use separate namespaces. For each feature, weight its count with `tf = 1 + ln(count)` and `idf = ln((N + 1) / (document_frequency + 1)) + 1`, where N is the number of reference prompts. Vocabulary and document frequencies are fitted only on reference prompt text. Drop target features outside that vocabulary, L2-normalize the vectors, and rank by cosine similarity. Break equal similarities by ascending reference ID, including all-zero overlap cases.
+
+This is a lexical similarity baseline, not a semantic embedding model. It uses no model service, crowd counts, distributions, or target labels. Keep this limitation visible when interpreting agent gains over it.
+
+### Interactive adapter: learn-room-agent-v1
+
+The adapter uses a plain JSON action protocol over chat completions. Each successful model response is parsed as one action and passed to `agent.act`. Its raw response is added as an assistant message and the environment observation as a user message. Native provider tool-calling support is not required. The initial request contains the target prompt, quadrant definitions, action schema, inspection/action limits, and model-turn limit. The agent must call `list_examples` to see the catalog.
+
+Malformed JSON and empty successful responses become invalid actions and receive feedback; they consume one model turn and one action. This allows bounded recovery on a later turn. A structurally valid submit action with invalid probabilities is terminal, as in every strategy. Transport failures are handled separately and never masquerade as evidence or actions. The model never receives target scores, rewards, or evaluation reports.
+
+The default maximum is eight model turns, independent of the eight-action environment limit. `--max-model-calls` can lower the turn limit to 1–8. Each turn has at most three HTTP attempts, so an agent episode makes at most 24 HTTP attempts under the defaults. Failed and interrupted attempts count toward that bound. Once the model-turn limit is reached without submission, the host ends the episode with zero points. Early successful submission stops immediately.
+
+`--max-tokens <positive-integer>` sets the output-token allowance for every baseline request and every agent turn; it defaults to 2,000. It does not change the number of turns. Providers may consume this allowance on reasoning before returning final content. A truncated response follows the existing validation rules: empty baseline content is missing, malformed baseline content is invalid, and empty or malformed agent actions receive bounded feedback. A length stop does not trigger a transport retry. Changing the allowance creates a separate run because `settings.maxTokens` is fingerprinted, and the spending guard reserves the configured output allowance before each attempt. Existing default runs retain their identities and resume behavior.
+
+### Optional agent policy: transfer-check-v1
+
+`--agent-policy default` uses the original agent instructions. Omitting the flag is identical, including the run identity and exact request contents. The original prompt and hash remain unchanged so saved runs continue to replay and resume.
+
+`--agent-policy transfer-check-v1` appends an experimental evidence-use instruction to the original system prompt. It asks the agent to assess transfer separately for performed versus honest and uniting versus dividing, consider differences between an individual and a broader category, preserve alternative interpretations of ambiguous targets, and discount references that support only a weak analogy. It explicitly retains a joint four-quadrant prediction rather than assuming statistical independence between axes. The policy contains no target-specific examples or crowd outcomes.
+
+The flag requires `--strategy agent` or `--strategy all`. It changes only the agent system message; automatic baseline requests, available actions, evidence pool, budgets, generation settings, and scoring are unchanged. The adapter protocol remains `learn-room-agent-v1`. The optional `agentPolicy` field and policy-specific `agentPromptHash` distinguish the new run and are checked during offline replay. Unknown policies and mismatched hashes are rejected rather than falling back. Reports label the selected non-default policy. Explicit `default` is normalized to the original manifest format.
+
+With `--strategy all`, selecting the new policy creates a new run for all four conditions; identical baseline requests are fresh calls in API mode, not automatically reused from an older run. Resume avoids rerolling completed responses only within the matching run configuration.
+
+Exercise the policy offline with the existing fixture scenarios:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-four-strategies.json --strategy all --limit 5 --max-tokens 4000 --agent-policy transfer-check-v1
+```
+
+Fixtures verify prompt routing, unchanged action and inspection limits, failures, early stopping, resume, and isolation of target labels. They cannot show that the instruction improves model behavior. In the completed [three-run development comparison](../results/learn-room/comparisons/glm-5-3-flash-policy-repeats/README.md), the new agent averaged 85.67 (observed range 85.48–85.85), versus the original policy's mean of 83.66. It still trailed its matching no-example control in every run, by 1.67 points on average, and individual regressions remain. These sequential development runs do not establish a causal policy effect or held-out performance. The original policy remains the default. The [first final evaluation](../results/learn-room/z-ai-glm-5-3-flash-e9d0327447ef1856/README.md) is now complete on all 45 held-out targets: direct distribution scored 87.93 and the transfer-check agent 85.57, with 180/180 valid predictions. The agent gain was -2.37 points (exploratory paired 95% interval -5.14 to +0.12); this run did not demonstrate an average benefit from evidence gathering. These final outcomes must not be used for tuning and then described as an untouched holdout. The motivation is documented in the [offline evidence audit](../results/learn-room/comparisons/glm-5-3-flash-4000-repeats/audit.md).
+
+## Scores and comparison
+
+For each completed episode, use the shared `100 * (1 - error / 2)` score. Invalid and missing predictions receive error 2. There is no inspection or compute penalty. Aggregate targets equally, and keep each track's result separate from the original RTR score.
+
+`--strategy all` evaluates all four strategies on identical declared targets, reference windows, model/provider, and generation settings. For compatibility, the default remains `both` (None and Random). Reports include each strategy's final score, error, validity counts, secondary winning-quadrant accuracy, evidence usage, logical model turns, API attempts, known token/cost totals, and unknown-usage counts. Agent uses the same audience and quadrant definitions, but its action instructions and multi-turn context differ from the one-shot baselines; comparisons do not isolate selection quality from additional inference compute.
+
+The paired comparison is:
+
+```text
+score_gain = strategy_score - control_score
+error_reduction = control_error - strategy_error
+score_gain = 50 * error_reduction
+```
+
+A gain is in points and may be negative. Reports compare Random, Similar, and Agent against None, and Agent against Random and Similar. Confidence intervals resample matching target differences by entity group, using 2,000 samples and seed 7. These exploratory intervals are not corrected for multiple comparisons. Both final quality and gain are shown. Uniform and constant reference-distribution baselines are also shown. No gains are reported until the declared conditions are complete. Incomplete conditions have no headline score.
+
+A fixture comparison demonstrates arithmetic and trace handling; it provides no evidence that random examples improve a real model.
+
+## Runs, retries, and traces
+
+Each baseline episode permits one logical prediction request and at most three HTTP attempts. Agent permits the bounded turns described above, with at most three HTTP attempts per turn. The shared client retries only transport failures or HTTP 429/500/502/503/504, with a 60-second timeout per attempt. Backoff is the larger of the provider's Retry-After hint and the default one-/two-second delay. Retry hints can come from HTTP headers or saved OpenRouter error metadata. Malformed baseline predictions are terminal and are not rerolled; malformed agent actions receive bounded feedback instead. Provider pinning disables fallback. Unsupported settings fail explicitly.
+
+Each run manifest fingerprints the dataset, split, target/reference IDs, model/provider, strategies, budget, seed, prompts, environment/scorer/runner versions, generation/retry settings, and fixture contents. Runs live under `results/learn-room/`. Each episode records its exact prediction request, every API attempt, the action/observation trace, and its outcome. Unknown usage from interrupted calls remains explicitly unknown. Fixture mode makes zero API calls.
+
+### Optional spending limit
+
+Use `--max-usd <amount>` with `--provider <name-or-endpoint-tag>` to enforce a per-run API usage budget. Before execution, the runner fetches public endpoint metadata, chooses an available compatible endpoint, and records its exact tag, context limit, and token price bounds in the manifest. Requests pin that tag with fallback disabled and send OpenRouter maximum-price ceilings. An exact endpoint tag is preferable for a reviewed cost estimate. Nonzero per-request charges, unknown nonzero pricing fields, malformed rates, and unsupported pricing tiers are rejected.
+
+Existing fixed-price endpoints retain the `reserved-context-v1` policy and their original run identities. The `reserved-text-v2` policy supports prompt-length price tiers and endpoints that lack temperature control. It records the catalog pricing, model, temperature capability, and output limit. Reservations use the highest input/cache-read/cache-write rate and highest completion/reasoning rate across all tiers; discounts are not applied a second time. Reasoning counts toward output usage. This follows OpenRouter's [reasoning accounting](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) and conservatively covers [cache writes](https://openrouter.ai/docs/guides/best-practices/prompt-caching). The policy permits only plain-text messages and the runner's bounded request fields; tools, plugins, media, explicit caching controls, and additional request options are rejected before dispatch. Catalog prices for unused media and web search do not apply to these requests.
+
+For budgeted runs, supported endpoints still receive temperature zero. If the selected endpoint does not advertise temperature support, all four conditions omit that parameter and the manifest records `settings.temperature: null`. Markdown and HTML reports explicitly label the provider default and its difference from temperature-zero runs. This is a generation-setting exception, not a change to prompts, evidence selection, scoring, or the frozen split. The November 14, 2026 catalog snapshots in `tests/fixtures/provider-pricing-2026-11-14.json` exercise this exception for GPT-6 Luna and Claude Haiku 5.5; Gemini 3.8 Flash on Google AI Studio retains temperature zero. Offline tests do not establish live inference availability.
+
+Before each HTTP attempt, the runner durably reserves the endpoint's full input-context allowance plus the request's maximum output tokens at those prices. This deliberately overestimates the short text prompts. Reported cost replaces the reservation when available; unknown costs and interrupted requests retain the full reservation. Explicit pre-generation HTTP 429 rate-limit rejections, with no generation ID or choices, release the reservation as unbilled under [OpenRouter's pricing policy](https://openrouter.ai/pricing); that basis is saved in the ledger. This exception never treats network failures or empty successful responses as free. Accounting rounds upward to billionths of a dollar and persists in `spending.json` across resumes. The run stops before dispatch if the next reservation would exceed the cap. An unexpected provider charge above the reservation stops further calls for review.
+
+Budget stops leave the evaluation incomplete and write an incomplete report; they do not convert unattempted targets into benchmark failures. The cap covers this run's API usage, not unrelated account activity or credit-purchase fees. Changing configuration creates a new run with a separate ledger; do not interpret multiple run caps as a shared campaign budget. Fixture runs require no spending policy.
+
+Identical commands resume saved terminal episodes, including failures. Each agent turn stores its exact request, all attempts, and whether its action has been applied. On resume, the adapter reconstructs prior messages and observations, checks request/trace identity, and applies a durably saved but unapplied response exactly once. It does not rerun completed model turns. Interrupted attempts consume the fixed attempt budget. Atomic record writes and an exclusive run lock prevent partial writes and concurrent writers. Remove a stale lock only after confirming that its recorded process has exited.
+
+Use `--replicate <id>` to record a fresh run at otherwise identical settings. IDs contain 1–64 letters, numbers, underscores, or hyphens. The ID enters the manifest and run fingerprint only; it does not enter model requests, change the random-example seed, or alter evidence selection. Reusing the same ID and settings resumes that replicate without rerolling saved results. Omitting the flag preserves existing run identities. Separate replicates have separate spending ledgers: for a combined budget, assign per-run caps whose sum does not exceed the authorized total. Do not count repeat predictions on the same targets as additional independent target observations.
+
+For example, after authorizing two repeats with a combined $0.25 cap, run each sequentially with its own $0.125 cap:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --replicate repeat-2 --max-usd 0.125
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --replicate repeat-3 --max-usd 0.125
+```
+
+Offline reporting reconstructs evidence access and predictions from the frozen configuration and saved raw responses, verifies the saved request and trace, and recomputes scores rather than trusting cached outcomes. A missing episode file remains pending, with no invented observation trace.
+
+## Commands
+
+Source commands require Node.js 22.18 or newer. The bundled CLI supports the same commands after `npm run build`.
+
+Run the five-target, four-strategy fixture comparison with no API spending:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-four-strategies.json --strategy all --limit 5
+```
+
+The fixture intentionally contains valid, invalid, and missing predictions. Agent scripts exercise early stopping, invalid-action recovery, denied over-budget inspections, repeated inspections, and exhausted action budgets. Each baseline strategy maps target day to one raw response string. Agent maps each target day to an array of raw JSON action strings; an exhausted script supplies empty responses until the episode ends. No script is executed as code. Reports are explicitly labeled synthetic. All conditions use the same five development targets; fixture gains are not evidence of real model improvement.
+
+Select a single condition or a different evidence budget and random seed:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-pilot.json --strategy random --budget 2 --seed 42 --limit 5
+```
+
+For a future authorized model pilot:
+
+```bash
+node src/cli.ts learn-room <model-id> --provider <provider-name> --strategy all --split development --limit 5
+```
+
+For example, a five-target GLM pilot capped at $0.25:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Tenstorrent --strategy all --split development --limit 5 --max-usd 0.25
+```
+
+For a separately authorized development comparison with a 4,000-token allowance:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --max-usd 0.25
+```
+
+Keep the other settings fixed when comparing output allowances. A single run at each allowance cannot separate the allowance's effect from run-to-run variation; use paired results as development evidence, not a causal estimate.
+
+Use `--output <directory>` to choose an output root. Use `--split evaluation` only once development decisions are frozen. Omitting `--limit` runs the entire selected partition and requires corresponding fixture entries in fixture mode.
+
+Regenerate the JSON, Markdown, and searchable HTML reports offline:
+
+```bash
+node src/cli.ts learn-room-report <run-directory>
+```
+
+Run all tests and build:
+
+```bash
+npm test
+npm run build
+```
+
+## Embedding an interactive policy
+
+In the source checkout, a host can import `createEpisode` from `src/tracks/learn-room/environment.ts`. Construct the episode from the frozen split, then pass only `episode.agent` to a policy. The policy can inspect the catalog, choose references based on previous observations, and submit early. Keep the evaluator and dataset in the host. Persist all policy/model calls alongside `episode.evaluator.trace()`, and use `episode.evaluator.outcome()` for the final score. The included adapter in `agent.ts` demonstrates bounded model turns and replay; custom policies must enforce equivalent call limits.

package/docs/npm-release.md

@@ -0,0 +1,37 @@
+# Publishing RTR-bench 0.2.0
+
+This release adds the public `read-the-room`, `predict`, and `learn` commands to the npm package, along with the frozen dataset, reports, and spending controls. It includes compatibility for tiered text pricing and records when a provider does not support temperature control. Node.js 22.18 or newer is required.
+
+Publishing to npm is separate from committing and pushing to GitHub. These instructions are for the package maintainer, using an npm account with permission to publish `rtr-bench`.
+
+## Prepare and check
+
+From the repository folder:
+
+```bash
+git pull --ff-only
+npm ci
+npm test
+npm pack --dry-run
+```
+
+The package version in both `package.json` and `package-lock.json` must be `0.2.0`. `npm pack --dry-run` builds the bundled CLI and lists what would be included without publishing anything. The existing `prepack` script runs `npm run build` automatically for both packing and publishing.
+
+The package includes `dist/cli.js`, the dataset and frozen split, protocol documentation, and synthetic fixtures. API keys, `.env`, local evaluation outputs, and `node_modules` must not be included. Tests and synthetic runs make no paid model calls.
+
+## Publish
+
+```bash
+npm login
+npm publish --access public
+```
+
+Follow npm's browser sign-in and any authentication prompts. The second command publishes version 0.2.0 to the public registry and makes it the default `latest` release. An already-published name/version cannot be overwritten. See npm's [login](https://docs.npmjs.com/cli/v11/commands/npm-login/) and [publish](https://docs.npmjs.com/cli/v11/commands/npm-publish/) documentation.
+
+## Verify
+
+```bash
+npm view rtr-bench@latest version
+```
+
+This should print `0.2.0` once the registry has updated. Users can then run `npx rtr-bench@0.2.0 read-the-room ...` using the model, provider, split, and spending limit described in the README. A real model evaluation requires an OpenRouter key and incurs API charges; publishing the package does not start an evaluation.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rtr-bench",
-  "version": "0.1.4",
+  "version": "0.2.0",
   "description": "Can a language model predict how a crowd reads a word? A cultural fluency benchmark built from Mixed Messages.",
   "type": "module",
   "license": "MIT",
@@ -17,7 +17,7 @@
     "generative-art"
   ],
   "engines": {
-    "node": ">=18"
+    "node": ">=22.18.0"
   },
   "bin": {
     "rtr-bench": "dist/cli.js"
@@ -25,11 +25,18 @@
   "files": [
     "dist/",
     "data/",
+    "docs/distribution.md",
+    "docs/learn-room.md",
+    "docs/npm-release.md",
+    "tests/fixtures/distribution-pilot.json",
+    "tests/fixtures/learn-room-pilot.json",
+    "tests/fixtures/learn-room-four-strategies.json",
     "results/leaderboard-scatter.png",
     "README.md",
     "LICENSE"
   ],
   "scripts": {
+    "test": "node --test tests/*.test.ts tests/distribution/*.test.ts tests/learn-room/*.test.ts",
     "build": "esbuild src/cli.ts --bundle --platform=node --format=esm --outfile=dist/cli.js --external:node:*",
     "prepack": "npm run build",
     "export": "node scripts/export.mjs",

package/tests/fixtures/distribution-pilot.json

@@ -0,0 +1,7 @@
+{
+  "100": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.3, \"performed+dividing\": 0.4, \"honest+dividing\": 0.1}",
+  "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+  "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+  "103": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.2, \"performed+dividing\": 0.2, \"honest+dividing\": 0.2}",
+  "104": ""
+}

package/tests/fixtures/learn-room-four-strategies.json

@@ -0,0 +1,58 @@
+{
+  "none": {
+    "100": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.3, \"performed+dividing\": 0.4, \"honest+dividing\": 0.1}",
+    "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+    "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+    "103": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.2, \"performed+dividing\": 0.2, \"honest+dividing\": 0.2}",
+    "104": ""
+  },
+  "random": {
+    "100": "{\"performed+uniting\": 0.35, \"honest+uniting\": 0.15, \"performed+dividing\": 0.35, \"honest+dividing\": 0.15}",
+    "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+    "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+    "103": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.6, \"performed+dividing\": 0.1, \"honest+dividing\": 0.2}",
+    "104": ""
+  },
+  "similar": {
+    "100": "{\"performed+uniting\": 0.35, \"honest+uniting\": 0.15, \"performed+dividing\": 0.35, \"honest+dividing\": 0.15}",
+    "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+    "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+    "103": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.6, \"performed+dividing\": 0.1, \"honest+dividing\": 0.2}",
+    "104": ""
+  },
+  "agent": {
+    "100": [
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"inspect_example\", \"id\": 80}",
+      "{\"type\": \"submit_prediction\", \"prediction\": {\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}}"
+    ],
+    "101": [
+      "{\"type\": \"submit_prediction\", \"prediction\": {\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}}"
+    ],
+    "102": [
+      "bad json",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"inspect_example\", \"id\": 24}",
+      "{\"type\": \"inspect_example\", \"id\": 30}",
+      "{\"type\": \"inspect_example\", \"id\": 35}",
+      "{\"type\": \"inspect_example\", \"id\": 36}",
+      "{\"type\": \"submit_prediction\", \"prediction\": {\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}}"
+    ],
+    "103": [
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"list_examples\"}"
+    ],
+    "104": [
+      "{\"type\": \"list_examples\"}",
+      "{\"type\": \"inspect_example\", \"id\": 1}",
+      "{\"type\": \"inspect_example\", \"id\": 1}",
+      "{\"type\": \"submit_prediction\", \"prediction\": {}}"
+    ]
+  }
+}

package/tests/fixtures/learn-room-pilot.json

@@ -0,0 +1,16 @@
+{
+  "none": {
+    "100": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.3, \"performed+dividing\": 0.4, \"honest+dividing\": 0.1}",
+    "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+    "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+    "103": "{\"performed+uniting\": 0.2, \"honest+uniting\": 0.2, \"performed+dividing\": 0.2, \"honest+dividing\": 0.2}",
+    "104": ""
+  },
+  "random": {
+    "100": "{\"performed+uniting\": 0.35, \"honest+uniting\": 0.15, \"performed+dividing\": 0.35, \"honest+dividing\": 0.15}",
+    "101": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.2, \"performed+dividing\": 0.3, \"honest+dividing\": 0.4}",
+    "102": "{\"performed+uniting\": 0.25, \"honest+uniting\": 0.25, \"performed+dividing\": 0.25, \"honest+dividing\": 0.25}",
+    "103": "{\"performed+uniting\": 0.1, \"honest+uniting\": 0.6, \"performed+dividing\": 0.1, \"honest+dividing\": 0.2}",
+    "104": ""
+  }
+}