rtr-bench 0.1.5 → 0.2.1

package/docs/final-run-plan-8000.md

@@ -0,0 +1,73 @@
+# Final Predict / Learn run plan — 8,000 tokens
+
+**Prepared for approval; no final campaign calls have been made under this plan.** The campaign contains eleven models: the ten remaining models plus a new GLM 5.3 Flash run under the same 8,000-token allowance. All models run sequentially.
+
+## Scope and fixed settings
+
+Each model receives all 45 frozen evaluation targets (days 130–174) in four conditions: Predict, random examples, similar examples, and Learn. That is 180 final predictions per model and 1,980 across the campaign. The controls are included in the costs below.
+
+Every run explicitly sets 8,000 output tokens per response, evidence budget 3, seed 7, maximum eight model turns, maximum eight environment actions, and policy `transfer-check-v1`. The existing three-attempt retry limit, 60-second request timeout, prompts, scoring, reference pool, and split remain unchanged. The [machine-readable plan](final-run-plan-8000.json) records the exact target/reference IDs, dataset fingerprints, provider prices, spending policies, and command arguments.
+
+Temperature remains zero where supported; GPT-6 Luna and Claude Haiku 5.5 use their provider default because those endpoints do not support temperature control. Reasoning effort remains each provider’s default. This fixes the output allowance across models, not identical reasoning effort or compute use.
+
+## Budget and time
+
+The measured-use estimate is **$1.97**. The proposed individual caps total **$10.00**. Running sequentially is estimated to take **about 5.9 hours** (roughly 3,827 HTTP attempts); provider speed, retries, reasoning length, and the larger allowance can change this substantially.
+
+| Model | Pinned provider | Precision | Estimated cost | Spending cap | Estimated minutes |
+|---|---|---|---:|---:|---:|
+| z-ai/glm-5.3-flash | relace/fp4 | fp4 | $0.034 | $0.25 | 30 |
+| z-ai/glm-5.3 | wafer/us | fp4 | $0.051 | $0.25 | 4 |
+| minimax/minimax-m3.1-flash | inferact-vllm/nvfp4 | nvfp4 | $0.096 | $0.50 | 19 |
+| xiaomi/mimo-v2.6-flash | xiaomi/fp8 | fp8 | $0.070 | $0.50 | 36 |
+| deepseek/deepseek-v4.1-flash-1017 | sail-research/fp4/us | fp4 | $0.101 | $0.50 | 44 |
+| tencent/hy3 | gmicloud/bf16 | bf16 | $0.315 | $1.00 | 112 |
+| google/gemini-3.8-flash | google-ai-studio | unknown | $0.848 | $3.50 | 27 |
+| openai/gpt-6-luna | openai | unknown | $0.072 | $0.50 | 56 |
+| anthropic/claude-haiku-5.5 | anthropic | unknown | $0.089 | $1.50 | 11 |
+| thinkingmachines/inkling-small | deepinfra/fp4 | fp4 | $0.283 | $1.25 | 7 |
+| openai/gpt-oss-120b | primeintellect/fp4 | fp4 | $0.012 | $0.25 | 7 |
+
+Estimates reprice saved prompt and completion tokens at the refreshed base rates without cache discounts. MiniMax and DeepSeek use their 8,000-token development check. The other eight new models use their earlier 4,000-token development pilot, scaled from three to 45 targets; GLM Flash uses its earlier 45-target final run. Reasoning tokens are already included in completion usage and are not added again. There is no claim that increasing the token allowance leaves actual usage unchanged.
+
+Each cap is three times its measured-use cost estimate plus one conservative request reservation, rounded up to the next $0.25. Reservations cover the provider’s full context allowance and 8,000 output tokens at bounded rates, including applicable text/cache tiers. That is why some inexpensive models still need relatively large caps. These are per-run limits, not a guarantee against provider billing errors or unrelated account spending. Known costs settle reservations; unknown costs retain them. A cap stop can therefore occur before the reported charges reach the cap.
+
+The selected endpoints were listed available when this plan was prepared and advertised at least 8,000 completion tokens. Prices and availability can change. The CLI records a launch-time pricing/capability snapshot for each new run; subsequent resumes reuse that snapshot. If the endpoint becomes unavailable or prices/capabilities change materially, stop for review rather than silently changing providers, settings, or budgets.
+
+## Why GLM is included again
+
+The existing GLM Flash evaluation remains valid under its original 4,000-token setting. It is not a measurement under the new 8,000-token setting. This plan reruns it once so all eleven final results share the same declared allowance. The old run remains intact and linked as historical evidence. Never choose whichever of the old and new scores looks better.
+
+The allowance change follows development compatibility observations: four one-shot predictions failed at 4,000 tokens; the authorized follow-up produced 24/24 valid predictions at 8,000. DeepSeek used up to 7,665 tokens. This is a small compatibility check, not a guarantee that all final responses will fit. Truncated, invalid, or missing final predictions retain the declared failure penalties; no selective rerolls or post-evaluation allowance changes. The evaluation data is already public and the GLM final outcomes have been seen, so this is not an untouched holdout.
+
+## Execution after budget approval
+
+Use the tested source checkout for this campaign. The prepared release is 0.2.1; npm still served 0.2.0 when the plan was prepared. Publishing the patch is separate from running these source commands.
+
+From the repository folder, execute the [campaign script](https://github.com/1999labs/RTR-bench/blob/main/scripts/run-final-8000.sh):
+
+```bash
+bash scripts/run-final-8000.sh
+```
+
+This is a paid command. It runs all eleven model commands sequentially, with explicit 8,000-token settings and the caps above. It stops at the first command error. It does not automatically raise a cap, switch provider, delete attempts, or launch a replacement run. Review a stopped run before proceeding.
+
+For example, the first command is:
+
+```bash
+node src/cli.ts read-the-room z-ai/glm-5.3-flash \
+  --controls --provider relace/fp4 --split evaluation --limit 45 \
+  --budget 3 --seed 7 --max-model-calls 8 --max-tokens 8000 \
+  --agent-policy transfer-check-v1 --max-usd 0.25 \
+  --output results/final-8000/z-ai__glm-5.3-flash
+```
+
+All outputs go under `results/final-8000/`, separated by model and run fingerprint. Repeating the same script resumes completed and partial runs under the existing recovery rules; it does not reroll terminal predictions. Do not change the script’s budgets or output directories during recovery, because that can create separate paid runs. Remove a stale run lock only after confirming its process is no longer active.
+
+Before publishing scores, replay the saved traces and scoring, inspect failed/pending targets, verify actual serving providers and settings, reconcile spending, and confirm every result in the comparison uses this protocol. Incomplete runs have no headline score. Keep the existing README results unchanged until the corresponding verified replacements are available.
+
+## Release and compatibility checks
+
+The prepared 0.2.1 public commands (`predict`, `learn`, and `read-the-room`) now default to 8,000 tokens. Legacy command defaults remain unchanged. Resuming an older public-command run requires an explicit `--max-tokens 4000` plus its original other settings. Tests verify that this resumes the old run while the new default creates a separate run without altering the old report. The campaign script pins the allowance explicitly so it does not depend on future default changes.
+
+Verification passed: all 63 automated tests, package build/packing, fresh and resumed synthetic runs of all four public command variants from the extracted package, shell syntax validation, and comparison of every campaign command against the machine-readable plan. No paid calls were made during this preparation.

package/docs/learn-room.md

@@ -0,0 +1,188 @@
+# Learn the Room protocol v2
+
+Learn the Room is a separate evaluation track built on the [distribution protocol](distribution.md). It uses the same chronological split, probability validation, failure penalties, equal-window weighting, and score out of 100. This implementation provides a controlled interactive environment, four evidence strategies, resumable runs, and comparison reports. Strategies are version 2; the runner is version 4 with durable budget recovery, explicit setup-error stops, and parameter-support enforcement. The environment and distribution scorer remain version 1. Saved earlier reports remain replayable. See the [preflight audit](preflight-audit.md) for verification and remaining launch checks.
+
+## Public commands
+
+Use `node src/cli.ts read-the-room <model>` for Predict (`none`) and Learn (`agent`), `predict <model>` for Predict alone, or `learn <model>` for Learn alone. Add the boolean `--controls` flag to `read-the-room` for all four strategies. In version 0.2.1, these commands default to 8,000 output tokens per response and, when running Learn, `transfer-check-v1`. They share the existing runner, spending enforcement, reports, and `results/learn-room/` storage. Use `--max-usd` and `--provider` for a capped paid run; see the [Quickstart](../README.md#quickstart).
+
+To resume an earlier public-command run made at the old default, specify `--max-tokens 4000` and preserve the rest of its original command. Omitting it now selects 8,000 tokens and therefore a different run. The [final campaign plan](final-run-plan-8000.md) pins all settings explicitly.
+
+The legacy `learn-room` command retains its defaults: 2,000 tokens, the original agent policy, and `--strategy both` meaning `none` plus `random`. Its new `--strategy tracks` selects `none` plus `agent`; `--strategy all` still selects all four conditions. New commands do not accept `--strategy`. A combined run reports Learn’s paired gain; Learn alone does not calculate a gain without a Predict baseline. Separate strategy sets have separate run identities and do not share saved predictions or budgets.
+
+## Environment contract
+
+`createEpisode(target, reference, options)` returns separate `agent` and `evaluator` capabilities. A policy receives only `agent`. The evaluator owns the hidden target distribution, scoring, and trace access. Each episode copies its inputs, has independent state, and rejects references that repeat the target entity, are not earlier than the target, or duplicate an ID.
+
+`agent.observe()` supplies the target ID and prompt, quadrant names, inspection budget, and action limit. `agent.act(action)` accepts these JSON actions:
+
+```json
+{"type":"list_examples"}
+{"type":"inspect_example","id":1}
+{"type":"submit_prediction","prediction":{"performed+uniting":0.2,"honest+uniting":0.3,"performed+dividing":0.4,"honest+dividing":0.1}}
+```
+
+| Operation | Observation | Budget behavior |
+|---|---|---|
+| `list_examples` | Reference IDs and prompt text only | Uses one action |
+| `inspect_example` | Selected reference distribution and submission count | Uses one action and, on first successful inspection, one inspection |
+| `submit_prediction` | Validation acknowledgement and terminal status | Uses one action and ends the episode |
+
+Default limits are three distinct inspections and eight total actions. The CLI permits inspection budgets from zero to three. Invalid actions, unknown IDs, repeated catalog requests, and denied inspections all consume an action. A duplicate successful inspection returns cached evidence, consumes another action, and does not consume another inspection. An over-budget request reveals no additional evidence.
+
+A valid submit action with invalid probabilities ends the episode with zero points. Reaching the action limit without a prediction also ends it with zero points. A valid prediction on the final allowed action is accepted. A policy may submit before using its inspection budget. Once ended, further actions are rejected without changing the outcome or trace.
+
+Action observations contain validation status and remaining budgets, but never the target distribution, score, reward, or correctness. The host reads `evaluator.outcome()` only for reporting. `evaluator.abort(reason)` records host termination when a prediction cannot be obtained; it is logged separately from policy actions. Every accepted in-episode action attempt and its observation is recorded. Returned values are copies, so changing an observation cannot alter environment state.
+
+This is an API boundary for model evaluation, not a sandbox for arbitrary untrusted local code. A model adapter must receive only the agent-facing observations and must not have filesystem access, the full dataset, evaluator methods, or reports in its context.
+
+## Four strategies
+
+- **None:** no catalog or examples are supplied to the model. Its prediction messages exactly match direct distribution prediction, and the controller submits the resulting probabilities to the environment.
+- **Random:** the controller lists the catalog, selects the configured number of distinct examples without replacement, inspects them through the environment, and supplies only those observations with the prediction instructions. The model makes one final prediction.
+- **Similar:** the controller ranks reference prompt text by the fixed TF–IDF cosine method below, inspects the highest-ranked examples, and supplies them with the same final prediction instructions as Random.
+- **Agent:** the model chooses one environment action per turn, sees the observation, and chooses its next action. It may stop early or submit without inspecting any examples. Each episode starts with a fresh conversation, and only permitted observations enter that conversation.
+
+The random selector uses SHA-256 of `seed:targetDay` to initialize a 32-bit linear congruential generator, followed by a Fisher–Yates shuffle of sorted reference IDs. The default seed is 7. Selection is independent of execution order, run limits, and other episodes. The random algorithm is unchanged from `learn-room-strategies-v1`; the expanded strategy set is versioned as `learn-room-strategies-v2`.
+
+None, Random, and Similar are automatic baseline controllers. Agent selects evidence interactively. None of the selectors receives target outcomes.
+
+### Similarity method: word-char3-tfidf-v1
+
+Normalize prompt text with Unicode NFKC and lowercase it. Tokenize contiguous Unicode letters/numbers. Features are whole tokens plus length-three character sequences from each token padded with `^` and `$`. Word and character features use separate namespaces. For each feature, weight its count with `tf = 1 + ln(count)` and `idf = ln((N + 1) / (document_frequency + 1)) + 1`, where N is the number of reference prompts. Vocabulary and document frequencies are fitted only on reference prompt text. Drop target features outside that vocabulary, L2-normalize the vectors, and rank by cosine similarity. Break equal similarities by ascending reference ID, including all-zero overlap cases.
+
+This is a lexical similarity baseline, not a semantic embedding model. It uses no model service, crowd counts, distributions, or target labels. Keep this limitation visible when interpreting agent gains over it.
+
+### Interactive adapter: learn-room-agent-v1
+
+The adapter uses a plain JSON action protocol over chat completions. Each successful model response is parsed as one action and passed to `agent.act`. Its raw response is added as an assistant message and the environment observation as a user message. Native provider tool-calling support is not required. The initial request contains the target prompt, quadrant definitions, action schema, inspection/action limits, and model-turn limit. The agent must call `list_examples` to see the catalog.
+
+Malformed JSON and empty successful responses become invalid actions and receive feedback; they consume one model turn and one action. This allows bounded recovery on a later turn. A structurally valid submit action with invalid probabilities is terminal, as in every strategy. Transport failures are handled separately and never masquerade as evidence or actions. The model never receives target scores, rewards, or evaluation reports.
+
+The default maximum is eight model turns, independent of the eight-action environment limit. `--max-model-calls` can lower the turn limit to 1–8. Each turn has at most three HTTP attempts, so an agent episode makes at most 24 HTTP attempts under the defaults. Failed and interrupted attempts count toward that bound. Once the model-turn limit is reached without submission, the host ends the episode with zero points. Early successful submission stops immediately.
+
+`--max-tokens <positive-integer>` sets the output-token allowance for every baseline request and every agent turn; it defaults to 2,000. It does not change the number of turns. Providers may consume this allowance on reasoning before returning final content. A truncated response follows the existing validation rules: empty baseline content is missing, malformed baseline content is invalid, and empty or malformed agent actions receive bounded feedback. A length stop does not trigger a transport retry. Changing the allowance creates a separate run because `settings.maxTokens` is fingerprinted, and the spending guard reserves the configured output allowance before each attempt. Saved earlier runs remain replayable; see the version-upgrade and budgeted-resume rules below.
+
+### Optional agent policy: transfer-check-v1
+
+`--agent-policy default` uses the original agent instructions. Omitting the flag is identical, including the run identity and exact request contents. The original prompt and hash remain unchanged so saved runs continue to replay and resume.
+
+`--agent-policy transfer-check-v1` appends an experimental evidence-use instruction to the original system prompt. It asks the agent to assess transfer separately for performed versus honest and uniting versus dividing, consider differences between an individual and a broader category, preserve alternative interpretations of ambiguous targets, and discount references that support only a weak analogy. It explicitly retains a joint four-quadrant prediction rather than assuming statistical independence between axes. The policy contains no target-specific examples or crowd outcomes.
+
+The flag requires `--strategy agent` or `--strategy all`. It changes only the agent system message; automatic baseline requests, available actions, evidence pool, budgets, generation settings, and scoring are unchanged. The adapter protocol remains `learn-room-agent-v1`. The optional `agentPolicy` field and policy-specific `agentPromptHash` distinguish the new run and are checked during offline replay. Unknown policies and mismatched hashes are rejected rather than falling back. Reports label the selected non-default policy. Explicit `default` is normalized to the original manifest format.
+
+With `--strategy all`, selecting the new policy creates a new run for all four conditions; identical baseline requests are fresh calls in API mode, not automatically reused from an older run. Resume avoids rerolling completed responses only within the matching run configuration.
+
+Exercise the policy offline with the existing fixture scenarios:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-four-strategies.json --strategy all --limit 5 --max-tokens 4000 --agent-policy transfer-check-v1
+```
+
+Fixtures verify prompt routing, unchanged action and inspection limits, failures, early stopping, resume, and isolation of target labels. They cannot show that the instruction improves model behavior. In the completed [three-run development comparison](../results/learn-room/comparisons/glm-5-3-flash-policy-repeats/README.md), the new agent averaged 85.67 (observed range 85.48–85.85), versus the original policy's mean of 83.66. It still trailed its matching no-example control in every run, by 1.67 points on average, and individual regressions remain. These sequential development runs do not establish a causal policy effect or held-out performance. The original policy remains the default. The [first final evaluation](../results/learn-room/z-ai-glm-5-3-flash-e9d0327447ef1856/README.md) is now complete on all 45 held-out targets: direct distribution scored 87.93 and the transfer-check agent 85.57, with 180/180 valid predictions. The agent gain was -2.37 points (exploratory paired 95% interval -5.14 to +0.12); this run did not demonstrate an average benefit from evidence gathering. These final outcomes must not be used for tuning and then described as an untouched holdout. The motivation is documented in the [offline evidence audit](../results/learn-room/comparisons/glm-5-3-flash-4000-repeats/audit.md).
+
+## Scores and comparison
+
+For each completed episode, use the shared `100 * (1 - error / 2)` score. Invalid and missing predictions receive error 2. There is no inspection or compute penalty. Aggregate targets equally, and keep each track's result separate from the original RTR score.
+
+`--strategy all` evaluates all four strategies on identical declared targets, reference windows, model/provider, and generation settings. For compatibility, the default remains `both` (None and Random). Reports include each strategy's final score, error, validity counts, secondary winning-quadrant accuracy, evidence usage, logical model turns, API attempts, known token/cost totals, and unknown-usage counts. Agent uses the same audience and quadrant definitions, but its action instructions and multi-turn context differ from the one-shot baselines; comparisons do not isolate selection quality from additional inference compute.
+
+The paired comparison is:
+
+```text
+score_gain = strategy_score - control_score
+error_reduction = control_error - strategy_error
+score_gain = 50 * error_reduction
+```
+
+A gain is in points and may be negative. Reports compare Random, Similar, and Agent against None, and Agent against Random and Similar. Confidence intervals resample matching target differences by entity group, using 2,000 samples and seed 7. These exploratory intervals are not corrected for multiple comparisons. Both final quality and gain are shown. Uniform and constant reference-distribution baselines are also shown. No gains are reported until the declared conditions are complete. Incomplete conditions have no headline score.
+
+A fixture comparison demonstrates arithmetic and trace handling; it provides no evidence that random examples improve a real model.
+
+## Runs, retries, and traces
+
+Each baseline episode permits one logical prediction request and at most three HTTP attempts. Agent permits the bounded turns described above, with at most three HTTP attempts per turn. The shared client retries transport failures, transient typed provider errors, and untyped error codes 429/500/502/503/504, including error envelopes returned with HTTP 200, with a 60-second timeout per attempt. Backoff is the larger of the provider's Retry-After hint and the default one-/two-second delay. Retry hints can come from HTTP headers or saved OpenRouter error metadata. Malformed baseline predictions are terminal and are not rerolled; malformed agent actions receive bounded feedback instead. Provider pinning disables fallback. Runner v4 also requests `require_parameters: true` for pinned providers, so unsupported settings fail explicitly. Authentication, billing, invalid-request, and other recognized setup errors stop the run immediately with pending targets and no headline score. Correcting setup and rerunning the identical command can retry within the original three-attempt limit; exhausting that limit requires review. Refusals, malformed predictions, and exhausted output budgets remain model outcomes.
+
+Each run manifest fingerprints the dataset, split, target/reference IDs, model/provider, strategies, budget, seed, prompts, environment/scorer/runner versions, generation/retry settings, and fixture contents. Runs live under `results/learn-room/`. Each episode records its exact prediction request, every API attempt, the action/observation trace, and its outcome. Unknown usage from interrupted calls remains explicitly unknown. Fixture mode makes zero API calls.
+
+### Optional spending limit
+
+Use `--max-usd <amount>` with `--provider <name-or-endpoint-tag>` to enforce a per-run API usage budget. Before execution, the runner fetches public endpoint metadata, chooses an available compatible endpoint, and records its exact tag, context limit, and token price bounds in the manifest. Requests pin that tag with fallback disabled and send OpenRouter maximum-price ceilings. An exact endpoint tag is preferable for a reviewed cost estimate. Nonzero per-request charges, unknown nonzero pricing fields, malformed rates, and unsupported pricing tiers are rejected.
+
+Existing fixed-price endpoints retain the `reserved-context-v1` policy and their original run identities. The `reserved-text-v2` policy supports prompt-length price tiers and endpoints that lack temperature control. It records the catalog pricing, model, temperature capability, and output limit. Reservations use the highest input/cache-read/cache-write rate and highest completion/reasoning rate across all tiers; discounts are not applied a second time. Reasoning counts toward output usage. This follows OpenRouter's [reasoning accounting](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) and conservatively covers [cache writes](https://openrouter.ai/docs/guides/best-practices/prompt-caching). The policy permits only plain-text messages and the runner's bounded request fields; tools, plugins, media, explicit caching controls, and additional request options are rejected before dispatch. Catalog prices for unused media and web search do not apply to these requests.
+
+For budgeted runs, supported endpoints still receive temperature zero. If the selected endpoint does not advertise temperature support, all four conditions omit that parameter and the manifest records `settings.temperature: null`. Markdown and HTML reports explicitly label the provider default and its difference from temperature-zero runs. This is a generation-setting exception, not a change to prompts, evidence selection, scoring, or the frozen split. The November 14, 2026 catalog snapshots in `tests/fixtures/provider-pricing-2026-11-14.json` exercise this exception for GPT-6 Luna and Claude Haiku 5.5; Gemini 3.8 Flash on Google AI Studio retains temperature zero. Offline tests do not establish live inference availability.
+
+Before each HTTP attempt, the runner durably reserves the endpoint's full input-context allowance plus the request's maximum output tokens at those prices. This deliberately overestimates the short text prompts. Reported cost replaces the reservation when available; unknown costs and interrupted requests retain the full reservation. Explicit pre-generation HTTP 429 rate-limit rejections, with no generation ID or choices, release the reservation as unbilled under [OpenRouter's pricing policy](https://openrouter.ai/pricing); that basis is saved in the ledger. This exception never treats network failures or empty successful responses as free. Accounting rounds upward to billionths of a dollar and persists in `spending.json` across resumes. The run stops before dispatch if the next reservation would exceed the cap. An unexpected provider charge above the reservation stops further calls for review.
+
+Budget stops leave the evaluation incomplete and write an incomplete report; they do not convert unattempted targets into benchmark failures. The cap covers this run's API usage, not unrelated account activity or credit-purchase fees. Changing configuration creates a new run with a separate ledger; do not interpret multiple run caps as a shared campaign budget. Fixture runs require no spending policy.
+
+Identical commands resume saved terminal episodes, including failures. Budgeted commands first look up their saved configuration before consulting the provider catalog, so later pricing changes cannot silently create duplicate paid runs. Saved prices and endpoint capabilities remain frozen; a changed budget or other configuration still creates a separate run. A matching completed v3 run is reused, while a matching incomplete v3 run requires review before upgrade. New v4 attempts are linked to durable spending reservations. Resume settles known saved charges, preserves unknown reservations, rejects missing ledgers, and continues to block after an overcharge. Existing request and trace validation happens before further calls. Each agent turn stores its exact request, all attempts, and whether its action has been applied. On resume, the adapter reconstructs prior messages and observations, checks request/trace identity, and applies a durably saved but unapplied response exactly once. It does not rerun completed model turns. Interrupted attempts consume the fixed attempt budget. Atomic record writes and an exclusive run lock prevent partial writes and concurrent writers. Remove a stale lock only after confirming that its recorded process has exited.
+
+Use `--replicate <id>` to record a fresh run at otherwise identical settings. IDs contain 1–64 letters, numbers, underscores, or hyphens. The ID enters the manifest and run fingerprint only; it does not enter model requests, change the random-example seed, or alter evidence selection. Reusing the same ID and settings resumes that replicate without rerolling saved results. Omitting the flag preserves existing run identities. Separate replicates have separate spending ledgers: for a combined budget, assign per-run caps whose sum does not exceed the authorized total. Do not count repeat predictions on the same targets as additional independent target observations.
+
+For example, after authorizing two repeats with a combined $0.25 cap, run each sequentially with its own $0.125 cap:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --replicate repeat-2 --max-usd 0.125
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --replicate repeat-3 --max-usd 0.125
+```
+
+Offline reporting reconstructs evidence access and predictions from the frozen configuration and saved raw responses, verifies the saved request and trace, and recomputes scores rather than trusting cached outcomes. A missing episode file remains pending, with no invented observation trace.
+
+## Commands
+
+Source commands require Node.js 22.18 or newer. The bundled CLI supports the same commands after `npm run build`.
+
+Run the five-target, four-strategy fixture comparison with no API spending:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-four-strategies.json --strategy all --limit 5
+```
+
+The fixture intentionally contains valid, invalid, and missing predictions. Agent scripts exercise early stopping, invalid-action recovery, denied over-budget inspections, repeated inspections, and exhausted action budgets. Each baseline strategy maps target day to one raw response string. Agent maps each target day to an array of raw JSON action strings; an exhausted script supplies empty responses until the episode ends. No script is executed as code. Reports are explicitly labeled synthetic. All conditions use the same five development targets; fixture gains are not evidence of real model improvement.
+
+Select a single condition or a different evidence budget and random seed:
+
+```bash
+node src/cli.ts learn-room --fixture tests/fixtures/learn-room-pilot.json --strategy random --budget 2 --seed 42 --limit 5
+```
+
+For a future authorized model pilot:
+
+```bash
+node src/cli.ts learn-room <model-id> --provider <provider-name> --strategy all --split development --limit 5
+```
+
+For example, a five-target GLM pilot capped at $0.25:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Tenstorrent --strategy all --split development --limit 5 --max-usd 0.25
+```
+
+For a separately authorized development comparison with a 4,000-token allowance:
+
+```bash
+node src/cli.ts learn-room z-ai/glm-5.3-flash --provider Relace --strategy all --split development --limit 30 --max-tokens 4000 --max-usd 0.25
+```
+
+Keep the other settings fixed when comparing output allowances. A single run at each allowance cannot separate the allowance's effect from run-to-run variation; use paired results as development evidence, not a causal estimate.
+
+Use `--output <directory>` to choose an output root. Use `--split evaluation` only once development decisions are frozen. Omitting `--limit` runs the entire selected partition and requires corresponding fixture entries in fixture mode.
+
+Regenerate the JSON, Markdown, and searchable HTML reports offline:
+
+```bash
+node src/cli.ts learn-room-report <run-directory>
+```
+
+Run all tests and build:
+
+```bash
+npm test
+npm run build
+```
+
+## Embedding an interactive policy
+
+In the source checkout, a host can import `createEpisode` from `src/tracks/learn-room/environment.ts`. Construct the episode from the frozen split, then pass only `episode.agent` to a policy. The policy can inspect the catalog, choose references based on previous observations, and submit early. Keep the evaluator and dataset in the host. Persist all policy/model calls alongside `episode.evaluator.trace()`, and use `episode.evaluator.outcome()` for the final score. The included adapter in `agent.ts` demonstrates bounded model turns and replay; custom policies must enforce equivalent call limits.

package/docs/npm-release.md

@@ -0,0 +1,37 @@
+# Publishing RTR-bench 0.2.1
+
+This patch fixes paid-run resume identity, setup-error handling, transient-error retries, and spending-ledger recovery. Pinned providers must support the requested parameters. Public commands now default to 8,000 output tokens following development compatibility checks. Resume older 4,000-token runs with an explicit `--max-tokens 4000`. The runner is version 4; the frozen data, Predict/Learn prompts, evidence rules, and score formula are unchanged. Node.js 22.18 or newer is required. See the [preflight audit](preflight-audit.md).
+
+Publishing to npm is separate from committing and pushing to GitHub. These instructions are for the package maintainer, using an npm account with permission to publish `rtr-bench`.
+
+## Prepare and check
+
+From the repository folder:
+
+```bash
+git pull --ff-only
+npm ci
+npm test
+npm pack --dry-run
+```
+
+The package version in both `package.json` and `package-lock.json` must be `0.2.1`. `npm pack --dry-run` builds the bundled CLI and lists what would be included without publishing anything. The existing `prepack` script runs `npm run build` automatically for both packing and publishing.
+
+The package includes `dist/cli.js`, the dataset and frozen split, protocol documentation, and synthetic fixtures. API keys, `.env`, local evaluation outputs, and `node_modules` must not be included. Tests and synthetic runs make no paid model calls.
+
+## Publish
+
+```bash
+npm login
+npm publish --access public
+```
+
+Follow npm's browser sign-in and any authentication prompts. The second command publishes version 0.2.1 to the public registry and makes it the default `latest` release. An already-published name/version cannot be overwritten. See npm's [login](https://docs.npmjs.com/cli/v11/commands/npm-login/) and [publish](https://docs.npmjs.com/cli/v11/commands/npm-publish/) documentation.
+
+## Verify
+
+```bash
+npm view rtr-bench@latest version
+```
+
+This should print `0.2.1` once the registry has updated. Users can then run `npx rtr-bench@0.2.1 read-the-room ...` using the model, provider, split, and spending limit described in the README. A real model evaluation requires an OpenRouter key and incurs API charges; publishing the package does not start an evaluation.

package/docs/preflight-audit.md

@@ -0,0 +1,81 @@
+# Predict / Learn preflight audit — November 14, 2026
+
+## Decision
+
+The benchmark's data separation, prediction scoring, and Learn environment passed this audit. No change to the questions, frozen split, prompts, evidence rules, or score formula is indicated. The full paid campaign should wait for a small development-only compatibility pilot using the corrected runner. Published npm version 0.2.0 does not contain these audit fixes; use the corrected checkout or publish a patch release before running.
+
+This audit began from commit `9c6bbc108dd74b93a1cc0def70279126b7eb236a`. It used code review, mocked API failures, synthetic evaluations, public provider metadata, and replay of existing saved responses. It made no paid inference calls. Passing offline checks cannot guarantee a hosted provider's availability, output format, or future behavior.
+
+## Problems reproduced and corrected
+
+| Finding | Consequence before the fix | Runner v4 behavior |
+|---|---|---|
+| Mutable provider prices entered the run fingerprint before resume lookup | Repeating an identical command after a catalog change could create a new paid run | Looks up the saved budgeted run first and reuses its frozen pricing and capabilities |
+| Authentication and billing errors were treated as final predictions | A setup problem could sweep through targets and produce a misleading zero-score completed run | Stops at the affected target, leaves the run incomplete, and permits bounded recovery |
+| HTTP 200 responses could contain transient error envelopes | Retryable provider failures could become immediate benchmark failures | Classifies typed errors and embedded error codes, preserving the three-attempt limit |
+| An overcharge stop did not survive reopening the ledger | Resume could send another call after a reservation had been exceeded | Blocks further requests until reviewed |
+| Saved response and spending settlement were separate crash points | Known charges could remain reserved; a missing ledger could reset accounting | Links attempts to reservations, reconciles saved costs, and rejects missing or mismatched accounting |
+| Completed episodes were not all validated before sending further calls | Corruption could be discovered only after more spending | Replays saved requests and traces before dispatch |
+| Pinned routing did not explicitly require parameter support | A provider could silently ignore a generation setting | Sends `require_parameters: true` for new pinned runs |
+
+These changes concern execution and accounting. They do not alter successful prediction requests' message contents, evidence selection, scoring, or output validation. The runner version changes to v4 to record the operational differences. Completed v3 budgeted runs are recognized; incomplete v3 runs require review rather than automatic migration. Arbitrarily changing output directories, budgets, strategies, or replicate IDs still creates separate runs and separate budgets.
+
+## Data and experiment checks
+
+- The frozen dataset contains 165 windows and 39,828 submissions. Reference has 90 windows, development 30, and evaluation 45. The loader verifies integer counts, SHA-256 fingerprints, complete disjoint coverage, chronology, and normalized prompt-group separation.
+- Grouping uses trimmed, lowercased, whitespace-normalized names; the reviewed alias map is empty. Both ELON MUSK windows remain in reference. The split is chronological, not chosen from crowd outcomes.
+- Predict receives the target word and task instructions, with no example or target labels. Learn can inspect only reference examples; development and evaluation labels are inaccessible through the environment's model-facing methods.
+- An added test changes every non-reference crowd count and submission count, then compares all 45 targets' requests across all four conditions. It also compares Learn conversations after listing examples, inspecting three references, and submitting a prediction. The model-visible content stays identical.
+- An added 180-episode synthetic evaluation exercises all 45 targets in all four conditions. An independent arithmetic calculation agrees with every aggregate score and paired gain, including invalid-output penalties.
+- Existing tests cover exact probability keys, normalization tolerance, invalid and missing output, equal target weighting, tied winners, paired group bootstrap intervals, reference-only baselines, deterministic example selection, inspection limits, action limits, model-turn limits, denied actions, retries, replay, and CLI aliases.
+- All 62 automated tests pass. `npm run build`, `npm pack`, and `git diff --check` pass. From the extracted package, `predict`, `learn`, `read-the-room`, and `read-the-room --controls` each pass fresh-run and resume checks outside the repository, with the correct strategies, 4,000-token default, Learn policy, and zero API calls. The audit documents are included in the package.
+
+## Existing GLM evaluation
+
+The completed GLM 5.3 Flash / Relace run needs no rerun because of these fixes. Its JSON, Markdown, and HTML reports reproduce exactly using the updated code. A separate calculation from the saved raw prediction JSON and original quadrant counts confirms:
+
+| Condition | Score / 100 |
+|---|---:|
+| Predict (`none`) | 87.93467443347016 |
+| Random examples | 86.47321119295975 |
+| Similar examples | 85.42762611497108 |
+| Learn (`agent`) | 85.56778927968283 |
+
+All 180 predictions are valid. The saved attempts total 347 API calls and $0.0314992195 reported cost. Learn's paired gain over Predict is −2.36688515378733 points. Successful historical results remain comparable on the same prediction protocol; the execution-version difference should remain disclosed.
+
+## Freeze before the campaign
+
+Use `read-the-room --controls` so each model gets Predict, random examples, similar examples, and Learn on the same 45 evaluation targets. The original proposal used 4,000 output tokens per turn. Following the development checks below, the [final plan](final-run-plan-8000.md) supersedes that allowance with 8,000 for every model. Fix seed 7, evidence budget 3, maximum eight model turns, maximum eight environment actions, and agent policy `transfer-check-v1`. Keep one predeclared run per model/provider combination. Run models sequentially to make interruptions and per-run budget stops easy to inspect.
+
+Record the exact model, provider tag, quantization, runner version, pricing/capability snapshot, and spending cap before each run. Include failed model outputs in the scores. Do not increase token limits or change prompts in response to final evaluation scores. Protocol changes prompted by development compatibility failures must be decided and documented before the remaining evaluation runs; assess historical comparability explicitly if such changes are needed.
+
+The ten-provider catalog check found the planned MiMo `gmicloud/bf16` endpoint marked unavailable (`status: -2`). Its native `xiaomi/fp8` endpoint was listed available and is a candidate replacement, subject to refreshed pricing, an explicit provider/precision declaration, and a development pilot. The other nine selected endpoints were listed available; that is not proof that a paid request will succeed. See the [dated provider snapshot](preflight-providers-2026-11-14.json).
+
+## Limits that should remain explicit
+
+- **What Learn measures:** in-context use of historical examples, not model training or weight updates. It gets more calls and different instructions than Predict. The gain measures the whole interactive procedure. The controls help compare evidence choices but do not fully isolate the causal value of autonomous selection from extra computation.
+- **Generation settings:** use temperature zero where supported. GPT-6 Luna and Claude Haiku 5.5 omit unsupported temperature and record `null`. Reasoning effort is not explicitly set; provider defaults differ. This is a comparison under a common output-token cap, not equal reasoning effort or equal compute. Reasoning can consume the allowance before visible JSON appears.
+- **Evidence strength:** 45 targets and one run per model support an exploratory comparison. Bootstrap intervals describe target variation, not all variability from model sampling, participant selection, or service changes. Avoid strong rankings from small score differences or unadjusted multiple comparisons.
+- **Public data:** this dataset was already public and used by the original benchmark. The split prevents runtime label leakage; it cannot establish absence from model training. The existing final GLM results also mean the evaluation set is no longer an untouched holdout for future policy development.
+- **Audience:** results describe this self-selected Mixed Messages audience and historical period, not a universal correct political or cultural judgment. Chronological audience changes may affect transfer from reference examples.
+- **Accounting:** unknown charges retain conservative reservations. Reservations can stop a run well below its eventual actual cost limit. Provider price ceilings and local accounting cannot control unrelated account spending or guarantee provider billing behavior.
+- **Reproduction:** hosted model weights and serving implementations cannot be frozen by this repository. Exact saved responses and scores can be replayed; new API outputs need not match.
+
+## Remaining launch checks
+
+1. Use the corrected source checkout or prepare and publish a patch release. Preserve the existing final GLM artifacts.
+2. Resolve the unavailable MiMo endpoint, refresh provider metadata and prices, and freeze the provider/precision list and individual caps. The earlier cost estimate is a planning estimate, not a guaranteed campaign price.
+3. After authorizing a small separate budget, run three **development** targets per model with all four conditions. Check valid JSON, output truncation, requested settings, action handling, reported usage, and resume behavior. Do not use these few scores to rank models. This is the remaining live compatibility gate; it has not been run during this audit.
+4. If those checks pass, run the predeclared final evaluation sequentially with per-model caps. Use the public Predict/Learn commands, not legacy batch scripts for the original winning-quadrant benchmark. Preserve incomplete runs for review rather than deleting them and rerunning to obtain a better score.
+
+Provider behavior references: [OpenRouter errors](https://openrouter.ai/docs/api_reference/errors-and-debugging), [provider routing and parameter support](https://openrouter.ai/docs/guides/routing/provider-selection), and [reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
+
+## Pilot prepared after the audit
+
+The [development pilot plan](development-pilot-2026-11-14.json) fixes the ten model/provider pairs, three development targets, protocol, pricing snapshots, and individual spending caps totaling $3.45. MiMo uses the available native `xiaomi/fp8` endpoint. The estimate is approximately $0.19 by repricing the earlier GLM token usage; it is not a guaranteed price. This pilot does not authorize or launch final evaluation.
+
+The pilot subsequently completed: 116/120 predictions were valid for $0.11486105754 in reported cost. MiniMax and DeepSeek exhausted the 4,000-token allowance on four one-shot predictions; all Learn episodes were valid. The full campaign remains on hold pending the token-allowance decision. See the [saved pilot findings](https://github.com/1999labs/RTR-bench/blob/main/results/development-preflight-2026-11-14/README.md).
+
+An authorized follow-up at 8,000 tokens then produced 24/24 valid predictions across MiniMax and DeepSeek for $0.00999899252, with no truncation. DeepSeek used up to 7,665 output tokens. This was a development check only; CLI defaults and final protocol settings were not changed. See the [8,000-token findings](https://github.com/1999labs/RTR-bench/blob/main/results/development-token-check-8000-2026-11-14/README.md).
+
+The final campaign plan now fixes 8,000 tokens for all eleven models, including a matching GLM Flash rerun. Public command defaults in the prepared 0.2.1 release have been aligned to it. The historical audit and pilot measurements above retain their original settings. No final campaign calls were made while preparing this plan.