rtr-bench 0.2.0 → 0.2.1

package/docs/final-run-plan-8000.md

@@ -0,0 +1,73 @@
+# Final Predict / Learn run plan — 8,000 tokens
+
+**Prepared for approval; no final campaign calls have been made under this plan.** The campaign contains eleven models: the ten remaining models plus a new GLM 5.3 Flash run under the same 8,000-token allowance. All models run sequentially.
+
+## Scope and fixed settings
+
+Each model receives all 45 frozen evaluation targets (days 130–174) in four conditions: Predict, random examples, similar examples, and Learn. That is 180 final predictions per model and 1,980 across the campaign. The controls are included in the costs below.
+
+Every run explicitly sets 8,000 output tokens per response, evidence budget 3, seed 7, maximum eight model turns, maximum eight environment actions, and policy `transfer-check-v1`. The existing three-attempt retry limit, 60-second request timeout, prompts, scoring, reference pool, and split remain unchanged. The [machine-readable plan](final-run-plan-8000.json) records the exact target/reference IDs, dataset fingerprints, provider prices, spending policies, and command arguments.
+
+Temperature remains zero where supported; GPT-6 Luna and Claude Haiku 5.5 use their provider default because those endpoints do not support temperature control. Reasoning effort remains each provider’s default. This fixes the output allowance across models, not identical reasoning effort or compute use.
+
+## Budget and time
+
+The measured-use estimate is **$1.97**. The proposed individual caps total **$10.00**. Running sequentially is estimated to take **about 5.9 hours** (roughly 3,827 HTTP attempts); provider speed, retries, reasoning length, and the larger allowance can change this substantially.
+
+| Model | Pinned provider | Precision | Estimated cost | Spending cap | Estimated minutes |
+|---|---|---|---:|---:|---:|
+| z-ai/glm-5.3-flash | relace/fp4 | fp4 | $0.034 | $0.25 | 30 |
+| z-ai/glm-5.3 | wafer/us | fp4 | $0.051 | $0.25 | 4 |
+| minimax/minimax-m3.1-flash | inferact-vllm/nvfp4 | nvfp4 | $0.096 | $0.50 | 19 |
+| xiaomi/mimo-v2.6-flash | xiaomi/fp8 | fp8 | $0.070 | $0.50 | 36 |
+| deepseek/deepseek-v4.1-flash-1017 | sail-research/fp4/us | fp4 | $0.101 | $0.50 | 44 |
+| tencent/hy3 | gmicloud/bf16 | bf16 | $0.315 | $1.00 | 112 |
+| google/gemini-3.8-flash | google-ai-studio | unknown | $0.848 | $3.50 | 27 |
+| openai/gpt-6-luna | openai | unknown | $0.072 | $0.50 | 56 |
+| anthropic/claude-haiku-5.5 | anthropic | unknown | $0.089 | $1.50 | 11 |
+| thinkingmachines/inkling-small | deepinfra/fp4 | fp4 | $0.283 | $1.25 | 7 |
+| openai/gpt-oss-120b | primeintellect/fp4 | fp4 | $0.012 | $0.25 | 7 |
+
+Estimates reprice saved prompt and completion tokens at the refreshed base rates without cache discounts. MiniMax and DeepSeek use their 8,000-token development check. The other eight new models use their earlier 4,000-token development pilot, scaled from three to 45 targets; GLM Flash uses its earlier 45-target final run. Reasoning tokens are already included in completion usage and are not added again. There is no claim that increasing the token allowance leaves actual usage unchanged.
+
+Each cap is three times its measured-use cost estimate plus one conservative request reservation, rounded up to the next $0.25. Reservations cover the provider’s full context allowance and 8,000 output tokens at bounded rates, including applicable text/cache tiers. That is why some inexpensive models still need relatively large caps. These are per-run limits, not a guarantee against provider billing errors or unrelated account spending. Known costs settle reservations; unknown costs retain them. A cap stop can therefore occur before the reported charges reach the cap.
+
+The selected endpoints were listed available when this plan was prepared and advertised at least 8,000 completion tokens. Prices and availability can change. The CLI records a launch-time pricing/capability snapshot for each new run; subsequent resumes reuse that snapshot. If the endpoint becomes unavailable or prices/capabilities change materially, stop for review rather than silently changing providers, settings, or budgets.
+
+## Why GLM is included again
+
+The existing GLM Flash evaluation remains valid under its original 4,000-token setting. It is not a measurement under the new 8,000-token setting. This plan reruns it once so all eleven final results share the same declared allowance. The old run remains intact and linked as historical evidence. Never choose whichever of the old and new scores looks better.
+
+The allowance change follows development compatibility observations: four one-shot predictions failed at 4,000 tokens; the authorized follow-up produced 24/24 valid predictions at 8,000. DeepSeek used up to 7,665 tokens. This is a small compatibility check, not a guarantee that all final responses will fit. Truncated, invalid, or missing final predictions retain the declared failure penalties; no selective rerolls or post-evaluation allowance changes. The evaluation data is already public and the GLM final outcomes have been seen, so this is not an untouched holdout.
+
+## Execution after budget approval
+
+Use the tested source checkout for this campaign. The prepared release is 0.2.1; npm still served 0.2.0 when the plan was prepared. Publishing the patch is separate from running these source commands.
+
+From the repository folder, execute the [campaign script](https://github.com/1999labs/RTR-bench/blob/main/scripts/run-final-8000.sh):
+
+```bash
+bash scripts/run-final-8000.sh
+```
+
+This is a paid command. It runs all eleven model commands sequentially, with explicit 8,000-token settings and the caps above. It stops at the first command error. It does not automatically raise a cap, switch provider, delete attempts, or launch a replacement run. Review a stopped run before proceeding.
+
+For example, the first command is:
+
+```bash
+node src/cli.ts read-the-room z-ai/glm-5.3-flash \
+  --controls --provider relace/fp4 --split evaluation --limit 45 \
+  --budget 3 --seed 7 --max-model-calls 8 --max-tokens 8000 \
+  --agent-policy transfer-check-v1 --max-usd 0.25 \
+  --output results/final-8000/z-ai__glm-5.3-flash
+```
+
+All outputs go under `results/final-8000/`, separated by model and run fingerprint. Repeating the same script resumes completed and partial runs under the existing recovery rules; it does not reroll terminal predictions. Do not change the script’s budgets or output directories during recovery, because that can create separate paid runs. Remove a stale run lock only after confirming its process is no longer active.
+
+Before publishing scores, replay the saved traces and scoring, inspect failed/pending targets, verify actual serving providers and settings, reconcile spending, and confirm every result in the comparison uses this protocol. Incomplete runs have no headline score. Keep the existing README results unchanged until the corresponding verified replacements are available.
+
+## Release and compatibility checks
+
+The prepared 0.2.1 public commands (`predict`, `learn`, and `read-the-room`) now default to 8,000 tokens. Legacy command defaults remain unchanged. Resuming an older public-command run requires an explicit `--max-tokens 4000` plus its original other settings. Tests verify that this resumes the old run while the new default creates a separate run without altering the old report. The campaign script pins the allowance explicitly so it does not depend on future default changes.
+
+Verification passed: all 63 automated tests, package build/packing, fresh and resumed synthetic runs of all four public command variants from the extracted package, shell syntax validation, and comparison of every campaign command against the machine-readable plan. No paid calls were made during this preparation.

package/docs/learn-room.md

@@ -1,10 +1,12 @@
 # Learn the Room protocol v2
 
-Learn the Room is a separate evaluation track built on the [distribution protocol](distribution.md). It uses the same chronological split, probability validation, failure penalties, equal-window weighting, and score out of 100. This implementation provides a controlled interactive environment, four evidence strategies, resumable runs, and comparison reports. Strategies are version 2; the runner is version 3 with provider Retry-After handling and optional spending enforcement. The environment and distribution scorer remain version 1. Saved earlier reports remain replayable.
+Learn the Room is a separate evaluation track built on the [distribution protocol](distribution.md). It uses the same chronological split, probability validation, failure penalties, equal-window weighting, and score out of 100. This implementation provides a controlled interactive environment, four evidence strategies, resumable runs, and comparison reports. Strategies are version 2; the runner is version 4 with durable budget recovery, explicit setup-error stops, and parameter-support enforcement. The environment and distribution scorer remain version 1. Saved earlier reports remain replayable. See the [preflight audit](preflight-audit.md) for verification and remaining launch checks.
 
 ## Public commands
 
-Use `node src/cli.ts read-the-room <model>` for Predict (`none`) and Learn (`agent`), `predict <model>` for Predict alone, or `learn <model>` for Learn alone. Add the boolean `--controls` flag to `read-the-room` for all four strategies. These commands default to 4,000 output tokens per response and, when running Learn, `transfer-check-v1`. They share the existing runner, spending enforcement, reports, and `results/learn-room/` storage. Use `--max-usd` and `--provider` for a capped paid run; see the [Quickstart](../README.md#quickstart).
+Use `node src/cli.ts read-the-room <model>` for Predict (`none`) and Learn (`agent`), `predict <model>` for Predict alone, or `learn <model>` for Learn alone. Add the boolean `--controls` flag to `read-the-room` for all four strategies. In version 0.2.1, these commands default to 8,000 output tokens per response and, when running Learn, `transfer-check-v1`. They share the existing runner, spending enforcement, reports, and `results/learn-room/` storage. Use `--max-usd` and `--provider` for a capped paid run; see the [Quickstart](../README.md#quickstart).
+
+To resume an earlier public-command run made at the old default, specify `--max-tokens 4000` and preserve the rest of its original command. Omitting it now selects 8,000 tokens and therefore a different run. The [final campaign plan](final-run-plan-8000.md) pins all settings explicitly.
 
 The legacy `learn-room` command retains its defaults: 2,000 tokens, the original agent policy, and `--strategy both` meaning `none` plus `random`. Its new `--strategy tracks` selects `none` plus `agent`; `--strategy all` still selects all four conditions. New commands do not accept `--strategy`. A combined run reports Learn’s paired gain; Learn alone does not calculate a gain without a Predict baseline. Separate strategy sets have separate run identities and do not share saved predictions or budgets.
 
@@ -59,7 +61,7 @@ Malformed JSON and empty successful responses become invalid actions and receive
 
 The default maximum is eight model turns, independent of the eight-action environment limit. `--max-model-calls` can lower the turn limit to 1–8. Each turn has at most three HTTP attempts, so an agent episode makes at most 24 HTTP attempts under the defaults. Failed and interrupted attempts count toward that bound. Once the model-turn limit is reached without submission, the host ends the episode with zero points. Early successful submission stops immediately.
 
-`--max-tokens <positive-integer>` sets the output-token allowance for every baseline request and every agent turn; it defaults to 2,000. It does not change the number of turns. Providers may consume this allowance on reasoning before returning final content. A truncated response follows the existing validation rules: empty baseline content is missing, malformed baseline content is invalid, and empty or malformed agent actions receive bounded feedback. A length stop does not trigger a transport retry. Changing the allowance creates a separate run because `settings.maxTokens` is fingerprinted, and the spending guard reserves the configured output allowance before each attempt. Existing default runs retain their identities and resume behavior.
+`--max-tokens <positive-integer>` sets the output-token allowance for every baseline request and every agent turn; it defaults to 2,000. It does not change the number of turns. Providers may consume this allowance on reasoning before returning final content. A truncated response follows the existing validation rules: empty baseline content is missing, malformed baseline content is invalid, and empty or malformed agent actions receive bounded feedback. A length stop does not trigger a transport retry. Changing the allowance creates a separate run because `settings.maxTokens` is fingerprinted, and the spending guard reserves the configured output allowance before each attempt. Saved earlier runs remain replayable; see the version-upgrade and budgeted-resume rules below.
 
 ### Optional agent policy: transfer-check-v1
 
@@ -99,7 +101,7 @@ A fixture comparison demonstrates arithmetic and trace handling; it provides no
 
 ## Runs, retries, and traces
 
-Each baseline episode permits one logical prediction request and at most three HTTP attempts. Agent permits the bounded turns described above, with at most three HTTP attempts per turn. The shared client retries only transport failures or HTTP 429/500/502/503/504, with a 60-second timeout per attempt. Backoff is the larger of the provider's Retry-After hint and the default one-/two-second delay. Retry hints can come from HTTP headers or saved OpenRouter error metadata. Malformed baseline predictions are terminal and are not rerolled; malformed agent actions receive bounded feedback instead. Provider pinning disables fallback. Unsupported settings fail explicitly.
+Each baseline episode permits one logical prediction request and at most three HTTP attempts. Agent permits the bounded turns described above, with at most three HTTP attempts per turn. The shared client retries transport failures, transient typed provider errors, and untyped error codes 429/500/502/503/504, including error envelopes returned with HTTP 200, with a 60-second timeout per attempt. Backoff is the larger of the provider's Retry-After hint and the default one-/two-second delay. Retry hints can come from HTTP headers or saved OpenRouter error metadata. Malformed baseline predictions are terminal and are not rerolled; malformed agent actions receive bounded feedback instead. Provider pinning disables fallback. Runner v4 also requests `require_parameters: true` for pinned providers, so unsupported settings fail explicitly. Authentication, billing, invalid-request, and other recognized setup errors stop the run immediately with pending targets and no headline score. Correcting setup and rerunning the identical command can retry within the original three-attempt limit; exhausting that limit requires review. Refusals, malformed predictions, and exhausted output budgets remain model outcomes.
 
 Each run manifest fingerprints the dataset, split, target/reference IDs, model/provider, strategies, budget, seed, prompts, environment/scorer/runner versions, generation/retry settings, and fixture contents. Runs live under `results/learn-room/`. Each episode records its exact prediction request, every API attempt, the action/observation trace, and its outcome. Unknown usage from interrupted calls remains explicitly unknown. Fixture mode makes zero API calls.
 
@@ -115,7 +117,7 @@ Before each HTTP attempt, the runner durably reserves the endpoint's full input-
 
 Budget stops leave the evaluation incomplete and write an incomplete report; they do not convert unattempted targets into benchmark failures. The cap covers this run's API usage, not unrelated account activity or credit-purchase fees. Changing configuration creates a new run with a separate ledger; do not interpret multiple run caps as a shared campaign budget. Fixture runs require no spending policy.
 
-Identical commands resume saved terminal episodes, including failures. Each agent turn stores its exact request, all attempts, and whether its action has been applied. On resume, the adapter reconstructs prior messages and observations, checks request/trace identity, and applies a durably saved but unapplied response exactly once. It does not rerun completed model turns. Interrupted attempts consume the fixed attempt budget. Atomic record writes and an exclusive run lock prevent partial writes and concurrent writers. Remove a stale lock only after confirming that its recorded process has exited.
+Identical commands resume saved terminal episodes, including failures. Budgeted commands first look up their saved configuration before consulting the provider catalog, so later pricing changes cannot silently create duplicate paid runs. Saved prices and endpoint capabilities remain frozen; a changed budget or other configuration still creates a separate run. A matching completed v3 run is reused, while a matching incomplete v3 run requires review before upgrade. New v4 attempts are linked to durable spending reservations. Resume settles known saved charges, preserves unknown reservations, rejects missing ledgers, and continues to block after an overcharge. Existing request and trace validation happens before further calls. Each agent turn stores its exact request, all attempts, and whether its action has been applied. On resume, the adapter reconstructs prior messages and observations, checks request/trace identity, and applies a durably saved but unapplied response exactly once. It does not rerun completed model turns. Interrupted attempts consume the fixed attempt budget. Atomic record writes and an exclusive run lock prevent partial writes and concurrent writers. Remove a stale lock only after confirming that its recorded process has exited.
 
 Use `--replicate <id>` to record a fresh run at otherwise identical settings. IDs contain 1–64 letters, numbers, underscores, or hyphens. The ID enters the manifest and run fingerprint only; it does not enter model requests, change the random-example seed, or alter evidence selection. Reusing the same ID and settings resumes that replicate without rerolling saved results. Omitting the flag preserves existing run identities. Separate replicates have separate spending ledgers: for a combined budget, assign per-run caps whose sum does not exceed the authorized total. Do not count repeat predictions on the same targets as additional independent target observations.
 

package/docs/npm-release.md

@@ -1,6 +1,6 @@
-# Publishing RTR-bench 0.2.0
+# Publishing RTR-bench 0.2.1
 
-This release adds the public `read-the-room`, `predict`, and `learn` commands to the npm package, along with the frozen dataset, reports, and spending controls. It includes compatibility for tiered text pricing and records when a provider does not support temperature control. Node.js 22.18 or newer is required.
+This patch fixes paid-run resume identity, setup-error handling, transient-error retries, and spending-ledger recovery. Pinned providers must support the requested parameters. Public commands now default to 8,000 output tokens following development compatibility checks. Resume older 4,000-token runs with an explicit `--max-tokens 4000`. The runner is version 4; the frozen data, Predict/Learn prompts, evidence rules, and score formula are unchanged. Node.js 22.18 or newer is required. See the [preflight audit](preflight-audit.md).
 
 Publishing to npm is separate from committing and pushing to GitHub. These instructions are for the package maintainer, using an npm account with permission to publish `rtr-bench`.
 
@@ -15,7 +15,7 @@ npm test
 npm pack --dry-run
 ```
 
-The package version in both `package.json` and `package-lock.json` must be `0.2.0`. `npm pack --dry-run` builds the bundled CLI and lists what would be included without publishing anything. The existing `prepack` script runs `npm run build` automatically for both packing and publishing.
+The package version in both `package.json` and `package-lock.json` must be `0.2.1`. `npm pack --dry-run` builds the bundled CLI and lists what would be included without publishing anything. The existing `prepack` script runs `npm run build` automatically for both packing and publishing.
 
 The package includes `dist/cli.js`, the dataset and frozen split, protocol documentation, and synthetic fixtures. API keys, `.env`, local evaluation outputs, and `node_modules` must not be included. Tests and synthetic runs make no paid model calls.
 
@@ -26,7 +26,7 @@ npm login
 npm publish --access public
 ```
 
-Follow npm's browser sign-in and any authentication prompts. The second command publishes version 0.2.0 to the public registry and makes it the default `latest` release. An already-published name/version cannot be overwritten. See npm's [login](https://docs.npmjs.com/cli/v11/commands/npm-login/) and [publish](https://docs.npmjs.com/cli/v11/commands/npm-publish/) documentation.
+Follow npm's browser sign-in and any authentication prompts. The second command publishes version 0.2.1 to the public registry and makes it the default `latest` release. An already-published name/version cannot be overwritten. See npm's [login](https://docs.npmjs.com/cli/v11/commands/npm-login/) and [publish](https://docs.npmjs.com/cli/v11/commands/npm-publish/) documentation.
 
 ## Verify
 
@@ -34,4 +34,4 @@ Follow npm's browser sign-in and any authentication prompts. The second command
 npm view rtr-bench@latest version
 ```
 
-This should print `0.2.0` once the registry has updated. Users can then run `npx rtr-bench@0.2.0 read-the-room ...` using the model, provider, split, and spending limit described in the README. A real model evaluation requires an OpenRouter key and incurs API charges; publishing the package does not start an evaluation.
+This should print `0.2.1` once the registry has updated. Users can then run `npx rtr-bench@0.2.1 read-the-room ...` using the model, provider, split, and spending limit described in the README. A real model evaluation requires an OpenRouter key and incurs API charges; publishing the package does not start an evaluation.

package/docs/preflight-audit.md

@@ -0,0 +1,81 @@
+# Predict / Learn preflight audit — November 14, 2026
+
+## Decision
+
+The benchmark's data separation, prediction scoring, and Learn environment passed this audit. No change to the questions, frozen split, prompts, evidence rules, or score formula is indicated. The full paid campaign should wait for a small development-only compatibility pilot using the corrected runner. Published npm version 0.2.0 does not contain these audit fixes; use the corrected checkout or publish a patch release before running.
+
+This audit began from commit `9c6bbc108dd74b93a1cc0def70279126b7eb236a`. It used code review, mocked API failures, synthetic evaluations, public provider metadata, and replay of existing saved responses. It made no paid inference calls. Passing offline checks cannot guarantee a hosted provider's availability, output format, or future behavior.
+
+## Problems reproduced and corrected
+
+| Finding | Consequence before the fix | Runner v4 behavior |
+|---|---|---|
+| Mutable provider prices entered the run fingerprint before resume lookup | Repeating an identical command after a catalog change could create a new paid run | Looks up the saved budgeted run first and reuses its frozen pricing and capabilities |
+| Authentication and billing errors were treated as final predictions | A setup problem could sweep through targets and produce a misleading zero-score completed run | Stops at the affected target, leaves the run incomplete, and permits bounded recovery |
+| HTTP 200 responses could contain transient error envelopes | Retryable provider failures could become immediate benchmark failures | Classifies typed errors and embedded error codes, preserving the three-attempt limit |
+| An overcharge stop did not survive reopening the ledger | Resume could send another call after a reservation had been exceeded | Blocks further requests until reviewed |
+| Saved response and spending settlement were separate crash points | Known charges could remain reserved; a missing ledger could reset accounting | Links attempts to reservations, reconciles saved costs, and rejects missing or mismatched accounting |
+| Completed episodes were not all validated before sending further calls | Corruption could be discovered only after more spending | Replays saved requests and traces before dispatch |
+| Pinned routing did not explicitly require parameter support | A provider could silently ignore a generation setting | Sends `require_parameters: true` for new pinned runs |
+
+These changes concern execution and accounting. They do not alter successful prediction requests' message contents, evidence selection, scoring, or output validation. The runner version changes to v4 to record the operational differences. Completed v3 budgeted runs are recognized; incomplete v3 runs require review rather than automatic migration. Arbitrarily changing output directories, budgets, strategies, or replicate IDs still creates separate runs and separate budgets.
+
+## Data and experiment checks
+
+- The frozen dataset contains 165 windows and 39,828 submissions. Reference has 90 windows, development 30, and evaluation 45. The loader verifies integer counts, SHA-256 fingerprints, complete disjoint coverage, chronology, and normalized prompt-group separation.
+- Grouping uses trimmed, lowercased, whitespace-normalized names; the reviewed alias map is empty. Both ELON MUSK windows remain in reference. The split is chronological, not chosen from crowd outcomes.
+- Predict receives the target word and task instructions, with no example or target labels. Learn can inspect only reference examples; development and evaluation labels are inaccessible through the environment's model-facing methods.
+- An added test changes every non-reference crowd count and submission count, then compares all 45 targets' requests across all four conditions. It also compares Learn conversations after listing examples, inspecting three references, and submitting a prediction. The model-visible content stays identical.
+- An added 180-episode synthetic evaluation exercises all 45 targets in all four conditions. An independent arithmetic calculation agrees with every aggregate score and paired gain, including invalid-output penalties.
+- Existing tests cover exact probability keys, normalization tolerance, invalid and missing output, equal target weighting, tied winners, paired group bootstrap intervals, reference-only baselines, deterministic example selection, inspection limits, action limits, model-turn limits, denied actions, retries, replay, and CLI aliases.
+- All 62 automated tests pass. `npm run build`, `npm pack`, and `git diff --check` pass. From the extracted package, `predict`, `learn`, `read-the-room`, and `read-the-room --controls` each pass fresh-run and resume checks outside the repository, with the correct strategies, 4,000-token default, Learn policy, and zero API calls. The audit documents are included in the package.
+
+## Existing GLM evaluation
+
+The completed GLM 5.3 Flash / Relace run needs no rerun because of these fixes. Its JSON, Markdown, and HTML reports reproduce exactly using the updated code. A separate calculation from the saved raw prediction JSON and original quadrant counts confirms:
+
+| Condition | Score / 100 |
+|---|---:|
+| Predict (`none`) | 87.93467443347016 |
+| Random examples | 86.47321119295975 |
+| Similar examples | 85.42762611497108 |
+| Learn (`agent`) | 85.56778927968283 |
+
+All 180 predictions are valid. The saved attempts total 347 API calls and $0.0314992195 reported cost. Learn's paired gain over Predict is −2.36688515378733 points. Successful historical results remain comparable on the same prediction protocol; the execution-version difference should remain disclosed.
+
+## Freeze before the campaign
+
+Use `read-the-room --controls` so each model gets Predict, random examples, similar examples, and Learn on the same 45 evaluation targets. The original proposal used 4,000 output tokens per turn. Following the development checks below, the [final plan](final-run-plan-8000.md) supersedes that allowance with 8,000 for every model. Fix seed 7, evidence budget 3, maximum eight model turns, maximum eight environment actions, and agent policy `transfer-check-v1`. Keep one predeclared run per model/provider combination. Run models sequentially to make interruptions and per-run budget stops easy to inspect.
+
+Record the exact model, provider tag, quantization, runner version, pricing/capability snapshot, and spending cap before each run. Include failed model outputs in the scores. Do not increase token limits or change prompts in response to final evaluation scores. Protocol changes prompted by development compatibility failures must be decided and documented before the remaining evaluation runs; assess historical comparability explicitly if such changes are needed.
+
+The ten-provider catalog check found the planned MiMo `gmicloud/bf16` endpoint marked unavailable (`status: -2`). Its native `xiaomi/fp8` endpoint was listed available and is a candidate replacement, subject to refreshed pricing, an explicit provider/precision declaration, and a development pilot. The other nine selected endpoints were listed available; that is not proof that a paid request will succeed. See the [dated provider snapshot](preflight-providers-2026-11-14.json).
+
+## Limits that should remain explicit
+
+- **What Learn measures:** in-context use of historical examples, not model training or weight updates. It gets more calls and different instructions than Predict. The gain measures the whole interactive procedure. The controls help compare evidence choices but do not fully isolate the causal value of autonomous selection from extra computation.
+- **Generation settings:** use temperature zero where supported. GPT-6 Luna and Claude Haiku 5.5 omit unsupported temperature and record `null`. Reasoning effort is not explicitly set; provider defaults differ. This is a comparison under a common output-token cap, not equal reasoning effort or equal compute. Reasoning can consume the allowance before visible JSON appears.
+- **Evidence strength:** 45 targets and one run per model support an exploratory comparison. Bootstrap intervals describe target variation, not all variability from model sampling, participant selection, or service changes. Avoid strong rankings from small score differences or unadjusted multiple comparisons.
+- **Public data:** this dataset was already public and used by the original benchmark. The split prevents runtime label leakage; it cannot establish absence from model training. The existing final GLM results also mean the evaluation set is no longer an untouched holdout for future policy development.
+- **Audience:** results describe this self-selected Mixed Messages audience and historical period, not a universal correct political or cultural judgment. Chronological audience changes may affect transfer from reference examples.
+- **Accounting:** unknown charges retain conservative reservations. Reservations can stop a run well below its eventual actual cost limit. Provider price ceilings and local accounting cannot control unrelated account spending or guarantee provider billing behavior.
+- **Reproduction:** hosted model weights and serving implementations cannot be frozen by this repository. Exact saved responses and scores can be replayed; new API outputs need not match.
+
+## Remaining launch checks
+
+1. Use the corrected source checkout or prepare and publish a patch release. Preserve the existing final GLM artifacts.
+2. Resolve the unavailable MiMo endpoint, refresh provider metadata and prices, and freeze the provider/precision list and individual caps. The earlier cost estimate is a planning estimate, not a guaranteed campaign price.
+3. After authorizing a small separate budget, run three **development** targets per model with all four conditions. Check valid JSON, output truncation, requested settings, action handling, reported usage, and resume behavior. Do not use these few scores to rank models. This is the remaining live compatibility gate; it has not been run during this audit.
+4. If those checks pass, run the predeclared final evaluation sequentially with per-model caps. Use the public Predict/Learn commands, not legacy batch scripts for the original winning-quadrant benchmark. Preserve incomplete runs for review rather than deleting them and rerunning to obtain a better score.
+
+Provider behavior references: [OpenRouter errors](https://openrouter.ai/docs/api_reference/errors-and-debugging), [provider routing and parameter support](https://openrouter.ai/docs/guides/routing/provider-selection), and [reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
+
+## Pilot prepared after the audit
+
+The [development pilot plan](development-pilot-2026-11-14.json) fixes the ten model/provider pairs, three development targets, protocol, pricing snapshots, and individual spending caps totaling $3.45. MiMo uses the available native `xiaomi/fp8` endpoint. The estimate is approximately $0.19 by repricing the earlier GLM token usage; it is not a guaranteed price. This pilot does not authorize or launch final evaluation.
+
+The pilot subsequently completed: 116/120 predictions were valid for $0.11486105754 in reported cost. MiniMax and DeepSeek exhausted the 4,000-token allowance on four one-shot predictions; all Learn episodes were valid. The full campaign remains on hold pending the token-allowance decision. See the [saved pilot findings](https://github.com/1999labs/RTR-bench/blob/main/results/development-preflight-2026-11-14/README.md).
+
+An authorized follow-up at 8,000 tokens then produced 24/24 valid predictions across MiniMax and DeepSeek for $0.00999899252, with no truncation. DeepSeek used up to 7,665 output tokens. This was a development check only; CLI defaults and final protocol settings were not changed. See the [8,000-token findings](https://github.com/1999labs/RTR-bench/blob/main/results/development-token-check-8000-2026-11-14/README.md).
+
+The final campaign plan now fixes 8,000 tokens for all eleven models, including a matching GLM Flash rerun. Public command defaults in the prepared 0.2.1 release have been aligned to it. The historical audit and pilot measurements above retain their original settings. No final campaign calls were made while preparing this plan.