rightmodeler 0.2.1 → 0.4.0

package/docs/gateways.md

@@ -0,0 +1,109 @@
+# Gateways
+
+Rightmodeler replays and judges through any OpenAI-compatible base URL, so a gateway in front of your models can be the replay route. This guide covers open-source gateways verified at pinned releases.
+
+## What a replay route needs
+
+- `--base-url` is the gateway's OpenAI-compatible root, ending in `/v1`, and `--api-key-env` names the variable holding the key the gateway expects.
+- The gateway's `/v1/models` lists the replay models. When it lists ids without prices or capabilities, pass `--catalog-reference` with the upstream's public model list; see [Catalogs without pricing or capabilities](getting-started.md#catalogs-without-pricing-or-capabilities).
+- Headers the gateway reads per request are passed with `--header 'name: value'`; see [Gateways that route by header](getting-started.md#gateways-that-route-by-header).
+- The replay route runs no fallbacks, model aliases, response caching or request plugins for the replay models. Rightmodeler never asks a gateway for them, and it checks each replayed response (in Mode B, each response to a step whose model it sets): one that names another model, reports a cache hit, or reports a changed request is left out of the evidence as `attribution_substituted`; see [Which model answered](getting-started.md#which-model-answered).
+- Replay latency includes the gateway hop; rightmodeler does not separate the gateway's share.
+- Mode B on the cloud backend calls the base URL from a remote sandbox, so it cannot reach a gateway on localhost or a private network. Use the Docker backend for a local gateway.
+
+## Portkey
+
+Verified on the open-source Portkey gateway 1.15.2 (MIT, `portkeyai/gateway:1.15.2`), the latest tagged release as of 2026-09-22. Portkey picks the upstream for each request from two headers and needs no server configuration:
+
+```sh
+docker run -d --name portkey -p 127.0.0.1:8787:8787 portkeyai/gateway:1.15.2
+npx rightmodeler init --traces ./traces.jsonl --repo . \
+  --base-url http://127.0.0.1:8787/v1 --api-key-env AI_GATEWAY_API_KEY \
+  --header 'x-portkey-provider: openai' \
+  --header 'x-portkey-custom-host: https://ai-gateway.vercel.sh/v1' \
+  --max-cost-usd 25
+```
+
+- Use the `openai` provider with a custom host for any OpenAI-compatible upstream (Vercel AI Gateway, OpenRouter, a LiteLLM proxy). Portkey's `openrouter` provider rewrites requests and does not serve the model catalog.
+- `--api-key-env` holds the upstream's key: Portkey forwards `Authorization` to the upstream and has no key of its own.
+- The catalog, per-call cost, and rate-limit and credit errors come from the upstream through Portkey unchanged.
+- Rightmodeler sends no `x-portkey-config`, so fallbacks, load balancing, retries, caching and guardrail or mutator hooks stay off. If you pass one with `--header`, a replayed response whose model differs (`override_params`, targets), that reports `x-portkey-cache-status: HIT`, or whose hook results show `transformed: true` is left out as `attribution_substituted`.
+- Bind the container to 127.0.0.1, or start it headless (`docker run ... portkeyai/gateway:1.15.2 run start:node -- --headless`): the 1.15.2 console and live log stream are served without authentication and show provider keys.
+- Portkey 1.15.2 keeps no traces or logs that rightmodeler can read. Export traces from your application: OpenTelemetry GenAI, AI SDK telemetry, or the OpenAI SDK JSONL shape.
+
+## Envoy AI Gateway (Agent Router)
+
+Envoy AI Gateway was renamed Agent Router on 2026-09-09; its resources, `aigw` CLI and images keep their names. Verified on v1.1.0 (Apache 2.0) running standalone with `aigw run` (`envoyproxy/ai-gateway-cli:v1.1.0`); always use a release tag, because the `latest` image follows the development branch.
+
+As a replay route:
+
+- Declare each replay model as an `Exact` `x-ai-eg-model` match under the upstream's own id, one `backendRef`, no `modelNameOverride`, no priority fallback and no `BackendTrafficPolicy` retries on the replay route. A fallback or an override answers with another model, so a replayed response it answers is left out as `attribution_substituted`.
+- Raise the Gateway's `ClientTrafficPolicy` `bufferLimit` (Envoy Gateway's 32 KiB default is too small for real prompts) and the route's `timeouts.request` for slow models.
+- The gateway lists only the declared ids at `/v1/models`, so pass `--catalog-reference` with the upstream's public list.
+- The gateway replaces `Authorization` with the route's key, so `--api-key-env` may name any non-empty variable.
+
+```sh
+docker run --rm -p 127.0.0.1:1975:1975 -e AI_GATEWAY_API_KEY \
+  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 \
+  -e 'OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES=agent-session-id:session.id,x-rightmodeler-family:rightmodeler.family,x-rightmodeler-replay:rightmodeler.replay' \
+  -v "$PWD/aigw.yaml:/config.yaml:ro" envoyproxy/ai-gateway-cli:v1.1.0 run /config.yaml
+export AIGW_CLIENT_KEY=unused
+npx rightmodeler init --traces ./spans.jsonl --repo . \
+  --base-url http://127.0.0.1:1975/v1 --api-key-env AIGW_CLIENT_KEY \
+  --catalog-reference https://ai-gateway.vercel.sh/v1/models \
+  --header 'x-rightmodeler-replay: 1' --max-cost-usd 25
+```
+
+As a trace source, rightmodeler reads the gateway's default OpenInference spans from an OpenTelemetry collector's file export:
+
+- Set `OTEL_EXPORTER_OTLP_ENDPOINT`. Map headers to span attributes with `OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES` as comma-separated `header:attribute` pairs; setting it replaces the default `agent-session-id:session.id`, so repeat that pair.
+- Send `agent-session-id` from your application so the steps of one conversation form one ordered run, and `x-rightmodeler-family: <name>` so each call has its family. Send `x-rightmodeler-replay: 1` with rightmodeler's replays (`--header`) so a later export leaves them out.
+- Each span's request body gives the model your application asked for and the conversation as sent; the response gives the output; the token counts give usage.
+- Failed calls, replay-tagged calls, and calls whose prompt or output the gateway hid (`OPENINFERENCE_HIDE_INPUTS`, `OPENINFERENCE_HIDE_OUTPUTS`) are left out of the corpus with a `trace_steps_excluded` warning that names each reason.
+- A span records the model that answered but not whether a priority fallback chose it (a `modelNameOverride` alias looks the same), so a call that fell back is read as an answer to the model your application asked for, with the fallback's output and usage. Keep fallback routes off the traffic you export, or expect those answers among the recorded outputs rightmodeler compares candidates against.
+- Steps whose recorded conversation contains tool calls are read but not replayed yet (`recorded_messages_not_replayable`).
+- With `AI_GATEWAY_TRACING_SEMCONV=gen_ai`, the spans are read by the OTel GenAI reader instead. Also set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`, because that convention records no messages without it and rightmodeler cannot read a span with none. Those spans are grouped by trace (propagate `traceparent` from your client), carry no `response_format` or `tool_choice`, and record images by type only.
+- Access logs carry no message content and are not a trace source.
+
+On Kubernetes (Kubernetes 1.32 or newer, Envoy Gateway 1.8.1 or newer, Helm charts `ai-gateway-crds-helm` and `ai-gateway-helm` v1.1.0), the same resources apply; set `OTEL_EXPORTER_OTLP_ENDPOINT` through the `ai-gateway-helm` chart's `extProc.extraEnvVars` and the header mapping through its `controller.spanRequestHeaderAttributes`. Mode B on the cloud backend cannot reach an in-cluster gateway.
+
+## Bifrost
+
+Verified on the open-source Bifrost gateway transports/v2.2.1 (Apache 2.0, `maximhq/bifrost:v2.2.1`); pin the image, because releases arrive weekly. Enterprise features ship in a separate image; this guide uses the open-source one.
+
+As a replay route:
+
+- Model ids are `<provider>/<upstream id>`, for example `vercel/openai/gpt-4o-mini` for Vercel AI Gateway configured as an OpenAI-typed custom provider. Bifrost answers with the upstream id, which rightmodeler accepts as the requested model.
+- Set every `compat` flag to `false` in the `client` block: a `client` block that omits them turns them all on, and the compat plugin can drop parameters such as `response_format` while answering 200.
+- Configure no key `aliases` or routing rules for replay models; an alias answers with another model and is left out as `attribution_substituted`.
+- Pass `--catalog-reference` with the upstream's public list. A custom provider lists ids and context only, with no prices or capabilities, and through Vercel AI Gateway every model carries the same `created` date: Bifrost copies Vercel's `created`, one placeholder for all models, and drops the real `released` date. Rightmodeler ranks judges partly by release date, and the reference's release dates win over the gateway's, so judges rank as they do on Vercel itself instead of the most expensive first. Bifrost's own pricing sheet is not used.
+- The billed cost comes from the upstream through Bifrost's `usage.cost.total_cost`.
+- Send `x-bf-cache-no-store: true` and `x-bf-dim-rightmodeler: replay` with rightmodeler's replays (`--header`): the first keeps replays out of a semantic cache (a cache hit is still detected and left out), the second tags them so a later log export leaves them out.
+
+```sh
+docker volume create bifrost-data
+docker create --name bifrost -p 127.0.0.1:8080:8080 -e AI_GATEWAY_API_KEY -v bifrost-data:/app/data maximhq/bifrost:v2.2.1
+docker cp config.json bifrost:/app/data/config.json && docker start bifrost
+export BIFROST_KEY=unused
+npx rightmodeler init --traces ./bifrost-logs.jsonl --repo . \
+  --base-url http://127.0.0.1:8080/v1 --api-key-env BIFROST_KEY \
+  --catalog-reference https://ai-gateway.vercel.sh/v1/models \
+  --header 'x-bf-cache-no-store: true' --header 'x-bf-dim-rightmodeler: replay' \
+  --max-cost-usd 25
+```
+
+As a trace source, rightmodeler reads Bifrost's own request logs, exported from its management API (add credentials once an admin exists):
+
+```sh
+curl -s 'http://127.0.0.1:8080/api/logs?objects=chat_completion,chat_completion_stream&order=asc&limit=1000&offset=0' | jq -r '.logs[].id' \
+  | while read -r id; do curl -s "http://127.0.0.1:8080/api/logs/$id"; echo; done > bifrost-logs.jsonl
+```
+
+With more than 1000 rows, repeat with `offset=1000`, `2000` and so on, appending (`>>`) to the same file, until a page returns fewer than 1000 rows; `order=asc` keeps earlier pages in place while new calls are logged. Do not pass `roots_only=true`: it hides fallback rows.
+
+- Send `x-bf-session-id` from your application so the steps of one conversation form one ordered run, and `x-bf-dim-rightmodeler-family: <name>` so each call has its family.
+- Each row gives the model the application asked for (`provider/model`, or the alias it sent), the conversation as sent (`input_history`), the output (`output_message`), usage, cost, latency and retries.
+- Failed calls, answers from a configured fallback, rightmodeler's own replays and calls whose content was not logged are left out of the corpus with a `trace_steps_excluded` warning that names each reason. Steps whose conversation contains tool calls are read but not replayed yet.
+- Bifrost's OpenTelemetry plugin export reads through the OTel GenAI reader, but it summarizes messages and loses tool-call ids: use the log export for anything but plain text calls.
+- Streamed calls through Mode B are checked on the stream's model and headers only; Bifrost reports a semantic-cache hit only in the response body, so such a hit on a streamed Mode B call is not detected.
+- Rightmodeler records every replay's latency through Bifrost, gateway hop included, and does not separate Bifrost's own overhead. Bifrost's published benchmark reports 11 microseconds of overhead on a t3.xlarge at 5,000 requests per second, excluding JSON marshalling and the HTTP call; measure it on your own traffic.

package/docs/getting-started.md

@@ -1,16 +1,23 @@
 # Getting started
 
-Rightmodeler analyzes recorded model calls, replays them against cheaper candidates, evaluates the outputs, and writes a recommendation report. The intended public package name is `rightmodeler`.
+Rightmodeler analyzes recorded model calls, replays them against cheaper candidates, evaluates the outputs, and writes a recommendation report. Published on npm as `rightmodeler`.
 
 ## Requirements
 
 - Node.js 24 or newer.
 - A Git repository to analyze.
 - Trace input in a supported format.
-- An OpenAI-compatible provider base URL and the name of an environment variable containing its API key before replay begins.
+- An OpenAI-compatible provider base URL and the name of an environment variable containing its API key before replay begins. Its `/v1/models` catalog should publish per-token pricing. OpenRouter and Vercel AI Gateway do. For a LiteLLM endpoint, Rightmodeler can fall back to `GET /model/info`; for a gateway that lists bare model ids or a direct OpenAI or Anthropic key, pass `--catalog-reference`; for another unpriced endpoint, pass `--pricing-file`.
 
-Supported trace sources are OTel GenAI, OpenAI JSONL, Langfuse, Braintrust,
-LangSmith, OpenInference, Helicone, W&B Weave, Claude Code, and Codex.
+Instead of an API key, candidates or the judge can run through the `claude`
+CLI you are signed in to on this machine, under your Claude plan:
+`--route claude-login` with a `--judge-route` from another vendor. Run
+`rightmodeler docs model-routes` for what a plan route measures, what it costs
+your plan, and its safeguards.
+
+Supported trace sources are OTel GenAI, AI SDK telemetry, OpenAI JSONL,
+Langfuse, Braintrust, LangSmith, OpenInference, Helicone, Bifrost, W&B Weave,
+Claude Code, and Codex.
 
 ## Start with automatic discovery
 
@@ -20,6 +27,13 @@ Run this from the repository you want to analyze:
 npx rightmodeler init
 ```
 
+Before it looks for traces, an interactive `init` or `estimate` that will reach
+replay asks how to call models: through the `claude` or `codex` CLI signed in on
+this machine, or through an API key for OpenRouter, Vercel AI Gateway, OpenAI,
+Anthropic or another OpenAI-compatible endpoint. It prints the equivalent flags
+and saves the answer as the next default, which `--yes` applies without asking.
+`npx rightmodeler docs model-routes` describes the choice.
+
 Rightmodeler checks conventional local trace files, Claude Code transcripts for
 the repository, and Codex sessions whose recorded working directory matches the
 repository. In an interactive terminal it lists matches newest-first with an
@@ -40,6 +54,27 @@ npx rightmodeler init --plan --output json --repo /path/to/repository
 npx rightmodeler init --through corpus --traces /path/to/traces.json --output json --repo /path/to/repository
 ```
 
+`--traces` accepts a single file or a directory. A directory is read non-recursively as its `.json` and `.jsonl` files in name order; every file must use the same trace format.
+
+Each family is replayed only against the call sites its own traces came from. Traced cases that cannot be tied to one such call site, because several call sites use the traced model or none matches it, are left out of the replay sample with a `family_cases_left_out` warning. A family with no case left abstains before any spend with `ambiguous_call_site_binding` or `unmatched_call_site_binding`. The AI SDK telemetry `functionId` (see below) is the way to tie an AI SDK call site to its family.
+
+## AI SDK telemetry
+
+The AI SDK emits telemetry in two dialects, and Rightmodeler reads both:
+
+- The `ai.*` dialect comes from AI SDK 5 and 6 with `experimental_telemetry: { isEnabled: true }` on each call, and from AI SDK 7 with `registerTelemetry(new LegacyOpenTelemetry())`. The AI SDK reader reads it.
+- The GenAI semantic conventions dialect comes from AI SDK 7 with `registerTelemetry(new OpenTelemetry())`. The OTel GenAI reader reads it and treats the agent, step and tool spans as structure, so each model call counts once.
+
+`registerTelemetry` comes from `ai`; `LegacyOpenTelemetry` and `OpenTelemetry` come from `@ai-sdk/otel`. Register only one of them: an export that holds both dialects is ambiguous. Keep `recordInputs` and `recordOutputs` on, which is the default, because a model call without its prompt or output cannot become a corpus case. Set a string-literal `functionId` on every call (`telemetry: { functionId: "summarize" }` in AI SDK 7, `experimental_telemetry: { isEnabled: true, functionId: "summarize" }` before it); it becomes the call's family.
+
+The scanner records that `functionId` on the call site, and a family binds to exactly the call sites whose `functionId` equals its name: its evidence and any swap stay on those call sites, and no other family borrows them. Two call sites that do the same job may share one `functionId`. A family bound to a single call site can still be recommended. The `functionId` must be a string literal in the call; a variable or a template literal is not read, and the call site then binds by model id only. Replay cannot run a call site that needs tools or structured output: its cases are left out of the family's replay sample with a `family_cases_left_out` warning, and a family whose traced cases all come from such call sites abstains with `bound_call_sites_not_replayable` before any spend.
+
+Export the spans through the OpenTelemetry NodeSDK or `@vercel/otel` to an OTLP collector, and pass the collector's file exporter output with `--traces`. A model call that ended without a finish reason, because it was aborted or errored, is left out of the corpus with a `trace_steps_excluded` warning, and the rest of the input is read. Token usage from AI SDK 4 exports (`ai.usage.promptTokens`) is not read, so those calls carry no usage.
+
+## OpenInference spans
+
+When an OpenInference span carries the request body in `input.value` (Envoy AI Gateway, and the OpenAI instrumentation), rightmodeler reads the model your application asked for and the conversation exactly as sent from it, and the output from `output.value`. Spans that share a `session.id` form one run ordered by start time. A `rightmodeler.family` attribute names the family; otherwise the span name does. Failed calls, calls tagged `rightmodeler.replay`, and calls whose content was hidden are left out with a `trace_steps_excluded` warning.
+
 ## Run the complete pipeline
 
 ```sh
@@ -51,6 +86,8 @@ npx rightmodeler init --traces /path/to/traces.json --base-url https://provider.
 `--api-key-env <name>` to use a different exported variable. The CLI does not ask
 for a secret value.
 
+Replay resends each recorded conversation as text. A recorded case whose conversation contains tool calls, non-text parts, or tool definitions is left out of the replay sample with a `recorded_messages_not_replayable` warning; the family's other cases replay, and a family left with too few cases abstains under the usual sample-size reasons.
+
 ## Estimate replay spend
 
 ```sh
@@ -59,7 +96,134 @@ npx rightmodeler estimate --traces /path/to/traces.json --base-url https://provi
 
 Estimate projects candidate replay spend from recorded token usage and the current
 model catalog before paid model calls begin.
+`--max-cost-usd` caps candidate replays and judge calls together: each call reserves
+its worst case before it is sent, and a call the cap cannot cover is not sent.
+
+## Static code context (Graphify)
+
+rightmodeler can read a code graph built by the open-source Graphify CLI (PyPI package `graphifyy`, Apache-2.0, tested with 0.9.65). Graphify builds it locally from your source, with no account and no model call.
+
+```sh
+uv tool install graphifyy
+graphify update .
+npx rightmodeler report --code-graph graphify-out/graph.json --repo .
+```
+
+`init --code-graph <path>` renders the same section at the end of a run. For each call site the scanner found, the section lists the enclosing symbol, its callers, the tests that reach it, and the owners of those files, which are listed only. It also lists files that import an AI SDK where the scanner found no call site.
+
+Graph edges are never replay trials, runtime proof, or quality evidence, and the flag never changes a stage before the report, a verdict, a gate, or confirmation. Each finding is labelled EXTRACTED, or INFERRED or AMBIGUOUS to verify, by its weakest hop. A graph built at another commit is shown file-level with a stale note. An unusable graph produces one warning, the section says why it is not shown, and the rest of the report is unchanged. The scan ignores `graphify-out/`, so building a graph never makes finished stages stale. Only `graphify update` and `graphify extract --code-only` are needed; other Graphify commands can call a language model.
+
+`apply --code-graph <path>` appends the same section to the draft pull request body, limited to the call sites the pull request swaps and to five findings of each kind per call site. Owners there are listed only and are never requested as reviewers; reviewers still come from CODEOWNERS and blame. The graph never changes the swap, its digest, or its reviewers. `apply --dry-run` prints the exact body it would post, with or without `--code-graph`.
+
+## Release policy
+
+`--policy <path>` is accepted by `init`, `estimate`, `replay`, and `confirm`. The JSON object can set the quality floor, shortlist size, and model allow and deny lists:
+
+```json
+{
+  "qualityFloor": 0.9,
+  "shortlistTop": 5,
+  "allowModels": ["acme/small-1"],
+  "denyModels": ["acme/large-1"]
+}
+```
+
+`qualityFloor` must be greater than 0.8 and less than 1, and `shortlistTop` must be a positive integer. Changing the policy changes the stamped gate policy version, so shortlist and replay run again instead of pooling evidence gathered under the old policy.
+
+## Catalogs without pricing or capabilities
+
+Rightmodeler reads per-token pricing from the model catalog. When every catalog
+entry has null pricing and no `--pricing-file` is set, it requests LiteLLM
+`GET /model/info` on the same host. Use `--pricing-file` when `/model/info` has
+no usable per-token costs; file entries override provider pricing.
+
+Some gateways answer `/v1/models` with bare model ids: Envoy AI Gateway lists
+the models a route declares, and Bifrost lists custom providers with ids and
+context only. Pass `--catalog-reference <url|path>` to name the upstream's own
+model list, for example `https://ai-gateway.vercel.sh/v1/models` or
+`https://openrouter.ai/api/v1/models`. Rightmodeler reads it without your key or
+headers and joins it to the gateway's entries by id, or by the reference id a
+gateway id ends with (`openai/gpt-4o-mini` for `vercel/openai/gpt-4o-mini`). A
+gateway entry takes from its match only what it does not declare itself (price,
+context window, output ceiling, tool and structured-output support); a match
+that is not a language model removes the entry; `--pricing-file` values win over
+both. Entries still unpriced after the join are named in a
+`catalog_reference_unmatched` warning, and a reference that cannot be read stops
+the run with `invalid_catalog_reference`.
+
+The release date is the one field where the reference wins over the gateway:
+when the matched reference entry has a release date, it replaces the date the
+gateway declares, and the gateway's date is kept only when the reference gives
+none. Rightmodeler ranks judges partly by how recent a model is, and a gateway
+can give every model the same placeholder date (Bifrost does for Vercel AI
+Gateway's models), which would leave context and price to rank the judges and
+favor the most expensive.
+
+```sh
+npx rightmodeler estimate --base-url https://provider.example/v1 --pricing-file /path/to/pricing.json --repo /path/to/repository
+```
+
+The pricing file maps each model id to input and output USD per token and may
+include the model's output ceiling:
+
+```json
+{
+  "acme/model": {
+    "input": 0.000001,
+    "output": 0.000002,
+    "maxOutputTokens": 4096
+  }
+}
+```
+
+Without usable pricing from the catalog, a catalog reference, LiteLLM
+`/model/info`, or a pricing file, the run refuses with `no_priced_candidates`
+instead of reporting zero cost. The judge must be priced too, so price at least
+one model from a family other than the current model's and the candidate's.
+
+### Direct OpenAI and Anthropic keys
+
+Point `--base-url` at `https://api.openai.com/v1` or
+`https://api.anthropic.com/v1` and name the variable that holds the key with
+`--api-key-env`. Neither vendor's model list publishes prices, so also pass
+`--catalog-reference https://ai-gateway.vercel.sh/v1/models`. Rightmodeler takes
+the vendor from these two hosts: their bare ids are that vendor's models and
+join the reference as `openai/<id>` or `anthropic/<id>`. Dated and dotted forms
+of a name match within one vendor, in the reference and in your traces, so
+`claude-haiku-4-5-20251001` matches `anthropic/claude-haiku-4.5`. For Anthropic,
+rightmodeler sends `anthropic-version: 2023-06-01` (a
+`--header 'anthropic-version: ...'` value wins), reads every page of the model
+list, and never asks for structured output, which Anthropic's OpenAI-compatible
+endpoint ignores. For OpenAI, rightmodeler caps a reply's length with
+`max_completion_tokens`, which OpenAI's chat reference names in place of the
+deprecated `max_tokens`; every other host still gets `max_tokens`.
+
+The built-in judge must come from a vendor other than both the candidate's and
+the recorded model's, and rightmodeler checks this before any paid call, also
+when an unreachable `--evaluator` falls back to the built-in judge. One vendor's
+key lists only that vendor's models, so a run on it alone stops with
+`no_neutral_judge` unless the judge runs through the other vendor's CLI you are
+signed in to (`--judge-route`, see `rightmodeler docs model-routes`) or your own
+evaluator grades the replays. Bare ids from any other host name no vendor, and
+a run on them stops with `judge_family_unknown`; use a gateway whose ids carry
+their vendor (`vendor/model`), or `--evaluator`.
+
+## Which model answered
+
+Rightmodeler checks every replay and judge response, and in Mode B every response to a step whose model it sets, before it counts. It records the model the response names, and it leaves a response out of the evidence when that model is not the one it asked for, when a gateway reports that it answered from its cache (Portkey's `x-portkey-cache-status: HIT`, Bifrost's `cache_debug.cache_hit`), or when a gateway reports that it changed the request (a Portkey hook with `transformed: true`, Bifrost's compat plugin dropping parameters). Such a response is recorded with `attribution: "substituted"`, is never graded, and counts as `attribution_substituted`; more than 5% of a family's replays substituted abstains the family. A `replay_responses_substituted` warning counts them.
+
+A response names the requested model when it echoes it, when it drops a gateway provider prefix (`openai/gpt-4o-mini` for `vercel/openai/gpt-4o-mini`), or when it adds a dated snapshot (`gpt-4o-mini-2024-07-18` for `gpt-4o-mini`). An alias whose answering model has another name counts as substituted, so name replay models by their upstream ids and turn off fallbacks, model aliases, response caching and request plugins for the replay route. Completed replay cells are reused, so after fixing the route rerun with a fresh store (`--store <directory>`). For streamed Mode B calls only the model a stream names and the response headers are checked.
+
+A judge whose response names another model is retired at that first response: rightmodeler starts no new call to it, cancels its calls still waiting for budget, and re-judges the affected replays with the next-ranked judge once the calls already under way return; the `judge_unusable` warning names the model that answered. Some catalogs also list a faster service tier of a model under its own id, which the gateway answers as the base model: on Vercel AI Gateway, `<id>-fast` is `<id>` at its fast tier. Rightmodeler never ranks such an id as a judge or shortlists it as a candidate while `<id>` is in the catalog too; an id that ends in `-fast` with no base model listed, such as `morph/morph-v3-fast`, is treated like any other model.
+
+## Gateways that route by header
+
+Some gateways choose the upstream, the cache policy, or a trace tag from request headers. Pass each one with `--header 'name: value'`; repeat the option for more. Rightmodeler sends them with every request it makes to the provider base URL: the model catalog, candidate replays, judge calls, and the calls Mode B makes from your application, where a configured header replaces one your application sends. `authorization` comes only from `--api-key-env`, and `content-type`, `content-length` and `host` are set by rightmodeler, so none of them can be passed as a header. Completed replay cells are reused when the headers change, so after changing the route rerun with a fresh store (`--store <directory>`). A detached replay run is keyed to the header values by their SHA-256 digests; the values themselves are passed to the detached worker on its command line and are not written to the store. Do not put secrets in headers.
+
+The default store is `.rightmodeler/` inside the analyzed repository. Completed stages resume when their inputs and outputs are still current. A complete run writes `.rightmodeler/project/reports/report.md`. The JSON report is kept inside the versioned store and is never written as a plain file, so read the final `result` event from `--output json` or `--output jsonl` for the machine-readable outcome.
+
+Read the generated [command reference](commands.md), the [evaluator guide](evaluators.md), [Mode B configuration](modeb.md), the [gateway guide](gateways.md), and the [exit-code convention](exit-codes.md) before automating a full run.
 
-The default store is `.rightmodeler/` inside the analyzed repository. Completed stages resume when their inputs and outputs are still current. A complete run writes `.rightmodeler/project/reports/report.md` and `.rightmodeler/project/reports/report.json`.
+To open the draft pull request and keep it reconciled, read the [GitHub guide](github.md).
 
-Read the generated [command reference](commands.md), the [evaluator guide](evaluators.md), [Mode B configuration](modeb.md), and the [exit-code convention](exit-codes.md) before automating a full run.
+Run `rightmodeler docs <name>` to print any of these documents from the installed package.