rightmodeler 0.2.1 → 0.3.0

package/docs/evaluators.md

@@ -1,16 +1,95 @@
 # Evaluators
 
-The default evaluator is the built-in judge selected from the configured provider catalog. Candidate and reference families are excluded when choosing the judge.
+The default evaluator is the built-in judge selected from the configured provider catalog. Candidate and reference families are excluded when choosing the judge. The exclusion applies per replayed call site: when the call sites of one traced family use models of different vendors (for example `openai/...` and `anthropic/...`), each call site's cases are judged by a model from neither the candidate's model family nor that call site's. Confirmation judges the recorded final output, so its judge also comes from outside the final step's model family.
 
-An external evaluator can be requested with `--evaluator braintrust`. Configure it with:
+An external evaluator is requested with `--evaluator <provider>` on `init`, `estimate`, `replay`, and `confirm`, where the provider is `braintrust`, `langfuse`, `langsmith`, or `promptfoo`. Every other `--evaluator-*` option is a usage error without `--evaluator`.
 
-- `--evaluator-project-id <id>` (required)
-- one or more `--evaluator-scorer <name>` options (required)
-- `--evaluator-api-key-env <name>` (defaults to `BRAINTRUST_API_KEY`)
-- `--evaluator-base-url <url>` (defaults to the provider API)
-- `--evaluator-gate-metric <name>` when more than one scorer is configured
-- `--evaluator-gate-threshold <value>` when the evaluator does not return a pass decision
+## Options shared by every provider
 
-The gate metric must name one of the configured scorers. If the external evaluator is unavailable, the pipeline warns and uses the built-in judge. If a reachable evaluator run fails or omits required case results, the pipeline records the missing assessments instead of fabricating scores.
+- `--evaluator-scorer <name>` (required, repeatable): the scorer names the provider must return for every case.
+- `--evaluator-gate-metric <name>`: the scorer used for release gates. Required when more than one scorer is configured; with a single scorer it defaults to that scorer. It must name a configured scorer.
+- `--evaluator-gate-threshold <value>`: pass threshold applied only when the evaluator omits a pass decision for a metric.
+
+## Provider options
+
+One table per provider. A rejected option is a usage error with that provider (exit code 2, `invalid_option`).
+
+### braintrust
+
+`--evaluator-project-id` names the Braintrust project that receives the experiment.
+
+| Option                              | Requirement | Default                      |
+| ----------------------------------- | ----------- | ---------------------------- |
+| `--evaluator-project-id <id>`       | required    |                              |
+| `--evaluator-api-key-env <name>`    | optional    | `BRAINTRUST_API_KEY`         |
+| `--evaluator-base-url <url>`        | optional    | `https://api.braintrust.dev` |
+| `--evaluator-public-key-env <name>` | rejected    |                              |
+| `--evaluator-command <path>`        | rejected    |                              |
+| `--evaluator-config <path>`         | rejected    |                              |
+
+### langfuse
+
+Langfuse authenticates with a public key and a secret key read from environment variables. The keys select the project, so there is no project option.
+
+| Option                              | Requirement | Default                      |
+| ----------------------------------- | ----------- | ---------------------------- |
+| `--evaluator-api-key-env <name>`    | optional    | `LANGFUSE_SECRET_KEY`        |
+| `--evaluator-public-key-env <name>` | optional    | `LANGFUSE_PUBLIC_KEY`        |
+| `--evaluator-base-url <url>`        | optional    | `https://cloud.langfuse.com` |
+| `--evaluator-project-id <id>`       | rejected    |                              |
+| `--evaluator-command <path>`        | rejected    |                              |
+| `--evaluator-config <path>`         | rejected    |                              |
+
+### langsmith
+
+`--evaluator-project-id` names the LangSmith dataset the experiment references. A scorer may be written `metric=rule-id` to bind a metric name to an evaluator rule; a bare name is used for both.
+
+| Option                              | Requirement | Default                           |
+| ----------------------------------- | ----------- | --------------------------------- |
+| `--evaluator-project-id <id>`       | required    |                                   |
+| `--evaluator-api-key-env <name>`    | optional    | `LANGSMITH_API_KEY`               |
+| `--evaluator-base-url <url>`        | optional    | `https://api.smith.langchain.com` |
+| `--evaluator-public-key-env <name>` | rejected    |                                   |
+| `--evaluator-command <path>`        | rejected    |                                   |
+| `--evaluator-config <path>`         | rejected    |                                   |
+
+### promptfoo
+
+promptfoo runs locally. `--evaluator-config` is the assertions file passed to `promptfoo eval --assertions`; `--evaluator-command` is the executable, one path or one command on `PATH` (so not `npx promptfoo`: install it with `npm install -g promptfoo@0.123.1` or pass the path to its binary). No API or project option applies.
+
+| Option                              | Requirement | Default     |
+| ----------------------------------- | ----------- | ----------- |
+| `--evaluator-config <path>`         | required    |             |
+| `--evaluator-command <path>`        | optional    | `promptfoo` |
+| `--evaluator-base-url <url>`        | rejected    |             |
+| `--evaluator-api-key-env <name>`    | rejected    |             |
+| `--evaluator-project-id <id>`       | rejected    |             |
+| `--evaluator-public-key-env <name>` | rejected    |             |
+
+rightmodeler is verified against promptfoo 0.123.1. Other releases work as long as they write the same results layout; one that does not stops the run with an error naming its version and the layout rightmodeler reads. A missing executable counts as unreachable (see below); any other promptfoo failure, such as a broken `promptfooconfig.*` next to the assertions file, stops the run with promptfoo's own message.
+
+Each evaluation runs one command in the assertions file's directory, with standard input closed:
+
+`<command> eval --assertions <assertions file> --model-outputs <model outputs file, relative to that directory> --output <temporary results file> --no-write --no-share --no-table --no-progress-bar`
+
+The command gets your environment plus `PROMPTFOO_DISABLE_UPDATE=true`, `PROMPTFOO_DISABLE_VAR_EXPANSION=true`, `PROMPTFOO_FAILED_TEST_EXIT_CODE=100`, `PROMPTFOO_SHORT_CIRCUIT_TEST_FAILURES=false`, `PROMPTFOO_STRIP_GRADING_RESULT=false`, `PROMPTFOO_STRIP_RESPONSE_OUTPUT=false`, and `PROMPTFOO_STRIP_TEST_VARS=false`. So every output is graded exactly once against every assertion, the batch is neither saved to promptfoo's history nor shared, promptfoo skips its startup update check, and your own promptfoo settings cannot change the exit code, turn a failed assertion into an error, or strip the fields rightmodeler reads. Everything else, including your promptfoo login, cache, telemetry setting, and model-graded assertion providers, works as when you run promptfoo yourself. promptfoo receives each candidate output and its execution id only; the input, messages, and reference answer are never sent.
+
+Put `metric: <name>` on every assertion that feeds an `--evaluator-scorer`. A metric's score is promptfoo's named score for it, and it passes only when every assertion carrying it passes. promptfoo always decides pass or fail, so `--evaluator-gate-threshold` never applies. Each assessment's rubric version is `promptfoo@<version>/<metric>/<digest>`, where the digest is the first 16 hex characters of the canonical SHA-256 of the assertions carrying that metric, so editing one of them, or upgrading promptfoo, changes it. A scorer that no assertion carries stops the run with an error naming it.
+
+Before grading, promptfoo renders `{{ }}` templates in an output, reads a `file://` path or loads a `package:` module it names, and strips one trailing newline. rightmodeler compares the output promptfoo graded with the output it sent. A case whose graded output differs beyond that one newline is recorded absent as `external_output_mismatch`, because the grade is not of the candidate's output; a case promptfoo could not grade at all, such as a `file://` path that does not exist, is recorded absent as `external_evaluator_error`, and the other cases in its batch keep their grades. Neither counts as evidence. Nor does a metric whose grader failed, which promptfoo marks `graderError` (for example when a model-graded assertion's grading call fails): it is left out rather than recorded as a failure, and a case that loses its gate metric this way is recorded absent as `external_evaluator_error`. Replayed outputs are kept in the store, so a rerun after fixing promptfoo does not repeat model calls.
+
+promptfoo also loads a `promptfooconfig.*` and a `.env` file from the directory it runs in, so any next to the assertions file apply: a `defaultTest` there adds its assertions to every case, a `defaultTest` transform makes every case `external_output_mismatch`, and its `env` block can override the settings above. Every grade, and a detached replay's identity (`replay --detach`), covers the assertions file and every `promptfooconfig.*` next to it (see [Changing an evaluator](#changing-an-evaluator)); a `.env` there is not covered. Keep the assertions file in a directory of its own unless you want those files to apply.
+
+## Changing an evaluator
+
+Every grade records the evaluator configuration it was made under: the provider, every evaluator option as given, and for promptfoo the bytes of the assertions file and of every `promptfooconfig.*` beside it. When that configuration changes, the next `init` or `replay` re-grades every stored candidate output that was graded under another configuration. The outputs are reused rather than replayed, so no candidate model is called again, and an `evaluator_regrade` warning counts the outputs being re-graded and says why. The earlier grades stay in the store, but no verdict or report uses them, and `export` leaves them out. A replay run after upgrading from a version that did not record the configuration re-grades once for the same reason.
+
+`--evaluator-scorer`, `--evaluator-gate-metric` and `--evaluator-gate-threshold` are different: they define the question the evidence answers, so changing one replays the candidates again. Whenever replay runs, a model the provider catalog now selects that was never replayed is replayed like any new candidate, within `--max-cost-usd`.
+
+rightmodeler cannot see a change it is not given: a scorer edited inside Braintrust, an evaluator edited inside Langfuse, a LangSmith rule edited in place, a file an assertion loads with `file://`, a `.env` beside the assertions file, or a promptfoo upgrade behind the same `--evaluator-command`. To re-grade after one of these, edit the assertions file, or bind the LangSmith metric to the new rule's id (`metric=rule-id`). For Braintrust and Langfuse, the grades already held stay in use until an evaluator option changes.
+
+## Reachability, polling, and absences
+
+If the external evaluator is unreachable, the pipeline warns and uses the built-in judge. Hosted scorers run asynchronously, so after launching a run the pipeline polls for up to 5 minutes, starting 250 ms after launch and doubling the wait between polls up to 10 seconds. If a reachable run fails, is still pending when the budget ends, or omits required case results, the pipeline records the missing assessments instead of fabricating scores.
 
 See [Commands](commands.md) for the complete option text and [Getting started](getting-started.md) for provider setup.

package/docs/exit-codes.md

@@ -14,13 +14,53 @@ Rightmodeler reserves exit codes `0` through `3` for machine-readable outcomes.
 
 - `0`: changes were applied, or a dry run was clean.
 - `1`: the change was refused by a machine gate.
+- `2`: no completed run in the store (for example `stage_not_completed`); the JSON error on standard error names the remedy.
+- `10` or greater: runtime failure.
+
+See [GitHub](github.md) for tokens, reviewers and refusal codes.
+
+## Rollback
+
+- `0`: the rollback pull request was opened, or already exists.
+- `1`: the rollback was refused by a machine gate.
+- `10` or greater: runtime failure.
+
+## Drift
+
+- `0`: the drift check completed.
+- `2`: `--traces` is missing.
 - `10` or greater: runtime failure.
 
 ## Watch
 
 - `0`: no action was required.
 - `1`: review or continuous-integration actions were taken.
-- `2`: another watcher holds the lock.
+- `2`: another watcher holds the lock, or the store has no completed run. A held lock prints a result with `"status":"lock_held"` on standard output; a missing run prints an error such as `stage_not_completed` on standard error and nothing on standard output.
 - `10` or greater: runtime failure.
 
 Use `--output json` for one result object or `--output jsonl` for stage events followed by the result. Errors use the selected machine-readable mode on standard error. See [Commands](commands.md) for command-specific options.
+
+## Error codes
+
+- `active_corpus_usage_unavailable` (exit `2`): publish a corpus version built from traces that include token usage, then rerun.
+- `ambiguous_trace_format` (exit `2`): pass a trace file that unambiguously matches one supported format.
+- `budget_cap_refusal` (exit `3`): the run reached the cost boundary; raise `--max-cost-usd` to the cap named in the remedy and rerun.
+- `coverage_gate_failed` (exit `2`): add matcher coverage for the listed AI dependency surfaces, or pass `--matchers <file>` with declarative matchers that close them.
+- `empty_traces_directory` (exit `2`): point `--traces` at a directory containing `.json` or `.jsonl` trace files, or at a single trace file.
+- `git_repository_has_no_commits` (exit `2`): create the first commit, then rerun the command.
+- `invalid_catalog_reference` (exit `2`): pass `--catalog-reference` an http(s) URL or a readable file that returns an OpenAI-compatible `/models` document, or remove it, then rerun.
+- `invalid_matchers_file` (exit `2`): fix the listed matcher definitions in the `--matchers` file and rerun.
+- `invalid_modeb_config` (exit `2`): fix the named field in the `--modeb-config` file and rerun.
+- `invalid_option` (exit `2`): correct the option and rerun; use `rightmodeler <command> --help` for accepted values.
+- `invalid_policy_file` (exit `2`): fix the named field in the `--policy` file and rerun; `qualityFloor` must be greater than 0.8 and less than 1, `shortlistTop` a positive integer, `allowModels` and `denyModels` arrays of model ids.
+- `invalid_pricing_file` (exit `2`): fix `--pricing-file` to map each model id to non-negative `input` and `output` USD per token and, optionally, a positive integer `maxOutputTokens`, then rerun.
+- `missing_provider_configuration` (exit `2`): pass `--base-url <url>` and, if needed, `--api-key-env <environment-variable-name>` naming a populated variable.
+- `missing_traces_path` (exit `2`): pass `--traces <path>` pointing to an existing trace file or directory.
+- `mixed_trace_formats` (exit `2`): split the directory so every file uses the same trace format, or pass one file with `--traces`.
+- `modeb_cloud_unavailable` (exit `2`): install the optional sandbox SDK and set its credentials, or set `"backend": "docker"` in the `--modeb-config` file, then rerun.
+- `no_priced_candidates` (exit `2`): point `--base-url` at a catalog that publishes per-token pricing, pass `--catalog-reference <url>`, expose priced LiteLLM `GET /model/info`, or pass `--pricing-file <path>`, then rerun.
+- `no_replayable_call_sites` (exit `2`): point `--repo` at a service with plain text completions, or add a matcher for a text call site, then rerun.
+- `not_git_repository` (exit `2`): run the command again from a Git repository with at least one commit.
+- `stage_not_completed` (exit `2`): run `rightmodeler init --through <stage>` first, then rerun the command.
+- `unusable_trace_input` (exit `2`): the selected discovered trace could not be adapted; rerun and choose a different trace file.
+- `usage_error` (exit `10`): the command line is invalid; `message` carries the parser text.

package/docs/gateways.md

@@ -0,0 +1,109 @@
+# Gateways
+
+Rightmodeler replays and judges through any OpenAI-compatible base URL, so a gateway in front of your models can be the replay route. This guide covers open-source gateways verified at pinned releases.
+
+## What a replay route needs
+
+- `--base-url` is the gateway's OpenAI-compatible root, ending in `/v1`, and `--api-key-env` names the variable holding the key the gateway expects.
+- The gateway's `/v1/models` lists the replay models. When it lists ids without prices or capabilities, pass `--catalog-reference` with the upstream's public model list; see [Catalogs without pricing or capabilities](getting-started.md#catalogs-without-pricing-or-capabilities).
+- Headers the gateway reads per request are passed with `--header 'name: value'`; see [Gateways that route by header](getting-started.md#gateways-that-route-by-header).
+- The replay route runs no fallbacks, model aliases, response caching or request plugins for the replay models. Rightmodeler never asks a gateway for them, and it checks each replayed response (in Mode B, each response to a step whose model it sets): one that names another model, reports a cache hit, or reports a changed request is left out of the evidence as `attribution_substituted`; see [Which model answered](getting-started.md#which-model-answered).
+- Replay latency includes the gateway hop; rightmodeler does not separate the gateway's share.
+- Mode B on the cloud backend calls the base URL from a remote sandbox, so it cannot reach a gateway on localhost or a private network. Use the Docker backend for a local gateway.
+
+## Portkey
+
+Verified on the open-source Portkey gateway 1.15.2 (MIT, `portkeyai/gateway:1.15.2`), the latest tagged release as of 2026-09-22. Portkey picks the upstream for each request from two headers and needs no server configuration:
+
+```sh
+docker run -d --name portkey -p 127.0.0.1:8787:8787 portkeyai/gateway:1.15.2
+npx rightmodeler init --traces ./traces.jsonl --repo . \
+  --base-url http://127.0.0.1:8787/v1 --api-key-env AI_GATEWAY_API_KEY \
+  --header 'x-portkey-provider: openai' \
+  --header 'x-portkey-custom-host: https://ai-gateway.vercel.sh/v1' \
+  --max-cost-usd 25
+```
+
+- Use the `openai` provider with a custom host for any OpenAI-compatible upstream (Vercel AI Gateway, OpenRouter, a LiteLLM proxy). Portkey's `openrouter` provider rewrites requests and does not serve the model catalog.
+- `--api-key-env` holds the upstream's key: Portkey forwards `Authorization` to the upstream and has no key of its own.
+- The catalog, per-call cost, and rate-limit and credit errors come from the upstream through Portkey unchanged.
+- Rightmodeler sends no `x-portkey-config`, so fallbacks, load balancing, retries, caching and guardrail or mutator hooks stay off. If you pass one with `--header`, a replayed response whose model differs (`override_params`, targets), that reports `x-portkey-cache-status: HIT`, or whose hook results show `transformed: true` is left out as `attribution_substituted`.
+- Bind the container to 127.0.0.1, or start it headless (`docker run ... portkeyai/gateway:1.15.2 run start:node -- --headless`): the 1.15.2 console and live log stream are served without authentication and show provider keys.
+- Portkey 1.15.2 keeps no traces or logs that rightmodeler can read. Export traces from your application: OpenTelemetry GenAI, AI SDK telemetry, or the OpenAI SDK JSONL shape.
+
+## Envoy AI Gateway (Agent Router)
+
+Envoy AI Gateway was renamed Agent Router on 2026-09-09; its resources, `aigw` CLI and images keep their names. Verified on v1.1.0 (Apache 2.0) running standalone with `aigw run` (`envoyproxy/ai-gateway-cli:v1.1.0`); always use a release tag, because the `latest` image follows the development branch.
+
+As a replay route:
+
+- Declare each replay model as an `Exact` `x-ai-eg-model` match under the upstream's own id, one `backendRef`, no `modelNameOverride`, no priority fallback and no `BackendTrafficPolicy` retries on the replay route. A fallback or an override answers with another model, so a replayed response it answers is left out as `attribution_substituted`.
+- Raise the Gateway's `ClientTrafficPolicy` `bufferLimit` (Envoy Gateway's 32 KiB default is too small for real prompts) and the route's `timeouts.request` for slow models.
+- The gateway lists only the declared ids at `/v1/models`, so pass `--catalog-reference` with the upstream's public list.
+- The gateway replaces `Authorization` with the route's key, so `--api-key-env` may name any non-empty variable.
+
+```sh
+docker run --rm -p 127.0.0.1:1975:1975 -e AI_GATEWAY_API_KEY \
+  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 \
+  -e 'OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES=agent-session-id:session.id,x-rightmodeler-family:rightmodeler.family,x-rightmodeler-replay:rightmodeler.replay' \
+  -v "$PWD/aigw.yaml:/config.yaml:ro" envoyproxy/ai-gateway-cli:v1.1.0 run /config.yaml
+export AIGW_CLIENT_KEY=unused
+npx rightmodeler init --traces ./spans.jsonl --repo . \
+  --base-url http://127.0.0.1:1975/v1 --api-key-env AIGW_CLIENT_KEY \
+  --catalog-reference https://ai-gateway.vercel.sh/v1/models \
+  --header 'x-rightmodeler-replay: 1' --max-cost-usd 25
+```
+
+As a trace source, rightmodeler reads the gateway's default OpenInference spans from an OpenTelemetry collector's file export:
+
+- Set `OTEL_EXPORTER_OTLP_ENDPOINT`. Map headers to span attributes with `OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES` as comma-separated `header:attribute` pairs; setting it replaces the default `agent-session-id:session.id`, so repeat that pair.
+- Send `agent-session-id` from your application so the steps of one conversation form one ordered run, and `x-rightmodeler-family: <name>` so each call has its family. Send `x-rightmodeler-replay: 1` with rightmodeler's replays (`--header`) so a later export leaves them out.
+- Each span's request body gives the model your application asked for and the conversation as sent; the response gives the output; the token counts give usage.
+- Failed calls, replay-tagged calls, and calls whose prompt or output the gateway hid (`OPENINFERENCE_HIDE_INPUTS`, `OPENINFERENCE_HIDE_OUTPUTS`) are left out of the corpus with a `trace_steps_excluded` warning that names each reason.
+- A span records the model that answered but not whether a priority fallback chose it (a `modelNameOverride` alias looks the same), so a call that fell back is read as an answer to the model your application asked for, with the fallback's output and usage. Keep fallback routes off the traffic you export, or expect those answers among the recorded outputs rightmodeler compares candidates against.
+- Steps whose recorded conversation contains tool calls are read but not replayed yet (`recorded_messages_not_replayable`).
+- With `AI_GATEWAY_TRACING_SEMCONV=gen_ai`, the spans are read by the OTel GenAI reader instead. Also set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`, because that convention records no messages without it and rightmodeler cannot read a span with none. Those spans are grouped by trace (propagate `traceparent` from your client), carry no `response_format` or `tool_choice`, and record images by type only.
+- Access logs carry no message content and are not a trace source.
+
+On Kubernetes (Kubernetes 1.32 or newer, Envoy Gateway 1.8.1 or newer, Helm charts `ai-gateway-crds-helm` and `ai-gateway-helm` v1.1.0), the same resources apply; set `OTEL_EXPORTER_OTLP_ENDPOINT` through the `ai-gateway-helm` chart's `extProc.extraEnvVars` and the header mapping through its `controller.spanRequestHeaderAttributes`. Mode B on the cloud backend cannot reach an in-cluster gateway.
+
+## Bifrost
+
+Verified on the open-source Bifrost gateway transports/v2.2.1 (Apache 2.0, `maximhq/bifrost:v2.2.1`); pin the image, because releases arrive weekly. Enterprise features are a separate image and are not covered here.
+
+As a replay route:
+
+- Model ids are `<provider>/<upstream id>`, for example `vercel/openai/gpt-4o-mini` for Vercel AI Gateway configured as an OpenAI-typed custom provider. Bifrost answers with the upstream id, which rightmodeler accepts as the requested model.
+- Set every `compat` flag to `false` in the `client` block: a `client` block that omits them turns them all on, and the compat plugin can drop parameters such as `response_format` while answering 200.
+- Configure no key `aliases` or routing rules for replay models; an alias answers with another model and is left out as `attribution_substituted`.
+- Pass `--catalog-reference` with the upstream's public list. A custom provider lists ids and context only, with no prices or capabilities, and through Vercel AI Gateway every model carries the same `created` date: Bifrost copies Vercel's `created`, one placeholder for all models, and drops the real `released` date. Rightmodeler ranks judges partly by release date, and the reference's release dates win over the gateway's, so judges rank as they do on Vercel itself instead of the most expensive first. Bifrost's own pricing sheet is not used.
+- The billed cost comes from the upstream through Bifrost's `usage.cost.total_cost`.
+- Send `x-bf-cache-no-store: true` and `x-bf-dim-rightmodeler: replay` with rightmodeler's replays (`--header`): the first keeps replays out of a semantic cache (a cache hit is still detected and left out), the second tags them so a later log export leaves them out.
+
+```sh
+docker volume create bifrost-data
+docker create --name bifrost -p 127.0.0.1:8080:8080 -e AI_GATEWAY_API_KEY -v bifrost-data:/app/data maximhq/bifrost:v2.2.1
+docker cp config.json bifrost:/app/data/config.json && docker start bifrost
+export BIFROST_KEY=unused
+npx rightmodeler init --traces ./bifrost-logs.jsonl --repo . \
+  --base-url http://127.0.0.1:8080/v1 --api-key-env BIFROST_KEY \
+  --catalog-reference https://ai-gateway.vercel.sh/v1/models \
+  --header 'x-bf-cache-no-store: true' --header 'x-bf-dim-rightmodeler: replay' \
+  --max-cost-usd 25
+```
+
+As a trace source, rightmodeler reads Bifrost's own request logs, exported from its management API (add credentials once an admin exists):
+
+```sh
+curl -s 'http://127.0.0.1:8080/api/logs?objects=chat_completion,chat_completion_stream&order=asc&limit=1000&offset=0' | jq -r '.logs[].id' \
+  | while read -r id; do curl -s "http://127.0.0.1:8080/api/logs/$id"; echo; done > bifrost-logs.jsonl
+```
+
+With more than 1000 rows, repeat with `offset=1000`, `2000` and so on, appending (`>>`) to the same file, until a page returns fewer than 1000 rows; `order=asc` keeps earlier pages in place while new calls are logged. Do not pass `roots_only=true`: it hides fallback rows.
+
+- Send `x-bf-session-id` from your application so the steps of one conversation form one ordered run, and `x-bf-dim-rightmodeler-family: <name>` so each call has its family.
+- Each row gives the model the application asked for (`provider/model`, or the alias it sent), the conversation as sent (`input_history`), the output (`output_message`), usage, cost, latency and retries.
+- Failed calls, answers from a configured fallback, rightmodeler's own replays and calls whose content was not logged are left out of the corpus with a `trace_steps_excluded` warning that names each reason. Steps whose conversation contains tool calls are read but not replayed yet.
+- Bifrost's OpenTelemetry plugin export reads through the OTel GenAI reader, but it summarizes messages and loses tool-call ids: use the log export for anything but plain text calls.
+- Streamed calls through Mode B are checked on the stream's model and headers only; Bifrost reports a semantic-cache hit only in the response body, so such a hit on a streamed Mode B call is not detected.
+- Rightmodeler records every replay's latency through Bifrost, gateway hop included, and does not separate Bifrost's own overhead. Bifrost's published benchmark reports 11 microseconds of overhead on a t3.xlarge at 5,000 requests per second, excluding JSON marshalling and the HTTP call; measure it on your own traffic.

package/docs/getting-started.md

@@ -1,16 +1,17 @@
 # Getting started
 
-Rightmodeler analyzes recorded model calls, replays them against cheaper candidates, evaluates the outputs, and writes a recommendation report. The intended public package name is `rightmodeler`.
+Rightmodeler analyzes recorded model calls, replays them against cheaper candidates, evaluates the outputs, and writes a recommendation report. Published on npm as `rightmodeler`.
 
 ## Requirements
 
 - Node.js 24 or newer.
 - A Git repository to analyze.
 - Trace input in a supported format.
-- An OpenAI-compatible provider base URL and the name of an environment variable containing its API key before replay begins.
+- An OpenAI-compatible provider base URL and the name of an environment variable containing its API key before replay begins. Its `/v1/models` catalog should publish per-token pricing. OpenRouter and Vercel AI Gateway do. For a LiteLLM endpoint, Rightmodeler can fall back to `GET /model/info`; for a gateway that lists bare model ids, pass `--catalog-reference`; for bare OpenAI or another unpriced endpoint, pass `--pricing-file`.
 
-Supported trace sources are OTel GenAI, OpenAI JSONL, Langfuse, Braintrust,
-LangSmith, OpenInference, Helicone, W&B Weave, Claude Code, and Codex.
+Supported trace sources are OTel GenAI, AI SDK telemetry, OpenAI JSONL,
+Langfuse, Braintrust, LangSmith, OpenInference, Helicone, Bifrost, W&B Weave,
+Claude Code, and Codex.
 
 ## Start with automatic discovery
 
@@ -40,6 +41,27 @@ npx rightmodeler init --plan --output json --repo /path/to/repository
 npx rightmodeler init --through corpus --traces /path/to/traces.json --output json --repo /path/to/repository
 ```
 
+`--traces` accepts a single file or a directory. A directory is read non-recursively as its `.json` and `.jsonl` files in name order; every file must use the same trace format.
+
+Each family is replayed only against the call sites its own traces came from. Traced cases that cannot be tied to one such call site, because several call sites use the traced model or none matches it, are left out of the replay sample with a `family_cases_left_out` warning. A family with no case left abstains before any spend with `ambiguous_call_site_binding` or `unmatched_call_site_binding`. The AI SDK telemetry `functionId` (see below) is the way to tie an AI SDK call site to its family.
+
+## AI SDK telemetry
+
+The AI SDK emits telemetry in two dialects, and Rightmodeler reads both:
+
+- The `ai.*` dialect comes from AI SDK 5 and 6 with `experimental_telemetry: { isEnabled: true }` on each call, and from AI SDK 7 with `registerTelemetry(new LegacyOpenTelemetry())`. The AI SDK reader reads it.
+- The GenAI semantic conventions dialect comes from AI SDK 7 with `registerTelemetry(new OpenTelemetry())`. The OTel GenAI reader reads it and treats the agent, step and tool spans as structure, so each model call counts once.
+
+`registerTelemetry` comes from `ai`; `LegacyOpenTelemetry` and `OpenTelemetry` come from `@ai-sdk/otel`. Register only one of them: an export that holds both dialects is ambiguous. Keep `recordInputs` and `recordOutputs` on, which is the default, because a model call without its prompt or output cannot become a corpus case. Set a string-literal `functionId` on every call (`telemetry: { functionId: "summarize" }` in AI SDK 7, `experimental_telemetry: { isEnabled: true, functionId: "summarize" }` before it); it becomes the call's family.
+
+The scanner records that `functionId` on the call site, and a family binds to exactly the call sites whose `functionId` equals its name: its evidence and any swap stay on those call sites, and no other family borrows them. Two call sites that do the same job may share one `functionId`. A family bound to a single call site can still be recommended. The `functionId` must be a string literal in the call; a variable or a template literal is not read, and the call site then binds by model id only. Replay cannot run a call site that needs tools or structured output: its cases are left out of the family's replay sample with a `family_cases_left_out` warning, and a family whose traced cases all come from such call sites abstains with `bound_call_sites_not_replayable` before any spend.
+
+Export the spans through the OpenTelemetry NodeSDK or `@vercel/otel` to an OTLP collector, and pass the collector's file exporter output with `--traces`. A model call that ended without a finish reason, because it was aborted or errored, is left out of the corpus with a `trace_steps_excluded` warning, and the rest of the input is read. Token usage from AI SDK 4 exports (`ai.usage.promptTokens`) is not read, so those calls carry no usage.
+
+## OpenInference spans
+
+When an OpenInference span carries the request body in `input.value` (Envoy AI Gateway, and the OpenAI instrumentation), rightmodeler reads the model your application asked for and the conversation exactly as sent from it, and the output from `output.value`. Spans that share a `session.id` form one run ordered by start time. A `rightmodeler.family` attribute names the family; otherwise the span name does. Failed calls, calls tagged `rightmodeler.replay`, and calls whose content was hidden are left out with a `trace_steps_excluded` warning.
+
 ## Run the complete pipeline
 
 ```sh
@@ -51,6 +73,8 @@ npx rightmodeler init --traces /path/to/traces.json --base-url https://provider.
 `--api-key-env <name>` to use a different exported variable. The CLI does not ask
 for a secret value.
 
+Replay resends each recorded conversation as text. A recorded case whose conversation contains tool calls, non-text parts, or tool definitions is left out of the replay sample with a `recorded_messages_not_replayable` warning; the family's other cases replay, and a family left with too few cases abstains under the usual sample-size reasons.
+
 ## Estimate replay spend
 
 ```sh
@@ -59,7 +83,108 @@ npx rightmodeler estimate --traces /path/to/traces.json --base-url https://provi
 
 Estimate projects candidate replay spend from recorded token usage and the current
 model catalog before paid model calls begin.
+`--max-cost-usd` caps candidate replays and judge calls together: each call reserves
+its worst case before it is sent, and a call the cap cannot cover is not sent.
+
+## Static code context (Graphify)
+
+rightmodeler can read a code graph built by the open-source Graphify CLI (PyPI package `graphifyy`, Apache-2.0, tested with 0.9.65). Graphify builds it locally from your source, with no account and no model call.
+
+```sh
+uv tool install graphifyy
+graphify update .
+npx rightmodeler report --code-graph graphify-out/graph.json --repo .
+```
+
+`init --code-graph <path>` renders the same section at the end of a run. For each call site the scanner found, the section lists the enclosing symbol, its callers, the tests that reach it, and the owners of those files, which are listed only. It also lists files that import an AI SDK where the scanner found no call site.
+
+Graph edges are never replay trials, runtime proof, or quality evidence, and the flag never changes a stage before the report, a verdict, a gate, or confirmation. Each finding is labelled EXTRACTED, or INFERRED or AMBIGUOUS to verify, by its weakest hop. A graph built at another commit is shown file-level with a stale note. An unusable graph produces one warning, the section says why it is not shown, and the rest of the report is unchanged. The scan ignores `graphify-out/`, so building a graph never makes finished stages stale. Only `graphify update` and `graphify extract --code-only` are needed; other Graphify commands can call a language model.
+
+`apply --code-graph <path>` appends the same section to the draft pull request body, limited to the call sites the pull request swaps and to five findings of each kind per call site. Owners there are listed only and are never requested as reviewers; reviewers still come from CODEOWNERS and blame. The graph never changes the swap, its digest, or its reviewers. `apply --dry-run` prints the exact body it would post, with or without `--code-graph`.
+
+## Release policy
+
+`--policy <path>` is accepted by `init`, `estimate`, `replay`, and `confirm`. The JSON object can set the quality floor, shortlist size, and model allow and deny lists:
+
+```json
+{
+  "qualityFloor": 0.9,
+  "shortlistTop": 5,
+  "allowModels": ["acme/small-1"],
+  "denyModels": ["acme/large-1"]
+}
+```
+
+`qualityFloor` must be greater than 0.8 and less than 1, and `shortlistTop` must be a positive integer. Changing the policy changes the stamped gate policy version, so shortlist and replay run again instead of pooling evidence gathered under the old policy.
+
+## Catalogs without pricing or capabilities
+
+Rightmodeler reads per-token pricing from the model catalog. When every catalog
+entry has null pricing and no `--pricing-file` is set, it requests LiteLLM
+`GET /model/info` on the same host. Use `--pricing-file` for bare OpenAI
+endpoints or when `/model/info` has no usable per-token costs; file entries
+override provider pricing.
+
+Some gateways answer `/v1/models` with bare model ids: Envoy AI Gateway lists
+the models a route declares, and Bifrost lists custom providers with ids and
+context only. Pass `--catalog-reference <url|path>` to name the upstream's own
+model list, for example `https://ai-gateway.vercel.sh/v1/models` or
+`https://openrouter.ai/api/v1/models`. Rightmodeler reads it without your key or
+headers and joins it to the gateway's entries by id, or by the reference id a
+gateway id ends with (`openai/gpt-4o-mini` for `vercel/openai/gpt-4o-mini`). A
+gateway entry takes from its match only what it does not declare itself (price,
+context window, output ceiling, tool and structured-output support); a match
+that is not a language model removes the entry; `--pricing-file` values win over
+both. Entries still unpriced after the join are named in a
+`catalog_reference_unmatched` warning, and a reference that cannot be read stops
+the run with `invalid_catalog_reference`.
+
+The release date is the one field where the reference wins over the gateway:
+when the matched reference entry has a release date, it replaces the date the
+gateway declares, and the gateway's date is kept only when the reference gives
+none. Rightmodeler ranks judges partly by how recent a model is, and a gateway
+can give every model the same placeholder date (Bifrost does for Vercel AI
+Gateway's models), which would leave context and price to rank the judges and
+favor the most expensive.
+
+```sh
+npx rightmodeler estimate --base-url https://provider.example/v1 --pricing-file /path/to/pricing.json --repo /path/to/repository
+```
+
+The pricing file maps each model id to input and output USD per token and may
+include the model's output ceiling:
+
+```json
+{
+  "acme/model": {
+    "input": 0.000001,
+    "output": 0.000002,
+    "maxOutputTokens": 4096
+  }
+}
+```
+
+Without usable pricing from the catalog, a catalog reference, LiteLLM
+`/model/info`, or a pricing file, the run refuses with `no_priced_candidates`
+instead of reporting zero cost. The judge must be priced too, so price at least
+one model from a family other than the current model's and the candidate's.
+
+## Which model answered
+
+Rightmodeler checks every replay and judge response, and in Mode B every response to a step whose model it sets, before it counts. It records the model the response names, and it leaves a response out of the evidence when that model is not the one it asked for, when a gateway reports that it answered from its cache (Portkey's `x-portkey-cache-status: HIT`, Bifrost's `cache_debug.cache_hit`), or when a gateway reports that it changed the request (a Portkey hook with `transformed: true`, Bifrost's compat plugin dropping parameters). Such a response is recorded with `attribution: "substituted"`, is never graded, and counts as `attribution_substituted`; more than 5% of a family's replays substituted abstains the family. A `replay_responses_substituted` warning counts them.
+
+A response names the requested model when it echoes it, when it drops a gateway provider prefix (`openai/gpt-4o-mini` for `vercel/openai/gpt-4o-mini`), or when it adds a dated snapshot (`gpt-4o-mini-2024-07-18` for `gpt-4o-mini`). An alias whose answering model has another name counts as substituted, so name replay models by their upstream ids and turn off fallbacks, model aliases, response caching and request plugins for the replay route. Completed replay cells are reused, so after fixing the route rerun with a fresh store (`--store <directory>`). For streamed Mode B calls only the model a stream names and the response headers are checked.
+
+A judge whose response names another model is retired at that first response: rightmodeler starts no new call to it, cancels its calls still waiting for budget, and re-judges the affected replays with the next-ranked judge once the calls already under way return; the `judge_unusable` warning names the model that answered. Some catalogs also list a faster service tier of a model under its own id, which the gateway answers as the base model: on Vercel AI Gateway, `<id>-fast` is `<id>` at its fast tier. Rightmodeler never ranks such an id as a judge or shortlists it as a candidate while `<id>` is in the catalog too; an id that ends in `-fast` with no base model listed, such as `morph/morph-v3-fast`, is treated like any other model.
+
+## Gateways that route by header
+
+Some gateways choose the upstream, the cache policy, or a trace tag from request headers. Pass each one with `--header 'name: value'`; repeat the option for more. Rightmodeler sends them with every request it makes to the provider base URL: the model catalog, candidate replays, judge calls, and the calls Mode B makes from your application, where a configured header replaces one your application sends. `authorization` comes only from `--api-key-env`, and `content-type`, `content-length` and `host` are set by rightmodeler, so none of them can be passed as a header. Completed replay cells are reused when the headers change, so after changing the route rerun with a fresh store (`--store <directory>`). A detached replay run is keyed to the header values by their SHA-256 digests; the values themselves are passed to the detached worker on its command line and are not written to the store. Do not put secrets in headers.
+
+The default store is `.rightmodeler/` inside the analyzed repository. Completed stages resume when their inputs and outputs are still current. A complete run writes `.rightmodeler/project/reports/report.md`. The JSON report is kept inside the versioned store and is never written as a plain file, so read the final `result` event from `--output json` or `--output jsonl` for the machine-readable outcome.
+
+Read the generated [command reference](commands.md), the [evaluator guide](evaluators.md), [Mode B configuration](modeb.md), the [gateway guide](gateways.md), and the [exit-code convention](exit-codes.md) before automating a full run.
 
-The default store is `.rightmodeler/` inside the analyzed repository. Completed stages resume when their inputs and outputs are still current. A complete run writes `.rightmodeler/project/reports/report.md` and `.rightmodeler/project/reports/report.json`.
+To open the draft pull request and keep it reconciled, read the [GitHub guide](github.md).
 
-Read the generated [command reference](commands.md), the [evaluator guide](evaluators.md), [Mode B configuration](modeb.md), and the [exit-code convention](exit-codes.md) before automating a full run.
+Run `rightmodeler docs <name>` to print any of these documents from the installed package.