rightmodeler 0.2.1 → 0.4.0

package/docs/commands.md

@@ -38,8 +38,9 @@ Commands:
   rollback [options]   open a draft pull request restoring a prior model swap
   drift [options]      detect drift against the active replay corpus
   watch [options]      reconcile one open model-swap pull request
-  report               write report.md and report.json
+  report [options]     write report.md and report.json
   status [options]     summarize the current store
+  docs [name]          print documentation packaged with this CLI
   help [command]       display help for command
 ```
 
@@ -51,7 +52,8 @@ Usage: rightmodeler init [options]
 run the resumable Phase A pipeline
 
 Options:
-  --traces <path>                     trace input file
+  --traces <path>                     trace input file or directory
+  --matchers <path>                   declarative matcher definitions JSON file
   --include-free                      include zero-priced models in candidate
                                       shortlists
   --modeb-config <path>               versioned Mode B runtime configuration
@@ -59,9 +61,30 @@ Options:
   --base-url <url>                    OpenAI-compatible provider base URL
   --api-key-env <name>                environment variable containing the
                                       provider API key
+  --route <kind>                      where candidate replays run: api (the
+                                      --base-url endpoint), claude-login or
+                                      codex-login (the claude or codex CLI
+                                      signed in on this machine); a plan route
+                                      needs --judge-route (choices: "api",
+                                      "claude-login", "codex-login")
+  --judge-route <kind>                where the built-in judge runs: api,
+                                      claude-login or codex-login; use a vendor
+                                      other than the candidates' (choices:
+                                      "api", "claude-login", "codex-login")
   --max-cost-usd <amount>             optional hard spend cap in USD; omit to
                                       run uncapped so every case and judge cell
                                       completes
+  --max-concurrency <n>               maximum concurrent provider requests
+  --pricing-file <path>               JSON map from model id to per-token input
+                                      and output USD, for catalogs without
+                                      pricing
+  --header <header>                   extra HTTP header for every provider
+                                      request, as 'name: value' (repeatable)
+  --catalog-reference <url-or-path>   upstream /models URL or file that fills
+                                      pricing, context and capabilities the
+                                      provider catalog lacks
+  --policy <path>                     release policy JSON file: quality floor,
+                                      shortlist size, model allow and deny lists
   --evaluator <provider>              external evaluator provider (choices:
                                       "braintrust", "langfuse", "langsmith",
                                       "promptfoo")
@@ -84,8 +107,10 @@ Options:
                                       "ingest", "reconcile", "scrub", "corpus",
                                       "audit-sample", "shortlist", "replay",
                                       "aggregate", "confirm", "report")
-  --yes                               accept the newest discovered trace without
-                                      prompting
+  --code-graph <path>                 Graphify graph.json for static code
+                                      context in the report; never evidence
+  --yes                               accept the newest discovered trace and the
+                                      saved model route without prompting
   -h, --help                          display help for command
 ```
 
@@ -97,7 +122,8 @@ Usage: rightmodeler estimate [options]
 project replay spend before paid model calls
 
 Options:
-  --traces <path>                     trace input file
+  --traces <path>                     trace input file or directory
+  --matchers <path>                   declarative matcher definitions JSON file
   --include-free                      include zero-priced models in candidate
                                       shortlists
   --modeb-config <path>               versioned Mode B runtime configuration
@@ -105,9 +131,30 @@ Options:
   --base-url <url>                    OpenAI-compatible provider base URL
   --api-key-env <name>                environment variable containing the
                                       provider API key
+  --route <kind>                      where candidate replays run: api (the
+                                      --base-url endpoint), claude-login or
+                                      codex-login (the claude or codex CLI
+                                      signed in on this machine); a plan route
+                                      needs --judge-route (choices: "api",
+                                      "claude-login", "codex-login")
+  --judge-route <kind>                where the built-in judge runs: api,
+                                      claude-login or codex-login; use a vendor
+                                      other than the candidates' (choices:
+                                      "api", "claude-login", "codex-login")
   --max-cost-usd <amount>             optional hard spend cap in USD; omit to
                                       run uncapped so every case and judge cell
                                       completes
+  --max-concurrency <n>               maximum concurrent provider requests
+  --pricing-file <path>               JSON map from model id to per-token input
+                                      and output USD, for catalogs without
+                                      pricing
+  --header <header>                   extra HTTP header for every provider
+                                      request, as 'name: value' (repeatable)
+  --catalog-reference <url-or-path>   upstream /models URL or file that fills
+                                      pricing, context and capabilities the
+                                      provider catalog lacks
+  --policy <path>                     release policy JSON file: quality floor,
+                                      shortlist size, model allow and deny lists
   --evaluator <provider>              external evaluator provider (choices:
                                       "braintrust", "langfuse", "langsmith",
                                       "promptfoo")
@@ -125,8 +172,8 @@ Options:
   --evaluator-gate-metric <name>      scorer metric used for release gates
   --evaluator-gate-threshold <value>  fallback pass threshold when the evaluator
                                       omits a pass decision
-  --yes                               accept the newest discovered trace without
-                                      prompting
+  --yes                               accept the newest discovered trace and the
+                                      saved model route without prompting
   --approved-run <digest>             scope projection to one merged approved
                                       swap
   -h, --help                          display help for command
@@ -140,7 +187,8 @@ Usage: rightmodeler scan [options]
 run through the scan stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -154,7 +202,8 @@ Usage: rightmodeler ingest [options]
 run through the ingest stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -168,7 +217,8 @@ Usage: rightmodeler reconcile [options]
 run through the reconcile stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -182,7 +232,8 @@ Usage: rightmodeler scrub [options]
 run through the scrub stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -196,7 +247,8 @@ Usage: rightmodeler shortlist [options]
 run through the shortlist stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -210,7 +262,8 @@ Usage: rightmodeler replay [options]
 run through the replay stage
 
 Options:
-  --traces <path>                     trace input file
+  --traces <path>                     trace input file or directory
+  --matchers <path>                   declarative matcher definitions JSON file
   --include-free                      include zero-priced models in candidate
                                       shortlists
   --modeb-config <path>               versioned Mode B runtime configuration
@@ -218,9 +271,30 @@ Options:
   --base-url <url>                    OpenAI-compatible provider base URL
   --api-key-env <name>                environment variable containing the
                                       provider API key
+  --route <kind>                      where candidate replays run: api (the
+                                      --base-url endpoint), claude-login or
+                                      codex-login (the claude or codex CLI
+                                      signed in on this machine); a plan route
+                                      needs --judge-route (choices: "api",
+                                      "claude-login", "codex-login")
+  --judge-route <kind>                where the built-in judge runs: api,
+                                      claude-login or codex-login; use a vendor
+                                      other than the candidates' (choices:
+                                      "api", "claude-login", "codex-login")
   --max-cost-usd <amount>             optional hard spend cap in USD; omit to
                                       run uncapped so every case and judge cell
                                       completes
+  --max-concurrency <n>               maximum concurrent provider requests
+  --pricing-file <path>               JSON map from model id to per-token input
+                                      and output USD, for catalogs without
+                                      pricing
+  --header <header>                   extra HTTP header for every provider
+                                      request, as 'name: value' (repeatable)
+  --catalog-reference <url-or-path>   upstream /models URL or file that fills
+                                      pricing, context and capabilities the
+                                      provider catalog lacks
+  --policy <path>                     release policy JSON file: quality floor,
+                                      shortlist size, model allow and deny lists
   --evaluator <provider>              external evaluator provider (choices:
                                       "braintrust", "langfuse", "langsmith",
                                       "promptfoo")
@@ -252,7 +326,8 @@ Usage: rightmodeler aggregate [options]
 run through the aggregate stage
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -266,7 +341,8 @@ Usage: rightmodeler confirm [options]
 run through the confirm stage
 
 Options:
-  --traces <path>                     trace input file
+  --traces <path>                     trace input file or directory
+  --matchers <path>                   declarative matcher definitions JSON file
   --include-free                      include zero-priced models in candidate
                                       shortlists
   --modeb-config <path>               versioned Mode B runtime configuration
@@ -274,9 +350,30 @@ Options:
   --base-url <url>                    OpenAI-compatible provider base URL
   --api-key-env <name>                environment variable containing the
                                       provider API key
+  --route <kind>                      where candidate replays run: api (the
+                                      --base-url endpoint), claude-login or
+                                      codex-login (the claude or codex CLI
+                                      signed in on this machine); a plan route
+                                      needs --judge-route (choices: "api",
+                                      "claude-login", "codex-login")
+  --judge-route <kind>                where the built-in judge runs: api,
+                                      claude-login or codex-login; use a vendor
+                                      other than the candidates' (choices:
+                                      "api", "claude-login", "codex-login")
   --max-cost-usd <amount>             optional hard spend cap in USD; omit to
                                       run uncapped so every case and judge cell
                                       completes
+  --max-concurrency <n>               maximum concurrent provider requests
+  --pricing-file <path>               JSON map from model id to per-token input
+                                      and output USD, for catalogs without
+                                      pricing
+  --header <header>                   extra HTTP header for every provider
+                                      request, as 'name: value' (repeatable)
+  --catalog-reference <url-or-path>   upstream /models URL or file that fills
+                                      pricing, context and capabilities the
+                                      provider catalog lacks
+  --policy <path>                     release policy JSON file: quality floor,
+                                      shortlist size, model allow and deny lists
   --evaluator <provider>              external evaluator provider (choices:
                                       "braintrust", "langfuse", "langsmith",
                                       "promptfoo")
@@ -305,7 +402,8 @@ Usage: rightmodeler corpus [options] [command]
 build or import the replay corpus
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -375,7 +473,8 @@ Usage: rightmodeler audit sample [options]
 write the audit worksheet without blocking
 
 Options:
-  --traces <path>        trace input file
+  --traces <path>        trace input file or directory
+  --matchers <path>      declarative matcher definitions JSON file
   --include-free         include zero-priced models in candidate shortlists
   --modeb-config <path>  versioned Mode B runtime configuration JSON file
   -h, --help             display help for command
@@ -402,9 +501,14 @@ open a draft pull request for proven model swaps
 
 Options:
   --owner <owner>            GitHub repository owner
-  --github-base-url <url>    GitHub API base URL
+  --github-repo <repo>       GitHub repository name (default: the repository
+                             directory name)
+  --github-base-url <url>    GitHub API base URL (default:
+                             "https://api.github.com")
   --github-token-env <name>  environment variable containing the GitHub token
   --dry-run                  run all machine gates without writing GitHub state
+  --code-graph <path>        Graphify graph.json for static code context in the
+                             pull request body; never evidence
   -h, --help                 display help for command
 ```
 
@@ -417,8 +521,11 @@ open a draft pull request restoring a prior model swap
 
 Options:
   --owner <owner>            GitHub repository owner
+  --github-repo <repo>       GitHub repository name (default: the repository
+                             directory name)
   --pr <number>              merged pull request number
-  --github-base-url <url>    GitHub API base URL
+  --github-base-url <url>    GitHub API base URL (default:
+                             "https://api.github.com")
   --github-token-env <name>  environment variable containing the GitHub token
   -h, --help                 display help for command
 ```
@@ -474,9 +581,11 @@ reconcile one open model-swap pull request
 
 Options:
   --owner <owner>            GitHub repository owner
-  --github-repo <repo>       GitHub repository name
+  --github-repo <repo>       GitHub repository name (default: the repository
+                             directory name)
   --pr <number>              pull request number
-  --github-base-url <url>    GitHub API base URL
+  --github-base-url <url>    GitHub API base URL (default:
+                             "https://api.github.com")
   --github-token-env <name>  environment variable containing the GitHub token
   -h, --help                 display help for command
 ```
@@ -489,7 +598,9 @@ Usage: rightmodeler report [options]
 write report.md and report.json
 
 Options:
-  -h, --help  display help for command
+  --code-graph <path>  Graphify graph.json for static code context in the
+                       report; never evidence
+  -h, --help           display help for command
 ```
 
 ## `rightmodeler status`
@@ -503,3 +614,19 @@ Options:
   --run <runId>  report one detached replay run
   -h, --help     display help for command
 ```
+
+## `rightmodeler docs`
+
+```text
+Usage: rightmodeler docs [options] [name]
+
+print documentation packaged with this CLI
+
+Arguments:
+  name        packaged document name (choices: "commands", "evaluators",
+              "exit-codes", "gateways", "getting-started", "github",
+              "github-actions", "modeb", "model-routes")
+
+Options:
+  -h, --help  display help for command
+```

package/docs/evaluators.md

@@ -1,16 +1,95 @@
 # Evaluators
 
-The default evaluator is the built-in judge selected from the configured provider catalog. Candidate and reference families are excluded when choosing the judge.
+The default evaluator is the built-in judge selected from the configured provider catalog. Candidate and reference families are excluded when choosing the judge. The exclusion applies per replayed call site: when the call sites of one traced family use models of different vendors (for example `openai/...` and `anthropic/...`), each call site's cases are judged by a model from neither the candidate's model family nor that call site's. Confirmation judges the recorded final output, so its judge also comes from outside the final step's model family. In replay, when the recorded answer is one assistant message of text, the judge compares the candidate's answer with that text; any other recorded answer, such as one that calls tools, has parts other than text or holds several messages, is shown to the judge as recorded.
 
-An external evaluator can be requested with `--evaluator braintrust`. Configure it with:
+An external evaluator is requested with `--evaluator <provider>` on `init`, `estimate`, `replay`, and `confirm`, where the provider is `braintrust`, `langfuse`, `langsmith`, or `promptfoo`. Every other `--evaluator-*` option is a usage error without `--evaluator`.
 
-- `--evaluator-project-id <id>` (required)
-- one or more `--evaluator-scorer <name>` options (required)
-- `--evaluator-api-key-env <name>` (defaults to `BRAINTRUST_API_KEY`)
-- `--evaluator-base-url <url>` (defaults to the provider API)
-- `--evaluator-gate-metric <name>` when more than one scorer is configured
-- `--evaluator-gate-threshold <value>` when the evaluator does not return a pass decision
+## Options shared by every provider
 
-The gate metric must name one of the configured scorers. If the external evaluator is unavailable, the pipeline warns and uses the built-in judge. If a reachable evaluator run fails or omits required case results, the pipeline records the missing assessments instead of fabricating scores.
+- `--evaluator-scorer <name>` (required, repeatable): the scorer names the provider must return for every case.
+- `--evaluator-gate-metric <name>`: the scorer used for release gates. Required when more than one scorer is configured; with a single scorer it defaults to that scorer. It must name a configured scorer.
+- `--evaluator-gate-threshold <value>`: pass threshold applied only when the evaluator omits a pass decision for a metric.
+
+## Provider options
+
+One table per provider. A rejected option is a usage error with that provider (exit code 2, `invalid_option`).
+
+### braintrust
+
+`--evaluator-project-id` names the Braintrust project that receives the experiment.
+
+| Option                              | Requirement | Default                      |
+| ----------------------------------- | ----------- | ---------------------------- |
+| `--evaluator-project-id <id>`       | required    |                              |
+| `--evaluator-api-key-env <name>`    | optional    | `BRAINTRUST_API_KEY`         |
+| `--evaluator-base-url <url>`        | optional    | `https://api.braintrust.dev` |
+| `--evaluator-public-key-env <name>` | rejected    |                              |
+| `--evaluator-command <path>`        | rejected    |                              |
+| `--evaluator-config <path>`         | rejected    |                              |
+
+### langfuse
+
+Langfuse authenticates with a public key and a secret key read from environment variables. The keys select the project, so there is no project option.
+
+| Option                              | Requirement | Default                      |
+| ----------------------------------- | ----------- | ---------------------------- |
+| `--evaluator-api-key-env <name>`    | optional    | `LANGFUSE_SECRET_KEY`        |
+| `--evaluator-public-key-env <name>` | optional    | `LANGFUSE_PUBLIC_KEY`        |
+| `--evaluator-base-url <url>`        | optional    | `https://cloud.langfuse.com` |
+| `--evaluator-project-id <id>`       | rejected    |                              |
+| `--evaluator-command <path>`        | rejected    |                              |
+| `--evaluator-config <path>`         | rejected    |                              |
+
+### langsmith
+
+`--evaluator-project-id` names the LangSmith dataset the experiment references. A scorer may be written `metric=rule-id` to bind a metric name to an evaluator rule; a bare name is used for both.
+
+| Option                              | Requirement | Default                           |
+| ----------------------------------- | ----------- | --------------------------------- |
+| `--evaluator-project-id <id>`       | required    |                                   |
+| `--evaluator-api-key-env <name>`    | optional    | `LANGSMITH_API_KEY`               |
+| `--evaluator-base-url <url>`        | optional    | `https://api.smith.langchain.com` |
+| `--evaluator-public-key-env <name>` | rejected    |                                   |
+| `--evaluator-command <path>`        | rejected    |                                   |
+| `--evaluator-config <path>`         | rejected    |                                   |
+
+### promptfoo
+
+promptfoo runs locally. `--evaluator-config` is the assertions file passed to `promptfoo eval --assertions`; `--evaluator-command` is the executable, one path or one command on `PATH` (so not `npx promptfoo`: install it with `npm install -g promptfoo@0.123.1` or pass the path to its binary). No API or project option applies.
+
+| Option                              | Requirement | Default     |
+| ----------------------------------- | ----------- | ----------- |
+| `--evaluator-config <path>`         | required    |             |
+| `--evaluator-command <path>`        | optional    | `promptfoo` |
+| `--evaluator-base-url <url>`        | rejected    |             |
+| `--evaluator-api-key-env <name>`    | rejected    |             |
+| `--evaluator-project-id <id>`       | rejected    |             |
+| `--evaluator-public-key-env <name>` | rejected    |             |
+
+rightmodeler is verified against promptfoo 0.123.1. Other releases work as long as they write the same results layout; one that does not stops the run with an error naming its version and the layout rightmodeler reads. A missing executable counts as unreachable (see below); any other promptfoo failure, such as a broken `promptfooconfig.*` next to the assertions file, stops the run with promptfoo's own message.
+
+Each evaluation runs one command in the assertions file's directory, with standard input closed:
+
+`<command> eval --assertions <assertions file> --model-outputs <model outputs file, relative to that directory> --output <temporary results file> --no-write --no-share --no-table --no-progress-bar`
+
+The command gets your environment plus `PROMPTFOO_DISABLE_UPDATE=true`, `PROMPTFOO_DISABLE_VAR_EXPANSION=true`, `PROMPTFOO_FAILED_TEST_EXIT_CODE=100`, `PROMPTFOO_SHORT_CIRCUIT_TEST_FAILURES=false`, `PROMPTFOO_STRIP_GRADING_RESULT=false`, `PROMPTFOO_STRIP_RESPONSE_OUTPUT=false`, and `PROMPTFOO_STRIP_TEST_VARS=false`. So every output is graded exactly once against every assertion, the batch is neither saved to promptfoo's history nor shared, promptfoo skips its startup update check, and your own promptfoo settings cannot change the exit code, turn a failed assertion into an error, or strip the fields rightmodeler reads. Everything else, including your promptfoo login, cache, telemetry setting, and model-graded assertion providers, works as when you run promptfoo yourself. promptfoo receives each candidate output and its execution id only; the input, messages, and reference answer are never sent.
+
+Put `metric: <name>` on every assertion that feeds an `--evaluator-scorer`. A metric's score is promptfoo's named score for it, and it passes only when every assertion carrying it passes. promptfoo always decides pass or fail, so `--evaluator-gate-threshold` never applies. Each assessment's rubric version is `promptfoo@<version>/<metric>/<digest>`, where the digest is the first 16 hex characters of the canonical SHA-256 of the assertions carrying that metric, so editing one of them, or upgrading promptfoo, changes it. A scorer that no assertion carries stops the run with an error naming it.
+
+Before grading, promptfoo renders `{{ }}` templates in an output, reads a `file://` path or loads a `package:` module it names, and strips one trailing newline. rightmodeler compares the output promptfoo graded with the output it sent. A case whose graded output differs beyond that one newline is recorded absent as `external_output_mismatch`, because the grade is not of the candidate's output; a case promptfoo could not grade at all, such as a `file://` path that does not exist, is recorded absent as `external_evaluator_error`, and the other cases in its batch keep their grades. Neither counts as evidence. Nor does a metric whose grader failed, which promptfoo marks `graderError` (for example when a model-graded assertion's grading call fails): it is left out rather than recorded as a failure, and a case that loses its gate metric this way is recorded absent as `external_evaluator_error`. Replayed outputs are kept in the store, so a rerun after fixing promptfoo does not repeat model calls.
+
+promptfoo also loads a `promptfooconfig.*` and a `.env` file from the directory it runs in, so any next to the assertions file apply: a `defaultTest` there adds its assertions to every case, a `defaultTest` transform makes every case `external_output_mismatch`, and its `env` block can override the settings above. Every grade, and a detached replay's identity (`replay --detach`), covers the assertions file and every `promptfooconfig.*` next to it (see [Changing an evaluator](#changing-an-evaluator)); a `.env` there is not covered. Keep the assertions file in a directory of its own unless you want those files to apply.
+
+## Changing an evaluator
+
+Every grade records the evaluator configuration it was made under: the provider, every evaluator option as given, and for promptfoo the bytes of the assertions file and of every `promptfooconfig.*` beside it. When that configuration changes, the next `init` or `replay` re-grades every stored candidate output that was graded under another configuration. The outputs are reused rather than replayed, so no candidate model is called again, and an `evaluator_regrade` warning counts the outputs being re-graded and says why. The earlier grades stay in the store, but no verdict or report uses them, and `export` leaves them out. A replay run after upgrading from a version that did not record the configuration re-grades once for the same reason.
+
+`--evaluator-scorer`, `--evaluator-gate-metric` and `--evaluator-gate-threshold` are different: they define the question the evidence answers, so changing one replays the candidates again. Whenever replay runs, a model the provider catalog now selects that was never replayed is replayed like any new candidate, within `--max-cost-usd`.
+
+rightmodeler cannot see a change it is not given: a scorer edited inside Braintrust, an evaluator edited inside Langfuse, a LangSmith rule edited in place, a file an assertion loads with `file://`, a `.env` beside the assertions file, or a promptfoo upgrade behind the same `--evaluator-command`. To re-grade after one of these, edit the assertions file, or bind the LangSmith metric to the new rule's id (`metric=rule-id`). For Braintrust and Langfuse, the grades already held stay in use until an evaluator option changes.
+
+## Reachability, polling, and absences
+
+If the external evaluator is unreachable, the pipeline warns and uses the built-in judge. Hosted scorers run asynchronously, so after launching a run the pipeline polls for up to 5 minutes, starting 250 ms after launch and doubling the wait between polls up to 10 seconds. If a reachable run fails, is still pending when the budget ends, or omits required case results, the pipeline records the missing assessments instead of fabricating scores.
 
 See [Commands](commands.md) for the complete option text and [Getting started](getting-started.md) for provider setup.

package/docs/exit-codes.md

@@ -6,7 +6,7 @@ Rightmodeler reserves exit codes `0` through `3` for machine-readable outcomes.
 
 - `0`: the command completed and no recommendation is being reported. Planning and partial `--through` runs also return `0` when successful.
 - `1`: a complete `init` or `report` found an actionable recommendation.
-- `2`: the run needs input at a resumable boundary, such as missing traces, a cancelled trace prompt, provider configuration, or required confirmation configuration.
+- `2`: the run needs input at a resumable boundary, such as missing traces, a cancelled trace prompt, provider configuration, required confirmation configuration, or a plan's usage limit (rerun after it resets).
 - `3`: the cost budget was reached at a resumable boundary.
 - `10` or greater: command-line or runtime failure.
 
@@ -14,13 +14,58 @@ Rightmodeler reserves exit codes `0` through `3` for machine-readable outcomes.
 
 - `0`: changes were applied, or a dry run was clean.
 - `1`: the change was refused by a machine gate.
+- `2`: no completed run in the store (for example `stage_not_completed`); the JSON error on standard error names the remedy.
+- `10` or greater: runtime failure.
+
+See [GitHub](github.md) for tokens, reviewers and refusal codes.
+
+## Rollback
+
+- `0`: the rollback pull request was opened, or already exists.
+- `1`: the rollback was refused by a machine gate.
+- `10` or greater: runtime failure.
+
+## Drift
+
+- `0`: the drift check completed.
+- `2`: `--traces` is missing.
 - `10` or greater: runtime failure.
 
 ## Watch
 
 - `0`: no action was required.
 - `1`: review or continuous-integration actions were taken.
-- `2`: another watcher holds the lock.
+- `2`: another watcher holds the lock, or the store has no completed run. A held lock prints a result with `"status":"lock_held"` on standard output; a missing run prints an error such as `stage_not_completed` on standard error and nothing on standard output.
 - `10` or greater: runtime failure.
 
 Use `--output json` for one result object or `--output jsonl` for stage events followed by the result. Errors use the selected machine-readable mode on standard error. See [Commands](commands.md) for command-specific options.
+
+## Error codes
+
+- `active_corpus_usage_unavailable` (exit `2`): publish a corpus version built from traces that include token usage, then rerun.
+- `ambiguous_trace_format` (exit `2`): pass a trace file that unambiguously matches one supported format.
+- `budget_cap_refusal` (exit `3`): the run reached the cost boundary; raise `--max-cost-usd` to the cap named in the remedy and rerun.
+- `coverage_gate_failed` (exit `2`): add matcher coverage for the listed AI dependency surfaces, or pass `--matchers <file>` with declarative matchers that close them.
+- `empty_traces_directory` (exit `2`): point `--traces` at a directory containing `.json` or `.jsonl` trace files, or at a single trace file.
+- `git_repository_has_no_commits` (exit `2`): create the first commit, then rerun the command.
+- `invalid_catalog_reference` (exit `2`): pass `--catalog-reference` an http(s) URL or a readable file that returns an OpenAI-compatible `/models` document, or remove it, then rerun.
+- `invalid_matchers_file` (exit `2`): fix the listed matcher definitions in the `--matchers` file and rerun.
+- `invalid_modeb_config` (exit `2`): fix the named field in the `--modeb-config` file and rerun.
+- `invalid_option` (exit `2`): correct the option and rerun; use `rightmodeler <command> --help` for accepted values.
+- `invalid_policy_file` (exit `2`): fix the named field in the `--policy` file and rerun; `qualityFloor` must be greater than 0.8 and less than 1, `shortlistTop` a positive integer, `allowModels` and `denyModels` arrays of model ids.
+- `invalid_pricing_file` (exit `2`): fix `--pricing-file` to map each model id to non-negative `input` and `output` USD per token and, optionally, a positive integer `maxOutputTokens`, then rerun.
+- `judge_family_unknown` (exit `2`): the catalog's model ids name no vendor, so the built-in judge could share a vendor with the candidate or the recorded model; use a gateway whose ids carry their vendor (`vendor/model`), or grade with `--evaluator`.
+- `missing_provider_configuration` (exit `2`): pass `--base-url <url>` and, if needed, `--api-key-env <environment-variable-name>` naming a populated variable.
+- `missing_traces_path` (exit `2`): pass `--traces <path>` pointing to an existing trace file or directory.
+- `mixed_trace_formats` (exit `2`): split the directory so every file uses the same trace format, or pass one file with `--traces`.
+- `modeb_cloud_unavailable` (exit `2`): install the optional sandbox SDK and set its credentials, or set `"backend": "docker"` in the `--modeb-config` file, then rerun.
+- `no_neutral_judge` (exit `2`): the catalog has no priced model from a vendor other than both the candidate's and the recorded model's, which the built-in judge needs; list or price one (a multi-vendor gateway, `--catalog-reference` or `--pricing-file`), run the judge through another vendor's CLI with `--judge-route`, or grade with `--evaluator`.
+- `no_priced_candidates` (exit `2`): point `--base-url` at a catalog that publishes per-token pricing, pass `--catalog-reference <url>`, expose priced LiteLLM `GET /model/info`, or pass `--pricing-file <path>`, then rerun.
+- `no_replayable_call_sites` (exit `2`): point `--repo` at a service with plain text completions, or add a matcher for a text call site, then rerun.
+- `not_git_repository` (exit `2`): run the command again from a Git repository with at least one commit.
+- `plan_cli_unavailable` (exit `2`): the CLI a plan route runs is missing, older than the verified version, or changed an output shape, or `CI` is set; install or update it (`claude update`, or `npm install -g @openai/codex@latest` for `codex`), unset `CI` on your own machine, or use an API route with `--base-url`.
+- `plan_login_required` (exit `2`): sign the CLI in to your plan (`claude auth login`, or `codex login` with ChatGPT) and remove any API key setting it would use, or use an API route with `--base-url`; finished calls are kept.
+- `plan_usage_limit` (exit `2`): a plan you are signed in to reached its usage limit; rerun the same command after the reset time in the message, and completed replay and judge calls are kept and not repeated; or choose a route that does not use this plan with `--route` or `--judge-route`.
+- `stage_not_completed` (exit `2`): run `rightmodeler init --through <stage>` first, then rerun the command.
+- `unusable_trace_input` (exit `2`): the selected discovered trace could not be adapted; rerun and choose a different trace file.
+- `usage_error` (exit `10`): the command line is invalid; `message` carries the parser text.