rightmodeler 0.1.0

package/docs/commands.md

@@ -0,0 +1,503 @@
+# Commands
+
+This file is generated from the CLI definitions. Run `pnpm docs:generate` after changing a command.
+
+See [Getting started](getting-started.md) for the shortest complete workflow and [Exit codes](exit-codes.md) for automation behavior.
+
+## `rightmodeler`
+
+```text
+Usage: rightmodeler [options] [command]
+
+Find and prove safe model substitutions.
+
+Options:
+  -V, --version        output the version number
+  --repo <dir>         repository to analyze (default:
+                       ".")
+  --store <dir>        store directory
+  --output <mode>      output mode (choices: "human", "json", "jsonl", default:
+                       "human")
+  -h, --help           display help for command
+
+Commands:
+  init [options]       run the resumable Phase A pipeline
+  estimate [options]   project replay spend before paid model calls
+  scan [options]       run through the scan stage
+  ingest [options]     run through the ingest stage
+  reconcile [options]  run through the reconcile stage
+  scrub [options]      run through the scrub stage
+  shortlist [options]  run through the shortlist stage
+  replay [options]     run through the replay stage
+  aggregate [options]  run through the aggregate stage
+  confirm [options]    run through the confirm stage
+  corpus [options]     build or import the replay corpus
+  export [options]     export trials and verdicts to an evaluation provider
+  audit                manage the reference audit
+  apply [options]      open a draft pull request for proven model swaps
+  rollback [options]   open a draft pull request restoring a prior model swap
+  drift [options]      detect drift against the active replay corpus
+  watch [options]      reconcile one open model-swap pull request
+  report               write report.md and report.json
+  status [options]     summarize the current store
+  help [command]       display help for command
+```
+
+## `rightmodeler init`
+
+```text
+Usage: rightmodeler init [options]
+
+run the resumable Phase A pipeline
+
+Options:
+  --traces <path>                     trace input file
+  --include-free                      include zero-priced models in candidate
+                                      shortlists
+  --modeb-config <path>               versioned Mode B runtime configuration
+                                      JSON file
+  --base-url <url>                    OpenAI-compatible provider base URL
+  --api-key-env <name>                environment variable containing the
+                                      provider API key
+  --max-cost-usd <amount>             optional hard spend cap in USD; omit to
+                                      run uncapped so every case and judge cell
+                                      completes
+  --evaluator <provider>              external evaluator provider (choices:
+                                      "braintrust", "langfuse", "langsmith",
+                                      "promptfoo")
+  --evaluator-base-url <url>          external evaluator API base URL
+  --evaluator-api-key-env <name>      environment variable containing the
+                                      evaluator API key
+  --evaluator-public-key-env <name>   environment variable containing the
+                                      Langfuse public key
+  --evaluator-project-id <id>         Braintrust project or LangSmith dataset
+                                      identifier
+  --evaluator-command <path>          promptfoo executable path or command
+  --evaluator-config <path>           promptfoo assertions configuration file
+  --evaluator-scorer <name>           external evaluator scorer name
+                                      (repeatable)
+  --evaluator-gate-metric <name>      scorer metric used for release gates
+  --evaluator-gate-threshold <value>  fallback pass threshold when the evaluator
+                                      omits a pass decision
+  --plan                              print stage states without executing
+  --through <stage>                   stop after this stage (choices: "scan",
+                                      "ingest", "reconcile", "scrub", "corpus",
+                                      "audit-sample", "shortlist", "replay",
+                                      "aggregate", "confirm", "report")
+  --yes                               accept defaults (reserved for future
+                                      prompts)
+  -h, --help                          display help for command
+```
+
+## `rightmodeler estimate`
+
+```text
+Usage: rightmodeler estimate [options]
+
+project replay spend before paid model calls
+
+Options:
+  --traces <path>                     trace input file
+  --include-free                      include zero-priced models in candidate
+                                      shortlists
+  --modeb-config <path>               versioned Mode B runtime configuration
+                                      JSON file
+  --base-url <url>                    OpenAI-compatible provider base URL
+  --api-key-env <name>                environment variable containing the
+                                      provider API key
+  --max-cost-usd <amount>             optional hard spend cap in USD; omit to
+                                      run uncapped so every case and judge cell
+                                      completes
+  --evaluator <provider>              external evaluator provider (choices:
+                                      "braintrust", "langfuse", "langsmith",
+                                      "promptfoo")
+  --evaluator-base-url <url>          external evaluator API base URL
+  --evaluator-api-key-env <name>      environment variable containing the
+                                      evaluator API key
+  --evaluator-public-key-env <name>   environment variable containing the
+                                      Langfuse public key
+  --evaluator-project-id <id>         Braintrust project or LangSmith dataset
+                                      identifier
+  --evaluator-command <path>          promptfoo executable path or command
+  --evaluator-config <path>           promptfoo assertions configuration file
+  --evaluator-scorer <name>           external evaluator scorer name
+                                      (repeatable)
+  --evaluator-gate-metric <name>      scorer metric used for release gates
+  --evaluator-gate-threshold <value>  fallback pass threshold when the evaluator
+                                      omits a pass decision
+  --approved-run <digest>             scope projection to one merged approved
+                                      swap
+  -h, --help                          display help for command
+```
+
+## `rightmodeler scan`
+
+```text
+Usage: rightmodeler scan [options]
+
+run through the scan stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler ingest`
+
+```text
+Usage: rightmodeler ingest [options]
+
+run through the ingest stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler reconcile`
+
+```text
+Usage: rightmodeler reconcile [options]
+
+run through the reconcile stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler scrub`
+
+```text
+Usage: rightmodeler scrub [options]
+
+run through the scrub stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler shortlist`
+
+```text
+Usage: rightmodeler shortlist [options]
+
+run through the shortlist stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler replay`
+
+```text
+Usage: rightmodeler replay [options]
+
+run through the replay stage
+
+Options:
+  --traces <path>                     trace input file
+  --include-free                      include zero-priced models in candidate
+                                      shortlists
+  --modeb-config <path>               versioned Mode B runtime configuration
+                                      JSON file
+  --base-url <url>                    OpenAI-compatible provider base URL
+  --api-key-env <name>                environment variable containing the
+                                      provider API key
+  --max-cost-usd <amount>             optional hard spend cap in USD; omit to
+                                      run uncapped so every case and judge cell
+                                      completes
+  --evaluator <provider>              external evaluator provider (choices:
+                                      "braintrust", "langfuse", "langsmith",
+                                      "promptfoo")
+  --evaluator-base-url <url>          external evaluator API base URL
+  --evaluator-api-key-env <name>      environment variable containing the
+                                      evaluator API key
+  --evaluator-public-key-env <name>   environment variable containing the
+                                      Langfuse public key
+  --evaluator-project-id <id>         Braintrust project or LangSmith dataset
+                                      identifier
+  --evaluator-command <path>          promptfoo executable path or command
+  --evaluator-config <path>           promptfoo assertions configuration file
+  --evaluator-scorer <name>           external evaluator scorer name
+                                      (repeatable)
+  --evaluator-gate-metric <name>      scorer metric used for release gates
+  --evaluator-gate-threshold <value>  fallback pass threshold when the evaluator
+                                      omits a pass decision
+  --detach                            enqueue replay and return its run
+                                      identifier
+  --approved-run <digest>             regression-test one merged approved swap
+  -h, --help                          display help for command
+```
+
+## `rightmodeler aggregate`
+
+```text
+Usage: rightmodeler aggregate [options]
+
+run through the aggregate stage
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler confirm`
+
+```text
+Usage: rightmodeler confirm [options]
+
+run through the confirm stage
+
+Options:
+  --traces <path>                     trace input file
+  --include-free                      include zero-priced models in candidate
+                                      shortlists
+  --modeb-config <path>               versioned Mode B runtime configuration
+                                      JSON file
+  --base-url <url>                    OpenAI-compatible provider base URL
+  --api-key-env <name>                environment variable containing the
+                                      provider API key
+  --max-cost-usd <amount>             optional hard spend cap in USD; omit to
+                                      run uncapped so every case and judge cell
+                                      completes
+  --evaluator <provider>              external evaluator provider (choices:
+                                      "braintrust", "langfuse", "langsmith",
+                                      "promptfoo")
+  --evaluator-base-url <url>          external evaluator API base URL
+  --evaluator-api-key-env <name>      environment variable containing the
+                                      evaluator API key
+  --evaluator-public-key-env <name>   environment variable containing the
+                                      Langfuse public key
+  --evaluator-project-id <id>         Braintrust project or LangSmith dataset
+                                      identifier
+  --evaluator-command <path>          promptfoo executable path or command
+  --evaluator-config <path>           promptfoo assertions configuration file
+  --evaluator-scorer <name>           external evaluator scorer name
+                                      (repeatable)
+  --evaluator-gate-metric <name>      scorer metric used for release gates
+  --evaluator-gate-threshold <value>  fallback pass threshold when the evaluator
+                                      omits a pass decision
+  -h, --help                          display help for command
+```
+
+## `rightmodeler corpus`
+
+```text
+Usage: rightmodeler corpus [options] [command]
+
+build or import the replay corpus
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+
+Commands:
+  import [options]       import a curated provider dataset
+```
+
+## `rightmodeler corpus import`
+
+```text
+Usage: rightmodeler corpus import [options]
+
+import a curated provider dataset
+
+Options:
+  --from <provider:dataset>  provider and dataset
+  --base-url <url>           dataset provider API base URL
+  --api-key-env <name>       environment variable containing the dataset
+                             provider API key
+  --public-key-env <name>    environment variable containing the Langfuse public
+                             key
+  -h, --help                 display help for command
+```
+
+## `rightmodeler export`
+
+```text
+Usage: rightmodeler export [options]
+
+export trials and verdicts to an evaluation provider
+
+Options:
+  --to <provider>          result sink provider (choices: "braintrust",
+                           "langfuse")
+  --base-url <url>         result sink API base URL
+  --api-key-env <name>     environment variable containing the result sink API
+                           key
+  --public-key-env <name>  environment variable containing the Langfuse public
+                           key
+  --project-id <id>        Braintrust project identifier
+  --dataset-id <id>        Langfuse dataset identifier
+  -h, --help               display help for command
+```
+
+## `rightmodeler audit`
+
+```text
+Usage: rightmodeler audit [options] [command]
+
+manage the reference audit
+
+Options:
+  -h, --help          display help for command
+
+Commands:
+  sample [options]    write the audit worksheet without blocking
+  tabulate [options]  tabulate a completed audit worksheet
+  help [command]      display help for command
+```
+
+## `rightmodeler audit sample`
+
+```text
+Usage: rightmodeler audit sample [options]
+
+write the audit worksheet without blocking
+
+Options:
+  --traces <path>        trace input file
+  --include-free         include zero-priced models in candidate shortlists
+  --modeb-config <path>  versioned Mode B runtime configuration JSON file
+  -h, --help             display help for command
+```
+
+## `rightmodeler audit tabulate`
+
+```text
+Usage: rightmodeler audit tabulate [options]
+
+tabulate a completed audit worksheet
+
+Options:
+  --worksheet <path>  completed worksheet JSON file
+  -h, --help          display help for command
+```
+
+## `rightmodeler apply`
+
+```text
+Usage: rightmodeler apply [options]
+
+open a draft pull request for proven model swaps
+
+Options:
+  --owner <owner>            GitHub repository owner
+  --github-base-url <url>    GitHub API base URL
+  --github-token-env <name>  environment variable containing the GitHub token
+  --dry-run                  run all machine gates without writing GitHub state
+  -h, --help                 display help for command
+```
+
+## `rightmodeler rollback`
+
+```text
+Usage: rightmodeler rollback [options]
+
+open a draft pull request restoring a prior model swap
+
+Options:
+  --owner <owner>            GitHub repository owner
+  --pr <number>              merged pull request number
+  --github-base-url <url>    GitHub API base URL
+  --github-token-env <name>  environment variable containing the GitHub token
+  -h, --help                 display help for command
+```
+
+## `rightmodeler drift`
+
+```text
+Usage: rightmodeler drift [options] [command]
+
+detect drift against the active replay corpus
+
+Options:
+  --traces <path>    new trace batch
+  -h, --help         display help for command
+
+Commands:
+  approve [options]  approve a stored corpus drift proposal
+  publish [options]  publish an approved corpus drift proposal
+```
+
+## `rightmodeler drift approve`
+
+```text
+Usage: rightmodeler drift approve [options]
+
+approve a stored corpus drift proposal
+
+Options:
+  --proposal <id>  drift proposal SHA-256 identifier
+  --actor <name>   approving actor
+  --reason <text>  approval reason
+  -h, --help       display help for command
+```
+
+## `rightmodeler drift publish`
+
+```text
+Usage: rightmodeler drift publish [options]
+
+publish an approved corpus drift proposal
+
+Options:
+  --proposal <id>  drift proposal SHA-256 identifier
+  -h, --help       display help for command
+```
+
+## `rightmodeler watch`
+
+```text
+Usage: rightmodeler watch [options]
+
+reconcile one open model-swap pull request
+
+Options:
+  --owner <owner>            GitHub repository owner
+  --github-repo <repo>       GitHub repository name
+  --pr <number>              pull request number
+  --github-base-url <url>    GitHub API base URL
+  --github-token-env <name>  environment variable containing the GitHub token
+  -h, --help                 display help for command
+```
+
+## `rightmodeler report`
+
+```text
+Usage: rightmodeler report [options]
+
+write report.md and report.json
+
+Options:
+  -h, --help  display help for command
+```
+
+## `rightmodeler status`
+
+```text
+Usage: rightmodeler status [options]
+
+summarize the current store
+
+Options:
+  --run <runId>  report one detached replay run
+  -h, --help     display help for command
+```

package/docs/evaluators.md

@@ -0,0 +1,16 @@
+# Evaluators
+
+The default evaluator is the built-in judge selected from the configured provider catalog. Candidate and reference families are excluded when choosing the judge.
+
+An external evaluator can be requested with `--evaluator braintrust`. Configure it with:
+
+- `--evaluator-project-id <id>` (required)
+- one or more `--evaluator-scorer <name>` options (required)
+- `--evaluator-api-key-env <name>` (defaults to `BRAINTRUST_API_KEY`)
+- `--evaluator-base-url <url>` (defaults to the provider API)
+- `--evaluator-gate-metric <name>` when more than one scorer is configured
+- `--evaluator-gate-threshold <value>` when the evaluator does not return a pass decision
+
+The gate metric must name one of the configured scorers. If the external evaluator is unavailable, the pipeline warns and uses the built-in judge. If a reachable evaluator run fails or omits required case results, the pipeline records the missing assessments instead of fabricating scores.
+
+See [Commands](commands.md) for the complete option text and [Getting started](getting-started.md) for provider setup.

package/docs/exit-codes.md

@@ -0,0 +1,26 @@
+# Exit codes
+
+Rightmodeler reserves exit codes `0` through `3` for machine-readable outcomes. Runtime and command-line failures use `10` or greater.
+
+## Pipeline commands
+
+- `0`: the command completed and no recommendation is being reported. Planning and partial `--through` runs also return `0` when successful.
+- `1`: a complete `init` or `report` found an actionable recommendation.
+- `2`: the run needs input at a resumable boundary, such as missing traces, provider configuration, or required confirmation configuration.
+- `3`: the cost budget was reached at a resumable boundary.
+- `10` or greater: command-line or runtime failure.
+
+## Apply
+
+- `0`: changes were applied, or a dry run was clean.
+- `1`: the change was refused by a machine gate.
+- `10` or greater: runtime failure.
+
+## Watch
+
+- `0`: no action was required.
+- `1`: review or continuous-integration actions were taken.
+- `2`: another watcher holds the lock.
+- `10` or greater: runtime failure.
+
+Use `--output json` for one result object or `--output jsonl` for stage events followed by the result. Errors use the selected machine-readable mode on standard error. See [Commands](commands.md) for command-specific options.

package/docs/getting-started.md

@@ -0,0 +1,33 @@
+# Getting started
+
+Rightmodeler analyzes recorded model calls, replays them against cheaper candidates, evaluates the outputs, and writes a recommendation report. The intended public package name is `rightmodeler`.
+
+## Requirements
+
+- Node.js 24 or newer.
+- A Git repository to analyze.
+- Trace input in a supported format.
+- An OpenAI-compatible provider base URL and the name of an environment variable containing its API key before replay begins.
+
+## Preview without changing the repository
+
+```sh
+npx rightmodeler init --plan --output json --repo /path/to/repository
+```
+
+## Run through the free corpus stage
+
+```sh
+npx rightmodeler init --through corpus --traces /path/to/traces.json --output json --repo /path/to/repository
+```
+
+## Run the complete pipeline
+
+```sh
+export RIGHTMODELER_API_KEY="provider-key"
+npx rightmodeler init --traces /path/to/traces.json --base-url https://provider.example/v1 --output json --repo /path/to/repository
+```
+
+The default store is `.rightmodeler/` inside the analyzed repository. Completed stages resume when their inputs and outputs are still current. A complete run writes `.rightmodeler/project/reports/report.md` and `.rightmodeler/project/reports/report.json`.
+
+Read the generated [command reference](commands.md), the [evaluator guide](evaluators.md), [Mode B configuration](modeb.md), and the [exit-code convention](exit-codes.md) before automating a full run.

package/docs/modeb.md

@@ -0,0 +1,31 @@
+# Mode B configuration
+
+Mode B runs confirmation cases inside a container when a recommendation can affect downstream model-authored steps. Pass a JSON file with `--modeb-config`.
+
+```json
+{
+  "version": "1",
+  "image": "my-agent:latest",
+  "appSpec": {
+    "mountPath": ".",
+    "command": ["node", "/rightmodeler/app/driver.mjs", "{caseFile}"],
+    "installCommand": ["pnpm", "install", "--offline"]
+  },
+  "stepMap": {
+    "canonical-step-id": "runtime-step-header"
+  },
+  "confirmMaxRunSets": 20
+}
+```
+
+## Contract
+
+- `version` must be the string `"1"`.
+- `image` is the non-empty container image name.
+- `appSpec.mountPath` is resolved relative to the configuration file and mounted read-only at `/rightmodeler/app`.
+- `appSpec.command` is a non-empty array of non-empty arguments. At least one argument must contain `{caseFile}`; the harness replaces every occurrence with the in-container case file path.
+- `appSpec.installCommand` is optional. When present, it is a non-empty array of non-empty arguments run before the workload.
+- `stepMap` maps at least one canonical scanner step ID to the runtime step header emitted by the application. Runtime headers must be unique.
+- `confirmMaxRunSets` is optional and must be a non-negative integer.
+
+See [Commands](commands.md) for where `--modeb-config` is accepted and [Exit codes](exit-codes.md) for blocked or failed runs.

package/package.json

@@ -0,0 +1,16 @@
+{
+  "name": "rightmodeler",
+  "version": "0.1.0",
+  "type": "module",
+  "bin": {
+    "rightmodeler": "./dist-bundle/cli.js"
+  },
+  "exports": "./dist-bundle/cli.js",
+  "files": [
+    "dist-bundle",
+    "docs"
+  ],
+  "engines": {
+    "node": ">=24"
+  }
+}