rightmodeler 0.2.0 → 0.3.0

package/docs/github-actions.md

@@ -0,0 +1,328 @@
+# GitHub Actions
+
+This workflow runs rightmodeler in GitHub Actions with the job's built-in `GITHUB_TOKEN`. It needs no GitHub App and no personal access token.
+
+## What it does
+
+- Runs `init` every Monday and whenever a push to `main` changes `traces/`.
+- Opens the draft pull request only when a person runs the workflow with `command=apply`.
+- Runs `watch` every 6 hours on each swap pull request that is still being watched.
+- Never merges. A person reviews and merges the draft.
+
+## Setup
+
+1. Let GitHub Actions open pull requests. In the repository settings, under Actions, General, Workflow permissions, turn on "Allow GitHub Actions to create and approve pull requests", or run `gh api -X PUT repos/<owner>/<repo>/actions/permissions/workflow -F can_approve_pull_request_reviews=true`. This is the only repository setting the workflow needs. In a repository owned by an organization, the organization must allow it first.
+2. Set the repository variable `RIGHTMODELER_PROVIDER_BASE_URL` and the repository secret `RIGHTMODELER_PROVIDER_API_KEY`, for example with `gh variable set` and `gh secret set`.
+3. Commit traces under `traces/`, or change `RIGHTMODELER_TRACES` and the `paths` filter together.
+4. Change `main` if the default branch has another name, and adjust `RIGHTMODELER_MAX_COST_USD`.
+5. Save the workflow below as `.github/workflows/rightmodeler.yml`.
+
+Each job grants `GITHUB_TOKEN` only what its command needs: `apply` gets `contents: write` and `pull-requests: write`, and `watch` gets `contents: read`, `pull-requests: write`, `checks: read` and `statuses: read`. See [GitHub](github.md) for what each command does with them.
+
+## The workflow
+
+```yaml
+name: rightmodeler
+
+on:
+  schedule:
+    - cron: "17 5 * * 1"
+    - cron: "43 */6 * * *"
+  push:
+    branches: [main]
+    paths:
+      - "traces/**"
+  workflow_dispatch:
+    inputs:
+      command:
+        description: Which rightmodeler command to run
+        type: choice
+        options: [init, apply, watch]
+        default: init
+
+permissions:
+  contents: read
+
+concurrency:
+  group: rightmodeler-store
+  cancel-in-progress: false
+
+env:
+  RIGHTMODELER_VERSION: "0.3.0"
+  RIGHTMODELER_TRACES: traces
+  RIGHTMODELER_MAX_COST_USD: "5"
+  RM_ANNOTATE: |
+    const { readFileSync } = require("node:fs");
+    const data = (s) => String(s).replace(/%/g, "%25").replace(/\r/g, "%0D").replace(/\n/g, "%0A");
+    const prop = (s) => data(s).replace(/:/g, "%3A").replace(/,/g, "%2C");
+    for (const file of process.argv.slice(1)) {
+      let text = "";
+      try { text = readFileSync(file, "utf8"); } catch { continue; }
+      for (const line of text.split("\n")) {
+        let value;
+        try { value = JSON.parse(line); } catch { continue; }
+        if (value === null || typeof value !== "object") continue;
+        if (value.event === "result") value = value.result;
+        if (value.event === "warning") {
+          console.log(`::warning title=${prop(`rightmodeler ${value.code}`)}::${data(value.message)}`);
+        } else if (value.status === "refused" && Array.isArray(value.reasons)) {
+          for (const reason of value.reasons) {
+            console.log(`::error title=${prop(`rightmodeler ${reason.code}`)}::${data(reason.message)}`);
+          }
+        } else if (typeof value.code === "string" && typeof value.message === "string") {
+          console.log(`::error title=${prop(`rightmodeler ${value.code}`)}::${data(`${value.message} Remedy: ${value.remedy ?? ""}`)}`);
+        }
+      }
+    }
+
+jobs:
+  init:
+    if: >-
+      github.event_name == 'push' ||
+      (github.event_name == 'schedule' && github.event.schedule == '17 5 * * 1') ||
+      (github.event_name == 'workflow_dispatch' && inputs.command == 'init')
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 24
+      - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+          restore-keys: rightmodeler-store-
+      - id: init
+        name: Find and prove cheaper models
+        env:
+          RIGHTMODELER_PROVIDER_BASE_URL: ${{ vars.RIGHTMODELER_PROVIDER_BASE_URL }}
+          RIGHTMODELER_PROVIDER_API_KEY: ${{ secrets.RIGHTMODELER_PROVIDER_API_KEY }}
+        run: |
+          out="$RUNNER_TEMP/rightmodeler"
+          mkdir -p "$out"
+          npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" --version
+          set +e
+          npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" init \
+            --traces "$RIGHTMODELER_TRACES" \
+            --base-url "$RIGHTMODELER_PROVIDER_BASE_URL" \
+            --api-key-env RIGHTMODELER_PROVIDER_API_KEY \
+            --max-cost-usd "$RIGHTMODELER_MAX_COST_USD" \
+            --output jsonl --repo "$GITHUB_WORKSPACE" \
+            >"$out/init.jsonl" 2>"$out/init.err"
+          code=$?
+          set -e
+          node -e "$RM_ANNOTATE" "$out/init.jsonl" "$out/init.err"
+          report="$GITHUB_WORKSPACE/.rightmodeler/project/reports/report.md"
+          if [ -f "$report" ]; then
+            cp "$report" "$out/report.md"
+            cat "$report" >>"$GITHUB_STEP_SUMMARY"
+          fi
+          case "$code" in
+            0) echo "recommendation=false" >>"$GITHUB_OUTPUT" ;;
+            1)
+              echo "recommendation=true" >>"$GITHUB_OUTPUT"
+              echo "::notice title=rightmodeler::A proven swap is ready. Run this workflow with command=apply to open the draft pull request."
+              ;;
+            *)
+              echo "::error title=rightmodeler::init exited $code; the annotations above name the cause and the fix."
+              exit 1
+              ;;
+          esac
+      - if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rightmodeler-init
+          path: ${{ runner.temp }}/rightmodeler/
+          if-no-files-found: ignore
+      - if: always()
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+
+  apply:
+    if: github.event_name == 'workflow_dispatch' && inputs.command == 'apply'
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: write
+      pull-requests: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 24
+      - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+          restore-keys: rightmodeler-store-
+      - id: apply
+        name: Open the draft pull request
+        env:
+          RIGHTMODELER_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          out="$RUNNER_TEMP/rightmodeler"
+          mkdir -p "$out"
+          npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" --version
+          run_apply() {
+            npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" apply "$@" \
+              --owner "$GITHUB_REPOSITORY_OWNER" \
+              --github-repo "${GITHUB_REPOSITORY#*/}" \
+              --github-base-url "$GITHUB_API_URL" \
+              --github-token-env RIGHTMODELER_GITHUB_TOKEN \
+              --output json --repo "$GITHUB_WORKSPACE"
+          }
+          for mode in dry-run apply; do
+            set +e
+            if [ "$mode" = "dry-run" ]; then
+              run_apply --dry-run >"$out/$mode.json" 2>"$out/$mode.err"
+            else
+              run_apply >"$out/$mode.json" 2>"$out/$mode.err"
+            fi
+            code=$?
+            set -e
+            node -e "$RM_ANNOTATE" "$out/$mode.json" "$out/$mode.err"
+            if [ "$code" -ne 0 ]; then
+              echo "::error title=rightmodeler::apply ($mode) exited $code; the annotations above name the cause and the fix."
+              exit 1
+            fi
+          done
+          pr="$(node -p 'JSON.parse(require("node:fs").readFileSync(process.argv[1], "utf8")).prNumber' "$out/apply.json")"
+          echo "pr-number=$pr" >>"$GITHUB_OUTPUT"
+          echo "Draft pull request #$pr is open for review. rightmodeler never merges it." >>"$GITHUB_STEP_SUMMARY"
+      - if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rightmodeler-apply
+          path: ${{ runner.temp }}/rightmodeler/
+          if-no-files-found: ignore
+      - if: always()
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+
+  watch:
+    if: >-
+      (github.event_name == 'schedule' && github.event.schedule == '43 */6 * * *') ||
+      (github.event_name == 'workflow_dispatch' && inputs.command == 'watch')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      pull-requests: write
+      checks: read
+      statuses: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 24
+      - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+          restore-keys: rightmodeler-store-
+      - id: watch
+        name: Reconcile open swap pull requests
+        env:
+          RIGHTMODELER_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          out="$RUNNER_TEMP/rightmodeler"
+          mkdir -p "$out"
+          npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" --version
+          set +e
+          npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" status --output json \
+            --repo "$GITHUB_WORKSPACE" >"$out/status.json" 2>"$out/status.err"
+          code=$?
+          set -e
+          node -e "$RM_ANNOTATE" "$out/status.err"
+          if [ "$code" -ne 0 ]; then
+            echo "::error title=rightmodeler::status exited $code"
+            exit 1
+          fi
+          failed=0
+          for pr in $(node -p 'JSON.parse(require("node:fs").readFileSync(process.argv[1], "utf8")).pullRequests.map((entry) => entry.prNumber).join(" ")' "$out/status.json"); do
+            set +e
+            npx --yes "rightmodeler@${RIGHTMODELER_VERSION}" watch --pr "$pr" \
+              --owner "$GITHUB_REPOSITORY_OWNER" \
+              --github-repo "${GITHUB_REPOSITORY#*/}" \
+              --github-base-url "$GITHUB_API_URL" \
+              --github-token-env RIGHTMODELER_GITHUB_TOKEN \
+              --output json --repo "$GITHUB_WORKSPACE" \
+              >"$out/watch-$pr.json" 2>"$out/watch-$pr.err"
+            code=$?
+            set -e
+            node -e "$RM_ANNOTATE" "$out/watch-$pr.json" "$out/watch-$pr.err"
+            case "$code" in
+              0) ;;
+              1) echo "::notice title=rightmodeler::watch acted on pull request #$pr" ;;
+              2)
+                held="$(node -p 'try { JSON.parse(require("node:fs").readFileSync(process.argv[1], "utf8")).status } catch { "" }' "$out/watch-$pr.json")"
+                if [ "$held" = "lock_held" ]; then
+                  echo "::warning title=rightmodeler::another watcher holds the lock for pull request #$pr; the next run retries"
+                else
+                  failed=1
+                fi
+                ;;
+              *) failed=1 ;;
+            esac
+          done
+          exit "$failed"
+      - if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rightmodeler-watch
+          path: ${{ runner.temp }}/rightmodeler/
+          if-no-files-found: ignore
+      - if: always()
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: .rightmodeler
+          key: rightmodeler-store-${{ github.run_id }}-${{ github.run_attempt }}
+```
+
+## How it behaves
+
+- **The store.** `.rightmodeler/` lives in the Actions cache. Each run saves it under a new key and restores the newest `rightmodeler-store-` entry. GitHub removes entries unused for 7 days, and the 6-hourly watch keeps the store in use. Anyone with read access to the repository can read cache contents and uploaded artifacts, so use this workflow in private repositories.
+- **One run at a time.** All runs share one concurrency group. If a run is already waiting, a newly queued run replaces it, so dispatch again if yours was replaced.
+- **Annotations.** Every error and warning the CLI prints becomes an annotation that names its code, and errors also carry the remedy. `init` adds the report to the job summary, and every job uploads its output files as an artifact.
+- **Stale evidence.** If `main` moves between `init` and `apply`, `apply` refuses with `stale_evidence`. Run `init` again.
+- **Other branches.** A dispatch from another branch works on that branch. It starts from the default branch's store, saves its own copy that only that branch's runs restore, and `apply` opens the draft against that branch.
+- **The token.** The draft is authored by `github-actions[bot]`, and the owners of the swapped files are requested as reviewers. GitHub starts no workflow for a push made with `GITHUB_TOKEN`. For a pull request that `GITHUB_TOKEN` opens, GitHub creates the `pull_request` workflow runs in an approval-required state, and a person with write access starts them with "Approve workflows to run" on the pull request.
+- **Schedules.** GitHub can delay scheduled runs at busy times, and disables schedules in public repositories after 60 days without activity.
+
+## Optional: a GitHub App token
+
+Not tested live: rightmodeler's acceptance runs of this workflow use `GITHUB_TOKEN` only. A GitHub App installation token lets the draft's `pull_request` workflows start without approval, and the draft is authored by `<app-slug>[bot]`. To use one:
+
+1. Create a GitHub App with the permissions in [GitHub](github.md) and install it on the repository.
+2. Store its client ID in the repository variable `RIGHTMODELER_APP_CLIENT_ID` and its private key in the repository secret `RIGHTMODELER_APP_PRIVATE_KEY`.
+3. In the `apply` and `watch` jobs, add this step before the rightmodeler step, and change that step's `RIGHTMODELER_GITHUB_TOKEN` to `${{ steps.app-token.outputs.token }}`:
+
+   ```
+   - id: app-token
+     uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
+     with:
+       client-id: ${{ vars.RIGHTMODELER_APP_CLIENT_ID }}
+       private-key: ${{ secrets.RIGHTMODELER_APP_PRIVATE_KEY }}
+   ```
+
+## Cloud Mode B
+
+To confirm in Vercel Sandbox, add `--modeb-config <file>` to the `init` command and map the `VERCEL_TOKEN`, `VERCEL_TEAM_ID` and `VERCEL_PROJECT_ID` secrets into that step's `env`. See [Mode B](modeb.md).
+
+## Upgrading
+
+Change `RIGHTMODELER_VERSION`. Each rightmodeler step first runs the CLI with `--version`, so a version npm cannot install fails the step with npm's error in its log instead of being mistaken for a rightmodeler exit code. Each release's copy of this guide pins that release, and `rightmodeler docs github-actions` prints the copy for the installed version.

package/docs/github.md

@@ -0,0 +1,110 @@
+# GitHub
+
+Three commands talk to GitHub. `apply` opens a draft pull request that changes model identifiers only and carries an evidence table. `watch` reconciles one of those pull requests per run. `rollback` opens a draft pull request that restores a merged swap. The CLI never merges a pull request: a person reviews and merges.
+
+## Tokens
+
+Pass the name of the environment variable that holds the token with `--github-token-env`, never the token itself.
+
+| Token                                                                    | What works                                                                                                                                                                                                                                                                                                |
+| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| GitHub App installation token (recommended)                              | `apply` and `watch` fully.                                                                                                                                                                                                                                                                                |
+| Classic personal access token, or `gh auth token`, with the `repo` scope | `apply` and `watch` fully.                                                                                                                                                                                                                                                                                |
+| Fine-grained personal access token                                       | `apply` fully. `watch` works but cannot read check runs, so each pass prints the `github_checks_unavailable` warning and uses commit statuses only.                                                                                                                                                       |
+| GitHub Actions `GITHUB_TOKEN`                                            | `apply` and `watch` when the job grants `contents: write`, `pull-requests: write` and `statuses: read` (add `checks: read` for check runs) and the repository allows GitHub Actions to create pull requests. CI on the pull requests it opens waits for approval, which an App installation token avoids. |
+
+Give a GitHub App these repository permissions: Contents read and write, Pull requests read and write, Checks read, Commit statuses read, and Metadata read. Pull requests it opens are authored by `<app-slug>[bot]`.
+
+Give a fine-grained personal access token access to the repository with Contents read and write, Pull requests read and write, Commit statuses read, and Metadata read. GitHub offers no check-run permission for these tokens.
+
+Commenting on a pull request needs only Pull requests write, so neither token needs the Issues permission.
+
+For a ready workflow that uses `GITHUB_TOKEN`, see [GitHub Actions](github-actions.md).
+
+## API host
+
+`--github-base-url` defaults to `https://api.github.com`. For GitHub Enterprise Server, pass its API URL, such as `https://github.example.com/api/v3`. Enterprise Server 3.21 or newer is needed, because the CLI sends REST API version `2026-03-10` and older servers answer every call with HTTP 400.
+
+`--github-repo` defaults to the name of the directory given to `--repo` on `apply`, `watch` and `rollback`. Pass it when the directory name differs from the repository name on GitHub.
+
+## Apply
+
+`apply` needs:
+
+- a completed `init` run whose report recommends at least one swap;
+- local `HEAD` on a branch and equal to the evidence revision, and that branch on GitHub at the same revision (the branch becomes the pull request base);
+- no uncommitted change to any file the swap touches.
+
+`--dry-run` runs every check and reads GitHub but writes nothing. Its result lists the branch, title, body, files and reviewers it would use. The reviewer list is shown before the pull request author is removed, because the author is known only once the pull request exists.
+
+Reviewers:
+
+- Owners of each swapped file come from the first of `.github/CODEOWNERS`, `CODEOWNERS` and `docs/CODEOWNERS` that exists, and the last matching rule wins.
+- A file with no matching rule falls back to its three most recent `git blame` authors, matched to GitHub users by commit email. Blame needs full history, so fetch with depth 0 in CI.
+- At most five users and five teams are requested, and never the pull request author.
+- If GitHub rejects the batch with HTTP 422, each reviewer is requested alone and the rejected ones are dropped.
+- Team reviewers need a repository owned by an organization, and every reviewer needs access to the repository.
+- GitHub does not request code owners on draft pull requests by itself, so every review request on the draft comes from `apply`.
+
+The branch is `<prefix>swap-<family>-<digest prefix>`, where the prefix is the one most common among the repository's recent branches, or `rightmodeler/` when no branch has one. The title is `perf(models): swap <families>` when the repository uses conventional commits, and `Swap <families> models` otherwise. The body is the repository's pull request template, if it has one, followed by the `## Rightmodeler evidence` table: revision, corpus version, and per family the decision, evaluators, cascade status, worst-case bound, models, cost per case, latency, caps and case IDs. Case IDs are SHA-256 digests of the replayed cases, never prompts. With `--code-graph <path>`, a `## Code context (Graphify)` section for the swapped call sites follows the table; the owners it lists are never requested as reviewers.
+
+Rerunning `apply` for the same evidence returns the same open pull request with `status: "existing"` and requests reviewers again only if the first attempt left no record of them.
+
+## Refusal codes
+
+A refusal exits `1` and prints a result with `status: "refused"` and one or more `reasons`, each with a `code`, a `message` and a `detail`.
+
+Apply:
+
+- `no_confirmed_recommendation`: no recommended family has confirmed or not-required cascade evidence.
+- `release_gate_failed`: at least one release gate is not green.
+- `inconsistent_evidence`: the selected families do not share one evidence revision, corpus and gate policy.
+- `previously_rejected`: this evidence and swap set was previously rejected and needs new evidence before it can be proposed again.
+- `stale_evidence`: the evidence revision does not match the repository `HEAD`, or the remote base moved beyond it; re-prove before applying.
+- `detached_head`: the repository has no current branch to use as the pull request base.
+- `dirty_worktree`: a file the swap touches has uncommitted changes; commit or stash them before applying.
+- `stale_location`: a proposed swap no longer matches its scan-time file digest or no longer has one fresh source location.
+- `diff_lint_failed`: the proposed diff changes something other than model identifiers.
+- `formatter_blocked`: the repository's formatter changed content outside the proposed swap.
+- `host_conventions_unreadable`: one or more repository instructions could not be read unambiguously.
+- `invalid_repository_revision`: the evidence revision is not a 40- or 64-character lowercase hex object ID.
+- `apply_branch_unowned`: the swap branch exists without a recorded apply start.
+- `apply_branch_scope_mismatch`: the existing swap branch changes files outside its declared scope.
+- `apply_resume_state_mismatch`: the recorded pre-apply state does not match the resumed change.
+- `apply_restore_failed`: after a failed apply, a file on the swap branch could not be restored.
+
+Rollback:
+
+- `missing_remediation_evidence`: the pull request has no valid recorded apply evidence.
+- `original_pr_not_merged`: only a merged swap pull request can be rolled back.
+- `pre_apply_revision_unavailable`: the recorded pre-apply file revision is unavailable or does not match its digest.
+- `post_apply_digest_mismatch`: the affected files no longer match the recorded post-apply state.
+- `rollback_branch_unowned`: the rollback branch exists without a recorded rollback start or base revision.
+- `rollback_branch_scope_mismatch`: the existing rollback branch changes files outside the recorded scope.
+- `rollback_restore_mismatch`: the rollback did not restore the recorded pre-apply state.
+- `rollback_restore_failed`: after a failed rollback, a file could not be restored to the base state.
+
+## Watch
+
+Each `watch` run makes one pass over one pull request under a lock kept in the store, so overlapping runs do not act twice. Run it on a schedule or from repository events. A pass:
+
+- records a merge and ends watching;
+- records a close without merge as a rejection and ends watching, after which `apply` refuses that swap with `previously_rejected`;
+- answers each human review and comment once with the stored evidence for the families it names, else for the families in the file it comments on, else for every family in the pull request;
+- marks families for re-proof when a reviewer requests changes, or when the base branch changes a swapped file, and says so in a comment; the re-proof itself happens on the next pipeline run;
+- on a failing check run or commit status, comments once, and if a check with the same name fails again under a new run on the same head commit, closes the pull request.
+
+Exit codes:
+
+- `0`: nothing needed doing.
+- `1`: the pass took an action, listed in the result's `actions`.
+- `2`: another watcher holds the lock, or the store has no completed run. With `--output json` or `jsonl`, a held lock prints a result with `"status":"lock_held"` on standard output, while a missing run prints an error with code `stage_not_completed` on standard error and nothing on standard output.
+- `10` or greater: runtime failure.
+
+When GitHub refuses to list check runs with HTTP 403, the pass still reconciles reviews, comments, commit statuses, merges and base-branch changes, and prints the `github_checks_unavailable` warning. Fine-grained personal access tokens always cause it. To include check runs, use a GitHub App installation token with Checks read or a classic token with the `repo` scope.
+
+## Rollback
+
+`rollback --pr <number>` opens a draft pull request that restores the pre-apply contents of the files a merged swap changed. It refuses unless the original pull request merged, and a rerun returns the same rollback pull request.
+
+See [Exit codes](exit-codes.md) and [Commands](commands.md) for every option.

package/docs/modeb.md

@@ -6,6 +6,7 @@ Mode B runs confirmation cases inside a container when a recommendation can affe
 {
   "version": "1",
   "image": "my-agent:latest",
+  "backend": "docker",
   "appSpec": {
     "mountPath": ".",
     "command": ["node", "/rightmodeler/app/driver.mjs", "{caseFile}"],
@@ -26,6 +27,39 @@ Mode B runs confirmation cases inside a container when a recommendation can affe
 - `appSpec.command` is a non-empty array of non-empty arguments. At least one argument must contain `{caseFile}`; the harness replaces every occurrence with the in-container case file path.
 - `appSpec.installCommand` is optional. When present, it is a non-empty array of non-empty arguments run before the workload.
 - `stepMap` maps at least one canonical scanner step ID to the runtime step header emitted by the application. Runtime headers must be unique.
+- `backend` is optional and is either `"docker"` (the default) or `"cloud"`. The cloud backend runs each case in a remote sandbox, so `image` must name an image that sandbox platform can pull, and the run fails before any case starts when the sandbox SDK or its credentials are absent.
 - `confirmMaxRunSets` is optional and must be a non-negative integer.
 
+## Runtime contract
+
+- The container receives `OPENAI_BASE_URL=http://127.0.0.1:8787/v1`, a placeholder `OPENAI_API_KEY`, and the `RM_RUN_ID`, `RM_CASE_ID`, and `RM_EXECUTION_ID` identifiers.
+- `/rightmodeler/scratch/driver/case.json` contains `{ "caseId", "input", "headers"? }`. The `headers` field is omitted when the case has no headers.
+- Every model request must carry `x-rm-step` exactly once. Its value is the runtime step header selected through `stepMap`.
+- Every model request must carry `x-rm-call`. Use one ID per logical call and reuse that ID when the SDK retries the call.
+- The request body must be a JSON object with a non-empty `model` string.
+- On a step the candidate does not replace, the request must name that step's current model by the id the provider catalog lists for it. The proxy holds prices only for the models the run's steps call, so a request for any other model is recorded as lost with `missing_pricing` and never forwarded.
+- `max_completion_tokens` or `max_tokens` is accepted when present. When both are absent, the proxy reserves against the catalog maximum output when known, or 4096 tokens otherwise, and does not add a limit to the forwarded body.
+- Streamed requests are sent with `stream_options.include_usage: true` and metered from the trailing usage chunk. A stream without a usage chunk is charged at its reservation.
+- The last non-empty stdout line must be `{ "runId", "caseId", "executionId", "finalOutput" }`.
+- A request is recorded as lost and never forwarded for `missing_correlation`, `duplicate_step_correlation`, `request_too_large` at 10 MiB, `malformed_json`, `invalid_request`, or `missing_pricing`.
+- A case with any lost request is a lost execution. Lost reason counts are reported on the Mode B result.
+- A request that the case lease cannot cover receives HTTP 402. The case is blocked on budget without an execution fact and is retried on rerun.
+- If the host cannot observe a container exit within the configured timeout plus ten seconds, it force-removes the container and records the case as lost under `container_lifecycle`.
+- The `docker` CLI must reach the Docker daemon. A missing daemon, a failed egress listener, or a failed container launch blocks affected cases with a named reason instead of recording executions, so a rerun retries them.
+- The in-container proxy runs on both backends. The workload always reaches it at `OPENAI_BASE_URL`, and it meters the case lease and records every attempt. Only the hop after it differs: the Docker backend forwards to a host listener over `host.docker.internal`, while the cloud backend forwards straight to the provider and the sandbox platform's egress firewall attaches the model credential in flight. The credential never enters the sandbox on either backend.
+- The workload is killed at the configured timeout by both the host and an in-container deadline.
+
+## Cloud backend
+
+- The cloud backend runs each confirmation case in a short-lived Vercel Sandbox microVM in your own Vercel project. It is separate from the Vercel AI Gateway: the `VERCEL_*` variables only authorize creating sandboxes, and the model credential is always the variable named by `--api-key-env`, whatever the provider.
+- Install: `npx rightmodeler` installs `@vercel/sandbox` as an optional dependency. Installing with `--omit=optional` leaves it out; the CLI still runs and the cloud backend reports `modeb_cloud_unavailable`.
+- Credentials: either `VERCEL_OIDC_TOKEN` (from `vercel link` then `vercel env pull`; it expires after 12 hours) or all of `VERCEL_TOKEN`, `VERCEL_TEAM_ID` and `VERCEL_PROJECT_ID`. Use the access token in CI.
+- Image: `image` must be a Vercel Container Registry reference or a managed image such as `vercel/sandbox/node:24`; a Docker Hub name does not work, and a custom image's entrypoint and command do not run. Commands run as the image's default user, which is not root (`ubuntu` on `vercel/sandbox/node:24`). An image the platform cannot start blocks its case with `launch-failed`, so a rerun retries it.
+- Provider: the base URL must be HTTPS, because the sandbox firewall matches the provider host by TLS server name before it attaches the credential. Its path is kept, so a base URL such as `https://openrouter.ai/api/v1` reaches the same endpoints as on the Docker backend.
+- Credential handling: the firewall adds the model credential to requests for the provider host only, and its policy lets every other host through, so this is credential brokering, not an egress allowlist.
+- Metering: the in-sandbox proxy asks the provider for uncompressed responses so it can read usage from every answer. There is no host listener to mark answers of its own, so every HTTP answer is attributed to the provider, including one the platform firewall produces itself; a connection that fails is attributed to the network path and its case is recorded as a lost execution.
+- Timing: the 60 second case deadline starts when the workload command starts and includes `appSpec.installCommand`; every case starts a fresh microVM, so bake dependencies into the image. The sandbox itself lives for the deadline plus 60 seconds, which also bounds how long an interrupted run can leave one running.
+- Cleanup: sandboxes are created non-persistent and deleted after each case.
+- Scoring: sandboxes never score. The host judges every case; sandbox output is only the candidate text.
+
 See [Commands](commands.md) for where `--modeb-config` is accepted and [Exit codes](exit-codes.md) for blocked or failed runs.

package/package.json

@@ -1,6 +1,8 @@
 {
   "name": "rightmodeler",
-  "version": "0.2.0",
+  "version": "0.3.0",
+  "description": "Find and prove safe model substitutions from the agent traces you already have.",
+  "homepage": "https://www.rightmodeler.com",
   "type": "module",
   "bin": {
     "rightmodeler": "./dist-bundle/cli.js"
@@ -13,6 +15,9 @@
   "engines": {
     "node": ">=24"
   },
+  "optionalDependencies": {
+    "@vercel/sandbox": "^3.3.0"
+  },
   "license": "MIT",
   "repository": {
     "type": "git",