browser_review_gate 0.2.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '09d5a330baa46a948f70191ba4537ba0d5c7097ed8c6d89e497a5f872eb6e671'
-  data.tar.gz: 48b41886a6ebbd39bb67c62af4e1f19b5c9ae52ae3dea622dfdfe8c0708336f6
+  metadata.gz: bf07fc0385a222895293b09a8000db74bf153a3b87731528b3e2d51c20cfd4e0
+  data.tar.gz: 5866378bb1acbfa17f03d44d98413c6f6222903858b7ec7dd6f71a9b6ba95788
 SHA512:
-  metadata.gz: 19564cbe10913982e3bf22af716e9e147ce1a38ffa5dfa47035e0083cbecb9bc63f7f78789a97fe1776a1f51065a3ab900d1cdcf854f527f40ae4327f6029a8b
-  data.tar.gz: b33fe20d59959593e0bb9a881f047065a89700f6c9f30e8a3ffe7aa6d69574180ef7b714261301e05c399b6bb43afaffc8dd9dc8761c529ad256ac2dfc19f84f
+  metadata.gz: de467b5092e7d8d3e9871e789b2c844712808f6af4916173d3ec903fb84407eac61218c5e6ee54e4c20e91e10ccdd3cb4d06010a355e2ddae869118b496c4791
+  data.tar.gz: 4ca8044e0cc223d3e269542ff1edb0ab635bdd710c549d017e48802c834eeef9b00fb2ff7ed71f9c9be37bd683185ea2d5450e95256574da7b12cf82723b7979

data/CHANGELOG.md

@@ -1,5 +1,53 @@
 # Changelog
 
+## 0.4.0
+
+- `provenance.trace` (off by default): the gem opens each case's Playwright trace and checks that a
+  response carried the tested commit in `revision_header`, that the case's page was requested, and
+  that requests fall within the run's time window. `warn` shows the result in the PR; `require`
+  counts a passing case only with a verified trace. Traces are read in memory with only zlib, with
+  caps on entries, sizes and compression ratio.
+- A HAR network log (`.har`) is accepted as a case's trace too, so the check works with any browser
+  tool that saves HAR, not only Playwright.
+- `doctor` warns when `provenance.trace` is on and no Playwright is configured in the project; `verify`
+  tells the agent a trace or HAR is needed per case.
+- `BrowserReviewGate::RevisionHeader`: Rack middleware for development that sends `X-App-Revision`
+  with the checked-out commit and follows new commits without a restart.
+- `doctor` checks that the app sends the revision header and that it matches the checkout.
+- The installer's next steps name the 0.3 settings.
+
+## 0.3.0
+
+Breaking: the report format and the settings change, and nothing from 0.2 is read. Move `url`,
+`start_command` and `sign_in` under `environments.local`, re-run `browser-review-gate install`, and
+publish a new run on PRs verified with a 0.2 report.
+
+- Reports (schema 2) name the environment, the browser tool and its capabilities, and the run's start
+  and finish times. Cases are `pass`, `fail`, `blocked` or `skipped`; the last two need a reason.
+  Validation is strict: unknown keys and malformed values make a report invalid.
+- Evidence per case: screenshots and traces are read from the report directory, confined to it,
+  checked by content, hashed and kept in the git directory; console errors and failed requests are
+  scrubbed of credential-like values and query strings; links only to hosts in `evidence.link_hosts`.
+- Named environments with health checks. `--environment` selects one. Nothing starts the app.
+- Only cases tested on a commit of the PR count; a case from elsewhere shows as stale.
+- Settings sections `verification`, `evidence`, `scenarios` and `gate`: fail a case on console or
+  network errors, require evidence (only what the tool can produce), keep earlier passing cases or
+  not, limit the scenario count, block on any failure or only on required scenarios.
+- Mandatory scenarios and a role/viewport matrix in the settings, added by the gem after the model
+  answers. A mandatory scenario scoped to paths makes a run required whatever the model decides.
+- Scenarios carry a priority, a category and why they are needed; ids are derived from behavior so
+  they stay stable across commits.
+- `verify`: one entry point that says what to run, checks the environment, requests a missing
+  assessment, and publishes or saves a finished run, with meaningful exit statuses and `--json`.
+- `doctor` (and `--json`): checks the setup end to end, with SKIP where something cannot be seen.
+- `stats` adds published and failing runs, failing scenarios, the median time to a verified run, and
+  token usage and cost as the provider reported them; `--json`.
+- PR comments show each scenario as passed, failed, blocked, skipped, missing or stale, and whether
+  human review can proceed. The report comment is a table with the evidence of every case.
+- Local commands read the settings from `origin/HEAD` when it has them, so a PR branch cannot point
+  the agent elsewhere or loosen the policy for its own run.
+- Removed: the 0.1.1 `hook --before` no-op, and the pre-scenario assessment fallback.
+
 ## 0.2.2
 
 - Requires json 2.7 or newer. Ruby 3.2 ships json 2.6, which ignores `script_safe`, so PR text in the

data/README.md

@@ -2,115 +2,212 @@
 
 Keeps human review of a pull request behind a browser run of its browser-visible changes.
 
-- CI reads each PR and decides whether it changes what a person sees or does in a browser. The decision goes into one PR comment, together with the scenarios a browser run must pass.
+- CI reads each PR and decides whether it changes what a person sees or does in a browser. The decision goes into one PR comment, together with the scenarios a browser run must pass and why each one is needed.
 - When a run is needed and none is published, reviewer requests are taken back and the PR says which command to run.
-- An AI coding agent (Claude Code, Cursor, Codex) runs the scenarios in a real browser and publishes what it observed: for each case, the page and what was seen there. A complete passing run adds the `browser-verified` label, and review requests stay.
-- CI compares the published cases with its own list. A run that leaves a scenario out does not open review.
+- An AI coding agent (Claude Code, Cursor, Codex) runs the scenarios in a real browser with its own browser tool and publishes what it observed, with evidence: screenshots, traces, console errors and failed requests.
+- The gem checks the report strictly, compares its cases with CI's list, and opens review only when every scenario has a case that counts.
+
+**What it is not.** The gem is an orchestration and enforcement layer. It does not drive a browser and ships no browser runtime: the agent does the run with Playwright, Chrome DevTools MCP or whatever browser tool it has. The gem decides what must be run, checks what the agent reports, and keeps review on hold until the report covers the change.
 
 ## Install
 
 ```sh
 bundle add browser_review_gate --group development
 bundle exec browser-review-gate install
+bundle exec browser-review-gate doctor
 ```
 
-In a Rails app the same thing is `bin/rails generate browser_review_gate:install`.
+In a Rails app the install step is also `bin/rails generate browser_review_gate:install`.
 
-One run sets up everything and can be repeated after a gem update:
+One run sets up everything and can be repeated after a gem update (it is idempotent and leaves files you wrote alone unless you pass `--force`):
 
 | File | Purpose |
 |---|---|
 | `.github/workflows/browser-review-gate.yml` | Assesses PRs and takes back premature reviewer requests |
-| `.github/browser-review-gate.yml` | Settings: how to start the app, what is never gated, which paths cannot touch the browser |
-| `.github/browser-review-gate/rules.md` | Your application's rules: what always gets checked, scenarios by area |
+| `.github/browser-review-gate.yml` | Settings (see below) |
+| `.github/browser-review-gate/rules.md` | Your application's rules in plain language, read by the model and the agent |
 | `docs/browser-review-gate.md` | The playbook every agent follows |
 | `.claude/skills/browser-pr-verification/SKILL.md`, hook in `.claude/settings.json` | Claude Code |
 | `.cursor/commands/browser-pr-verification.md`, hook in `.cursor/hooks.json` | Cursor |
-| Section in `AGENTS.md`, hook in `.codex/hooks.json` | Codex (the `AGENTS.md` section also serves other agents that read it) |
+| Section in `AGENTS.md`, hook in `.codex/hooks.json` | Codex and other agents that read `AGENTS.md` |
 
-The installer detects which agents the project uses (`.claude/` or `CLAUDE.md`, `.cursor/` or `.cursorrules`, `.codex/` or `AGENTS.md`) and sets up all of them. Choose explicitly with `--agents claude,cursor,codex`. Preview with `--dry-run`. Files you wrote yourself are never replaced without `--force`.
+The installer detects which agents the project uses and sets up all of them; choose with `--agents claude,cursor,codex`, preview with `--dry-run`.
 
 Then:
 
-1. Add one secret to the repository: `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY` (see "Who assesses a PR").
-2. Fill `start_command`, `url` and `sign_in` in `.github/browser-review-gate.yml`.
+1. Add one secret to the repository: `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY`.
+2. Describe your environments in `.github/browser-review-gate.yml`.
 3. Merge to the default branch. The workflow uses `pull_request_target`, so it runs from there.
+4. Optionally make the `browser-verification` status a required check so the gate also blocks merging.
 
-CI installs the gem from RubyGems. It can also take it from a git repository (`--gem-source "git:https://host/owner/browser_review_gate.git#v0.2.0"`) or from a copy kept in the project (`--gem-source path:vendor/browser_review_gate`).
+`browser-review-gate doctor` checks all of it and says what to fix.
 
-## Who assesses a PR
+## How a PR goes through
 
-The assessment is one model call without tools. Which model makes it follows from the secret you added:
+1. The author runs `/browser-pr-verification` (the agent command) on the branch. The agent runs `browser-review-gate verify --json`, which says what to run, in which environment, and whether the app answers.
+2. The agent runs the scenarios in its browser tool, writes a report with evidence next to it, and runs `browser-review-gate verify --report PATH`. Before the PR exists the run is saved for that commit; afterwards it is published.
+3. When the PR is opened, the agent's post-command hook publishes the saved run.
+4. CI assesses the PR, adds the project's mandatory scenarios, and matches the published cases against the list. When every scenario has a counting case, reviewer requests stay; reviewers taken back earlier are asked again automatically.
+
+Later commits keep the verification unless they add a scenario no counting case covers; then only the new scenarios need to run (`scenarios.reuse_passing`).
+
+## Settings
+
+`.github/browser-review-gate.yml`. Every key is optional; an unknown key or a malformed value is an error, never silently ignored. **Enforced** settings are checked by the gem; **agent** settings are instructions the agent receives through `verify` and `status`.
+
+```yaml
+environments:
+  local:
+    url: "http://localhost:3000"     # what the agent opens
+    healthcheck: "/up"               # path on url, or a full URL; what verify/doctor ask
+    start_command: "bin/dev"         # what the person runs; nothing starts the app automatically
+    sign_in: "Use the seed accounts from db/seeds.rb"
+  staging:
+    url: "https://staging.example.com"
+    healthcheck: "/up"
+
+verification:
+  environment: local                 # default; override with --environment
+  fail_on_console_errors: true       # enforced
+  fail_on_network_errors: false      # enforced
+
+evidence:
+  screenshots: on_failure            # agent: never | on_failure | always; enforced with gate.require_evidence
+  trace: never                       # agent: never | on_failure | always; enforced with gate.require_evidence
+  console_logs: true                 # agent; enforced with gate.require_evidence
+  network_logs: true                 # agent; enforced with gate.require_evidence
+  retention_days: 14                 # enforced: saved runs and kept evidence in .git are pruned
+  link_hosts: ["artifacts.example.com"]  # enforced: hosts evidence links may point to
+
+scenarios:
+  max_count: 12                      # enforced: most scenarios the model may ask for (1-30)
+  reuse_passing: true                # enforced: earlier passing cases on PR commits keep counting
+  mandatory:                         # enforced: added by the gem, not proposed by the model
+    - id: SIGN-IN
+      name: Sign in and land on the dashboard
+      why: Authentication changes must keep sign-in working.
+      paths: ["app/controllers/sessions_controller.rb", "app/views/sessions/**"]
+      matrix: true
+  matrix:                            # enforced: at most 24 mandatory cases after expansion
+    role: [admin, member]
+    viewport: [desktop, mobile]
+
+gate:
+  require_evidence: false            # enforced: passing cases need the evidence asked for above
+  block_on_failure: true             # enforced: any failing case keeps review on hold
+
+provenance:
+  trace: "off"                       # enforced: off | warn | require (see "Checking that a run happened")
+  revision_header: "X-App-Revision"
+  check_pages: true
+  clock_skew_seconds: 300
+```
 
-| Secret | What runs |
-|---|---|
-| `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | The Claude Code CLI, as before |
-| only `OPENAI_API_KEY` | The OpenAI API; set `model` in the settings file |
-| only `GEMINI_API_KEY` | The Gemini API; set `model` in the settings file |
+Other keys: `provider` and `model` (who assesses, see below), `workflow`, `label`, `waiver_label`, `waiver_by_author`, `status_context`, `command`, `skip_authors`, `skip_branch_prefixes`, `ignore_paths`.
 
-Set `provider` in `.github/browser-review-gate.yml` to `claude-cli`, `anthropic`, `openai` or `gemini` to choose explicitly. `anthropic` calls the Anthropic API directly with `ANTHROPIC_API_KEY`, without installing the CLI in CI.
+**Environments.** A local agent uses `local`. A cloud agent cannot reach `localhost`: give it an environment it can reach (staging, a preview deployment) and run `verify --environment staging`. `url` is what the agent opens; `healthcheck` may point elsewhere, for example a full health URL behind the same host. Put no credentials in this file: it is in the repository, and its values appear in `verify` output.
 
-## Fitting it to your application
+**Mandatory scenarios and matrices.** A mandatory entry with `paths` makes a run required whenever the change touches those paths, whatever the model decides. Without `paths` it is added to every run that is required. `matrix: true` repeats it for every combination of the matrix, as ids such as `SIGN-IN.admin.mobile`, so the gate can tell exactly which combination is missing.
 
-The gem ships with a general decision policy and a general playbook. There are two ways to make them yours.
+**Not configurable on purpose.** Run timeouts and retries are not settings: the browser run happens in the agent, and the gem cannot enforce them. Tell the agent in `rules.md` if you have expectations.
 
-**Add rules (recommended).** Write under the headings of `.github/browser-review-gate/rules.md`, in plain language:
+## Checking that a run happened
 
-- when a browser run is needed in this application;
-- what every run always checks, such as roles or window sizes;
-- which scenarios each area needs;
-- which accounts and data to use.
+Off by default. A report is the agent's account; `provenance.trace` lets the gem check it against a Playwright trace instead of taking it on trust.
 
-CI reads the rules when it decides whether a PR needs a run and whether a published run still covers it. The agent reads them when it picks scenarios. The built-in policy stays in force and gets updates with the gem; your rules are added on top and win where they disagree. The generated file contains only commented examples and changes nothing until you write in it.
+1. Make the app name the commit it serves, in development only:
 
-**Replace the built-in text.** `browser-review-gate eject policy` copies the decision policy to `.github/browser-review-gate/policy.md`; `browser-review-gate eject playbook` hands `docs/browser-review-gate.md` over to the project. From then on the gem uses your copy and stops updating it. The frame around the policy — the model's role, the rule that PR content is data, the answer format — is not ejectable.
+   ```ruby
+   # config/environments/development.rb
+   config.middleware.use BrowserReviewGate::RevisionHeader
+   ```
 
-CI reads all of these from the base branch, so a pull request cannot loosen its own rules.
+   It adds `X-App-Revision: <HEAD SHA>` to every response and follows new commits without a restart. Any other way of sending the header works too; name it in `provenance.revision_header`.
+2. Set `provenance.trace` to `warn` (results shown in the PR) or `require` (a passing case counts only with a verified trace). `doctor` checks that the app sends the header and that it matches the checkout.
+3. The agent records a Playwright trace (`.zip`) or a HAR network log (`.har`) per case and lists it as `evidence.trace`. HAR is a standard format that Playwright, Chrome DevTools ("Save all as HAR") and many other tools write, so the check does not need Playwright itself. A browser tool that can save neither cannot satisfy `require`; `doctor` and `verify` warn about it before a run.
 
-## How a PR goes through
+When the run is published, the gem opens each trace (a zip, or HAR JSON; read in memory with size, entry and compression-ratio caps, no extra dependency) and checks its network log: a response carried the revision header with the tested commit, the case's page was requested, and the request times fall within `started_at`..`finished_at` (± `clock_skew_seconds`). The PR shows "trace verified" or what was wrong.
 
-1. The author runs `/browser-pr-verification` on the branch before opening the PR. The agent picks the scenarios, runs them and saves the results for that commit inside the git directory.
-2. The PR is opened. In Claude Code, Cursor and Codex a hook publishes the saved results right after `gh pr create` (Codex asks you to trust the hook once in `/hooks`). Anywhere else, the command from the PR comment publishes them without a new run.
-3. CI writes its own list of scenarios and matches the published cases against it. When every scenario has a passing case, reviewer requests stay in place. Otherwise the label goes and the comment ticks what passed and lists what is left.
+What it catches: an agent that reports a run it did not do, ran against another commit or a stale server, or ran a different page. What it does not catch: someone who builds a trace by hand. It is a check on agents, not an attestation.
 
-If the PR is opened without a run, the comment shows the scenarios and the ready command, for example `/browser-pr-verification 42`. The agent runs exactly that list and reports each case under the scenario's id.
+## Reports
 
-A draft PR is not assessed until it is marked ready for review; run the command on a draft to have it assessed earlier.
+A report is JSON in the shape the playbook describes (`schema_version: 2`): the tested commit (full SHA), the environment, the browser tool and its capabilities, start and finish times, and one case per scenario with `result` (`pass`, `fail`, `blocked`, `skipped`), the page, what was observed, and evidence.
 
-Later commits keep the label unless they add a scenario the published run has no passing case for. Then only the new scenarios need to run.
+- Screenshots (`.png`, `.jpg`, `.webp`) and traces (`.zip` Playwright traces or `.har` network logs) are read from the report's directory. Paths that leave it, links, wrong file types (checked by content, not just name) and files over 20 MB are refused. Accepted files are hashed and kept in the git directory; the PR shows their names, sizes and SHA-256 digests, not the files.
+- Console errors and failed requests are trimmed, query strings are dropped, and values that look like credentials are masked.
+- Evidence links are accepted only on hosts listed in `evidence.link_hosts`; use them to point reviewers at screenshots you uploaded to your own artifact store.
+- A case counts when it passes, breaks no enforced policy, and was tested on a commit of the PR.
 
-A report counts only when it comes from someone who can push to the repository, and every submitted case must name the page it was exercised on (`url`) and what was observed (`details`).
+The PR comment shows every scenario as passed, failed, blocked, skipped, missing or stale, the environment and tool, the tested commit of each case, and the evidence. It says plainly that observations are the agent's account.
 
 ## Commands
 
 ```
-browser-review-gate install                     set the project up
+browser-review-gate install [--agents LIST] [--gem-source SOURCE] [--workflow NAME] [--force] [--dry-run]
 browser-review-gate eject policy|playbook       copy a built-in text into the project to rewrite it
-browser-review-gate status [--pr N]             what the PR needs and what to do next, as JSON
+browser-review-gate doctor [--json]             check the setup; PASS, WARN, FAIL or SKIP with a fix
+browser-review-gate verify [--pr N] [--environment NAME] [--report PATH] [--no-request] [--json]
+browser-review-gate status [--pr N] [--environment NAME]   the raw facts verify works from, as JSON
 browser-review-gate publish [--pr N] [--report PATH]
-browser-review-gate save --report PATH          keep results until the PR is opened
-browser-review-gate request-assessment N        assess a PR that has no decision yet
-browser-review-gate hook [claude|cursor|codex]  post-command hook
+browser-review-gate save --report PATH
+browser-review-gate request-assessment N
+browser-review-gate hook claude|cursor|codex    post-command hook
 browser-review-gate ci N                        what the workflow runs
-browser-review-gate stats [--days N]            what the gate did on recent PRs
+browser-review-gate stats [--days N] [--json]   what the gate did on recent PRs
 ```
 
+`verify` exit statuses: 0 nothing to do or published and verified; 2 a run is needed; 3 the environment does not answer; 4 an assessment was requested or is missing; 5 check out the PR head, or open the PR; 6 a saved run waits; 7 published but not verified; 1 error.
+
+`doctor` checks Ruby, the executable, the repository, the settings, `gh` and its sign-in, your push access, each environment's health check, agent files and hooks, a browser tool configured in the project, the workflow and its version, whether the status is a required check, and whether a model secret exists (by name). What it cannot see from your machine is SKIP, never PASS.
+
+`stats` counts PRs assessed, needing a run, verified, waived and still owing one; published runs and runs with a failure; scenarios that are failing; the median time from opening a PR to its verified run; and the token usage and cost of the latest assessment of each PR, only as the provider reported them. The Claude Code CLI reports cost; the APIs report tokens only, so no cost is shown for them.
+
 Local commands need the `gh` CLI, signed in.
 
-## Less friction
+## Who assesses a PR
+
+The assessment is one model call without tools. Which model makes it follows from the secret you added:
+
+| Secret | What runs |
+|---|---|
+| `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | The Claude Code CLI |
+| only `OPENAI_API_KEY` | The OpenAI API; set `model` |
+| only `GEMINI_API_KEY` | The Gemini API; set `model` |
+
+Set `provider` to `claude-cli`, `anthropic`, `openai` or `gemini` to choose explicitly. When the call fails, review stays on hold.
+
+## Fitting it to your application
+
+**Add rules.** Write under the headings of `.github/browser-review-gate/rules.md` in plain language: when a run is needed, what every run checks, which scenarios each area needs, which accounts and data to use. The model reads them when it assesses; the agent reads them when it picks scenarios. Use `scenarios.mandatory` and `scenarios.matrix` for anything that must not depend on a model following instructions.
+
+**Replace the built-in text.** `eject policy` copies the decision policy to `.github/browser-review-gate/policy.md`; `eject playbook` hands `docs/browser-review-gate.md` over to the project.
+
+## Security
+
+- The workflow runs on `pull_request_target` from the base branch with `contents: read`, `issues: write`, `pull-requests: write` and `statuses: write`. PR code is never checked out or executed; the diff is read through the API as data. Only the PR number reaches the shell.
+- Settings, rules and the policy are read from the base branch in CI, so a PR cannot change the rules of its own assessment. Local commands read the settings from `origin/HEAD` when it has them, so a PR branch cannot point the agent at another URL or loosen the policy for its own run (`settings_source` in `verify --json` says which was used).
+- The assessing model gets the PR as escaped data with no tools and answers in a fixed JSON schema; the gem validates the answer and does every write. An invalid answer or a failed call keeps review on hold.
+- The decision comment counts only from `github-actions[bot]`; a report counts only from someone with push access. Anyone else's comments are ignored as evidence.
+- Reports are validated strictly; evidence paths are confined to the report directory, symbolic links are refused, file types are checked by content, and logged URLs and messages are scrubbed.
+- Health checks go only to http(s) URLs from the settings, with a HEAD request, short timeouts and no redirects.
+
+Residual risks, stated plainly:
+
+- **A report is the agent's account, not an attestation.** Someone with push access can publish a passing report for a run that did not happen. With `provenance.trace: require` an agent cannot do so by accident or by guessing, but a hand-built trace still passes. Human review remains the control.
+- **Evidence files stay local.** Reviewers see digests, not screenshots, unless you upload them and link them through `evidence.link_hosts`.
+- **Scrubbing is pattern-based.** It catches common credential shapes, not everything. Agents are told not to record secrets; screenshots and traces are never inspected.
+- **A taken-back review request may still notify the reviewer**, because GitHub has no way to refuse a request.
+- **GitHub lists at most 250 commits of a PR**; on longer PRs, cases from older commits stop counting.
+- **The gate does not block merging by itself.** Make the status a required check for that.
 
-- **Reviewers come back on their own.** A request taken back is remembered; once the run is published, the same people are asked again.
-- **A waiver for what cannot be run.** Someone other than the author adds the `browser-verification-waived` label and review proceeds without a run.
-- **A status next to the checks.** `browser-verification` is pending while a run is owed and green otherwise. Make it a required check if you want it to block merging.
-- **Nothing is forced.** Opening a PR never requires a run. The PR comment says when one is needed and how to do it; it only becomes necessary when the author wants a reviewer.
-- **The agent does not start your app.** It uses the one that is running and asks you to start it otherwise.
+## Upgrading from 0.2
 
-## What it does not do
+0.3 changes the report format and the settings; nothing from 0.2 is read.
 
-- It cannot refuse a review request. GitHub has no such switch, so the request is taken back seconds after it is made, and the reviewer may still get a notification.
-- It does not watch the browser. The cases, their pages and what was observed are the agent's own account; the gem checks that every scenario CI asked for is reported as passed, not that it happened.
-- It does not block merging by itself. Make the `browser-verification` status a required check for that.
-- The model that assesses a PR sees the diff as data, has no tools, and answers in a fixed JSON shape. The gem validates the answer and does every write. PR code is never checked out or run in CI.
+- Move `url`, `start_command` and `sign_in` under `environments.local`.
+- Re-run `browser-review-gate install` to update the workflow, the playbook and the agent files.
+- Reports published by 0.2 do not count any more: PRs that were verified with one need a new run, or a waiver.
 
 ## Development
 

data/lib/browser_review_gate/assessment.rb

@@ -3,24 +3,41 @@ require "json"
 module BrowserReviewGate
   # The decision on whether a PR needs a browser run, and the scenarios that run must exercise. Kept in
   # one PR comment that the gate and the report publisher read back.
+  #
+  # Scenarios come from two places: the model proposes them from the diff, and the project settings add
+  # mandatory ones (`scenarios.mandatory`). Mandatory scenarios are added by this code after the model
+  # answered, so neither the model nor the PR can leave them out.
   class Assessment
     MARKER = /<!--\s*browser-verification-assessment:([A-Za-z0-9_-]+)\s*-->/
     # The gate workflow writes the comment with GITHUB_TOKEN; a marker from anyone else is ignored.
     AUTHOR = "github-actions[bot]"
     DECISIONS = %w[required not-required].freeze
+    PRIORITIES = %w[critical normal].freeze
     SHA_PATTERN = /\A\h{40}\z/
     MAX_REASON = 300
-    MAX_SCENARIOS = 12
     MAX_NAME = 200
+    MAX_WHY = 200
+    CATEGORY_PATTERN = /\A[a-z][a-z0-9-]{0,31}\z/
+    STATUS_MARKS = { "passed" => "x", "failed" => " ", "skipped" => " ", "blocked" => " ", "missing" => " ", "stale" => " " }.freeze
 
-    attr_reader :assessed_sha, :decision, :reason, :scenarios, :report_seen, :errors
+    attr_reader :assessed_sha, :decision, :reason, :scenarios, :report_seen, :usage, :errors
 
     # Builds an assessment from the model's JSON answer, which is untrusted. `report_seen` is the
     # fingerprint of the report the model was shown, so the same report is not judged twice.
-    def self.parse(text, assessed_sha:, report_seen: nil)
+    # `mandatory` are the scenarios the settings require for this change.
+    def self.parse(text, assessed_sha:, report_seen: nil, mandatory: [], max_count: 12, usage: nil)
       data = JSON.parse(text.to_s)
       data = {} unless data.is_a?(Hash)
-      assessment = new(data.slice("decision", "reason", "scenarios").merge("assessed_sha" => assessed_sha, "report_seen" => report_seen))
+      proposed = data["scenarios"]
+      if proposed.is_a?(Array) && proposed.size > max_count
+        return new({ "assessed_sha" => assessed_sha }).tap { |assessment| assessment.errors << "scenarios must have at most #{max_count} entries" }
+      end
+
+      if data["decision"] == "required" && proposed.is_a?(Array)
+        taken = proposed.map { |entry| entry.is_a?(Hash) ? entry["id"] : nil }
+        data["scenarios"] = proposed + mandatory.reject { |scenario| taken.include?(scenario["id"]) }
+      end
+      assessment = new(data.slice("decision", "reason", "scenarios").merge("assessed_sha" => assessed_sha, "report_seen" => report_seen, "usage" => usage))
       assessment.errors << "a required browser run needs at least one scenario" if assessment.required? && assessment.scenarios.empty?
       assessment
     rescue JSON::ParserError
@@ -53,9 +70,8 @@ module BrowserReviewGate
       @assessed_sha = data["assessed_sha"]
       @decision = data["decision"]
       @reason = data["reason"].is_a?(String) ? data["reason"].split.join(" ")[0, MAX_REASON] : nil
-      # Comments written before scenarios existed carry the model's own verdict on coverage instead.
-      @covered_by_report = data["covered_by_report"] == true
       @report_seen = data["report_seen"].is_a?(String) ? data["report_seen"] : nil
+      @usage = read_usage(data["usage"])
       @errors = []
       @scenarios = required? ? read_scenarios(data["scenarios"]) : []
       validate
@@ -65,62 +81,66 @@ module BrowserReviewGate
     def required? = decision == "required"
     def for?(sha) = assessed_sha == sha
 
-    # The scenarios no passing case of the report exercises yet.
-    def outstanding(report)
-      passed = report ? report.passed_ids : []
+    # The scenarios no counting case of the report exercises yet. With `commits`, only cases tested on
+    # those commits count; with `reuse_passing` false, only cases tested on the assessed commit.
+    def outstanding(report, commits: nil, reuse_passing: true)
+      allowed = reuse_passing ? commits : [ assessed_sha ]
+      passed = report ? report.passed_ids(commits: allowed) : []
       scenarios.reject { |scenario| passed.include?(scenario["id"]) }
     end
 
-    # A passing report covers the change when it has a passing case for every scenario.
-    def covered_by?(report)
+    # A report covers the change when it is complete, nothing failed (unless failures do not block),
+    # and every scenario has a counting case.
+    def covered_by?(report, commits: nil, reuse_passing: true)
       return false unless required? && report&.passed?
-      return @covered_by_report || report.tested_sha == assessed_sha if scenarios.empty?
 
-      outstanding(report).empty?
+      outstanding(report, commits: commits, reuse_passing: reuse_passing).empty?
     end
 
     # Whether this decision already accounts for the report, so the model need not be asked again.
     def settles?(report)
-      !required? || report.nil? || scenarios.empty? || outstanding(report).empty? || report_seen == report.fingerprint
+      !required? || report.nil? || outstanding(report).empty? || report_seen == report.fingerprint
     end
 
     def to_h
       { "assessed_sha" => assessed_sha, "decision" => decision, "reason" => reason, "scenarios" => scenarios,
-        "report_seen" => report_seen, "covered_by_report" => required? && @covered_by_report }
+        "report_seen" => report_seen, "usage" => usage }.compact
     end
 
-    # `verified` is true when a passing report covers the browser behavior of this commit.
-    # `command` is what the author runs in an AI coding agent, e.g. "/browser-pr-verification 42".
-    # `author` is who gets mentioned, and only when the comment asks them to do something: one login
-    # or several. `report` is the published run, used to tick the scenarios it already passed.
-    def markdown(verified:, command:, author: nil, report: nil)
+    # `verified` is true when a report covers the browser behavior of this commit. `command` is what the
+    # author runs in an AI coding agent. `author` is who gets mentioned, and only when the comment asks
+    # them to do something. `report` and `commits` give each scenario its status.
+    def markdown(verified:, command:, author: nil, report: nil, commits: nil)
       payload = Marker.encode(JSON.generate(to_h))
-      [ "## Browser verification", "", summary(verified, command, author, report), "",
+      [ "## Browser verification", "", summary(verified, command, author, report, commits), "",
         "Assessed commit `#{assessed_sha}`.", "", "<!-- browser-verification-assessment:#{payload} -->" ].join("\n")
     end
 
     private
 
-    def summary(verified, command, author, report)
+    def summary(verified, command, author, report, commits)
       why = Markdown.escape(reason)
       if !required?
         "Browser testing is not needed for this change: #{why} Human review can be requested."
       elsif verified
-        "Browser testing is required for this change, and the published browser report covers it: #{why} " \
-          "Human review can be requested."
+        [ "**Human review can proceed.** Browser testing is required for this change, and the published browser report covers it: #{why}",
+          "", *checklist(report, commits) ].join("\n").rstrip
       else
-        [ "#{Markdown.mention(author)}Browser testing is needed before requesting human review. #{why}", "",
-          *checklist(report),
+        [ "#{Markdown.mention(author)}**Human review is on hold.** Browser testing is needed before requesting human review. #{why}", "",
+          *checklist(report, commits),
           "To be able to request a reviewer, run this command in your AI coding agent from the PR branch:", "",
           "```", command, "```" ].join("\n")
       end
     end
 
-    def checklist(report)
-      return [] if scenarios.empty?
-
-      left = outstanding(report)
-      lines = scenarios.map { |scenario| "- [#{left.include?(scenario) ? " " : "x"}] `#{scenario["id"]}` #{Markdown.escape(scenario["name"])}" }
+    def checklist(report, commits)
+      lines = scenarios.map do |scenario|
+        status = report ? report.status_of(scenario["id"], commits: commits) : "missing"
+        flags = [ scenario["priority"] == "critical" ? "critical" : nil, scenario["category"] ].compact.join(", ")
+        why = scenario["why"] ? " — #{Markdown.escape(scenario["why"])}" : ""
+        "- [#{STATUS_MARKS.fetch(status)}] **#{status}** `#{scenario["id"]}` #{Markdown.escape(scenario["name"])}" \
+          "#{flags.empty? ? "" : " (#{Markdown.escape(flags)})"}#{why}"
+      end
       [ "Scenarios the run must pass:", "", *lines, "" ]
     end
 
@@ -133,15 +153,36 @@ module BrowserReviewGate
         return []
       end
 
-      scenarios = value.map { |entry| { "id" => entry["id"], "name" => entry["name"].split.join(" ")[0, MAX_NAME] } }
+      scenarios = value.map { |entry| read_scenario(entry) }
       ids = scenarios.map { |scenario| scenario["id"] }
-      errors << "scenarios must have at most #{MAX_SCENARIOS} entries" if scenarios.size > MAX_SCENARIOS
+      errors << "scenarios must have at most #{Config::MAX_SCENARIOS + Config::MAX_MATRIX_CASES} entries" if scenarios.size > Config::MAX_SCENARIOS + Config::MAX_MATRIX_CASES
       errors << "scenario ids must use letters, numbers, dots, underscores, or hyphens" unless ids.all? { |id| id.is_a?(String) && Report::CASE_ID_PATTERN.match?(id) }
       errors << "scenario ids must be unique" unless ids.uniq.size == ids.size
       errors << "scenario names must not be empty" if scenarios.any? { |scenario| scenario["name"].empty? }
+      errors << "scenario priority must be critical or normal" unless scenarios.all? { |scenario| PRIORITIES.include?(scenario["priority"]) }
+      errors << "scenario category must be a short lowercase word" unless scenarios.all? { |scenario| scenario["category"].nil? || CATEGORY_PATTERN.match?(scenario["category"]) }
       scenarios
     end
 
+    def read_scenario(entry)
+      scenario = { "id" => entry["id"], "name" => entry["name"].split.join(" ")[0, MAX_NAME],
+                   "priority" => entry["priority"] || "normal",
+                   "category" => entry["category"].is_a?(String) && !entry["category"].empty? ? entry["category"].downcase : nil,
+                   "why" => entry["why"].is_a?(String) && !entry["why"].strip.empty? ? entry["why"].split.join(" ")[0, MAX_WHY] : nil }
+      scenario["matrix"] = entry["matrix"] if entry["matrix"].is_a?(Hash)
+      scenario.compact
+    end
+
+    # Token counts and cost as the provider reported them; nothing is estimated here.
+    def read_usage(value)
+      return unless value.is_a?(Hash)
+
+      usage = value.slice("provider", "model", "input_tokens", "output_tokens", "cost_usd").select do |key, entry|
+        %w[provider model].include?(key) ? entry.is_a?(String) && entry.length <= 80 : entry.is_a?(Numeric) && entry >= 0
+      end
+      usage.empty? ? nil : usage
+    end
+
     def validate
       errors << "assessed_sha must be a full commit SHA" unless assessed_sha.is_a?(String) && SHA_PATTERN.match?(assessed_sha)
       errors << "decision must be required or not-required" unless DECISIONS.include?(decision)