browser_review_gate 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +38 -0
- data/LICENSE.txt +1 -1
- data/README.md +134 -62
- data/lib/browser_review_gate/assessment.rb +74 -33
- data/lib/browser_review_gate/assessor.rb +38 -10
- data/lib/browser_review_gate/cli.rb +44 -19
- data/lib/browser_review_gate/config.rb +223 -10
- data/lib/browser_review_gate/doctor.rb +216 -0
- data/lib/browser_review_gate/evidence.rb +170 -0
- data/lib/browser_review_gate/gate.rb +8 -5
- data/lib/browser_review_gate/github.rb +7 -0
- data/lib/browser_review_gate/model_api.rb +16 -0
- data/lib/browser_review_gate/model_client.rb +10 -0
- data/lib/browser_review_gate/prompts.rb +10 -5
- data/lib/browser_review_gate/publisher.rb +32 -9
- data/lib/browser_review_gate/report.rb +222 -47
- data/lib/browser_review_gate/saved_report.rb +15 -1
- data/lib/browser_review_gate/stats.rb +90 -6
- data/lib/browser_review_gate/status.rb +52 -21
- data/lib/browser_review_gate/templates/agents_block.md.erb +1 -1
- data/lib/browser_review_gate/templates/config.yml.erb +63 -5
- data/lib/browser_review_gate/templates/playbook.md.erb +63 -25
- data/lib/browser_review_gate/templates/rules.md.erb +4 -1
- data/lib/browser_review_gate/verify.rb +86 -0
- data/lib/browser_review_gate/version.rb +1 -1
- data/lib/browser_review_gate.rb +3 -0
- metadata +31 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a2e4e7dfaa70718cc4eaf691aa3f60023ae10968b34127b0b363ff15c42971c6
|
|
4
|
+
data.tar.gz: e29d1f6ba94e5d07ebba865d6faa838efe9222eba7f5ece6e286fc3d88895bbf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d9859bc00fe366be2568ee088c18cdd19431e80cf0f468fd8370ec9465ce7a2111cd761ea96b1c44693c5188c1c1c2073daec2de928b05265942c9babdd99cd1
|
|
7
|
+
data.tar.gz: c5a80a4802e8d9053d3f0c2fcf2f197152e6b8b9ac9d8e172c24c3fcdc57ce153dd15802afaaab8fe31c8908d26a25161caa09a922252f3313b0f2d7333e45a4
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,43 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.3.0
|
|
4
|
+
|
|
5
|
+
Breaking: the report format and the settings change, and nothing from 0.2 is read. Move `url`,
|
|
6
|
+
`start_command` and `sign_in` under `environments.local`, re-run `browser-review-gate install`, and
|
|
7
|
+
publish a new run on PRs verified with a 0.2 report.
|
|
8
|
+
|
|
9
|
+
- Reports (schema 2) name the environment, the browser tool and its capabilities, and the run's start
|
|
10
|
+
and finish times. Cases are `pass`, `fail`, `blocked` or `skipped`; the last two need a reason.
|
|
11
|
+
Validation is strict: unknown keys and malformed values make a report invalid.
|
|
12
|
+
- Evidence per case: screenshots and traces are read from the report directory, confined to it,
|
|
13
|
+
checked by content, hashed and kept in the git directory; console errors and failed requests are
|
|
14
|
+
scrubbed of credential-like values and query strings; links only to hosts in `evidence.link_hosts`.
|
|
15
|
+
- Named environments with health checks. `--environment` selects one. Nothing starts the app.
|
|
16
|
+
- Only cases tested on a commit of the PR count; a case from elsewhere shows as stale.
|
|
17
|
+
- Settings sections `verification`, `evidence`, `scenarios` and `gate`: fail a case on console or
|
|
18
|
+
network errors, require evidence (only what the tool can produce), keep earlier passing cases or
|
|
19
|
+
not, limit the scenario count, block on any failure or only on required scenarios.
|
|
20
|
+
- Mandatory scenarios and a role/viewport matrix in the settings, added by the gem after the model
|
|
21
|
+
answers. A mandatory scenario scoped to paths makes a run required whatever the model decides.
|
|
22
|
+
- Scenarios carry a priority, a category and why they are needed; ids are derived from behavior so
|
|
23
|
+
they stay stable across commits.
|
|
24
|
+
- `verify`: one entry point that says what to run, checks the environment, requests a missing
|
|
25
|
+
assessment, and publishes or saves a finished run, with meaningful exit statuses and `--json`.
|
|
26
|
+
- `doctor` (and `--json`): checks the setup end to end, with SKIP where something cannot be seen.
|
|
27
|
+
- `stats` adds published and failing runs, failing scenarios, the median time to a verified run, and
|
|
28
|
+
token usage and cost as the provider reported them; `--json`.
|
|
29
|
+
- PR comments show each scenario as passed, failed, blocked, skipped, missing or stale, and whether
|
|
30
|
+
human review can proceed. The report comment is a table with the evidence of every case.
|
|
31
|
+
- Local commands read the settings from `origin/HEAD` when it has them, so a PR branch cannot point
|
|
32
|
+
the agent elsewhere or loosen the policy for its own run.
|
|
33
|
+
- Removed: the 0.1.1 `hook --before` no-op, and the pre-scenario assessment fallback.
|
|
34
|
+
|
|
35
|
+
## 0.2.2
|
|
36
|
+
|
|
37
|
+
- Requires json 2.7 or newer. Ruby 3.2 ships json 2.6, which ignores `script_safe`, so PR text in the
|
|
38
|
+
assessment prompt was not escaped there.
|
|
39
|
+
- Links to the source code, changelog and issues.
|
|
40
|
+
|
|
3
41
|
## 0.2.1
|
|
4
42
|
|
|
5
43
|
- A reviewer who approved or asked for changes on the latest commit while their request was paused
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
|
@@ -2,115 +2,187 @@
|
|
|
2
2
|
|
|
3
3
|
Keeps human review of a pull request behind a browser run of its browser-visible changes.
|
|
4
4
|
|
|
5
|
-
- CI reads each PR and decides whether it changes what a person sees or does in a browser. The decision goes into one PR comment, together with the scenarios a browser run must pass.
|
|
5
|
+
- CI reads each PR and decides whether it changes what a person sees or does in a browser. The decision goes into one PR comment, together with the scenarios a browser run must pass and why each one is needed.
|
|
6
6
|
- When a run is needed and none is published, reviewer requests are taken back and the PR says which command to run.
|
|
7
|
-
- An AI coding agent (Claude Code, Cursor, Codex) runs the scenarios in a real browser and publishes what it observed
|
|
8
|
-
-
|
|
7
|
+
- An AI coding agent (Claude Code, Cursor, Codex) runs the scenarios in a real browser with its own browser tool and publishes what it observed, with evidence: screenshots, traces, console errors and failed requests.
|
|
8
|
+
- The gem checks the report strictly, compares its cases with CI's list, and opens review only when every scenario has a case that counts.
|
|
9
|
+
|
|
10
|
+
**What it is not.** The gem is an orchestration and enforcement layer. It does not drive a browser and ships no browser runtime: the agent does the run with Playwright, Chrome DevTools MCP or whatever browser tool it has. The gem decides what must be run, checks what the agent reports, and keeps review on hold until the report covers the change.
|
|
9
11
|
|
|
10
12
|
## Install
|
|
11
13
|
|
|
12
14
|
```sh
|
|
13
15
|
bundle add browser_review_gate --group development
|
|
14
16
|
bundle exec browser-review-gate install
|
|
17
|
+
bundle exec browser-review-gate doctor
|
|
15
18
|
```
|
|
16
19
|
|
|
17
|
-
In a Rails app the
|
|
20
|
+
In a Rails app the install step is also `bin/rails generate browser_review_gate:install`.
|
|
18
21
|
|
|
19
|
-
One run sets up everything and can be repeated after a gem update:
|
|
22
|
+
One run sets up everything and can be repeated after a gem update (it is idempotent and leaves files you wrote alone unless you pass `--force`):
|
|
20
23
|
|
|
21
24
|
| File | Purpose |
|
|
22
25
|
|---|---|
|
|
23
26
|
| `.github/workflows/browser-review-gate.yml` | Assesses PRs and takes back premature reviewer requests |
|
|
24
|
-
| `.github/browser-review-gate.yml` | Settings
|
|
25
|
-
| `.github/browser-review-gate/rules.md` | Your application's rules
|
|
27
|
+
| `.github/browser-review-gate.yml` | Settings (see below) |
|
|
28
|
+
| `.github/browser-review-gate/rules.md` | Your application's rules in plain language, read by the model and the agent |
|
|
26
29
|
| `docs/browser-review-gate.md` | The playbook every agent follows |
|
|
27
30
|
| `.claude/skills/browser-pr-verification/SKILL.md`, hook in `.claude/settings.json` | Claude Code |
|
|
28
31
|
| `.cursor/commands/browser-pr-verification.md`, hook in `.cursor/hooks.json` | Cursor |
|
|
29
|
-
| Section in `AGENTS.md`, hook in `.codex/hooks.json` | Codex
|
|
32
|
+
| Section in `AGENTS.md`, hook in `.codex/hooks.json` | Codex and other agents that read `AGENTS.md` |
|
|
30
33
|
|
|
31
|
-
The installer detects which agents the project uses
|
|
34
|
+
The installer detects which agents the project uses and sets up all of them; choose with `--agents claude,cursor,codex`, preview with `--dry-run`.
|
|
32
35
|
|
|
33
36
|
Then:
|
|
34
37
|
|
|
35
|
-
1. Add one secret to the repository: `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY
|
|
36
|
-
2.
|
|
38
|
+
1. Add one secret to the repository: `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY`.
|
|
39
|
+
2. Describe your environments in `.github/browser-review-gate.yml`.
|
|
37
40
|
3. Merge to the default branch. The workflow uses `pull_request_target`, so it runs from there.
|
|
41
|
+
4. Optionally make the `browser-verification` status a required check so the gate also blocks merging.
|
|
38
42
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
## Who assesses a PR
|
|
42
|
-
|
|
43
|
-
The assessment is one model call without tools. Which model makes it follows from the secret you added:
|
|
44
|
-
|
|
45
|
-
| Secret | What runs |
|
|
46
|
-
|---|---|
|
|
47
|
-
| `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | The Claude Code CLI, as before |
|
|
48
|
-
| only `OPENAI_API_KEY` | The OpenAI API; set `model` in the settings file |
|
|
49
|
-
| only `GEMINI_API_KEY` | The Gemini API; set `model` in the settings file |
|
|
50
|
-
|
|
51
|
-
Set `provider` in `.github/browser-review-gate.yml` to `claude-cli`, `anthropic`, `openai` or `gemini` to choose explicitly. `anthropic` calls the Anthropic API directly with `ANTHROPIC_API_KEY`, without installing the CLI in CI.
|
|
43
|
+
`browser-review-gate doctor` checks all of it and says what to fix.
|
|
52
44
|
|
|
53
|
-
##
|
|
54
|
-
|
|
55
|
-
The gem ships with a general decision policy and a general playbook. There are two ways to make them yours.
|
|
56
|
-
|
|
57
|
-
**Add rules (recommended).** Write under the headings of `.github/browser-review-gate/rules.md`, in plain language:
|
|
58
|
-
|
|
59
|
-
- when a browser run is needed in this application;
|
|
60
|
-
- what every run always checks, such as roles or window sizes;
|
|
61
|
-
- which scenarios each area needs;
|
|
62
|
-
- which accounts and data to use.
|
|
45
|
+
## How a PR goes through
|
|
63
46
|
|
|
64
|
-
|
|
47
|
+
1. The author runs `/browser-pr-verification` (the agent command) on the branch. The agent runs `browser-review-gate verify --json`, which says what to run, in which environment, and whether the app answers.
|
|
48
|
+
2. The agent runs the scenarios in its browser tool, writes a report with evidence next to it, and runs `browser-review-gate verify --report PATH`. Before the PR exists the run is saved for that commit; afterwards it is published.
|
|
49
|
+
3. When the PR is opened, the agent's post-command hook publishes the saved run.
|
|
50
|
+
4. CI assesses the PR, adds the project's mandatory scenarios, and matches the published cases against the list. When every scenario has a counting case, reviewer requests stay; reviewers taken back earlier are asked again automatically.
|
|
51
|
+
|
|
52
|
+
Later commits keep the verification unless they add a scenario no counting case covers; then only the new scenarios need to run (`scenarios.reuse_passing`).
|
|
53
|
+
|
|
54
|
+
## Settings
|
|
55
|
+
|
|
56
|
+
`.github/browser-review-gate.yml`. Every key is optional; an unknown key or a malformed value is an error, never silently ignored. **Enforced** settings are checked by the gem; **agent** settings are instructions the agent receives through `verify` and `status`.
|
|
57
|
+
|
|
58
|
+
```yaml
|
|
59
|
+
environments:
|
|
60
|
+
local:
|
|
61
|
+
url: "http://localhost:3000" # what the agent opens
|
|
62
|
+
healthcheck: "/up" # path on url, or a full URL; what verify/doctor ask
|
|
63
|
+
start_command: "bin/dev" # what the person runs; nothing starts the app automatically
|
|
64
|
+
sign_in: "Use the seed accounts from db/seeds.rb"
|
|
65
|
+
staging:
|
|
66
|
+
url: "https://staging.example.com"
|
|
67
|
+
healthcheck: "/up"
|
|
68
|
+
|
|
69
|
+
verification:
|
|
70
|
+
environment: local # default; override with --environment
|
|
71
|
+
fail_on_console_errors: true # enforced
|
|
72
|
+
fail_on_network_errors: false # enforced
|
|
73
|
+
|
|
74
|
+
evidence:
|
|
75
|
+
screenshots: on_failure # agent: never | on_failure | always; enforced with gate.require_evidence
|
|
76
|
+
trace: never # agent: never | on_failure | always; enforced with gate.require_evidence
|
|
77
|
+
console_logs: true # agent; enforced with gate.require_evidence
|
|
78
|
+
network_logs: true # agent; enforced with gate.require_evidence
|
|
79
|
+
retention_days: 14 # enforced: saved runs and kept evidence in .git are pruned
|
|
80
|
+
link_hosts: ["artifacts.example.com"] # enforced: hosts evidence links may point to
|
|
81
|
+
|
|
82
|
+
scenarios:
|
|
83
|
+
max_count: 12 # enforced: most scenarios the model may ask for (1-30)
|
|
84
|
+
reuse_passing: true # enforced: earlier passing cases on PR commits keep counting
|
|
85
|
+
mandatory: # enforced: added by the gem, not proposed by the model
|
|
86
|
+
- id: SIGN-IN
|
|
87
|
+
name: Sign in and land on the dashboard
|
|
88
|
+
why: Authentication changes must keep sign-in working.
|
|
89
|
+
paths: ["app/controllers/sessions_controller.rb", "app/views/sessions/**"]
|
|
90
|
+
matrix: true
|
|
91
|
+
matrix: # enforced: at most 24 mandatory cases after expansion
|
|
92
|
+
role: [admin, member]
|
|
93
|
+
viewport: [desktop, mobile]
|
|
94
|
+
|
|
95
|
+
gate:
|
|
96
|
+
require_evidence: false # enforced: passing cases need the evidence asked for above
|
|
97
|
+
block_on_failure: true # enforced: any failing case keeps review on hold
|
|
98
|
+
```
|
|
65
99
|
|
|
66
|
-
|
|
100
|
+
Other keys: `provider` and `model` (who assesses, see below), `workflow`, `label`, `waiver_label`, `waiver_by_author`, `status_context`, `command`, `skip_authors`, `skip_branch_prefixes`, `ignore_paths`.
|
|
67
101
|
|
|
68
|
-
|
|
102
|
+
**Environments.** A local agent uses `local`. A cloud agent cannot reach `localhost`: give it an environment it can reach (staging, a preview deployment) and run `verify --environment staging`. `url` is what the agent opens; `healthcheck` may point elsewhere, for example a full health URL behind the same host. Put no credentials in this file: it is in the repository, and its values appear in `verify` output.
|
|
69
103
|
|
|
70
|
-
|
|
104
|
+
**Mandatory scenarios and matrices.** A mandatory entry with `paths` makes a run required whenever the change touches those paths, whatever the model decides. Without `paths` it is added to every run that is required. `matrix: true` repeats it for every combination of the matrix, as ids such as `SIGN-IN.admin.mobile`, so the gate can tell exactly which combination is missing.
|
|
71
105
|
|
|
72
|
-
|
|
73
|
-
2. The PR is opened. In Claude Code, Cursor and Codex a hook publishes the saved results right after `gh pr create` (Codex asks you to trust the hook once in `/hooks`). Anywhere else, the command from the PR comment publishes them without a new run.
|
|
74
|
-
3. CI writes its own list of scenarios and matches the published cases against it. When every scenario has a passing case, reviewer requests stay in place. Otherwise the label goes and the comment ticks what passed and lists what is left.
|
|
106
|
+
**Not configurable on purpose.** Run timeouts and retries are not settings: the browser run happens in the agent, and the gem cannot enforce them. Tell the agent in `rules.md` if you have expectations.
|
|
75
107
|
|
|
76
|
-
|
|
108
|
+
## Reports
|
|
77
109
|
|
|
78
|
-
A
|
|
110
|
+
A report is JSON in the shape the playbook describes (`schema_version: 2`): the tested commit (full SHA), the environment, the browser tool and its capabilities, start and finish times, and one case per scenario with `result` (`pass`, `fail`, `blocked`, `skipped`), the page, what was observed, and evidence.
|
|
79
111
|
|
|
80
|
-
|
|
112
|
+
- Screenshots (`.png`, `.jpg`, `.webp`) and traces (`.zip`) are read from the report's directory. Paths that leave it, links, wrong file types (checked by content, not just name) and files over 20 MB are refused. Accepted files are hashed and kept in the git directory; the PR shows their names, sizes and SHA-256 digests, not the files.
|
|
113
|
+
- Console errors and failed requests are trimmed, query strings are dropped, and values that look like credentials are masked.
|
|
114
|
+
- Evidence links are accepted only on hosts listed in `evidence.link_hosts`; use them to point reviewers at screenshots you uploaded to your own artifact store.
|
|
115
|
+
- A case counts when it passes, breaks no enforced policy, and was tested on a commit of the PR.
|
|
81
116
|
|
|
82
|
-
|
|
117
|
+
The PR comment shows every scenario as passed, failed, blocked, skipped, missing or stale, the environment and tool, the tested commit of each case, and the evidence. It says plainly that observations are the agent's account.
|
|
83
118
|
|
|
84
119
|
## Commands
|
|
85
120
|
|
|
86
121
|
```
|
|
87
|
-
browser-review-gate install
|
|
122
|
+
browser-review-gate install [--agents LIST] [--gem-source SOURCE] [--workflow NAME] [--force] [--dry-run]
|
|
88
123
|
browser-review-gate eject policy|playbook copy a built-in text into the project to rewrite it
|
|
89
|
-
browser-review-gate
|
|
124
|
+
browser-review-gate doctor [--json] check the setup; PASS, WARN, FAIL or SKIP with a fix
|
|
125
|
+
browser-review-gate verify [--pr N] [--environment NAME] [--report PATH] [--no-request] [--json]
|
|
126
|
+
browser-review-gate status [--pr N] [--environment NAME] the raw facts verify works from, as JSON
|
|
90
127
|
browser-review-gate publish [--pr N] [--report PATH]
|
|
91
|
-
browser-review-gate save --report PATH
|
|
92
|
-
browser-review-gate request-assessment N
|
|
93
|
-
browser-review-gate hook
|
|
128
|
+
browser-review-gate save --report PATH
|
|
129
|
+
browser-review-gate request-assessment N
|
|
130
|
+
browser-review-gate hook claude|cursor|codex post-command hook
|
|
94
131
|
browser-review-gate ci N what the workflow runs
|
|
95
|
-
browser-review-gate stats [--days N]
|
|
132
|
+
browser-review-gate stats [--days N] [--json] what the gate did on recent PRs
|
|
96
133
|
```
|
|
97
134
|
|
|
135
|
+
`verify` exit statuses: 0 nothing to do or published and verified; 2 a run is needed; 3 the environment does not answer; 4 an assessment was requested or is missing; 5 check out the PR head, or open the PR; 6 a saved run waits; 7 published but not verified; 1 error.
|
|
136
|
+
|
|
137
|
+
`doctor` checks Ruby, the executable, the repository, the settings, `gh` and its sign-in, your push access, each environment's health check, agent files and hooks, a browser tool configured in the project, the workflow and its version, whether the status is a required check, and whether a model secret exists (by name). What it cannot see from your machine is SKIP, never PASS.
|
|
138
|
+
|
|
139
|
+
`stats` counts PRs assessed, needing a run, verified, waived and still owing one; published runs and runs with a failure; scenarios that are failing; the median time from opening a PR to its verified run; and the token usage and cost of the latest assessment of each PR, only as the provider reported them. The Claude Code CLI reports cost; the APIs report tokens only, so no cost is shown for them.
|
|
140
|
+
|
|
98
141
|
Local commands need the `gh` CLI, signed in.
|
|
99
142
|
|
|
100
|
-
##
|
|
143
|
+
## Who assesses a PR
|
|
144
|
+
|
|
145
|
+
The assessment is one model call without tools. Which model makes it follows from the secret you added:
|
|
146
|
+
|
|
147
|
+
| Secret | What runs |
|
|
148
|
+
|---|---|
|
|
149
|
+
| `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` | The Claude Code CLI |
|
|
150
|
+
| only `OPENAI_API_KEY` | The OpenAI API; set `model` |
|
|
151
|
+
| only `GEMINI_API_KEY` | The Gemini API; set `model` |
|
|
152
|
+
|
|
153
|
+
Set `provider` to `claude-cli`, `anthropic`, `openai` or `gemini` to choose explicitly. When the call fails, review stays on hold.
|
|
154
|
+
|
|
155
|
+
## Fitting it to your application
|
|
156
|
+
|
|
157
|
+
**Add rules.** Write under the headings of `.github/browser-review-gate/rules.md` in plain language: when a run is needed, what every run checks, which scenarios each area needs, which accounts and data to use. The model reads them when it assesses; the agent reads them when it picks scenarios. Use `scenarios.mandatory` and `scenarios.matrix` for anything that must not depend on a model following instructions.
|
|
158
|
+
|
|
159
|
+
**Replace the built-in text.** `eject policy` copies the decision policy to `.github/browser-review-gate/policy.md`; `eject playbook` hands `docs/browser-review-gate.md` over to the project.
|
|
160
|
+
|
|
161
|
+
## Security
|
|
162
|
+
|
|
163
|
+
- The workflow runs on `pull_request_target` from the base branch with `contents: read`, `issues: write`, `pull-requests: write` and `statuses: write`. PR code is never checked out or executed; the diff is read through the API as data. Only the PR number reaches the shell.
|
|
164
|
+
- Settings, rules and the policy are read from the base branch in CI, so a PR cannot change the rules of its own assessment. Local commands read the settings from `origin/HEAD` when it has them, so a PR branch cannot point the agent at another URL or loosen the policy for its own run (`settings_source` in `verify --json` says which was used).
|
|
165
|
+
- The assessing model gets the PR as escaped data with no tools and answers in a fixed JSON schema; the gem validates the answer and does every write. An invalid answer or a failed call keeps review on hold.
|
|
166
|
+
- The decision comment counts only from `github-actions[bot]`; a report counts only from someone with push access. Anyone else's comments are ignored as evidence.
|
|
167
|
+
- Reports are validated strictly; evidence paths are confined to the report directory, symbolic links are refused, file types are checked by content, and logged URLs and messages are scrubbed.
|
|
168
|
+
- Health checks go only to http(s) URLs from the settings, with a HEAD request, short timeouts and no redirects.
|
|
169
|
+
|
|
170
|
+
Residual risks, stated plainly:
|
|
171
|
+
|
|
172
|
+
- **A report is the agent's account, not an attestation.** Someone with push access, or an agent that misreports, can publish a passing report for a run that did not happen. The gem raises the cost (exact scenario ids, pages, observations, evidence digests, commit checks) but cannot prove a browser did anything. Human review remains the control.
|
|
173
|
+
- **Evidence files stay local.** Reviewers see digests, not screenshots, unless you upload them and link them through `evidence.link_hosts`.
|
|
174
|
+
- **Scrubbing is pattern-based.** It catches common credential shapes, not everything. Agents are told not to record secrets; screenshots and traces are never inspected.
|
|
175
|
+
- **A taken-back review request may still notify the reviewer**, because GitHub has no way to refuse a request.
|
|
176
|
+
- **GitHub lists at most 250 commits of a PR**; on longer PRs, cases from older commits stop counting.
|
|
177
|
+
- **The gate does not block merging by itself.** Make the status a required check for that.
|
|
101
178
|
|
|
102
|
-
|
|
103
|
-
- **A waiver for what cannot be run.** Someone other than the author adds the `browser-verification-waived` label and review proceeds without a run.
|
|
104
|
-
- **A status next to the checks.** `browser-verification` is pending while a run is owed and green otherwise. Make it a required check if you want it to block merging.
|
|
105
|
-
- **Nothing is forced.** Opening a PR never requires a run. The PR comment says when one is needed and how to do it; it only becomes necessary when the author wants a reviewer.
|
|
106
|
-
- **The agent does not start your app.** It uses the one that is running and asks you to start it otherwise.
|
|
179
|
+
## Upgrading from 0.2
|
|
107
180
|
|
|
108
|
-
|
|
181
|
+
0.3 changes the report format and the settings; nothing from 0.2 is read.
|
|
109
182
|
|
|
110
|
-
-
|
|
111
|
-
-
|
|
112
|
-
-
|
|
113
|
-
- The model that assesses a PR sees the diff as data, has no tools, and answers in a fixed JSON shape. The gem validates the answer and does every write. PR code is never checked out or run in CI.
|
|
183
|
+
- Move `url`, `start_command` and `sign_in` under `environments.local`.
|
|
184
|
+
- Re-run `browser-review-gate install` to update the workflow, the playbook and the agent files.
|
|
185
|
+
- Reports published by 0.2 do not count any more: PRs that were verified with one need a new run, or a waiver.
|
|
114
186
|
|
|
115
187
|
## Development
|
|
116
188
|
|
|
@@ -3,24 +3,41 @@ require "json"
|
|
|
3
3
|
module BrowserReviewGate
|
|
4
4
|
# The decision on whether a PR needs a browser run, and the scenarios that run must exercise. Kept in
|
|
5
5
|
# one PR comment that the gate and the report publisher read back.
|
|
6
|
+
#
|
|
7
|
+
# Scenarios come from two places: the model proposes them from the diff, and the project settings add
|
|
8
|
+
# mandatory ones (`scenarios.mandatory`). Mandatory scenarios are added by this code after the model
|
|
9
|
+
# answered, so neither the model nor the PR can leave them out.
|
|
6
10
|
class Assessment
|
|
7
11
|
MARKER = /<!--\s*browser-verification-assessment:([A-Za-z0-9_-]+)\s*-->/
|
|
8
12
|
# The gate workflow writes the comment with GITHUB_TOKEN; a marker from anyone else is ignored.
|
|
9
13
|
AUTHOR = "github-actions[bot]"
|
|
10
14
|
DECISIONS = %w[required not-required].freeze
|
|
15
|
+
PRIORITIES = %w[critical normal].freeze
|
|
11
16
|
SHA_PATTERN = /\A\h{40}\z/
|
|
12
17
|
MAX_REASON = 300
|
|
13
|
-
MAX_SCENARIOS = 12
|
|
14
18
|
MAX_NAME = 200
|
|
19
|
+
MAX_WHY = 200
|
|
20
|
+
CATEGORY_PATTERN = /\A[a-z][a-z0-9-]{0,31}\z/
|
|
21
|
+
STATUS_MARKS = { "passed" => "x", "failed" => " ", "skipped" => " ", "blocked" => " ", "missing" => " ", "stale" => " " }.freeze
|
|
15
22
|
|
|
16
|
-
attr_reader :assessed_sha, :decision, :reason, :scenarios, :report_seen, :errors
|
|
23
|
+
attr_reader :assessed_sha, :decision, :reason, :scenarios, :report_seen, :usage, :errors
|
|
17
24
|
|
|
18
25
|
# Builds an assessment from the model's JSON answer, which is untrusted. `report_seen` is the
|
|
19
26
|
# fingerprint of the report the model was shown, so the same report is not judged twice.
|
|
20
|
-
|
|
27
|
+
# `mandatory` are the scenarios the settings require for this change.
|
|
28
|
+
def self.parse(text, assessed_sha:, report_seen: nil, mandatory: [], max_count: 12, usage: nil)
|
|
21
29
|
data = JSON.parse(text.to_s)
|
|
22
30
|
data = {} unless data.is_a?(Hash)
|
|
23
|
-
|
|
31
|
+
proposed = data["scenarios"]
|
|
32
|
+
if proposed.is_a?(Array) && proposed.size > max_count
|
|
33
|
+
return new({ "assessed_sha" => assessed_sha }).tap { |assessment| assessment.errors << "scenarios must have at most #{max_count} entries" }
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
if data["decision"] == "required" && proposed.is_a?(Array)
|
|
37
|
+
taken = proposed.map { |entry| entry.is_a?(Hash) ? entry["id"] : nil }
|
|
38
|
+
data["scenarios"] = proposed + mandatory.reject { |scenario| taken.include?(scenario["id"]) }
|
|
39
|
+
end
|
|
40
|
+
assessment = new(data.slice("decision", "reason", "scenarios").merge("assessed_sha" => assessed_sha, "report_seen" => report_seen, "usage" => usage))
|
|
24
41
|
assessment.errors << "a required browser run needs at least one scenario" if assessment.required? && assessment.scenarios.empty?
|
|
25
42
|
assessment
|
|
26
43
|
rescue JSON::ParserError
|
|
@@ -53,9 +70,8 @@ module BrowserReviewGate
|
|
|
53
70
|
@assessed_sha = data["assessed_sha"]
|
|
54
71
|
@decision = data["decision"]
|
|
55
72
|
@reason = data["reason"].is_a?(String) ? data["reason"].split.join(" ")[0, MAX_REASON] : nil
|
|
56
|
-
# Comments written before scenarios existed carry the model's own verdict on coverage instead.
|
|
57
|
-
@covered_by_report = data["covered_by_report"] == true
|
|
58
73
|
@report_seen = data["report_seen"].is_a?(String) ? data["report_seen"] : nil
|
|
74
|
+
@usage = read_usage(data["usage"])
|
|
59
75
|
@errors = []
|
|
60
76
|
@scenarios = required? ? read_scenarios(data["scenarios"]) : []
|
|
61
77
|
validate
|
|
@@ -65,62 +81,66 @@ module BrowserReviewGate
|
|
|
65
81
|
def required? = decision == "required"
|
|
66
82
|
def for?(sha) = assessed_sha == sha
|
|
67
83
|
|
|
68
|
-
# The scenarios no
|
|
69
|
-
|
|
70
|
-
|
|
84
|
+
# The scenarios no counting case of the report exercises yet. With `commits`, only cases tested on
|
|
85
|
+
# those commits count; with `reuse_passing` false, only cases tested on the assessed commit.
|
|
86
|
+
def outstanding(report, commits: nil, reuse_passing: true)
|
|
87
|
+
allowed = reuse_passing ? commits : [ assessed_sha ]
|
|
88
|
+
passed = report ? report.passed_ids(commits: allowed) : []
|
|
71
89
|
scenarios.reject { |scenario| passed.include?(scenario["id"]) }
|
|
72
90
|
end
|
|
73
91
|
|
|
74
|
-
# A
|
|
75
|
-
|
|
92
|
+
# A report covers the change when it is complete, nothing failed (unless failures do not block),
|
|
93
|
+
# and every scenario has a counting case.
|
|
94
|
+
def covered_by?(report, commits: nil, reuse_passing: true)
|
|
76
95
|
return false unless required? && report&.passed?
|
|
77
|
-
return @covered_by_report || report.tested_sha == assessed_sha if scenarios.empty?
|
|
78
96
|
|
|
79
|
-
outstanding(report).empty?
|
|
97
|
+
outstanding(report, commits: commits, reuse_passing: reuse_passing).empty?
|
|
80
98
|
end
|
|
81
99
|
|
|
82
100
|
# Whether this decision already accounts for the report, so the model need not be asked again.
|
|
83
101
|
def settles?(report)
|
|
84
|
-
!required? || report.nil? ||
|
|
102
|
+
!required? || report.nil? || outstanding(report).empty? || report_seen == report.fingerprint
|
|
85
103
|
end
|
|
86
104
|
|
|
87
105
|
def to_h
|
|
88
106
|
{ "assessed_sha" => assessed_sha, "decision" => decision, "reason" => reason, "scenarios" => scenarios,
|
|
89
|
-
"report_seen" => report_seen, "
|
|
107
|
+
"report_seen" => report_seen, "usage" => usage }.compact
|
|
90
108
|
end
|
|
91
109
|
|
|
92
|
-
# `verified` is true when a
|
|
93
|
-
#
|
|
94
|
-
#
|
|
95
|
-
|
|
96
|
-
def markdown(verified:, command:, author: nil, report: nil)
|
|
110
|
+
# `verified` is true when a report covers the browser behavior of this commit. `command` is what the
|
|
111
|
+
# author runs in an AI coding agent. `author` is who gets mentioned, and only when the comment asks
|
|
112
|
+
# them to do something. `report` and `commits` give each scenario its status.
|
|
113
|
+
def markdown(verified:, command:, author: nil, report: nil, commits: nil)
|
|
97
114
|
payload = Marker.encode(JSON.generate(to_h))
|
|
98
|
-
[ "## Browser verification", "", summary(verified, command, author, report), "",
|
|
115
|
+
[ "## Browser verification", "", summary(verified, command, author, report, commits), "",
|
|
99
116
|
"Assessed commit `#{assessed_sha}`.", "", "<!-- browser-verification-assessment:#{payload} -->" ].join("\n")
|
|
100
117
|
end
|
|
101
118
|
|
|
102
119
|
private
|
|
103
120
|
|
|
104
|
-
def summary(verified, command, author, report)
|
|
121
|
+
def summary(verified, command, author, report, commits)
|
|
105
122
|
why = Markdown.escape(reason)
|
|
106
123
|
if !required?
|
|
107
124
|
"Browser testing is not needed for this change: #{why} Human review can be requested."
|
|
108
125
|
elsif verified
|
|
109
|
-
"Browser testing is required for this change, and the published browser report covers it: #{why}
|
|
110
|
-
"
|
|
126
|
+
[ "**Human review can proceed.** Browser testing is required for this change, and the published browser report covers it: #{why}",
|
|
127
|
+
"", *checklist(report, commits) ].join("\n").rstrip
|
|
111
128
|
else
|
|
112
|
-
[ "#{Markdown.mention(author)}Browser testing is needed before requesting human review. #{why}", "",
|
|
113
|
-
*checklist(report),
|
|
129
|
+
[ "#{Markdown.mention(author)}**Human review is on hold.** Browser testing is needed before requesting human review. #{why}", "",
|
|
130
|
+
*checklist(report, commits),
|
|
114
131
|
"To be able to request a reviewer, run this command in your AI coding agent from the PR branch:", "",
|
|
115
132
|
"```", command, "```" ].join("\n")
|
|
116
133
|
end
|
|
117
134
|
end
|
|
118
135
|
|
|
119
|
-
def checklist(report)
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
136
|
+
def checklist(report, commits)
|
|
137
|
+
lines = scenarios.map do |scenario|
|
|
138
|
+
status = report ? report.status_of(scenario["id"], commits: commits) : "missing"
|
|
139
|
+
flags = [ scenario["priority"] == "critical" ? "critical" : nil, scenario["category"] ].compact.join(", ")
|
|
140
|
+
why = scenario["why"] ? " — #{Markdown.escape(scenario["why"])}" : ""
|
|
141
|
+
"- [#{STATUS_MARKS.fetch(status)}] **#{status}** `#{scenario["id"]}` #{Markdown.escape(scenario["name"])}" \
|
|
142
|
+
"#{flags.empty? ? "" : " (#{Markdown.escape(flags)})"}#{why}"
|
|
143
|
+
end
|
|
124
144
|
[ "Scenarios the run must pass:", "", *lines, "" ]
|
|
125
145
|
end
|
|
126
146
|
|
|
@@ -133,15 +153,36 @@ module BrowserReviewGate
|
|
|
133
153
|
return []
|
|
134
154
|
end
|
|
135
155
|
|
|
136
|
-
scenarios = value.map { |entry|
|
|
156
|
+
scenarios = value.map { |entry| read_scenario(entry) }
|
|
137
157
|
ids = scenarios.map { |scenario| scenario["id"] }
|
|
138
|
-
errors << "scenarios must have at most #{MAX_SCENARIOS} entries" if scenarios.size > MAX_SCENARIOS
|
|
158
|
+
errors << "scenarios must have at most #{Config::MAX_SCENARIOS + Config::MAX_MATRIX_CASES} entries" if scenarios.size > Config::MAX_SCENARIOS + Config::MAX_MATRIX_CASES
|
|
139
159
|
errors << "scenario ids must use letters, numbers, dots, underscores, or hyphens" unless ids.all? { |id| id.is_a?(String) && Report::CASE_ID_PATTERN.match?(id) }
|
|
140
160
|
errors << "scenario ids must be unique" unless ids.uniq.size == ids.size
|
|
141
161
|
errors << "scenario names must not be empty" if scenarios.any? { |scenario| scenario["name"].empty? }
|
|
162
|
+
errors << "scenario priority must be critical or normal" unless scenarios.all? { |scenario| PRIORITIES.include?(scenario["priority"]) }
|
|
163
|
+
errors << "scenario category must be a short lowercase word" unless scenarios.all? { |scenario| scenario["category"].nil? || CATEGORY_PATTERN.match?(scenario["category"]) }
|
|
142
164
|
scenarios
|
|
143
165
|
end
|
|
144
166
|
|
|
167
|
+
def read_scenario(entry)
|
|
168
|
+
scenario = { "id" => entry["id"], "name" => entry["name"].split.join(" ")[0, MAX_NAME],
|
|
169
|
+
"priority" => entry["priority"] || "normal",
|
|
170
|
+
"category" => entry["category"].is_a?(String) && !entry["category"].empty? ? entry["category"].downcase : nil,
|
|
171
|
+
"why" => entry["why"].is_a?(String) && !entry["why"].strip.empty? ? entry["why"].split.join(" ")[0, MAX_WHY] : nil }
|
|
172
|
+
scenario["matrix"] = entry["matrix"] if entry["matrix"].is_a?(Hash)
|
|
173
|
+
scenario.compact
|
|
174
|
+
end
|
|
175
|
+
|
|
176
|
+
# Token counts and cost as the provider reported them; nothing is estimated here.
|
|
177
|
+
def read_usage(value)
|
|
178
|
+
return unless value.is_a?(Hash)
|
|
179
|
+
|
|
180
|
+
usage = value.slice("provider", "model", "input_tokens", "output_tokens", "cost_usd").select do |key, entry|
|
|
181
|
+
%w[provider model].include?(key) ? entry.is_a?(String) && entry.length <= 80 : entry.is_a?(Numeric) && entry >= 0
|
|
182
|
+
end
|
|
183
|
+
usage.empty? ? nil : usage
|
|
184
|
+
end
|
|
185
|
+
|
|
145
186
|
def validate
|
|
146
187
|
errors << "assessed_sha must be a full commit SHA" unless assessed_sha.is_a?(String) && SHA_PATTERN.match?(assessed_sha)
|
|
147
188
|
errors << "decision must be required or not-required" unless DECISIONS.include?(decision)
|
|
@@ -16,8 +16,12 @@ module BrowserReviewGate
|
|
|
16
16
|
type: "array",
|
|
17
17
|
items: {
|
|
18
18
|
type: "object",
|
|
19
|
-
properties: {
|
|
20
|
-
|
|
19
|
+
properties: {
|
|
20
|
+
id: { type: "string" }, name: { type: "string" },
|
|
21
|
+
priority: { type: "string", enum: Assessment::PRIORITIES },
|
|
22
|
+
category: { type: "string" }, why: { type: "string" }
|
|
23
|
+
},
|
|
24
|
+
required: %w[id name priority category why],
|
|
21
25
|
additionalProperties: false
|
|
22
26
|
}
|
|
23
27
|
}
|
|
@@ -27,6 +31,7 @@ module BrowserReviewGate
|
|
|
27
31
|
}.freeze
|
|
28
32
|
|
|
29
33
|
IGNORED_REASON = "Only files that cannot change browser behavior were touched."
|
|
34
|
+
MANDATORY_REASON = "The project settings require browser scenarios for the files this change touches."
|
|
30
35
|
|
|
31
36
|
def initialize(github:, model:, config: Config.new, prompts: Prompts.new, log: $stdout)
|
|
32
37
|
@github = github
|
|
@@ -40,12 +45,13 @@ module BrowserReviewGate
|
|
|
40
45
|
# when a report it has not judged lacks some of its scenarios; the comment is refreshed either way.
|
|
41
46
|
def assess(number, pull_request: @github.pull_request(number), comments: @github.trusted_comments(number))
|
|
42
47
|
head_sha = pull_request.fetch("head").fetch("sha")
|
|
43
|
-
report = Report.latest(comments)
|
|
48
|
+
report = Report.latest(comments, config: @config)
|
|
49
|
+
commits = @github.pull_request_commits(number)
|
|
44
50
|
|
|
45
51
|
existing = Assessment.latest(comments)
|
|
46
52
|
assessed_before = existing&.for?(head_sha)
|
|
47
53
|
assessment = assessed_before && existing.settles?(report) ? existing : propose(number, pull_request, head_sha, report)
|
|
48
|
-
publish(number, assessment, pull_request, comments, report, new_commit: !assessed_before)
|
|
54
|
+
publish(number, assessment, pull_request, comments, report, commits, new_commit: !assessed_before)
|
|
49
55
|
assessment
|
|
50
56
|
rescue KeyError => error
|
|
51
57
|
raise Error, "Browser assessment data is incomplete: #{error.message}"
|
|
@@ -53,27 +59,49 @@ module BrowserReviewGate
|
|
|
53
59
|
|
|
54
60
|
private
|
|
55
61
|
|
|
62
|
+
# Mandatory scenarios scoped to paths make a run required whatever the model says; the rest are
|
|
63
|
+
# added whenever a run is required.
|
|
56
64
|
def propose(number, pull_request, head_sha, report)
|
|
57
65
|
files = @github.files(number)
|
|
58
|
-
|
|
66
|
+
paths = files.map { |file| file["filename"] }
|
|
67
|
+
mandatory = @config.mandatory_scenarios(paths)
|
|
68
|
+
forced = mandatory.any? { |scenario| scoped?(scenario) }
|
|
69
|
+
return Assessment.not_required(IGNORED_REASON, assessed_sha: head_sha) if !forced && paths.all? { |path| @config.ignored?(path) }
|
|
59
70
|
|
|
60
|
-
text = @model.complete(system: @prompts.assessment, user: user_message(pull_request, files, report), schema: SCHEMA)
|
|
61
|
-
|
|
71
|
+
text = @model.complete(system: @prompts.assessment(max_count: @config.scenarios["max_count"]), user: user_message(pull_request, files, report), schema: SCHEMA)
|
|
72
|
+
text = force_required(text) if forced
|
|
73
|
+
usage = @model.respond_to?(:last_usage) ? @model.last_usage : nil
|
|
74
|
+
assessment = Assessment.parse(text, assessed_sha: head_sha, report_seen: report&.fingerprint, mandatory: mandatory,
|
|
75
|
+
max_count: @config.scenarios["max_count"], usage: usage)
|
|
62
76
|
raise Error, "Model returned an invalid browser assessment: #{assessment.errors.join("; ")}" unless assessment.valid?
|
|
63
77
|
|
|
64
78
|
assessment
|
|
65
79
|
end
|
|
66
80
|
|
|
81
|
+
def scoped?(scenario)
|
|
82
|
+
entry = @config.scenarios["mandatory"].find { |item| scenario["id"] == item["id"] || scenario["id"].start_with?("#{item["id"]}.") }
|
|
83
|
+
entry && entry["paths"]
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def force_required(text)
|
|
87
|
+
data = JSON.parse(text.to_s)
|
|
88
|
+
return text unless data.is_a?(Hash) && data["decision"] == "not-required"
|
|
89
|
+
|
|
90
|
+
JSON.generate(data.merge("decision" => "required", "reason" => MANDATORY_REASON, "scenarios" => []))
|
|
91
|
+
rescue JSON::ParserError
|
|
92
|
+
text
|
|
93
|
+
end
|
|
94
|
+
|
|
67
95
|
# Ordinary commits keep the label; it goes only when the report does not cover the commit.
|
|
68
|
-
def publish(number, assessment, pull_request, comments, report, new_commit:)
|
|
96
|
+
def publish(number, assessment, pull_request, comments, report, commits, new_commit:)
|
|
69
97
|
labelled = pull_request.fetch("labels").any? { |label| label["name"] == @config.label }
|
|
70
|
-
covered = assessment.covered_by?(report)
|
|
98
|
+
covered = assessment.covered_by?(report, commits: commits, reuse_passing: @config.scenarios["reuse_passing"])
|
|
71
99
|
run_owed = assessment.required? && !(labelled && covered)
|
|
72
100
|
@github.remove_label(number, @config.label) if run_owed && labelled
|
|
73
101
|
|
|
74
102
|
comment = Assessment.latest_comment(comments)
|
|
75
103
|
body = assessment.markdown(verified: labelled && covered, command: "#{@config.command} #{number}",
|
|
76
|
-
author: run_owed ? people_to_tell(pull_request, assessment.assessed_sha) : nil, report: report)
|
|
104
|
+
author: run_owed ? people_to_tell(pull_request, assessment.assessed_sha) : nil, report: report, commits: commits)
|
|
77
105
|
if comment.nil?
|
|
78
106
|
@github.create_comment(number, body)
|
|
79
107
|
elsif comment["body"] == body
|