lastlight 0.7.4 → 0.7.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/plugins/lastlight/.claude-plugin/plugin.json +1 -1
- package/plugins/lastlight/skills/lastlight-evals/SKILL.md +29 -7
- package/skills/README.md +8 -6
- package/skills/browser-qa/SKILL.md +15 -0
- package/skills/browser-qa/scripts/agent-browser.mjs +143 -7
- package/skills/code-review/SKILL.md +28 -5
- package/skills/demo/SKILL.md +10 -2
- package/skills/pr-review/SKILL.md +27 -19
- package/workflows/pr-review.yaml +4 -3
- package/workflows/prompts/demo.md +5 -2
- package/workflows/prompts/guardrails.md +15 -0
package/package.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"$schema": "https://json.schemastore.org/claude-code-plugin.json",
|
|
3
3
|
"name": "lastlight",
|
|
4
4
|
"displayName": "Last Light",
|
|
5
|
-
"version": "0.7.
|
|
5
|
+
"version": "0.7.5",
|
|
6
6
|
"description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
|
|
7
7
|
"author": { "name": "Clifton Cunningham" },
|
|
8
8
|
"homepage": "https://github.com/cliftonc/lastlight",
|
|
@@ -97,20 +97,36 @@ lastlight-evals run triage --model haiku # fuzzy match in models.json
|
|
|
97
97
|
lastlight-evals run triage --model openai/gpt-5.5,anthropic/claude-opus-4-8
|
|
98
98
|
lastlight-evals run --compare # cross-vendor set (only models whose envKey is present)
|
|
99
99
|
lastlight-evals run triage --runs 3 # repeat each case 3× (worst-case verdict, mean metrics)
|
|
100
|
+
lastlight-evals run pr-review --limit 3 # only the first 3 cases of the tier (controlled/cheap run)
|
|
101
|
+
lastlight-evals run triage --instance <id> # only this exact instance_id (or set EVAL_INSTANCE)
|
|
100
102
|
lastlight-evals run triage --no-open # don't open the report
|
|
101
103
|
# Plain layout: add --overlay . (e.g. lastlight-evals run triage --overlay .)
|
|
104
|
+
|
|
105
|
+
lastlight-evals serve # browse past runs in the dashboard (no models run)
|
|
106
|
+
lastlight-evals clean --dry-run # list killed/crashed runs that are stuck "running"
|
|
107
|
+
lastlight-evals clean # finalize them (mark interrupted; --delete to remove)
|
|
102
108
|
```
|
|
103
109
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
110
|
+
Each run lands in its own dir `./eval-results/<tiers>/<runId>/`: `scorecard.json`
|
|
111
|
+
+ `predictions.jsonl` (SWE-bench format). The report is a **JSON-driven dashboard
|
|
112
|
+
SPA**, not generated HTML — `run` starts a local server and opens it; browse every
|
|
113
|
+
past run later with `lastlight-evals serve`. In the dashboard a code-fix row's
|
|
114
|
+
**files** button opens the agent's captured diff and **log** shows the per-phase
|
|
115
|
+
agent session.
|
|
116
|
+
|
|
117
|
+
If a run is killed or crashes mid-flight it stays stuck showing "running" (its
|
|
118
|
+
scorecard never got its final write). `lastlight-evals clean` finalizes such runs
|
|
119
|
+
— marks them *interrupted* (default; keeps the partial scorecard + transcripts)
|
|
120
|
+
or `--delete` removes the run dir.
|
|
107
121
|
|
|
108
122
|
## 5. Author eval cases (optional)
|
|
109
123
|
|
|
110
|
-
|
|
111
|
-
workflow with held-out tests)
|
|
112
|
-
|
|
113
|
-
|
|
124
|
+
Three tiers ship: **triage** (cheap, issue-triage), **code-fix** (heavy, build
|
|
125
|
+
workflow with held-out tests), and **pr-review** (PR-review precision, graded by
|
|
126
|
+
an LLM judge against a gold set → precision / recall / **F0.5**). To add cases or
|
|
127
|
+
a custom tier, read **`references/instance-schema.md`** — it has the
|
|
128
|
+
`SweBenchInstance` schema, the exact files to create for each tier, and worked
|
|
129
|
+
examples.
|
|
114
130
|
|
|
115
131
|
Quick shape (paths are relative to the workspace's `evals/` dir):
|
|
116
132
|
- **Triage case:** append a `SweBenchInstance` to `evals/datasets/triage/instances.json`
|
|
@@ -118,6 +134,12 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
|
|
|
118
134
|
- **Code-fix case:** add the instance to `evals/datasets/code-fix/instances.json` **and**
|
|
119
135
|
create `evals/datasets/code-fix/repos/<instance_id>/` (fixture repo at base) +
|
|
120
136
|
`evals/datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
|
|
137
|
+
- **PR-review case:** a `SweBenchInstance` with a `pr` fixture (base/head refs +
|
|
138
|
+
commits, checked out at the PR head) and a `review_gold` set (severity +
|
|
139
|
+
description). The `pr-review` tier ships empty — populate the full Martian
|
|
140
|
+
[Code Review Bench](https://codereview.withmartian.com/) 50 with
|
|
141
|
+
`npx tsx scripts/import-martian.ts`. Grading needs a judge model
|
|
142
|
+
(`EVAL_JUDGE_MODEL`, else a strong default per provider key).
|
|
121
143
|
- **Custom tier:** a new `evals/datasets/<tier>/` with `tier.json` +
|
|
122
144
|
`instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
|
|
123
145
|
automatic — no code change.
|
package/skills/README.md
CHANGED
|
@@ -17,22 +17,24 @@ or editing a skill.
|
|
|
17
17
|
| [`chat`](chat/) | chat | Conversational assistant for Slack/Discord threads — answer repo/PR/issue questions, guide to natural-language triggers. |
|
|
18
18
|
| [`issue-triage`](issue-triage/) | `issue-triage.yaml`, chat | Move issues through the canonical triage state machine — categorise, label, dedupe, manage stale, write a problem-statement summary (not a solution design), re-triage on reporter replies. |
|
|
19
19
|
| [`issue-comment`](issue-comment/) | `issue-comment.yaml` | Handle non-build maintainer comments on issues/PRs — close, label, answer briefly, redirect build requests. |
|
|
20
|
-
| [`pr-review`](pr-review/) | `pr-review.yaml`, chat | Review a PR with
|
|
20
|
+
| [`pr-review`](pr-review/) | `pr-review.yaml`, chat | Review a PR with precision-first, tiered feedback and post a formal review. A pure code review — no building (CI validates the change). |
|
|
21
21
|
| [`pr-comment`](pr-comment/) | `pr-comment.yaml` | Answer a maintainer's question about an open PR with concrete, code-cited evidence (not a full review). |
|
|
22
22
|
| [`repo-health`](repo-health/) | `repo-health.yaml`, chat | Generate a repository health report — open issues, PR backlog, CI status, action items. |
|
|
23
23
|
| [`security-review`](security-review/) | `security-review.yaml` | Diff-scoped SDLC security review; file one dated summary issue with a task-list of findings. |
|
|
24
24
|
| [`security-feedback`](security-feedback/) | `security-feedback.yaml` | Process a maintainer's comment on a security scan summary — break findings into issues or record suppressions. |
|
|
25
|
-
| [`building`](building/) | build (executor, reviewer), `pr-fix.yaml
|
|
25
|
+
| [`building`](building/) | build (executor, reviewer), `pr-fix.yaml` | Shared craft: install-first + package-manager detection, the test/lint/typecheck gate, and TDD discipline in the sandbox. |
|
|
26
26
|
| [`code-review`](code-review/) | build (reviewer), `pr-review.yaml` | Shared rubric: finding tiers (Critical/Important/Suggestions/Nits) and what-to-check. Referenced by both the branch-diff reviewer and the PR reviewer. |
|
|
27
27
|
| [`verify`](verify/) | `verify.yaml` | Test a behaviour claim as an investigator — CONFIRMED / REFUTED / INCONCLUSIVE with bash-captured evidence. |
|
|
28
28
|
| [`qa-test`](qa-test/) | `qa-test.yaml` | Run an automated QA flow against a CLI or locally-served app and report step-level pass/fail with evidence. |
|
|
29
29
|
| [`browser-qa`](browser-qa/) | `verify.yaml`, `qa-test.yaml`, `demo.yaml` | Drive a real headless browser against a locally-served web UI and capture screenshot (and, for `demo`, video) evidence. Gated to the docker QA image. |
|
|
30
30
|
| [`demo`](demo/) | `demo.yaml` | Record a short demo VIDEO of a PR/feature — drive the web UI, screen-record, and composite a titled, size-capped mp4 with ffmpeg. Gated to the docker QA image. |
|
|
31
31
|
|
|
32
|
-
`building` and `code-review` are **shared building blocks**
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
32
|
+
`building` and `code-review` are **shared building blocks** so the install
|
|
33
|
+
gate and the review rubric each live in one place. `building` is staged by the
|
|
34
|
+
build cycle (executor + reviewer) and `pr-fix`; `code-review` by the build
|
|
35
|
+
reviewer and `pr-review` (which stages `code-review` but **not** `building` — a
|
|
36
|
+
pure code review). They're staged alongside the phase's primary skill/prompt and
|
|
37
|
+
read on demand.
|
|
36
38
|
|
|
37
39
|
The live set is enforced at startup: `validateAssets()` (`src/workflows/loader.ts`)
|
|
38
40
|
resolves every workflow `skill:` reference and `CHAT_SKILL_NAMES`
|
|
@@ -91,11 +91,26 @@ Step keys (a step may combine an action **plus** a trailing `screenshot`):
|
|
|
91
91
|
| `waitFor` | selector | Wait until the element is visible (~10s timeout). |
|
|
92
92
|
| `assertText` | string | Pass if the text is visible anywhere on the page; else step FAIL.|
|
|
93
93
|
| `text` | selector | Extract `textContent` into the step result (for you to read). |
|
|
94
|
+
| `pause` | milliseconds | Hold for N ms — a readable beat after a state change (demo videos).|
|
|
94
95
|
| `screenshot` | basename | Write `<out-dir>/<basename>.png` (full page). |
|
|
95
96
|
|
|
96
97
|
The whole flow runs in **one Chromium session** — login state, cookies, and
|
|
97
98
|
navigation persist across steps.
|
|
98
99
|
|
|
100
|
+
### Demo mode (recording only)
|
|
101
|
+
|
|
102
|
+
When the session is **recorded** (`--record-dir`, `"record": true`, or an
|
|
103
|
+
explicit `--demo`), the driver switches on "demo mode" so the capture is
|
|
104
|
+
watchable — headless Chromium otherwise paints no cursor and fires actions
|
|
105
|
+
instantly. Demo mode adds a **synthetic cursor overlay** that animates to each
|
|
106
|
+
target, types `type` steps **character-by-character**, and holds a beat
|
|
107
|
+
**between steps**. For screenshot QA (no recording) these are all no-ops, so
|
|
108
|
+
QA runs stay fast. Prefer `type` over `fill` in a recorded flow so the typing is
|
|
109
|
+
visible, and use `pause` for deliberate holds. Tunables:
|
|
110
|
+
`--step-delay MS` (default 700), `--type-delay MS` (default 70),
|
|
111
|
+
`--move-steps N` (default 25), `--no-cursor` to drop the overlay (env:
|
|
112
|
+
`LASTLIGHT_STEP_DELAY_MS`, `LASTLIGHT_TYPE_DELAY_MS`, `LASTLIGHT_MOVE_STEPS`).
|
|
113
|
+
|
|
99
114
|
## Run the flow
|
|
100
115
|
|
|
101
116
|
```bash
|
|
@@ -18,6 +18,8 @@
|
|
|
18
18
|
// The skill runs this first to decide browser-vs-text.
|
|
19
19
|
//
|
|
20
20
|
// run <flow.json> [--base-url URL] [--out-dir DIR] [--record-dir DIR]
|
|
21
|
+
// [--demo] [--no-cursor] [--step-delay MS] [--type-delay MS]
|
|
22
|
+
// [--move-steps N]
|
|
21
23
|
// Executes a FLOW in ONE Chromium session (state preserved across steps) and
|
|
22
24
|
// prints a single JSON report.
|
|
23
25
|
//
|
|
@@ -27,6 +29,17 @@
|
|
|
27
29
|
// reported as `video` in the JSON. Used by the `/demo` workflow, which then
|
|
28
30
|
// composites the raw webm into a titled mp4 with compose-demo.sh.
|
|
29
31
|
//
|
|
32
|
+
// DEMO MODE (auto-on whenever recording; also --demo or `"demo": true` in the
|
|
33
|
+
// flow) makes the capture human-watchable — headless Chromium otherwise
|
|
34
|
+
// paints no cursor and fires actions instantly:
|
|
35
|
+
// - a synthetic cursor overlay (opt out with --no-cursor) that animates to
|
|
36
|
+
// each target before acting, driven by real page.mouse events;
|
|
37
|
+
// - `type` steps that key in char-by-char (--type-delay, default 70ms);
|
|
38
|
+
// - a deliberate hold between steps (--step-delay, default 700ms).
|
|
39
|
+
// Outside demo mode every one of these is a no-op, so screenshot QA runs
|
|
40
|
+
// behave exactly as before. Env equivalents: LASTLIGHT_STEP_DELAY_MS,
|
|
41
|
+
// LASTLIGHT_TYPE_DELAY_MS, LASTLIGHT_MOVE_STEPS.
|
|
42
|
+
//
|
|
30
43
|
// Flow shape:
|
|
31
44
|
// { "baseUrl": "http://localhost:3000",
|
|
32
45
|
// "viewport": {"width":1280,"height":800},
|
|
@@ -38,10 +51,13 @@
|
|
|
38
51
|
// {"press":"Enter"},
|
|
39
52
|
// {"waitFor":"#dashboard"},
|
|
40
53
|
// {"assertText":"Welcome"},
|
|
54
|
+
// {"pause": 1200},
|
|
41
55
|
// {"text":"h1"},
|
|
42
56
|
// {"screenshot":"after-login"}
|
|
43
57
|
// ] }
|
|
44
|
-
// A step may combine an action plus a trailing `screenshot`.
|
|
58
|
+
// A step may combine an action plus a trailing `screenshot`. `pause` holds
|
|
59
|
+
// for N ms (a readable beat after a state change, for demo recordings);
|
|
60
|
+
// `type` shows visible per-character typing, `fill` sets the value instantly.
|
|
45
61
|
//
|
|
46
62
|
// Per-step semantics:
|
|
47
63
|
// - A step error (selector not found, assertion fail, timeout) is recorded
|
|
@@ -77,6 +93,81 @@ const WAIT_TIMEOUT = 10_000; // sane default for waitFor / assertText probes
|
|
|
77
93
|
// watchable tail. Override with --record-settle-ms / LASTLIGHT_RECORD_SETTLE_MS.
|
|
78
94
|
const RECORD_SETTLE_MS = Number(process.env.LASTLIGHT_RECORD_SETTLE_MS) || 1500;
|
|
79
95
|
|
|
96
|
+
// ── Demo-mode pacing ─────────────────────────────────────────────────────────
|
|
97
|
+
// Headless Chromium paints NO cursor and fires actions instantly, so a raw
|
|
98
|
+
// recording looks like the UI mutating on its own. "Demo mode" (auto-on whenever
|
|
99
|
+
// the session is recorded — see run()) makes the capture human-watchable: a
|
|
100
|
+
// synthetic cursor overlay that animates to each target, char-by-char typing,
|
|
101
|
+
// and a deliberate pause between steps. These are no-ops outside demo mode, so
|
|
102
|
+
// screenshot QA runs are untouched. All tunable via flags / env.
|
|
103
|
+
const DEMO_STEP_DELAY_MS = Number(process.env.LASTLIGHT_STEP_DELAY_MS) || 700;
|
|
104
|
+
const DEMO_TYPE_DELAY_MS = Number(process.env.LASTLIGHT_TYPE_DELAY_MS) || 70;
|
|
105
|
+
const DEMO_MOVE_STEPS = Number(process.env.LASTLIGHT_MOVE_STEPS) || 25;
|
|
106
|
+
|
|
107
|
+
// Synthetic cursor overlay, adapted from Puppeteer's mouse-helper (Apache-2.0),
|
|
108
|
+
// made self-installing so it can be injected via context.addInitScript and picked
|
|
109
|
+
// up by Playwright's recordVideo (it's page DOM, so it lands in the .webm). It
|
|
110
|
+
// follows the REAL mouse events that page.mouse.move/down/up dispatch.
|
|
111
|
+
const MOUSE_HELPER_SRC = `(() => {
|
|
112
|
+
const install = () => {
|
|
113
|
+
if (window.__mouseHelperInstalled || !document.body) return;
|
|
114
|
+
window.__mouseHelperInstalled = true;
|
|
115
|
+
const box = document.createElement('div');
|
|
116
|
+
box.classList.add('mouse-helper');
|
|
117
|
+
const style = document.createElement('style');
|
|
118
|
+
style.innerHTML = \`
|
|
119
|
+
.mouse-helper {
|
|
120
|
+
pointer-events: none;
|
|
121
|
+
position: absolute;
|
|
122
|
+
z-index: 2147483647;
|
|
123
|
+
width: 20px; height: 20px;
|
|
124
|
+
margin-left: -10px; margin-top: -10px;
|
|
125
|
+
border-radius: 10px;
|
|
126
|
+
border: 2px solid rgba(255,255,255,.9);
|
|
127
|
+
background: rgba(0,0,0,.35);
|
|
128
|
+
box-shadow: 0 0 0 1px rgba(0,0,0,.4);
|
|
129
|
+
transition: background .15s, border-radius .15s, transform .08s;
|
|
130
|
+
}
|
|
131
|
+
.mouse-helper.button-1 {
|
|
132
|
+
transition: none;
|
|
133
|
+
background: rgba(30,120,255,.7);
|
|
134
|
+
transform: scale(.75);
|
|
135
|
+
}
|
|
136
|
+
\`;
|
|
137
|
+
document.head.appendChild(style);
|
|
138
|
+
document.body.appendChild(box);
|
|
139
|
+
const move = (e) => { box.style.left = e.pageX + 'px'; box.style.top = e.pageY + 'px'; };
|
|
140
|
+
document.addEventListener('mousemove', move, true);
|
|
141
|
+
document.addEventListener('mousedown', (e) => { move(e); box.classList.add('button-1'); }, true);
|
|
142
|
+
document.addEventListener('mouseup', (e) => { move(e); box.classList.remove('button-1'); }, true);
|
|
143
|
+
};
|
|
144
|
+
if (document.readyState === 'loading') {
|
|
145
|
+
document.addEventListener('DOMContentLoaded', install, false);
|
|
146
|
+
} else {
|
|
147
|
+
install();
|
|
148
|
+
}
|
|
149
|
+
})();`;
|
|
150
|
+
|
|
151
|
+
// Animate the synthetic cursor to the centre of a selector, best-effort. Returns
|
|
152
|
+
// the target box when it landed (so the caller can click at those coords), or
|
|
153
|
+
// null to fall back to Playwright's own targeting.
|
|
154
|
+
async function moveCursorTo(page, selector, demo) {
|
|
155
|
+
try {
|
|
156
|
+
const loc = page.locator(selector).first();
|
|
157
|
+
await loc.waitFor({ state: 'visible', timeout: WAIT_TIMEOUT });
|
|
158
|
+
await loc.scrollIntoViewIfNeeded({ timeout: WAIT_TIMEOUT }).catch(() => {});
|
|
159
|
+
const box = await loc.boundingBox();
|
|
160
|
+
if (!box) return null;
|
|
161
|
+
const cx = box.x + box.width / 2;
|
|
162
|
+
const cy = box.y + box.height / 2;
|
|
163
|
+
await page.mouse.move(cx, cy, { steps: demo.moveSteps });
|
|
164
|
+
await page.waitForTimeout(140).catch(() => {});
|
|
165
|
+
return { cx, cy };
|
|
166
|
+
} catch {
|
|
167
|
+
return null;
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
|
|
80
171
|
function emit(obj) {
|
|
81
172
|
process.stdout.write(JSON.stringify(obj) + '\n');
|
|
82
173
|
}
|
|
@@ -140,6 +231,7 @@ function actionOf(step) {
|
|
|
140
231
|
'waitFor',
|
|
141
232
|
'assertText',
|
|
142
233
|
'text',
|
|
234
|
+
'pause',
|
|
143
235
|
]) {
|
|
144
236
|
if (k in step) return k;
|
|
145
237
|
}
|
|
@@ -147,7 +239,7 @@ function actionOf(step) {
|
|
|
147
239
|
return 'noop';
|
|
148
240
|
}
|
|
149
241
|
|
|
150
|
-
async function execStep(page, step, baseUrl, outDir, screenshots) {
|
|
242
|
+
async function execStep(page, step, baseUrl, outDir, screenshots, demo) {
|
|
151
243
|
const result = {};
|
|
152
244
|
let fatal = false;
|
|
153
245
|
|
|
@@ -164,15 +256,31 @@ async function execStep(page, step, baseUrl, outDir, screenshots) {
|
|
|
164
256
|
});
|
|
165
257
|
}
|
|
166
258
|
} else if ('click' in step) {
|
|
167
|
-
|
|
259
|
+
// Demo mode: animate the cursor to the target and click at its coordinates
|
|
260
|
+
// so the pointer travel + press are visible. Fall back to Playwright's own
|
|
261
|
+
// targeting when the box can't be resolved (or outside demo mode).
|
|
262
|
+
const at = demo.enabled ? await moveCursorTo(page, step.click, demo) : null;
|
|
263
|
+
if (at) {
|
|
264
|
+
await page.mouse.down();
|
|
265
|
+
await page.mouse.up();
|
|
266
|
+
} else {
|
|
267
|
+
await page.click(step.click, { timeout: WAIT_TIMEOUT });
|
|
268
|
+
}
|
|
168
269
|
} else if ('fill' in step) {
|
|
169
270
|
const [sel, val] = step.fill;
|
|
271
|
+
if (demo.enabled) await moveCursorTo(page, sel, demo);
|
|
170
272
|
await page.fill(sel, val, { timeout: WAIT_TIMEOUT });
|
|
171
273
|
} else if ('type' in step) {
|
|
172
274
|
const [sel, val] = step.type;
|
|
275
|
+
if (demo.enabled) await moveCursorTo(page, sel, demo);
|
|
173
276
|
await page.locator(sel).first().pressSequentially(val, {
|
|
174
277
|
timeout: WAIT_TIMEOUT,
|
|
278
|
+
delay: demo.enabled ? demo.typeDelay : 0,
|
|
175
279
|
});
|
|
280
|
+
} else if ('pause' in step) {
|
|
281
|
+
// Explicit hold — for reading a state change before the next action.
|
|
282
|
+
const ms = Number(step.pause) || 0;
|
|
283
|
+
if (ms > 0) await page.waitForTimeout(ms);
|
|
176
284
|
} else if ('press' in step) {
|
|
177
285
|
await page.keyboard.press(step.press);
|
|
178
286
|
} else if ('waitFor' in step) {
|
|
@@ -210,7 +318,7 @@ async function execStep(page, step, baseUrl, outDir, screenshots) {
|
|
|
210
318
|
return { result, fatal };
|
|
211
319
|
}
|
|
212
320
|
|
|
213
|
-
async function run(flowPath, baseUrlArg, outDirArg, recordDirArg) {
|
|
321
|
+
async function run(flowPath, baseUrlArg, outDirArg, recordDirArg, opts = {}) {
|
|
214
322
|
let flow;
|
|
215
323
|
try {
|
|
216
324
|
flow = JSON.parse(readFileSync(resolve(flowPath), 'utf8'));
|
|
@@ -229,6 +337,17 @@ async function run(flowPath, baseUrlArg, outDirArg, recordDirArg) {
|
|
|
229
337
|
// `"record": true`. The .webm lands in the record dir (default: out-dir).
|
|
230
338
|
const doRecord = !!recordDirArg || flow.record === true;
|
|
231
339
|
const videoDir = resolve(recordDirArg || outDir);
|
|
340
|
+
// Demo mode = recording, an explicit --demo, or flow.demo. It slows the run
|
|
341
|
+
// down (cursor animation, char-by-char typing, inter-step holds) so the
|
|
342
|
+
// capture is watchable; it's a no-op for plain screenshot QA. Cursor overlay
|
|
343
|
+
// is on by default in demo mode, opt out with --no-cursor.
|
|
344
|
+
const demo = {
|
|
345
|
+
enabled: opts.demo === true || flow.demo === true || doRecord,
|
|
346
|
+
cursor: opts.cursor !== false,
|
|
347
|
+
stepDelay: Number.isFinite(opts.stepDelay) ? opts.stepDelay : DEMO_STEP_DELAY_MS,
|
|
348
|
+
typeDelay: Number.isFinite(opts.typeDelay) ? opts.typeDelay : DEMO_TYPE_DELAY_MS,
|
|
349
|
+
moveSteps: Number.isFinite(opts.moveSteps) ? opts.moveSteps : DEMO_MOVE_STEPS,
|
|
350
|
+
};
|
|
232
351
|
try {
|
|
233
352
|
mkdirSync(outDir, { recursive: true });
|
|
234
353
|
} catch {
|
|
@@ -267,6 +386,11 @@ async function run(flowPath, baseUrlArg, outDirArg, recordDirArg) {
|
|
|
267
386
|
// the clip isn't letterboxed. The .webm is flushed on context.close().
|
|
268
387
|
...(doRecord ? { recordVideo: { dir: videoDir, size: viewport } } : {}),
|
|
269
388
|
});
|
|
389
|
+
// Inject the synthetic cursor before the first navigation so it's present on
|
|
390
|
+
// every page (addInitScript re-runs on each document). Only in demo mode.
|
|
391
|
+
if (demo.enabled && demo.cursor) {
|
|
392
|
+
await context.addInitScript(MOUSE_HELPER_SRC).catch(() => {});
|
|
393
|
+
}
|
|
270
394
|
const page = await context.newPage();
|
|
271
395
|
|
|
272
396
|
page.on('console', (msg) => {
|
|
@@ -288,7 +412,7 @@ async function run(flowPath, baseUrlArg, outDirArg, recordDirArg) {
|
|
|
288
412
|
|
|
289
413
|
const started = Date.now();
|
|
290
414
|
try {
|
|
291
|
-
const { result } = await execStep(page, step, baseUrl, outDir, screenshots);
|
|
415
|
+
const { result } = await execStep(page, step, baseUrl, outDir, screenshots, demo);
|
|
292
416
|
steps.push({ index: i, action, ok: true, ms: Date.now() - started, ...result });
|
|
293
417
|
} catch (err) {
|
|
294
418
|
const entry = {
|
|
@@ -301,6 +425,12 @@ async function run(flowPath, baseUrlArg, outDirArg, recordDirArg) {
|
|
|
301
425
|
steps.push(entry);
|
|
302
426
|
if (err && err.fatal) fatalHit = true; // navigation failure: skip the rest
|
|
303
427
|
}
|
|
428
|
+
|
|
429
|
+
// Demo mode: hold between steps so the video breathes (skip after an
|
|
430
|
+
// explicit `pause`, which already held, and after a fatal step).
|
|
431
|
+
if (demo.enabled && demo.stepDelay > 0 && action !== 'pause' && !fatalHit) {
|
|
432
|
+
await page.waitForTimeout(demo.stepDelay).catch(() => {});
|
|
433
|
+
}
|
|
304
434
|
}
|
|
305
435
|
|
|
306
436
|
// Finalize a recording (if any) BEFORE emitting: saving the video requires
|
|
@@ -370,18 +500,24 @@ async function main() {
|
|
|
370
500
|
let baseUrl;
|
|
371
501
|
let outDir;
|
|
372
502
|
let recordDir;
|
|
503
|
+
const opts = {};
|
|
373
504
|
for (let i = 0; i < rest.length; i++) {
|
|
374
505
|
const a = rest[i];
|
|
375
506
|
if (a === '--base-url') baseUrl = rest[++i];
|
|
376
507
|
else if (a === '--out-dir') outDir = rest[++i];
|
|
377
508
|
else if (a === '--record-dir') recordDir = rest[++i];
|
|
509
|
+
else if (a === '--demo') opts.demo = true;
|
|
510
|
+
else if (a === '--no-cursor') opts.cursor = false;
|
|
511
|
+
else if (a === '--step-delay') opts.stepDelay = Number(rest[++i]);
|
|
512
|
+
else if (a === '--type-delay') opts.typeDelay = Number(rest[++i]);
|
|
513
|
+
else if (a === '--move-steps') opts.moveSteps = Number(rest[++i]);
|
|
378
514
|
else positional.push(a);
|
|
379
515
|
}
|
|
380
516
|
if (!positional[0]) {
|
|
381
|
-
emit({ ok: false, error: 'usage: agent-browser.mjs run <flow.json> [--base-url URL] [--out-dir DIR] [--record-dir DIR]' });
|
|
517
|
+
emit({ ok: false, error: 'usage: agent-browser.mjs run <flow.json> [--base-url URL] [--out-dir DIR] [--record-dir DIR] [--demo] [--no-cursor] [--step-delay MS] [--type-delay MS] [--move-steps N]' });
|
|
382
518
|
process.exit(1);
|
|
383
519
|
}
|
|
384
|
-
await run(positional[0], baseUrl, outDir, recordDir);
|
|
520
|
+
await run(positional[0], baseUrl, outDir, recordDir, opts);
|
|
385
521
|
return;
|
|
386
522
|
}
|
|
387
523
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: code-review
|
|
3
|
-
description: The shared rubric for reviewing a code change —
|
|
4
|
-
version:
|
|
3
|
+
description: The shared rubric for reviewing a code change — precision-first, high-signal findings only (Critical/Important), plus what to check (correctness, security, edge cases, regression risk, test coverage). Use when reviewing a PR or a branch diff.
|
|
4
|
+
version: 2.0.0
|
|
5
5
|
tags: [review, code-quality]
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -17,17 +17,40 @@ modified functions, check callers of anything whose signature or behaviour
|
|
|
17
17
|
changed for regression risk, and check that tests cover the actual risk areas,
|
|
18
18
|
not just the happy path.
|
|
19
19
|
|
|
20
|
+
## Precision first — post only what matters
|
|
21
|
+
|
|
22
|
+
A review is only useful if people trust it. A noisy reviewer gets muted; every
|
|
23
|
+
low-value comment you post spends the credibility of the ones that matter. So
|
|
24
|
+
the bar is **high-signal only**:
|
|
25
|
+
|
|
26
|
+
- **Post only Critical and Important findings.** Suggestions and Nits (below)
|
|
27
|
+
are *not* worth a formal review comment on their own — drop them, or fold at
|
|
28
|
+
most one genuinely valuable line into the summary. When in doubt, leave it out.
|
|
29
|
+
- **If you cannot name the concrete impact — what breaks, and for which input or
|
|
30
|
+
caller — do not post it.** "This could be cleaner" is not a finding; "this
|
|
31
|
+
crashes when `items` is empty because line 42 indexes `[0]`" is.
|
|
32
|
+
- **Confidence gate.** Before you finalise, re-read each finding against the
|
|
33
|
+
actual code and try to *refute your own claim*. If you can't defend it against
|
|
34
|
+
what the code actually does (not what you assumed), drop it. A missed marginal
|
|
35
|
+
issue costs far less than a wrong one.
|
|
36
|
+
|
|
20
37
|
## Finding tiers
|
|
21
38
|
|
|
22
|
-
Categorise every finding into exactly one tier
|
|
39
|
+
Categorise every finding into exactly one tier. **Only Critical and Important
|
|
40
|
+
are posted** (see Precision first):
|
|
23
41
|
|
|
24
42
|
- **Critical** — security issues, data loss, breaking changes, silent
|
|
25
43
|
data-dropping (see Correctness below). Blocks merge.
|
|
26
44
|
- **Important** — missing tests, performance problems, type errors, **avoidable
|
|
27
45
|
duplication**, **excessive complexity**, **compiler-silencing assertions**.
|
|
28
46
|
Should fix.
|
|
29
|
-
- **Suggestions** — clarity, naming, minor DRY tidy-ups.
|
|
30
|
-
|
|
47
|
+
- **Suggestions** — clarity, naming, minor DRY tidy-ups. *Not posted* — noise
|
|
48
|
+
in a formal review.
|
|
49
|
+
- **Nits** — style, formatting. *Not posted* — this is the linter's job.
|
|
50
|
+
|
|
51
|
+
Every posted finding carries a **one-line concrete impact**: the consequence
|
|
52
|
+
(what breaks / for whom) and, where it helps, the fix. That local reasoning is
|
|
53
|
+
what makes a comment actionable rather than a vague worry.
|
|
31
54
|
|
|
32
55
|
## What to check
|
|
33
56
|
|
package/skills/demo/SKILL.md
CHANGED
|
@@ -82,10 +82,18 @@ Author a `flow.json` (same shape as **browser-qa**) that scripts the
|
|
|
82
82
|
interaction as a *director*, not an operator:
|
|
83
83
|
|
|
84
84
|
- **Record the baseline first** — the starting state is act one.
|
|
85
|
-
- **Hold after state changes** — add a `
|
|
85
|
+
- **Hold after state changes** — add a `{"pause": 1200}` (or `waitFor`) so the
|
|
86
|
+
result is readable before moving on.
|
|
86
87
|
- **Verify between steps** — `waitFor` the expected element before the next
|
|
87
88
|
action; don't blindly fire keys into an unsettled page.
|
|
88
89
|
|
|
90
|
+
Because recording auto-enables the driver's **demo mode**, you get a visible
|
|
91
|
+
**synthetic cursor** that animates to each target, and holds between steps —
|
|
92
|
+
you don't script those. To make typing visible on screen, prefer **`type`**
|
|
93
|
+
(character-by-character) over `fill` (instant) for any input the viewer should
|
|
94
|
+
watch being filled. Tune pacing with `--step-delay` / `--type-delay` if a flow
|
|
95
|
+
feels rushed or draggy (see the browser-qa skill).
|
|
96
|
+
|
|
89
97
|
Record the session by passing `--record-dir`:
|
|
90
98
|
|
|
91
99
|
```bash
|
|
@@ -134,7 +142,7 @@ Composite the raw recording(s) into the final mp4 with the bundled wrapper:
|
|
|
134
142
|
--output <artifact-dir>/demo.mp4 \
|
|
135
143
|
--title "PR #42 — Add dark mode toggle" \
|
|
136
144
|
--subtitle "Toggling persists the theme across reloads" \
|
|
137
|
-
--layout single --speed 1
|
|
145
|
+
--layout single --speed 1 \
|
|
138
146
|
/tmp/demo-cap/session.webm
|
|
139
147
|
|
|
140
148
|
# Before/after comparison — first clip = left/BEFORE, second = right/AFTER
|
|
@@ -1,18 +1,21 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: pr-review
|
|
3
|
-
description: Review a GitHub pull request and post one formal review — advance the existing discussion
|
|
4
|
-
version:
|
|
3
|
+
description: Review a GitHub pull request and post one formal review — advance the existing discussion and give precision-first, high-signal feedback. A pure code review — no building. Use when asked to review a PR or on a cron PR scan.
|
|
4
|
+
version: 5.0.0
|
|
5
5
|
tags: [github, review, code-quality]
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
# PR Review
|
|
9
9
|
|
|
10
|
-
Review an open PR and post **one formal review
|
|
11
|
-
|
|
10
|
+
Review an open PR and post **one formal review** — high-signal findings only.
|
|
11
|
+
This is a **pure code review**: read the change and reason about it. Do **not**
|
|
12
|
+
install dependencies, build, or run tests — that is CI's job, and it validates
|
|
13
|
+
whether the change actually works far more reliably than you re-running it here.
|
|
14
|
+
Your job is judgement on the diff, not a build gate. A noisy review gets muted,
|
|
15
|
+
so precision matters more than volume.
|
|
12
16
|
|
|
13
|
-
This skill is the PR-specific procedure. It uses
|
|
14
|
-
|
|
15
|
-
the **code-review** skill for the finding tiers and what-to-check rubric.
|
|
17
|
+
This skill is the PR-specific procedure. It uses the **code-review** skill for
|
|
18
|
+
the precision bar and what-to-check rubric.
|
|
16
19
|
|
|
17
20
|
## Workspace
|
|
18
21
|
|
|
@@ -63,27 +66,32 @@ git diff --stat origin/<baseRef>...HEAD # churn
|
|
|
63
66
|
git diff origin/<baseRef>...HEAD # the patch
|
|
64
67
|
```
|
|
65
68
|
|
|
66
|
-
### 4.
|
|
69
|
+
### 4. Assess and submit
|
|
67
70
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
71
|
+
Apply the **code-review** skill's rubric — read each changed file in context;
|
|
72
|
+
check correctness / edge-cases / security / regression-risk / test-coverage.
|
|
73
|
+
Reason about the code statically; **don't build or run it** — trust CI to catch
|
|
74
|
+
what only running reveals, and spend your effort on what a human reviewer sees.
|
|
75
|
+
Follow that skill's **precision-first** rule: post **only Critical and Important**
|
|
76
|
+
findings, each with a `path:line` reference and a one-line concrete impact (what
|
|
77
|
+
breaks, for which input or caller) plus an inline code suggestion where it helps.
|
|
78
|
+
Drop Suggestions and Nits.
|
|
72
79
|
|
|
73
|
-
|
|
80
|
+
Before submitting, run the **confidence gate**: re-read each finding against the
|
|
81
|
+
actual code and try to refute it; drop any you can't defend against what the code
|
|
82
|
+
really does. A clean PR should be approved with few or no comments — that is a
|
|
83
|
+
good review, not a lazy one.
|
|
74
84
|
|
|
75
|
-
|
|
76
|
-
check correctness/edge-cases/security/regression-risk/test-coverage, and
|
|
77
|
-
categorise findings into the tiers. Then write the review:
|
|
85
|
+
Then write the review:
|
|
78
86
|
|
|
79
87
|
- One or two sentences on what the PR does.
|
|
80
|
-
-
|
|
81
|
-
suggestion where it helps.
|
|
88
|
+
- The surviving Critical/Important findings, each with its `path:line` + impact.
|
|
82
89
|
- For a complex PR, an impact note (affected paths, regression risks).
|
|
83
90
|
- An overall assessment, and thanks to the contributor.
|
|
84
91
|
|
|
85
92
|
Submit with `github_create_pull_request_review` (a **formal** review, not a plain
|
|
86
|
-
issue comment), event `APPROVE` / `REQUEST_CHANGES` / `COMMENT` to match
|
|
93
|
+
issue comment), event `APPROVE` / `REQUEST_CHANGES` / `COMMENT` to match what
|
|
94
|
+
survived the gate.
|
|
87
95
|
|
|
88
96
|
## Verification
|
|
89
97
|
|
package/workflows/pr-review.yaml
CHANGED
|
@@ -13,8 +13,9 @@ description: |
|
|
|
13
13
|
phases:
|
|
14
14
|
- name: review
|
|
15
15
|
label: Review
|
|
16
|
-
# pr-review: the PR procedure (primary).
|
|
17
|
-
#
|
|
18
|
-
|
|
16
|
+
# pr-review: the PR procedure (primary). code-review: the finding-tier +
|
|
17
|
+
# what-to-check rubric. No `building` — pr-review is a pure code review; CI
|
|
18
|
+
# validates that the change builds/runs.
|
|
19
|
+
skills: [pr-review, code-review]
|
|
19
20
|
model: "{{models.review}}"
|
|
20
21
|
variant: "{{variants.review}}"
|
|
@@ -76,7 +76,10 @@ wrong — STOP and fix it; do not ship a side-by-side that proves nothing.{{/if}
|
|
|
76
76
|
|
|
77
77
|
Author a `flow.json` (shape in the `browser-qa` skill) that scripts the
|
|
78
78
|
interaction like a director — record the baseline first, hold after state
|
|
79
|
-
changes, verify between steps.
|
|
79
|
+
changes (use `{"pause": 1200}` steps), verify between steps. Recording
|
|
80
|
+
auto-enables the driver's demo mode, so a **visible cursor** animates to each
|
|
81
|
+
target and a beat is held between steps — prefer `type` over `fill` so typing
|
|
82
|
+
shows on screen. Record with `--record-dir`:
|
|
80
83
|
|
|
81
84
|
```
|
|
82
85
|
node <browser-qa skill dir>/scripts/agent-browser.mjs run flow.json \
|
|
@@ -89,7 +92,7 @@ writing it into **`{{issueDir}}/demo.mp4`** (the harness harvests that dir):
|
|
|
89
92
|
```
|
|
90
93
|
<demo skill dir>/scripts/compose-demo.sh \
|
|
91
94
|
--output {{issueDir}}/demo.mp4 --title "<PR # — what it does>" \
|
|
92
|
-
--subtitle "<one line>" --layout single --speed 1
|
|
95
|
+
--subtitle "<one line>" --layout single --speed 1 --target-size-mb 5 \
|
|
93
96
|
/tmp/demo-cap/session.webm
|
|
94
97
|
```
|
|
95
98
|
|
|
@@ -10,6 +10,21 @@ THE ISSUE THIS BUILD WILL IMPLEMENT (use it to judge the escape hatch below):
|
|
|
10
10
|
SKIP CHECK — if {{issueDir}}/status.md already exists and contains
|
|
11
11
|
guardrails_status: READY, output "READY — guardrails already verified" and stop.
|
|
12
12
|
|
|
13
|
+
INSTALL DEPENDENCIES FIRST (do this before running ANY check command):
|
|
14
|
+
The harness pre-clones the repo but does NOT install dependencies, so the
|
|
15
|
+
test/lint/typecheck binaries (vitest, oxlint, tsc, eslint, ruff, …) will not
|
|
16
|
+
resolve until you install them. Detect the ecosystem and run the install:
|
|
17
|
+
- Node — pick the package manager from the lockfile: `pnpm-lock.yaml` → `pnpm install`,
|
|
18
|
+
`yarn.lock` → `yarn install`, `package-lock.json` (or none) → `npm ci`
|
|
19
|
+
(fall back to `npm install` if `npm ci` fails for a lockfile mismatch).
|
|
20
|
+
- Python — `pip install -e .` / `pip install -r requirements.txt`, or `poetry install` / `uv sync`.
|
|
21
|
+
- Rust — cargo fetches on first build; no separate step.
|
|
22
|
+
Only AFTER a successful install should you judge whether a command "runs". A
|
|
23
|
+
binary that is missing *after* dependencies installed cleanly is a real gap; a
|
|
24
|
+
binary missing *because deps were never installed* is NOT — do not BLOCK on it.
|
|
25
|
+
If the install itself fails (bad lockfile, missing manifest), that IS a blocking
|
|
26
|
+
guardrail — report it as such.
|
|
27
|
+
|
|
13
28
|
CHECK THESE GUARDRAILS:
|
|
14
29
|
|
|
15
30
|
1. **Test Framework** — Does the repo have a test runner (vitest, jest, pytest, cargo test, etc.)?
|