@petukhovart/agent-view 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/README.md +27 -5
- package/agents/verify-runner.md +77 -17
- package/package.json +1 -1
- package/skills/verify/SKILL.md +15 -8
- package/skills/verify-recipe/SKILL.md +136 -117
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-view",
|
|
3
3
|
"description": "Visual verification CLI for desktop apps (Electron/Tauri/Browser) via Chrome DevTools Protocol. Ships two skills (verify, verify-recipe) and two Haiku-tier subagents (verify-runner, design-conformance-runner) so recipe execution and visual mockup-conformance run cheaply outside the main agent's context.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.8.0",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"cdp",
|
|
7
7
|
"electron",
|
package/README.md
CHANGED
|
@@ -262,15 +262,37 @@ I added a `saving` ref and bound it to :disabled. Verify it works:
|
|
|
262
262
|
Claude will pick the right skill (usually `verify` ad-hoc mode), run a handful of `eval` / `dom --filter` / `console`
|
|
263
263
|
calls, and only screenshot if the visual claim needs it.
|
|
264
264
|
|
|
265
|
-
###
|
|
265
|
+
### Three-phase preconditions (0.8.0+)
|
|
266
266
|
|
|
267
|
-
Recipes split setup into
|
|
267
|
+
Recipes split setup into three phases, each with a different runner:
|
|
268
268
|
|
|
269
|
-
|
|
269
|
+
| Phase | Who runs it | Purpose | Idempotent? |
|
|
270
|
+
|---|---|---|---|
|
|
271
|
+
| `## Manual Preconditions` | Human (or main agent) | Only what `agent-view` physically can't (USB token, hardware, multi-machine state). Usually empty. | n/a |
|
|
272
|
+
| `## Bringup` | verify-runner | Conditional + idempotent setup: `if <state-check> is falsy → run actions → wait for state to settle`. Logins, mounting widgets, navigation, anything `agent-view` can drive deterministically. | YES per step |
|
|
273
|
+
| `## Machine Preconditions` | verify-runner | Pure state queries that confirm Bringup actually landed. No actions here. | trivially |
|
|
270
274
|
|
|
271
|
-
|
|
275
|
+
Bringup steps look like:
|
|
272
276
|
|
|
273
|
-
|
|
277
|
+
```markdown
|
|
278
|
+
### B1. Login if on auth screen
|
|
279
|
+
- if `agent-view eval "typeof window.__dev"` is not `"object"`:
|
|
280
|
+
agent-view fill --filter "Логин" "root"
|
|
281
|
+
agent-view fill --filter "Пароль" "$AGENTVIEW_PASSWORD"
|
|
282
|
+
agent-view click --filter "Войти"
|
|
283
|
+
wait for `agent-view eval "typeof window.__dev"` to be `"object"`, timeout 15s
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
The runner always evaluates the IF first (cheap). If the system is already in the target state → skip the actions, advance. Otherwise → run actions in order, then poll the post-condition until truthy or timeout. **Re-running a recipe when the app is ready costs ~3 seconds** (only state-checks, zero actions). Starting from auth screen costs ~15-25 seconds (one-time login + mount + navigate). Same recipe, both starting states, no manual intervention.
|
|
287
|
+
|
|
288
|
+
Failure modes are distinct:
|
|
289
|
+
- `bringup_failed` → bringup spec is wrong (the action commands ran but the post-condition didn't land — likely the programmatic API doesn't exist or the button text changed). Author error.
|
|
290
|
+
- `precondition_failed` → bringup ran AND post-conditions landed, but Machine Preconditions still false. Real environment issue (degraded mode, wrong user role, missing data).
|
|
291
|
+
- `cascading_failures` → bringup + preconditions OK, but Evidence Commands fail. Verified feature is broken or recipe is stale.
|
|
292
|
+
|
|
293
|
+
Without the three-phase split, all three look like "step N failed" and you can't tell them apart.
|
|
294
|
+
|
|
295
|
+
**Credentials in Bringup:** always use env-var references (`$AGENTVIEW_PASSWORD`), never inline a literal password. Recipes get committed to git. The `verify-recipe` skill warns when it sees a literal-looking password and offers to convert.
|
|
274
296
|
|
|
275
297
|
### Anti-patterns to avoid
|
|
276
298
|
|
package/agents/verify-runner.md
CHANGED
|
@@ -14,16 +14,24 @@ You are NOT a debugger, NOT a recipe author, and NOT an investigator. Do not pro
|
|
|
14
14
|
The parent agent will give you:
|
|
15
15
|
- `recipe_path` — absolute path to the recipe file (required)
|
|
16
16
|
- `window_id` — value to substitute for `$W` in commands (optional; if recipe needs it and not provided, run `agent-view discover` once and pick the main window)
|
|
17
|
-
- `mode` — `full` (default) or `dry_run`. Dry-run executes only Machine Preconditions
|
|
17
|
+
- `mode` — `full` (default) or `dry_run`. Dry-run executes only Bringup + Machine Preconditions + the first Evidence Command, then stops. Use this to validate a recipe before a full run.
|
|
18
18
|
- `extra_context` — anything else relevant (optional)
|
|
19
19
|
|
|
20
20
|
## Hard budgets (non-negotiable)
|
|
21
21
|
|
|
22
22
|
These exist to prevent the failure mode where you flail trying to make a broken recipe work:
|
|
23
23
|
|
|
24
|
-
|
|
25
|
-
- **`
|
|
24
|
+
### Bringup phase (Phase 1)
|
|
25
|
+
- **`bringup_max_total_commands: 15`** — across all bringup steps combined.
|
|
26
|
+
- **`bringup_max_wall_time_seconds: 60`** — total bringup wall time. If exceeded → abort with `bringup_timeout`.
|
|
27
|
+
- **`bringup_max_step_seconds: 10`** — per-step wait timeout overrides allowed; the recipe author writes `timeout 30s` and you respect it, but never exceed 60s/step.
|
|
28
|
+
|
|
29
|
+
### Evidence phase (Phase 3)
|
|
30
|
+
- **`max_tool_calls_per_step: 2`** — exactly the commands listed in a recipe step + at most one re-run if the first command crashed (transient CDP error). Never a third call. Never a different command.
|
|
31
|
+
- **`max_tool_calls_total: 30`** — across the whole evidence section. If you hit this, abort with `budget_exhausted` and report what's done.
|
|
26
32
|
- **`max_consecutive_failures: 3`** — three steps fail back-to-back → abort with `cascading_failures: probable preconditions wrong or recipe stale`. Do not continue hoping later steps will recover.
|
|
33
|
+
|
|
34
|
+
### All phases
|
|
27
35
|
- **`no_exploration: hard`** — you may NEVER run a Bash command that is not literally written in the recipe. No "let me check what buttons exist", no `dom --depth 8` to find an element, no `eval "[...document.querySelectorAll('button')]"` to map UI. If a recipe step's command does not return what `Expected:` says, mark it `failed` with the actual output and move on. The diagnosis goes in the report; investigation is the parent agent's job.
|
|
28
36
|
|
|
29
37
|
If you find yourself thinking "let me try X to find out why Y failed" — stop. That is exploration. Mark `failed`, write one sentence in `diagnosis`, continue.
|
|
@@ -34,24 +42,55 @@ If you find yourself thinking "let me try X to find out why Y failed" — stop.
|
|
|
34
42
|
|
|
35
43
|
Read the recipe with `Read`. Identify these sections:
|
|
36
44
|
- `## Manual Preconditions` — instructions for a human / parent agent. **You DO NOT execute these.** They appear in your report as context for the user, nothing more.
|
|
37
|
-
- `##
|
|
45
|
+
- `## Bringup` — conditional + idempotent setup steps you DO execute (see Phase 1 below). Optional — older recipes (0.6/0.7) lack this section; skip Phase 1 and proceed to Phase 2 if absent.
|
|
46
|
+
- `## Machine Preconditions` — runnable `agent-view` state checks. You execute these AFTER Bringup, BEFORE Evidence.
|
|
38
47
|
- `## Evidence Commands` — the meat of the recipe. Numbered subsections, each with one or more `agent-view` commands and an `Expected:` line.
|
|
39
48
|
- `## Design Conformance` — IGNORE. Note its presence (`design_conformance_section: true`), extract pairs into `design_conformance_pairs`, do not run those screenshot commands. The design-conformance-runner handles them.
|
|
40
49
|
|
|
41
|
-
If `## Machine Preconditions` is absent: the recipe is older-format. Skip Phase
|
|
50
|
+
If `## Machine Preconditions` is absent: the recipe is older-format. Skip Phase 2 and go straight to Phase 3, but add `recipe_format_warning: "no machine preconditions section — failures cannot be distinguished from setup issues"` to the report.
|
|
51
|
+
|
|
52
|
+
### Phase 1 — Bringup (idempotent setup)
|
|
53
|
+
|
|
54
|
+
Each Bringup step has the form:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
### B<N>. <step title>
|
|
58
|
+
- if `<eval-command>` is `<falsy-criterion>`:
|
|
59
|
+
<action command 1>
|
|
60
|
+
<action command 2>
|
|
61
|
+
...
|
|
62
|
+
wait for `<post-condition-eval>` to be `<truthy-criterion>`, timeout <Ns>
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Execute each step strictly in order. Per step:
|
|
66
|
+
|
|
67
|
+
1. **Run the IF condition** (`eval` or `dom --filter`). Always cheap, always runs. This costs 1 command from your bringup budget.
|
|
68
|
+
2. **Evaluate the falsy criterion.** If condition is NOT falsy (i.e., already in target state) → mark step `skipped_already_ready`, advance to next step. Zero action commands run.
|
|
69
|
+
3. **Otherwise, run each action command in order.** Do not skip, do not reorder, do not substitute. Each costs one command from the bringup budget.
|
|
70
|
+
4. **Then run the post-condition wait.** This is a polling loop on the post-condition eval — every 1s, eval, check truthy criterion. Exit when truthy or timeout. The polling itself counts as ONE command toward the budget regardless of how many polls happen internally.
|
|
71
|
+
5. **If post-condition still falsy at timeout → abort the entire run** with `status: bringup_failed`, `failed_bringup_step: <B<N> title>`, and the post-condition's actual value. Do NOT proceed to Phase 2.
|
|
72
|
+
6. **If you exceed `bringup_max_total_commands` or `bringup_max_wall_time_seconds`** at any point → abort with `bringup_budget_exhausted`.
|
|
73
|
+
|
|
74
|
+
**No exploration in Bringup either.** If a recipe action command fails (e.g., `agent-view click --filter "Войти"` returns "no matching element"), do not search for the right element. Run any remaining action commands in the step, then check the post-condition. If post-condition fails → abort. The recipe author got the action wrong; surface that, don't paper over it.
|
|
75
|
+
|
|
76
|
+
A successful Bringup step's outcome is determined by the post-condition becoming truthy, not by the action commands succeeding. Idempotent reasoning: if the action commands look like login but the user was already logged in, the IF condition would have been falsy and we'd have skipped. We're here only because the system was NOT in the target state.
|
|
42
77
|
|
|
43
|
-
|
|
78
|
+
After all Bringup steps complete: if the recipe has a final `### B<last>` step that is just a screenshot (e.g., `agent-view screenshot --window $W --scale 0.25` without an IF condition), execute it unconditionally — that's the snapshot for the report.
|
|
44
79
|
|
|
45
|
-
|
|
80
|
+
### Phase 2 — Machine Preconditions (state checks only)
|
|
81
|
+
|
|
82
|
+
Run each Machine Precondition command. Compare to its `must be ...` criterion. **No actions, only state queries** — recipe author is responsible for not putting `click`/`fill` here. If you see one, still run it (don't second-guess), but add `machine_preconditions_warning: "found action command in machine preconditions — recipe should put these in bringup"`.
|
|
83
|
+
|
|
84
|
+
If ANY precondition fails:
|
|
46
85
|
- Stop immediately. Do not run any Evidence Commands.
|
|
47
86
|
- Set `status: precondition_failed`.
|
|
48
87
|
- Set `failed_precondition` to the exact line that failed and its actual value.
|
|
49
|
-
-
|
|
88
|
+
- If the recipe has a `## Manual Preconditions` block, echo it verbatim into `manual_preconditions_to_check` for the user. If empty, omit.
|
|
50
89
|
- Return the report.
|
|
51
90
|
|
|
52
91
|
If all preconditions pass, proceed.
|
|
53
92
|
|
|
54
|
-
### Phase
|
|
93
|
+
### Phase 3 — Evidence Commands
|
|
55
94
|
|
|
56
95
|
Substitute `$W` with `window_id`. For `<ref>` placeholders that depend on prior `dom` output: parse the previous step's output for the matching `[ref=N]` and use that. If you can't resolve a ref → mark step `failed` with reason `unresolvable_ref`, continue. **Do not run extra `dom` calls to find the ref.**
|
|
57
96
|
|
|
@@ -64,9 +103,9 @@ Run each command. Capture stdout, stderr, exit code. Compare to `Expected:`:
|
|
|
64
103
|
|
|
65
104
|
Track consecutive failures. After 3 in a row → abort with `cascading_failures`.
|
|
66
105
|
|
|
67
|
-
If `mode: dry_run` → after Machine Preconditions + the FIRST Evidence Command, stop. Set `dry_run: true` in the report.
|
|
106
|
+
If `mode: dry_run` → after Bringup + Machine Preconditions + the FIRST Evidence Command, stop. Set `dry_run: true` in the report.
|
|
68
107
|
|
|
69
|
-
### Phase
|
|
108
|
+
### Phase 4 — return report
|
|
70
109
|
|
|
71
110
|
Return EXACTLY one fenced JSON block. No prose before or after.
|
|
72
111
|
|
|
@@ -78,15 +117,35 @@ Return EXACTLY one fenced JSON block. No prose before or after.
|
|
|
78
117
|
"finished_at": "<ISO8601>",
|
|
79
118
|
"mode": "full | dry_run",
|
|
80
119
|
"window_id": "<resolved>",
|
|
81
|
-
"status": "completed | precondition_failed | cascading_failures | budget_exhausted | malformed_recipe",
|
|
120
|
+
"status": "completed | bringup_failed | bringup_budget_exhausted | bringup_timeout | precondition_failed | cascading_failures | budget_exhausted | malformed_recipe",
|
|
82
121
|
"design_conformance_section": false,
|
|
83
122
|
"design_conformance_pairs": [],
|
|
84
123
|
"recipe_format_warning": "<only present if no machine preconditions section>",
|
|
124
|
+
"machine_preconditions_warning": "<only present if action commands found in machine preconditions>",
|
|
125
|
+
"bringup": {
|
|
126
|
+
"executed": true,
|
|
127
|
+
"steps": [
|
|
128
|
+
{
|
|
129
|
+
"label": "B1. Login if on auth screen",
|
|
130
|
+
"if_check": "agent-view eval ...",
|
|
131
|
+
"if_actual": "undefined",
|
|
132
|
+
"triggered": true,
|
|
133
|
+
"actions_run": ["agent-view fill ...", "agent-view click ..."],
|
|
134
|
+
"post_condition": "typeof window.__dev === 'object'",
|
|
135
|
+
"post_actual": "object",
|
|
136
|
+
"result": "done | skipped_already_ready | failed_post_condition | action_command_error",
|
|
137
|
+
"wall_time_ms": 3200
|
|
138
|
+
}
|
|
139
|
+
],
|
|
140
|
+
"snapshot_screenshot_path": "<path or null>",
|
|
141
|
+
"commands_used": 7,
|
|
142
|
+
"wall_time_ms": 14500
|
|
143
|
+
},
|
|
85
144
|
"machine_preconditions": [
|
|
86
145
|
{ "command": "agent-view eval ...", "criterion": "must be true", "actual": "true", "passed": true }
|
|
87
146
|
],
|
|
88
147
|
"failed_precondition": null,
|
|
89
|
-
"manual_preconditions_to_check": "<verbatim text, only if precondition_failed>",
|
|
148
|
+
"manual_preconditions_to_check": "<verbatim text, only if precondition_failed and recipe has Manual section>",
|
|
90
149
|
"summary": {
|
|
91
150
|
"total": 0,
|
|
92
151
|
"passed": 0,
|
|
@@ -94,8 +153,8 @@ Return EXACTLY one fenced JSON block. No prose before or after.
|
|
|
94
153
|
"requires_visual_review": 0,
|
|
95
154
|
"subjective": 0,
|
|
96
155
|
"skipped": 0,
|
|
97
|
-
"
|
|
98
|
-
"
|
|
156
|
+
"evidence_tool_calls_used": 0,
|
|
157
|
+
"evidence_tool_calls_budget": 30
|
|
99
158
|
},
|
|
100
159
|
"steps": [
|
|
101
160
|
{
|
|
@@ -115,13 +174,14 @@ Return EXACTLY one fenced JSON block. No prose before or after.
|
|
|
115
174
|
"blocking_issues": [
|
|
116
175
|
"<one-line summary of each failure or abort reason; empty array if everything passed>"
|
|
117
176
|
],
|
|
118
|
-
"abort_reason": "<only present when status != completed: cascading_failures | budget_exhausted | malformed_recipe — one sentence>"
|
|
177
|
+
"abort_reason": "<only present when status != completed: bringup_failed | cascading_failures | budget_exhausted | precondition_failed | malformed_recipe — one sentence>"
|
|
119
178
|
}
|
|
120
179
|
```
|
|
121
180
|
|
|
122
181
|
## Boundaries (re-stated for clarity)
|
|
123
182
|
|
|
124
|
-
- **No exploration.**
|
|
183
|
+
- **No exploration. Anywhere.** Bringup, Machine Preconditions, Evidence — all bound by the same rule. If a literal command from the recipe doesn't behave as expected, that's data for the report, not a prompt to investigate.
|
|
184
|
+
- **Bringup is idempotent BY DESIGN.** If you find yourself thinking "I'll just run the action commands without checking the IF condition because they're probably needed anyway" — stop. Always run the IF check first. Skipping is a valid outcome and the cheapest path through bringup.
|
|
125
185
|
- **No fix suggestions.** `diagnosis` is descriptive only ("returned 0, expected > 0"). Never "you should change X" or "try Y instead".
|
|
126
186
|
- **Truncate aggressively.** Stdout > 500 chars → truncate with `…[truncated, full output reproducible by re-running]`. Parent agent can re-run cherry-picked commands itself.
|
|
127
187
|
- **One JSON block, nothing else.** Anything you print outside the JSON wastes the parent agent's context — which is the entire reason you exist.
|
package/package.json
CHANGED
package/skills/verify/SKILL.md
CHANGED
|
@@ -187,8 +187,13 @@ Why: recipe execution is mechanical (run commands, compare to `Expected:`, repor
|
|
|
187
187
|
**Pre-flight check (do this BEFORE spawning the runner):**
|
|
188
188
|
|
|
189
189
|
1. `Read` the recipe.
|
|
190
|
-
2.
|
|
191
|
-
|
|
190
|
+
2. Determine the recipe format:
|
|
191
|
+
- Has `## Bringup` section → 0.8.0+ format. Bringup runs automatically; no user setup needed unless `## Manual Preconditions` is non-empty.
|
|
192
|
+
- Has `## Machine Preconditions` but no `## Bringup` → 0.7.0 format. The user must do all setup manually before this call.
|
|
193
|
+
- Neither → 0.6.x format. Tell the user once: "this recipe predates the precondition contract — failures may not be diagnostic. Run anyway, or regenerate the recipe via `verify-recipe`?"
|
|
194
|
+
3. **Manual Preconditions handling:**
|
|
195
|
+
- 0.8.0 + Manual section empty → skip user confirmation, go straight to spawn.
|
|
196
|
+
- Manual section non-empty → read each item to the user (tight one-liners) and ask: "are these set up right now?" Wait for confirmation. If no — stop, don't waste a runner spawn.
|
|
192
197
|
4. Resolve the window id once: `agent-view discover` → pick the main window's id.
|
|
193
198
|
|
|
194
199
|
**Spawn:**
|
|
@@ -197,16 +202,18 @@ Why: recipe execution is mechanical (run commands, compare to `Expected:`, repor
|
|
|
197
202
|
- Absolute `recipe_path`
|
|
198
203
|
- Resolved `window_id`
|
|
199
204
|
- `mode: full` (or `dry_run` if the user asked to validate the recipe first)
|
|
200
|
-
- Any extra context
|
|
205
|
+
- Any extra context the user provided
|
|
201
206
|
6. Wait for the JSON report.
|
|
202
207
|
|
|
203
208
|
**Handle the report:**
|
|
204
209
|
|
|
205
|
-
7. **If `status:
|
|
206
|
-
8. **If `status:
|
|
207
|
-
9. **If `
|
|
208
|
-
10. **If `
|
|
209
|
-
11. **If
|
|
210
|
+
7. **If `status: bringup_failed`** → the bringup spec is wrong: an action command failed to land its post-condition (e.g., `selectLocationAndFly` doesn't exist, login button text changed, programmatic API renamed). Surface `failed_bringup_step` and the post-condition's actual value. Suggest re-authoring the broken Bringup step via `verify-recipe`. Do NOT click around manually trying to make it work.
|
|
211
|
+
8. **If `status: bringup_budget_exhausted` or `bringup_timeout`** → bringup is taking too long. Either an animation is slower than the recipe expects (extend `timeout` on the offending step), or bringup is fighting an unexpected app state. Show the bringup transcript and ask the user.
|
|
212
|
+
9. **If `status: precondition_failed`** → bringup completed but state still wrong. Could be: (a) bringup is incomplete (missing a step), (b) a Machine Precondition is overly strict, (c) real environment issue. Relay both the `bringup` block (showing what bringup did) and the `failed_precondition` so the user sees the full picture.
|
|
213
|
+
10. **If `status: cascading_failures` or `budget_exhausted`** → the recipe Evidence is likely stale (selectors/refs changed, UI restructured). Show the first 1-2 failed Evidence steps and ask whether to update the recipe via `verify-recipe` or investigate manually.
|
|
214
|
+
11. **If `design_conformance_section: true`** → also spawn `design-conformance-runner` in parallel with the `design_conformance_pairs` array. Merge both reports.
|
|
215
|
+
12. **If `requires_visual_review` steps** have no design ref attached → open each screenshot yourself with `Read` and decide pass/fail.
|
|
216
|
+
13. **If individual `failed` steps** in an otherwise-completed run → re-run that specific failing command yourself for richer evidence. Do not re-execute the whole recipe.
|
|
210
217
|
|
|
211
218
|
Output to the user: a tight summary — what passed, what failed, what needs visual review, and (if any) which design conformance issues to fix. Do not paste the raw JSON unless asked.
|
|
212
219
|
|
|
@@ -10,36 +10,43 @@ You help the developer author a disciplined, cheapest-first verification recipe
|
|
|
10
10
|
|
|
11
11
|
## What this produces
|
|
12
12
|
|
|
13
|
-
A file at `.claude/verify-recipes/<kebab-slug>.md`
|
|
13
|
+
A file at `.claude/verify-recipes/<kebab-slug>.md` with three precondition phases plus the verification body:
|
|
14
14
|
|
|
15
|
-
- **MANUAL PRECONDITIONS** —
|
|
16
|
-
- **
|
|
17
|
-
- **
|
|
18
|
-
- **EVIDENCE COMMANDS** — ordered `agent-view` calls, cheapest first, each annotated with what it proves
|
|
19
|
-
- **POSITIVE-CASE ASSERTIONS**
|
|
20
|
-
- **REGRESSION CHECKS** — adjacent paths that must not have broken
|
|
21
|
-
- **DESIGN CONFORMANCE** (optional) — screenshot ↔ design reference pairs for the design-conformance-runner
|
|
15
|
+
- **MANUAL PRECONDITIONS** — only what `agent-view` physically cannot do (rare; usually empty)
|
|
16
|
+
- **BRINGUP** — idempotent setup steps the verify-runner executes itself: `if <state-check> is falsy → run actions → wait for state to settle`. Drives the app from auth screen / fresh shell / mid-state into the exact state the recipe needs. Re-runs cost almost nothing because each step skips when its condition is already met.
|
|
17
|
+
- **MACHINE PRECONDITIONS** — pure state queries that confirm Bringup actually settled. No actions here. Any false → runner aborts.
|
|
18
|
+
- **EVIDENCE COMMANDS** — ordered `agent-view` calls, cheapest first, each annotated with what it proves.
|
|
19
|
+
- **POSITIVE-CASE ASSERTIONS** + **REGRESSION CHECKS** + optional **DESIGN CONFORMANCE**.
|
|
22
20
|
|
|
23
21
|
Create the directory if missing: `mkdir -p .claude/verify-recipes`
|
|
24
22
|
|
|
25
|
-
## Why
|
|
23
|
+
## Why three precondition phases
|
|
26
24
|
|
|
27
|
-
|
|
25
|
+
| Phase | Who runs it | What it does | Idempotent? |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| `## Manual Preconditions` | Human (or main agent) | Only what `agent-view` physically can't (USB token, hardware setup, multi-machine state) | n/a |
|
|
28
|
+
| `## Bringup` | verify-runner | `if state-check is falsy → action → wait`. Logins, mounting widgets, navigation — anything `agent-view` can do deterministically | YES (per step) |
|
|
29
|
+
| `## Machine Preconditions` | verify-runner | Pure `eval` / `dom --filter` state queries. NO actions. | trivially |
|
|
28
30
|
|
|
29
|
-
|
|
31
|
+
**Why split Bringup from Machine Preconditions?** A failure tells you a different thing in each:
|
|
32
|
+
- A failed Bringup post-condition → bringup spec is wrong (e.g., `selectLocationAndFly` doesn't exist; "Войти" button text changed). Author error.
|
|
33
|
+
- A failed Machine Precondition after Bringup succeeded → bringup THINKS it set up the state but the state isn't there (e.g., login appeared to succeed but the user role is wrong, the app is in a degraded mode). Real environment issue.
|
|
34
|
+
- A failed Evidence Command after both passed → real bug in the verified feature.
|
|
30
35
|
|
|
31
|
-
|
|
36
|
+
Without the split, all three look like "step N failed" and the developer can't tell which.
|
|
37
|
+
|
|
38
|
+
**Why keep Manual at all?** Some things really can't be automated cheaply: physical USB-key auth, multi-machine setups, long-running data migrations. List them — but be honest, this section should usually be empty.
|
|
32
39
|
|
|
33
40
|
## Methodology
|
|
34
41
|
|
|
35
42
|
Frame the recipe with **hard-debug** discipline. Apply this chain in authoring mode:
|
|
36
43
|
|
|
37
|
-
1. Start from
|
|
38
|
-
2. Convert vague expectations ("looks right") into measurable signals ("store.user.role === 'admin'")
|
|
39
|
-
3. Prefer the cheapest tool that can answer the question — a value check costs ~50 tokens, a screenshot costs ~6 000
|
|
40
|
-
4. Include at least one negative-case check (the old symptom must no longer appear)
|
|
41
|
-
5. Include at least one regression check (an adjacent flow must still work)
|
|
42
|
-
6. **Every
|
|
44
|
+
1. Start from the broadest possible starting state — recipe should run from auth screen, from logged-in shell, or from "already in the feature" without modification.
|
|
45
|
+
2. Convert vague expectations ("looks right") into measurable signals ("`store.user.role === 'admin'`").
|
|
46
|
+
3. Prefer the cheapest tool that can answer the question — a value check costs ~50 tokens, a screenshot costs ~6 000.
|
|
47
|
+
4. Include at least one negative-case check (the old symptom must no longer appear).
|
|
48
|
+
5. Include at least one regression check (an adjacent flow must still work).
|
|
49
|
+
6. **Bringup steps must be conditional + idempotent.** Every action wrapped in `if <state-check> is falsy`. Every action followed by `wait for <post-condition> to be truthy`. Re-running the recipe must be a no-op when the app is already ready.
|
|
43
50
|
|
|
44
51
|
## Tool-cost decision tree
|
|
45
52
|
|
|
@@ -56,29 +63,46 @@ Pick the first row that can answer the question. Only go lower when the row abov
|
|
|
56
63
|
| Canvas / WebGL scene state | `agent-view scene --diff` | DOM is empty for canvas apps |
|
|
57
64
|
|
|
58
65
|
**Anti-patterns to reject:**
|
|
59
|
-
-
|
|
60
|
-
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
-
|
|
65
|
-
-
|
|
66
|
+
- Putting UI clicks (`click`, `fill`) in `## Machine Preconditions`. Actions go in `## Bringup`. Machine Preconditions are state queries only.
|
|
67
|
+
- Bringup steps without an IF condition (unconditional actions). They break idempotency — re-running a "click Login" step when already logged in fails. Always wrap.
|
|
68
|
+
- Bringup steps without a post-condition wait. Without it, the runner advances before the action settled, and the next step's IF check sees stale state.
|
|
69
|
+
- Manual Preconditions for anything `agent-view` can do. Drag widget into cell? Often there's a programmatic API — interview the developer for it before defaulting to Manual.
|
|
70
|
+
- Hardcoded credentials in Bringup commands. Use env-var references (`"$AGENTVIEW_PASSWORD"`) — never inline a password, even for dev/masterkey accounts. Recipes get committed to git.
|
|
71
|
+
- Opening Evidence Commands with a screenshot to "see the state" — use `dom --filter` or `eval` first.
|
|
72
|
+
- "Check that it looks right" — every Evidence assertion must be a concrete pass/fail criterion. The single legitimate exception is the `## Design Conformance` section.
|
|
73
|
+
- Inventing design reference paths (`.figma-refs/...`) when the developer did not provide them. No refs → no Design Conformance section.
|
|
66
74
|
|
|
67
75
|
## Workflow
|
|
68
76
|
|
|
69
77
|
### Step 1 — gather context (interview the developer)
|
|
70
78
|
|
|
71
|
-
When invoked, ask
|
|
79
|
+
When invoked, ask in plain text. Wait for response before continuing.
|
|
80
|
+
|
|
81
|
+
**Block A — what's being verified:**
|
|
82
|
+
|
|
83
|
+
1. What was shipped or fixed? (feature name or bug description)
|
|
84
|
+
2. What was the original symptom or expected behavior?
|
|
85
|
+
3. Any known failure mode or edge case to cover?
|
|
86
|
+
|
|
87
|
+
**Block B — bringup: how the runner gets the app into a state where verification makes sense:**
|
|
88
|
+
|
|
89
|
+
4. **Authentication** — does the app require login before the feature is reachable? If yes:
|
|
90
|
+
- What identifies "logged out" vs "logged in" in JS state? (e.g., `typeof window.__dev`, `!!store.user`)
|
|
91
|
+
- Login flow: which fields/buttons (`fill --filter "..."`, `click --filter "..."`) and what credentials? **Use an env-var, never inline.** Suggest a name like `AGENTVIEW_PASSWORD` and tell the user to `export` it in their shell or `.env.local`.
|
|
92
|
+
5. **UI mode / view requirements** — does the feature live behind a specific app mode/view that must be active before it appears (e.g., "map view, not settings panel"; "edit mode, not view mode"; "modal X must be open")? For each:
|
|
93
|
+
- State check that proves this mode is active (e.g., `document.querySelector('.cesium-widget') !== null` for map mode).
|
|
94
|
+
- Action(s) to enter this mode if not active. Prefer programmatic API (e.g., `eval "store.setMode('map')"`) over UI clicks. If only UI works, list the click sequence.
|
|
95
|
+
6. **Setup actions** — beyond modes, what other setup must happen (drag widget into cell, navigate camera to a location, open a dialog)? For each:
|
|
96
|
+
- Programmatic API if it exists (e.g., `gis-widget-root.selectLocationAndFly(uuid)`, `workspace.addWidget(...)`). Prefer this — much more reliable than UI automation.
|
|
97
|
+
- State check that proves the setup happened.
|
|
98
|
+
- Wait timing: how long does this take to settle? (camera animation, async data load).
|
|
72
99
|
|
|
73
|
-
|
|
74
|
-
2. **What was the original symptom or expected behavior?**
|
|
75
|
-
3. **Any known failure mode or edge case to cover?**
|
|
76
|
-
4. **UI mode requirements**: Does this feature live behind a specific app mode/view that must be active before it appears (e.g., "map view, not settings panel"; "edit mode, not view mode"; "modal X must be open")? List every mode-toggle the user must have done.
|
|
77
|
-
5. **Manual setup steps**: Beyond modes, what physical actions must the user do before checks can run (drag a widget into a cell, search for and select a map location, open a specific dialog, log in as a particular role)? Be precise — these become Manual Preconditions verbatim.
|
|
78
|
-
6. **State assertions for each manual step**: For each item in (4) and (5), is there a JS expression or DOM selector that proves it happened? Examples: "after dragging the GIS widget — `pinia._s.get('gis-widget-root')?.cesiumReadyFlag` becomes true"; "after entering map mode — `document.querySelector('.cesium-widget')` exists; "after selecting a location — `selectedLocation` is truthy". If the developer doesn't know offhand, that's fine — ask them to point you at the store/composable/component where state lives and you can suggest expressions.
|
|
79
|
-
7. **(Optional) Design references**: Local image paths to compare screenshots against (Figma exports, hand-off PNGs). **Only local files are supported.** If none — skip the Design Conformance section entirely.
|
|
100
|
+
If the developer doesn't know whether a programmatic API exists, ask them to point you at the relevant store/composable/component file — you can read it and find one (or confirm none).
|
|
80
101
|
|
|
81
|
-
|
|
102
|
+
**Block C — verification body and visual:**
|
|
103
|
+
|
|
104
|
+
7. State assertions for the verified feature itself — JS expressions that prove it works (separate from bringup setup checks).
|
|
105
|
+
8. **(Optional) Design references** — local image paths to compare screenshots against (Figma exports, hand-off PNGs). **Only local files are supported.** If none — skip the Design Conformance section.
|
|
82
106
|
|
|
83
107
|
### Step 2 — draft the recipe
|
|
84
108
|
|
|
@@ -91,20 +115,54 @@ Generated: <date>
|
|
|
91
115
|
Scope: <one sentence describing what this covers>
|
|
92
116
|
|
|
93
117
|
## Manual Preconditions
|
|
94
|
-
<!--
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
118
|
+
<!-- ONLY for things agent-view physically cannot do. Usually empty. -->
|
|
119
|
+
<!-- The verify-runner does NOT execute these. -->
|
|
120
|
+
(none — bringup handles everything)
|
|
121
|
+
|
|
122
|
+
## Bringup
|
|
123
|
+
<!-- The verify-runner executes these. Each step is conditional + idempotent: skipped if already in target state. -->
|
|
124
|
+
<!-- Format per step: -->
|
|
125
|
+
<!-- ### B<N>. <one-line title> -->
|
|
126
|
+
<!-- - if `<eval>` is `<falsy criterion>`: -->
|
|
127
|
+
<!-- <action command 1> -->
|
|
128
|
+
<!-- <action command 2> -->
|
|
129
|
+
<!-- wait for `<post-condition eval>` to be `<truthy criterion>`, timeout <Ns> -->
|
|
130
|
+
|
|
131
|
+
### B1. Login if on auth screen
|
|
132
|
+
- if `agent-view eval "typeof window.__dev"` is not `"object"`:
|
|
133
|
+
agent-view fill --window $W --filter "Логин" "root"
|
|
134
|
+
agent-view fill --window $W --filter "Пароль" "$AGENTVIEW_PASSWORD"
|
|
135
|
+
agent-view click --window $W --filter "Войти"
|
|
136
|
+
wait for `agent-view eval "typeof window.__dev"` to be `"object"`, timeout 15s
|
|
137
|
+
|
|
138
|
+
### B2. Mount the GIS widget if not mounted
|
|
139
|
+
- if `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.cesiumReadyFlag"` is `false`:
|
|
140
|
+
agent-view eval "window.__dev.pinia._s.get('workspace').addWidget({type:'gis', cell:0})"
|
|
141
|
+
wait for `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.cesiumReadyFlag"` to be `true`, timeout 10s
|
|
142
|
+
|
|
143
|
+
### B3. Exit settings mode if active
|
|
144
|
+
- if `agent-view eval "document.querySelector('.cesium-widget') !== null"` is `false`:
|
|
145
|
+
agent-view click --window $W --filter "Готово"
|
|
146
|
+
wait for `agent-view eval "document.querySelector('.cesium-widget') !== null"` to be `true`, timeout 5s
|
|
147
|
+
|
|
148
|
+
### B4. Fly to first sublocation
|
|
149
|
+
- if `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.selectedLocation"` is `false`:
|
|
150
|
+
agent-view eval --await "window.__dev.pinia._s.get('gis-widget-root').selectLocationAndFly(window.__dev.mwStore.nvgn.sublocations.value[0].id)"
|
|
151
|
+
wait for `agent-view eval "(()=>{const w=document.querySelector('.cesium-widget'); const cw=Object.values(w).find(v=>v?.scene); return Math.round(window.Cesium.Cartographic.fromCartesian(cw.scene.camera.position).height);})()"` to be `< 5000`, timeout 30s
|
|
152
|
+
|
|
153
|
+
### B5. Snapshot for the report (always runs)
|
|
154
|
+
agent-view screenshot --window $W --scale 0.25
|
|
98
155
|
|
|
99
156
|
## Machine Preconditions
|
|
100
|
-
<!--
|
|
101
|
-
- `agent-view eval "window.__dev
|
|
157
|
+
<!-- Pure state checks. NO actions. The verify-runner aborts with `precondition_failed` if any return false. -->
|
|
158
|
+
- `agent-view eval "typeof window.__dev"` → must be `"object"`
|
|
102
159
|
- `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.cesiumReadyFlag"` → must be `true`
|
|
103
|
-
- `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.selectedLocation"` → must be `true`
|
|
104
160
|
- `agent-view eval "document.querySelector('.cesium-widget') !== null"` → must be `true`
|
|
161
|
+
- `agent-view eval "!!window.__dev.pinia._s.get('gis-widget-root')?.selectedLocation"` → must be `true`
|
|
162
|
+
- `agent-view eval "({locations:window.__dev.mwStore.nvgn.locations.value.length, sublocations:window.__dev.mwStore.nvgn.sublocations.value.length, zones:window.__dev.mwStore.nvgn.zones.value.length, objects:window.__dev.mwStore.nvgn.objects.value.length})"` → all four counts must be `> 0`
|
|
105
163
|
|
|
106
164
|
## Narrowed Signal
|
|
107
|
-
<!-- The one measurable thing that proves
|
|
165
|
+
<!-- The one measurable thing that proves the verified feature works -->
|
|
108
166
|
`<agent-view command>` must return `<expected value>`.
|
|
109
167
|
|
|
110
168
|
## Evidence Commands
|
|
@@ -126,114 +184,75 @@ Cost: ~<N> tokens
|
|
|
126
184
|
- [ ] <adjacent flow> — `agent-view <command>` → `<expected>`
|
|
127
185
|
|
|
128
186
|
## Design Conformance
|
|
129
|
-
<!--
|
|
130
|
-
<!-- Each row pairs a screenshot command with the expected reference image path. -->
|
|
131
|
-
<!-- The design-conformance-runner subagent reads this section, runs the screenshot commands, and visually compares against the expected refs. -->
|
|
132
|
-
<!-- Do NOT invent design ref paths. Use exactly what the developer provided. -->
|
|
187
|
+
<!-- Include ONLY if the developer provided design refs in question 8. -->
|
|
133
188
|
|
|
134
189
|
| Step Label | Screenshot Command | Expected Reference |
|
|
135
190
|
|---|---|---|
|
|
136
191
|
| <area name e.g. "filter panel collapsed"> | `agent-view screenshot --crop "<area>" --scale 0.5` | `<absolute path to expected PNG/JPEG>` |
|
|
137
|
-
| <area name> | `agent-view screenshot --window $W --scale 0.5` | `<absolute path>` |
|
|
138
192
|
|
|
139
|
-
Tolerance: `normal` (default
|
|
193
|
+
Tolerance: `normal` (default).
|
|
140
194
|
|
|
141
195
|
## Anti-patterns avoided
|
|
142
|
-
- <note any recipe-specific traps, e.g. "
|
|
196
|
+
- <note any recipe-specific traps, e.g. "B4 relies on `selectLocationAndFly` from gis-widget-root — verified to exist as of <date>; if it's removed, B4 falls back to Manual Precondition">
|
|
143
197
|
````
|
|
144
198
|
|
|
145
199
|
### Step 3 — save the file
|
|
146
200
|
|
|
147
|
-
Determine a kebab-slug from the feature/fix name
|
|
201
|
+
Determine a kebab-slug from the feature/fix name. Save to `.claude/verify-recipes/<slug>.md`. Create the directory first if it doesn't exist. Confirm the path to the developer.
|
|
202
|
+
|
|
203
|
+
### Step 4 — credentials warning (if Bringup contains login)
|
|
148
204
|
|
|
149
|
-
|
|
205
|
+
If Bringup includes a login step:
|
|
150
206
|
|
|
151
|
-
|
|
207
|
+
> Recipe uses `$AGENTVIEW_PASSWORD` for login. Make sure that env var is set in your shell or `.env.local` before running verify (`export AGENTVIEW_PASSWORD=<value>`). The recipe never contains the literal password.
|
|
152
208
|
|
|
153
|
-
|
|
209
|
+
If the developer insisted on inlining a literal password earlier, warn:
|
|
154
210
|
|
|
155
|
-
|
|
211
|
+
> ⚠️ Recipe contains a literal password on line N. This will be committed to git. Consider replacing with `$AGENTVIEW_PASSWORD` (env-var). Want me to convert it now? (yes/no)
|
|
156
212
|
|
|
157
|
-
|
|
213
|
+
### Step 5 — offer dry-run validation
|
|
214
|
+
|
|
215
|
+
After saving, ask:
|
|
216
|
+
|
|
217
|
+
> The recipe is saved at `<path>`. Is the app running? If yes, I can spawn `verify-runner` in `dry_run` mode — it'll execute Bringup + Machine Preconditions + the first Evidence Command (~10-20 commands total). That validates the recipe isn't broken before you commit a full run. Want me to run the dry-run? (yes/no)
|
|
158
218
|
|
|
159
219
|
If yes:
|
|
160
220
|
1. Get the window id with `agent-view discover`.
|
|
161
221
|
2. Spawn `verify-runner` via the Agent tool with `mode: dry_run`, the recipe path, and the window id.
|
|
162
222
|
3. Read the JSON report.
|
|
163
|
-
4.
|
|
164
|
-
5.
|
|
165
|
-
6.
|
|
223
|
+
4. **`status: bringup_failed`** → relay `failed_bringup_step` and the post-condition's actual value. Suggest revising the bringup step (most often: action commands wrong, programmatic API doesn't exist, or post-condition criterion mistuned).
|
|
224
|
+
5. **`status: precondition_failed`** → bringup ran but state isn't there. Either bringup is incomplete, or a state check is overly strict. Show both bringup outcome and the failing precondition.
|
|
225
|
+
6. **`status: completed` and dry-run passed** → recipe is healthy. Confirm and stop.
|
|
166
226
|
|
|
167
|
-
If no — confirm
|
|
227
|
+
If no — confirm path and stop.
|
|
168
228
|
|
|
169
|
-
## Worked example:
|
|
229
|
+
## Worked example: GIS feature with full bringup automation
|
|
170
230
|
|
|
171
231
|
**Developer input:**
|
|
172
|
-
>
|
|
232
|
+
> Verifying selective filter changes (commits 0c5f1b1ee, 4d0c89467, 4c99c1591) in the GIS widget. App requires login (root/masterkey via middleware). GIS widget mounts via `workspace.addWidget({type:'gis', cell:0})`. Camera flies via `gis-widget-root.selectLocationAndFly(id)`. Map view is the default after widget mount.
|
|
173
233
|
|
|
174
234
|
**Recipe produced:**
|
|
175
235
|
|
|
176
|
-
|
|
177
|
-
# Verify: Login Redirect Fix
|
|
178
|
-
|
|
179
|
-
Generated: 2026-04-27
|
|
180
|
-
Scope: Confirms that a successful login routes to /dashboard, not /home, and that the store mutation fires correctly.
|
|
236
|
+
(see the template above — it's already the worked example for this scenario)
|
|
181
237
|
|
|
182
|
-
|
|
183
|
-
1. App running, user logged out, browser at the `/login` route.
|
|
184
|
-
|
|
185
|
-
## Machine Preconditions
|
|
186
|
-
- `agent-view eval "router.currentRoute.path"` → must be `"/login"`
|
|
187
|
-
- `agent-view eval "!!store.state.auth.user"` → must be `false` (logged out)
|
|
238
|
+
**Saved to:** `.claude/verify-recipes/nvgn-gis-improvements.md`
|
|
188
239
|
|
|
189
|
-
|
|
190
|
-
`agent-view eval "router.currentRoute.path"` must return `"/dashboard"` after sign-in.
|
|
191
|
-
|
|
192
|
-
## Evidence Commands
|
|
193
|
-
|
|
194
|
-
### 0. Setup — baseline console
|
|
195
|
-
```bash
|
|
196
|
-
agent-view console --clear
|
|
240
|
+
**Dry-run output (first run after auth screen):**
|
|
197
241
|
```
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
agent-view dom --filter "Email" --depth 2
|
|
204
|
-
agent-view fill <email-ref> "admin@example.com"
|
|
205
|
-
agent-view fill <password-ref> "password"
|
|
206
|
-
agent-view click <signin-ref>
|
|
207
|
-
agent-view watch "router.currentRoute.path" --until "router.currentRoute.path === '/dashboard'"
|
|
242
|
+
status: completed
|
|
243
|
+
mode: dry_run
|
|
244
|
+
bringup: 5 steps, 12 commands, 18s — B1 (login) + B2 (mount) + B4 (fly) all triggered; B3 skipped (already map mode); B5 screenshot saved
|
|
245
|
+
machine_preconditions: 5/5 passed
|
|
246
|
+
steps: 1/1 passed
|
|
208
247
|
```
|
|
209
|
-
Expected: `replace / "/login" → "/dashboard"` in watch output
|
|
210
|
-
Cost: ~150 tokens
|
|
211
248
|
|
|
212
|
-
|
|
213
|
-
```bash
|
|
214
|
-
agent-view eval "store.state.auth.redirectPath"
|
|
249
|
+
**Dry-run output (second run, app already in target state):**
|
|
215
250
|
```
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
agent-view console --level error,warn
|
|
251
|
+
status: completed
|
|
252
|
+
mode: dry_run
|
|
253
|
+
bringup: 5 steps, 5 commands (4 IF-checks + 1 screenshot), 3s — all 4 conditional steps skipped_already_ready
|
|
254
|
+
machine_preconditions: 5/5 passed
|
|
255
|
+
steps: 1/1 passed
|
|
222
256
|
```
|
|
223
|
-
Expected: `(no console messages)`
|
|
224
|
-
Cost: ~30 tokens
|
|
225
|
-
|
|
226
|
-
## Positive-Case Assertions
|
|
227
|
-
- [ ] `router.currentRoute.path` === `/dashboard` after login
|
|
228
|
-
- [ ] `store.state.auth.redirectPath` === `/dashboard`
|
|
229
|
-
- [ ] No console errors during the flow
|
|
230
|
-
|
|
231
|
-
## Regression Checks
|
|
232
|
-
- [ ] Logout → `/login` still works (`agent-view click <logout-ref>` then `agent-view eval "router.currentRoute.path"` → `"/login"`)
|
|
233
|
-
|
|
234
|
-
## Anti-patterns avoided
|
|
235
|
-
- Not using screenshot to confirm route (route is a string — eval is 120× cheaper)
|
|
236
|
-
- watch used before eval so the route change is confirmed to have settled, not just sampled mid-transition
|
|
237
|
-
````
|
|
238
257
|
|
|
239
|
-
|
|
258
|
+
This is what idempotent bringup buys you: 5x speedup on re-runs, zero manual intervention either way.
|