@petukhovart/agent-view 0.15.0 → 0.17.0

package/skills/verify/references/commands.md

@@ -11,6 +11,7 @@ listing call — always expand from the most recent one.
 - [Interaction](#interaction) — `click`, `fill`, `drag`
 - [Modals & file pickers](#modals--file-pickers-dialog-upload) — `dialog`, `upload`
 - [Waiting](#waiting-wait) — `wait`
+- [Step protocol](#step-protocol-act) — `act`
 - [Screenshots](#screenshots) — `screenshot`
 - [Runtime state](#runtime-state-eval) — `eval`
 - [Reactive state](#reactive-state-watch) — `watch`
@@ -48,6 +49,15 @@ agent-view dom --diff                   # Lines changed since last dom call (+ a
 
 `--count` skips tree output and ref mutations — cheapest way to assert "element exists N times" without loading the full tree into context.
 
+**Test ids.** A node carrying a test id prints it: `button "Save" [testid=save-btn] [ref=12]`. An
+unnamed wrapper that carries one is printed too (`generic [testid=email] [ref=6]`), because
+component libraries put the id on the wrapper. The attribute is `testIdAttribute` from
+`agent-view.config.json`; without it `data-testid`, `data-test-id`, `data-test`, `data-qa` and
+`data-cy` are all read. A test id survives HMR, navigation and text changes; a ref does not. Once
+`dom` has shown a test id, address the element with `--testid` (below). Limit: an element absent
+from the AX tree prints no line. An inline `<span data-testid>` holding only text prints as its
+`StaticText`, without the id. `--testid` still finds it.
+
 ### Interaction
 ```bash
 agent-view click <ref>                  # Click element by ref from dom output
@@ -55,21 +65,40 @@ agent-view click --filter "Save"        # Find element by text and click
 agent-view click --pos 100,200          # Click by coordinates — CANVAS ONLY, see below
 agent-view click <ref> --double         # Double-click (fires dblclick handlers); works with --filter / --pos too
 agent-view click <ref> --right          # Right-click (fires contextmenu); works with --filter / --pos too
+agent-view click --testid save-btn      # By test id (see "Test ids" above)
+agent-view click --selector "tr:nth-child(3) button"  # By CSS selector
 agent-view fill <ref> "text"            # Type into input field
+agent-view fill --testid email "a@b.c"  # A test id on a wrapper resolves to the input/textarea inside
 agent-view drag --from <ref> --to <ref>          # Drag element to another element by ref
 agent-view drag --from-pos 50,80 --to-pos 200,300  # Drag by coordinates (for canvas / Pixi)
 agent-view drag --from <ref> --to <ref> --steps 25 --hold-ms 60  # Smoother movement, longer hold
+agent-view drag --from-pos 385,303 --to-pos 1250,589 --cancel  # HTML5: dragCancel instead of drop
+agent-view drag --from-pos 385,303 --to-pos 1250,589 --html5   # Fail unless an HTML5 drag started
 ```
 
-`drag` dispatches `mousePressed` → N × `mouseMoved` → `mouseReleased` via CDP. Endpoints can mix
+`drag` presses and moves via CDP; if Chromium starts an HTML5 drag (`draggable=true`) it takes it
+over with `Input.dispatchDragEvent` and drops with the app's real `dataTransfer`, otherwise it is a
+plain pointer drag. The first output line says which: `via html5 drop` followed by one line per
+MIME type (`application/json: "..."`) — check the payload is the app's, not a text selection — or
+`via pointer` with a warning when no HTML5 drag started. Start HTML5 drags on a cell without a
+control: an input/select/button under the cursor swallows `dragstart`. Endpoints can mix
 ref and coordinate (e.g. `--from <ref> --to-pos 400,300`). For canvas/Pixi targets always use
 `--from-pos`/`--to-pos` — derive the centroid via `agent-view eval` from the scene graph.
 Refs are resolved fresh on each call, so window resizes between snapshots are tolerated.
 Increase `--steps` for handlers using `globalpointermove` so intermediate frames are not skipped.
 
+`--testid` / `--selector` act on the **first visible** match; hidden copies (`v-show`, a closed
+popover) are skipped. When several match, the output says so — `Clicked testid "row" (first visible
+of 3)` — and the address is too broad: narrow it, e.g. a `--selector` with `:nth-child`. Exit 1 with
+`No element matches testid "x"` when nothing matches, and with `None of 2 element(s) matching
+testid "x" is visible` when every match is hidden. Pass exactly one of `<ref>`, `--filter`,
+`--testid`, `--selector`, `--pos`. `fill` exits 1 with `No input or textarea at or inside the
+element` when the element holds no field.
+
 **Coordinates are a last resort.** `--pos` / `--from-pos` / `--to-pos` exist for canvas and WebGL,
-where no ref exists. On DOM the correct order is `dom --filter "<text>"` → take the `[ref=N]` →
-`click <ref>`, or `click --filter "<text>"` in one step. A coordinate pair breaks on any layout
+where no ref exists. On DOM, use `--testid` when `dom` shows one. Otherwise run
+`dom --filter "<text>"` → take the `[ref=N]` → `click <ref>`, or `click --filter "<text>"` in one
+step. A coordinate pair breaks on any layout
 shift, scroll, zoom, or window resize, and it clicks whatever now sits at that point — silently.
 If you reach for `--pos` on a DOM element, first say why the ref was not usable.
 
@@ -141,25 +170,117 @@ an IPC channel) are out of reach — CDP does not see the main process.
 agent-view wait --filter "Saved"                    # until the text appears in the AX tree
 agent-view wait --filter "Saved" --timeout 20       # max wait in seconds (default 10)
 agent-view wait --filter "Row 5" --window "Main"    # specific window
+agent-view wait --testid order-saved                # until an element with this test id is visible
+agent-view wait --selector ".toast.success"         # until a selector match is visible
 ```
 
-Exits as soon as the element appears; exits non-zero on timeout — so `&&` after it is a real gate.
+Exactly one of `--filter`, `--testid`, `--selector`. Exits as soon as the element appears; exits
+non-zero on timeout — so `&&` after it is a real gate.
+A mounted but hidden match (`v-show`, `display:none`, `visibility:hidden`) does not end a
+`--testid` / `--selector` wait. On timeout the error says whether nothing matched or every match
+was hidden.
 
 **Never sleep for a fixed time** — see the waiting-signal table in `SKILL.md`. If no condition-based
 wait fits, poll `dom --filter X --count` with an explicit attempt cap and report the attempt count.
 
 
+### Step protocol (`act`)
+
+For a driver that only ever answers `op n [text]`. Each call acts, waits for the UI to settle,
+checks the done condition, and prints a fresh numbered control table. The done condition is
+agent-view's to check, never the driver's: `act start` needs exactly one of `--until-testid`,
+`--until-selector`.
+
+```bash
+agent-view act start --until-testid workspace-root   # new session for this CDP port; prints the table
+agent-view act start --until-selector ".dashboard" --max-steps 20   # budget, default 30
+agent-view act table                    # re-snapshot, print the table
+agent-view act click 3                  # row numbers refer to the LAST printed table
+agent-view act type 1 "admin"           # fill
+agent-view act select 4 "Monthly"       # native <select> only
+agent-view act scroll down              # wheel ~0.8 viewport at its centre; also `up`
+agent-view act do "type 2 root" "type 3 secret" "click 1"   # several ops, all numbered from ONE table; stops at the first BLOCKED/DONE/error
+agent-view act drag 5                   # pointer-drag row 5 onto the viewport centre
+agent-view act drag 5 2                 # … onto the centre of row 2
+agent-view act drag 7 testid=video-panel right   # … onto any visible element by test id, 20 px inside its right edge
+agent-view act drag 7 center left       # … near the viewport's left edge; edge: left|right|top|bottom|center (default)
+agent-view act wait                     # no action: settle up to 3 s, print the table (not a step)
+agent-view act save login               # writes ~/.agent-view/scratch/login.json, prints the path
+agent-view act start --until-testid x --save login   # same, written automatically on DONE (one turn fewer)
+AGENT_VIEW_SECRET=… agent-view act replay login   # re-run it with no model; exit 0 DONE, 1 FAIL, 3 STALE, 2 error
+```
+
+Table — interactive controls visible in the viewport, renumbered from 1 on every print, capped at
+150 rows (`… N more below — act scroll down`). Named `alert` / `status` nodes (a snackbar) are listed
+too, so the driver sees why a form came back. A field wrapper that boxes a field of its own role
+(Vuetify `v-field` around its input) is folded into the inner one. After the AX rows come `item`
+rows: elements with no interactive role but a test id and computed `cursor: pointer` (a
+`<div @pointerdown>` widget-bar entry), so they can be clicked or dragged; one inside a control is
+that control. Password values print as `••••`, or `empty`:
+
+```
+act step 2/30 · 14 controls · until testid "workspace-root"
+[1] textbox "Логин" testid=login-input · "admin"
+[2] textbox "Пароль" testid=password · ••••
+[3] button "Войти" testid=login-btn
+[4] checkbox "Запомнить" · checked
+```
+
+First line of an op's output:
+
+| Line | Meaning |
+|---|---|
+| `✓ click [3] button "Войти" · 240ms` | acted and settled; the new table follows |
+| `✓ click [3] button "Войти" · window reloaded · 900ms` | the window reloaded under the action (login); settled on the new document |
+| `✓ drag [7] item "ГИС" testid=nav__widgetbar__item-GisWidget → testid=video-panel right · 310ms` | dragged: `→` names the target and edge |
+| `DONE: until testid "workspace-root" · 3 steps · 4.1s` | until condition visible; nothing follows |
+| `BLOCKED: [3] button "Войти" is gone` | row no longer on screen, no unique match; fresh table follows, nothing was done |
+| `BLOCKED: [3] covered by div.overlay testid=spinner` | something else receives the click; fresh table follows, nothing was done |
+| `BLOCKED: step budget 30 exhausted` | `--max-steps` reached |
+
+`act wait` runs even after the budget is spent. Errors (non-zero exit):
+
+- `` run `agent-view act start` first ``, `act start --until-testid <id> | --until-selector <css> — exactly one`
+- `` this act run is DONE — `agent-view act start` for a new one `` (any step after DONE; `table` / `wait` still work)
+- `No row [n] in the last table (1-N)`
+- `[n] is a <role>, not a text field — pick a textbox/combobox row` (`type` on a non-text row);
+  `No input or textarea at or inside the element` (`type` on a native `<select>` — use `select`)
+- `not a native select — click it`, `No option "x" in [n]`
+- `act do: "…" — each step is click|type|select <n> [text]` (checked before any step runs; a
+  failing step prints the `✓` lines of the ones before it)
+- `act scroll <up|down>`, `act drag <n> [to] [edge] — edge is one of left|right|top|bottom|center`
+
+The saved script names each control by test id, else role + name — never by row number — and
+stores no password. `act replay` runs it inside the server: each step waits (50 ms polls, 10 s cap)
+only for its own control to be on screen, enabled and uncovered, then acts; a window reload
+mid-run is ridden over. Its one line, ending in the total time:
+
+| Line | Exit | Meaning |
+|---|---|---|
+| `DONE: replay login · 2 steps in 0.1s, then testid "x" · 1.8s` | 0 | until met; the steps took 0.1 s, the app the rest |
+| `FAIL: replay login — steps ran, testid "x" not visible after 15s · 16.2s` | 1 | the app did not answer — likely a bug |
+| `FAIL: step 2/2 click button "Войти" testid=login-btn — still disabled after 10s · 10.4s` | 1 | the control is there but stayed `disabled` / `covered by …` — likely a bug |
+| `STALE: step 2/2 click button "Войти" — not on screen within 10s · 10.3s` | 3 | the script no longer fits the app — re-record it |
+| `STALE: step 3/3 drag target testid=video-panel not found · 10.5s` | 3 | same, for a drop target |
+| `STALE: step 1/2 select combobox "Период" — no option "Monthly" · 0.2s` | 3 | same, for an option (or `not a native select`) |
+| `STALE: step 2/3 click button "Закрыть" — <CDP error> · 0.4s` | 3 | the control vanished between finding and acting |
+
+Exit 2, message on stderr: the replay could not run — `No saved script "x"`, `"x" types a password —
+set AGENT_VIEW_SECRET`, `"x" has no done condition`.
+
 ### Screenshots
 ```bash
-agent-view screenshot --scale 0.5              # Recommended: JPEG at half-res (~3× fewer vision tokens)
+agent-view screenshot --scale 0.5              # Recommended: half-res PNG (~3× fewer vision tokens)
 agent-view screenshot --scale 0.5 --window <id>  # Specific window
 agent-view screenshot --crop "Sidebar"         # Crop to element bounding box (~1.6k tokens — 12× win)
 agent-view screenshot --crop "Chart" --scale 0.5  # Crop + scale (stacks)
 agent-view screenshot --crop "Active bookings" --crop-up 1  # Crop the card, not just its heading
+agent-view screenshot --testid order-card      # Crop to the element with this test id
+agent-view screenshot --selector ".chart"      # Crop to the first visible selector match
 agent-view screenshot                          # Full-res PNG (expensive: ~19k tokens at 1920×1080)
 ```
 
-`--crop <filter>` resolves the element with the same filter syntax as `dom --filter`, then crops the screenshot to its bounding box. Prefer `--crop` over full-window screenshots whenever you only need to inspect a specific section. Falls back to full-window with a stderr warning if the filter matches nothing.
+`--crop <filter>` resolves the element with the same filter syntax as `dom --filter`, then crops the screenshot to its bounding box. Prefer `--crop` over full-window screenshots whenever you only need to inspect a specific section. Falls back to full-window with a stderr warning if the filter matches nothing. `--testid` / `--selector` crop the same way and take `--crop-up`, but a miss exits 1 instead: an exact address that misses is a wrong address, and a full-window capture would spend ~19k tokens on it. Pass at most one of `--crop`, `--testid`, `--selector`.
 
 A text filter usually matches the text-bearing node, so cropping on a section title returns a thin strip of that title. `--crop-up <n>` climbs `n` element ancestors before cropping — use `1` (sometimes `2`) to get the surrounding card/section. When a crop comes back text-sized, the command says so on stderr.