@vpxa/aikit 0.1.341 → 0.1.343

package/scaffold/dist/definitions/skills/browser-use.mjs

@@ -16,7 +16,7 @@ argument-hint: "URL or browser task description"
 
 Use AI Kit browser for JS-rendered pages, auth barriers, forms, screenshots, network capture, storage/cookies, and web app verification.
 
-> **Token efficiency**: agent-browser snapshots use ~200-400 tokens vs ~3000-5000 for full DOM. Always prefer read mode 'snapshot' over raw DOM or screenshots when the page content is sufficient.
+> **Token efficiency**: agent-browser snapshots use ~200-400 tokens vs ~3000-5000 for full DOM. Always prefer read mode 'snapshot' over raw DOM or screenshots when the page content is sufficient. \`includeSnapshot\` on open/navigate auto-returns the ARIA tree, saving one \`read\` round-trip.
 
 ## Tool Choice
 
@@ -28,18 +28,33 @@ Start with web_fetch/http for static content. Use browser when:
 
 Use headless by default. Use UI only for user-visible auth or visual debugging. Do not use system browser for agent-visible evidence.
 
+## Capabilities
+
+| Enhancement | Action | How it works |
+|---|---|---|
+| **Auto-return snapshot** | \`open\`, \`navigate\` | Default \`includeSnapshot: true\` — response includes ARIA tree, saving one \`read\` call. Set \`false\` for speed. |
+| **Annotated screenshots** | \`screenshot\` | Add \`annotate: true\` — overlays numbered \`[N]\` labels matching snapshot ref IDs. Bridges ARIA tree to visual. |
+| **Console + eval hybrid** | \`console\` | Add \`expression: 'document.title'\` to console.get — evaluates JS alongside console entries in one round-trip. |
+| **Ref ID snapshot format** | \`open\`, \`navigate\` | Accessibility tree shows \`[BUTTON] [e5] Sign In\` — prominent ref markers for agent targeting. |
+
 ## Recipes
 
 Open/read:
-1. browser open with url and waitUntil.
-2. browser read snapshot/text/markdown.
-3. Use eval only for data unavailable through accessibility/DOM reads.
+1. \`browser({ action: 'open', url, includeSnapshot: true })\` — opens page, auto-returns ARIA tree with ref IDs.
+2. Use \`browser read\` only when you need a different readMode (dom/markdown/text) or after the first auto-snapshot.
+3. Use \`eval\` only for data unavailable through accessibility/DOM reads.
 
 Interact:
-1. Read snapshot.
-2. Act by stable role/name/ref/selector.
+1. Read snapshot (or use auto-returned snapshot from open/navigate).
+2. Act by stable ref (e.g., \`[e5]\`), selector, or label.
 3. Re-read or diff.
-4. Capture screenshot when visual result matters.
+4. Capture annotated screenshot with \`annotate: true\` — numbered labels match snapshot ref IDs, bridging tree to visual.
+
+Console + Eval:
+1. \`browser({ action: 'console', consoleSubAction: 'enable' })\` to start capture.
+2. Perform page actions.
+3. \`browser({ action: 'console', consoleSubAction: 'get', expression: 'document.title' })\` — gets console entries AND evaluates JS in one round-trip.
+4. Use \`expression\` for quick page state queries without a separate eval call.
 
 Auth:
 1. Detect login/SSO/CAPTCHA.
@@ -84,16 +99,14 @@ Each recipe step declares:
 ### 1. Form Login
 
 ~~~text
-Step 1: Open login page
-  Action: browser({ action: 'open', url: '<login-url>', mode: 'ui' })
-  Verify: Page contains login form (read snapshot, check for username/password fields)
+Step 1: Open login page (auto-returns snapshot)
+  Action: browser({ action: 'open', url: '<login-url>', mode: 'ui', includeSnapshot: true })
+  Verify: Page contains login form (check auto-returned snapshot for username/password fields)
   On Failure: URL may have changed — check redirects
-  Extract: pageId
+  Extract: pageId, snapshot (auto-returned with ref IDs like [TEXTFIELD] [e3] [BUTTON] [e5])
 
-Step 2: Read form structure
-  Action: browser({ action: 'read', pageId, readMode: 'snapshot' })
-  Verify: Found username field, password field, submit button
-  On Failure: Try readMode 'dom' for hidden fields
+Step 2: Inspect form structure (from auto-returned snapshot)
+  Note: Snapshot already returned in Step 1 — no separate read needed unless different mode
   Extract: field refs (@username, @password, @submit)
 
 Step 3: Enter credentials
@@ -148,10 +161,10 @@ Step 5: Export session (optional)
 ### 3. Data Extraction
 
 ~~~text
-Step 1: Navigate to data page
-  Action: browser({ action: 'open', url: '<data-url>', mode: 'headless' })
-  Verify: Page loaded successfully
-  Extract: pageId
+Step 1: Navigate to data page (auto-returns snapshot)
+  Action: browser({ action: 'open', url: '<data-url>', mode: 'headless', includeSnapshot: true })
+  Verify: Page loaded successfully (check auto-returned snapshot)
+  Extract: pageId, snapshot
 
 Step 2: Wait for dynamic content
   Action: browser({ action: 'navigate', pageId, type: 'waitFor', selector: '<data-container>' })
@@ -159,7 +172,10 @@ Step 2: Wait for dynamic content
   On Failure: Increase timeout, check if JS is needed
 
 Step 3: Extract structured data
+  # Option A: standalone eval
   Action: browser({ action: 'eval', pageId, code: 'document.querySelectorAll("<selector>").map(...)' })
+  # Option B: console + eval hybrid (single round-trip)
+  Action: browser({ action: 'console', consoleSubAction: 'get', pageId, expression: 'JSON.stringify(dataObjects)' })
   Verify: Got expected data structure
   On Failure: Try different selector, use read(dom) to inspect structure
   Extract: structured data
@@ -244,9 +260,10 @@ Step 3: Check console for JS errors
   Verify: No critical JS errors
   Extract: error list (if any)
 
-Step 4: Screenshot for visual regression
-  Action: browser({ action: 'screenshot', pageId, fullPage: true })
-  Extract: screenshot for comparison
+Step 4: Annotated screenshot for visual regression
+  Action: browser({ action: 'screenshot', pageId, fullPage: true, annotate: true })
+  Note: Annotated screenshot overlays numbered [N] labels matching snapshot ref IDs
+  Extract: screenshot for visual comparison
 ~~~
 
 ### 7. Local File Preview
@@ -325,9 +342,9 @@ Browser-based authentication strategies for different auth mechanisms.
 **When:** Simple username/password form without SSO.
 
 **Steps:**
-1. \`open\` login page
-2. \`read\` snapshot to find form fields
-3. \`act\` type credentials (ask user for input via terminal)
+1. \`open\` login page with \`includeSnapshot: true\` — auto-returns ARIA tree with form field ref IDs
+2. Inspect auto-returned snapshot for form fields (ref IDs like \`[TEXTFIELD] [e3]\`)
+3. \`act\` type credentials (ask user for input)
 4. \`act\` click submit
 5. \`read\` to verify authenticated state
 
@@ -422,22 +439,26 @@ browser({ action: 'eval', pageId, code: 'localStorage.getItem("authToken")' })
 ## Quick Reference
 
 | Need | Action | Key params |
-|---|---|---|
-| Open a page | \`open\` | url, mode (ui/headless) |
+|---|---|---|---|
+| Open a page (with auto-snapshot) | \`open\` | url, mode, includeSnapshot (default true) |
 | Read page content | \`read\` | readMode: snapshot/dom/markdown/text |
 | Click/type/interact | \`act\` | kind: click/type/press/hover/select |
-| Wait for something | \`navigate\` | type: waitFor, selector |
+| Wait for something | \`navigate\` | type: waitFor, selector, includeSnapshot (default true) |
 | Check network calls | \`network\` | subAction: enable -> get |
 | Get cookies/storage | \`session\` | sessionAction: cookies/get-storage |
-| Take screenshot | \`screenshot\` | fullPage, selector |
+| Take screenshot (annotated) | \`screenshot\` | fullPage, selector, annotate (adds numbered labels) |
+| Console + eval hybrid | \`console\` | consoleSubAction: get, expression (evaluates JS) |
 | Compare changes | \`diff\` | (compares to previous snapshot) |
 
 For full parameter details: \`describe_tool('browser')\`
 
 ## Principles
 
+- Prefer \`includeSnapshot: true\` (default on open/navigate) — auto-returns ARIA tree, saves one round-trip vs explicit \`read\`.
 - Prefer \`read\` over \`screenshot\` — snapshots are structured (ARIA tree), searchable, token-efficient.
-- Always \`read\` after \`act\` — actions don't return page state.
+- Prefer \`annotate: true\` on screenshots — numbered labels match snapshot ref IDs, bridging tree to visual.
+- Prefer console + eval hybrid (\`console.get\` with \`expression\`) over separate \`console.get\` then \`eval\` — single round-trip.
+- Always \`read\` or use \`includeSnapshot\` after \`act\` — actions don't return page state.
 - Use \`diff\` instead of re-reading after an action — shows only what changed.
 - Network capture BEFORE navigation — enable then navigate. Captures start from enable time.
 - One page = one task — fresh pages avoid state contamination.
@@ -456,11 +477,17 @@ Always read(snapshot) first to get refs before interacting.
 Direct sequential browser() calls for one-off tasks and debugging:
 
 \`\`\`
-// Open -> Read -> Act -> Read loop
+// Open (with auto-snapshot) -> Act -> Read/Console
 browser({ action: 'open', url, mode: 'headless' })
-browser({ action: 'read', pageId })
+// snapshot auto-returned — no separate read needed unless different mode
 browser({ action: 'act', pageId, kind: 'click', ref: '@login-button' })
 browser({ action: 'read', pageId })  // verify state changed
+
+// Console + eval hybrid pattern:
+browser({ action: 'console', consoleSubAction: 'enable' })
+browser({ action: 'act', pageId, kind: 'click', ref: '@submit' })
+browser({ action: 'console', consoleSubAction: 'get', expression: 'document.title' })
+// returns console entries AND evaluated title in one response
 \`\`\`
 
 **Network Intelligence pattern:**
@@ -471,11 +498,14 @@ browser({ action: 'network', pageId, subAction: 'get' })
 browser({ action: 'network', pageId, subAction: 'export-har' })
 \`\`\`
 
-**Console capture:**
+**Console capture (with optional eval):**
 \`\`\`
 browser({ action: 'console', pageId, consoleSubAction: 'enable' })
 // ... trigger page actions ...
+// Get console entries only:
 browser({ action: 'console', pageId, consoleSubAction: 'get', level: 'error' })
+// Get console entries + evaluate JS (single round-trip):
+browser({ action: 'console', pageId, consoleSubAction: 'get', expression: 'JSON.stringify(window.__DATA__)' })
 \`\`\`
 
 ## Session Management