iframer-toolkit 0.10.6 → 0.11.0

package/skills/iframer.md

@@ -10,9 +10,12 @@ You are an expert at using the iframer browser automation toolkit. The user want
 - `iframer.credentials` — list or store login credentials (NEVER re-store as recovery)
 - `iframer.execute` — run a browser pipeline (navigate, login, screenshot, snapshot, etc.)
 - `iframer.browse` — fast headless page fetch (no pipeline needed)
-- `iframer.session` — stop (save state) or clear session data
+- `iframer.session` — stop (save state, close the browser), clear (wipe session data), restart (recover from a freeze or crash), capture-start / capture-stop (record XHR/fetch), get-cookies, get-auth
 - `iframer.status` — system health, modes, stored credentials
 - `iframer.reverse-engineer` — capture API calls a site makes
+- `iframer.tabs` — list/open tabs in the user's REAL Chrome (needs the extension)
+- `iframer.remember` — per-domain element anchors, targeted as `@a:<name>` in any selector
+- `iframer.clipboard` — read/write the machine's clipboard (same one the user's Chrome uses)
 
 ## Core rules
 
@@ -22,6 +25,8 @@ You are an expert at using the iframer browser automation toolkit. The user want
 4. **NEVER pass explicit selectors to the login step** unless the auto-detect failed AND you took a snapshot to find the right ones. The login step auto-detects email, password, submit, and OTP fields.
 5. If headless detects Imperva or Akamai, immediately stop headless and pivot to headed in the same run; do not wait for a block, retry headless, or defer the switch to a future request. Use headed from the outset when the provider is already known in the current task. On detection, switch to headed and use respectful pacing (at least 3 seconds between top-level actions; no parallel bursts or repeated blocked retries). Do not announce the provider or explain the switch unless the user asks why. Only when asked, explain that the site is covered by Imperva or Akamai and iframer adjusts its browser mode and pacing to respect it.
 6. **Let iframer handle mode escalation.** Don't hardcode modes. If you must pick one, prefer no mode (auto-select) or `binary-headful` for sites known to block headless.
+7. **"My browser" / "my tabs" / "I'm already logged in there" means extension mode, never a launched browser.** See the section below. A launched browser has none of the user's logins and opening one is a visible, unwanted window on their desktop.
+8. **Every browser you open is yours to close.** Call `session stop` when the work is done, and if a call opened a browser by mistake, close it immediately instead of leaving it. `status` lists live windows.
 
 ## How to build pipelines
 
@@ -33,27 +38,34 @@ The `execute` tool takes a `steps` array. Each step is an object with a `type` f
 |------|--------|-------------|
 | `navigate` | `url`, `waitUntil?` | Go to URL |
 | `login` | `domain` | Auto-detect form, fill stored credentials, handle 2FA. Handles email-first flows (Slack, Microsoft, Google) and standard email+password flows. |
-| `snapshot` | — | Get interactive elements as structured list with @e refs. **Do this BEFORE interacting.** |
+| `snapshot` | — | Get interactive elements as structured list with @e refs. **Do this BEFORE interacting.** Fields no human can reach (honeypots pushed offscreen/clipped/transparent by an ancestor) are omitted; bait-looking ones (`website`, `fax`, tabindex=-1) are marked `[honeypot?]` — leave those empty unless the task clearly needs them. |
 | `find` | `role?`, `name?`, `text?`, `placeholder?`, `label?` | Find a specific element, returns a ref |
 | `screenshot` | `annotate?` | Take a screenshot. `annotate: true` overlays numbered badges. |
-| `click` | `selector` | Click element (use @e refs from snapshot) |
-| `fill` | `selector`, `value` | Fill input (use @e refs) |
+| `click` | `selector` | Click element (use @e refs from snapshot/find). Text selectors (`text=`, `:has-text()`, `:text-is()`, `>>`) are rejected: `find` by text, then click the ref. |
+| `right-click` | `selector` or `x`, `y` | Right-click an element or coordinates |
+| `fill` | `selector`, `value`, `force?` | Fill input (use @e refs). Refuses honeypot fields (unreachable by a human) with an explanatory error — skip the field. `force: true` bypasses, only when you are certain it is real. |
 | `human-click` | `selector` or `x`, `y` | Human-like click with random offset |
-| `human-type` | `selector`, `value` | Human-like typing with variable delays |
+| `human-type` | `selector`, `value`, `clear?` | Human-like typing with variable delays. Appends by default; `clear: true` replaces the existing text. |
 | `scroll` | `deltaY?` | Scroll the page |
 | `wait` | `ms` | Wait N milliseconds |
 | `wait-for` | `selector`, `timeout?` | Wait for element to appear |
 | `evaluate` | `expression` | Run JS, return result |
 | `extract` | `expression` | Run JS, include result in response |
-| `keyboard` | `key` | Press a key (Enter, Escape, Tab, etc.) |
-| `solve-captcha` | — | Auto-detect and solve reCAPTCHA or hCaptcha |
+| `keyboard` | `key`, `meta?`, `ctrl?` | Press a key (Enter, Escape, Tab, etc.). Cmd/Ctrl+A selects all everywhere; other Cmd/Ctrl shortcuts are refused in extension mode. |
+| `select` | `selector`, `value` | Pick an option in a native `<select>`, with real change events. Never set `.value` via `evaluate`. |
+| `type-code` | `value`, `selector?` | Type a verification code digit by digit. Default target: `input[type="tel"]`. |
+| `upload` | `selector`, `files` | Set absolute local file paths on an `<input type=file>` |
+| `paste` | `selector?` | Paste the OS clipboard into a field, or the focused element if omitted. More reliable than a ⌘V `keyboard` step. |
+| `download` | `url`, `path?` | Fetch a file with the browser's cookies and save it server-side, no Save-As dialog. Default path: `~/.iframer/downloads/<name>`. |
+| `solve-captcha` | — | Best-effort reCAPTCHA or hCaptcha solve with vision AI. Works in any mode; needs `ANTHROPIC_API_KEY`. Fails the run with `captcha-unsolvable` if it can't solve it. |
+| `recaptcha` | `action`, `tiles?` | Manual captcha control: `info`, `click`, `answer`, `select`, `verify`, `solve`. The full workflow arrives in the result when a captcha blocks a run. |
 
 ### Pipeline options
 
 | Option | Default | Description |
 |--------|---------|-------------|
-| `mode` | auto | `headless`, `binary-headful`, `docker-headful`. Omit for auto-select + auto-escalation. |
-| `captureApi` | false | Record all XHR/fetch requests. Use for reverse-engineering. |
+| `mode` | auto | `headless`, `binary-headful`. Omit for auto-select + auto-escalation. |
+| `captureApi` | false | Record all XHR/fetch requests. Use for reverse-engineering, and when a form submit fails with only a vague on-page message: every HTTP >= 400 response is reported per step, so you read the server's real answer (e.g. `POST /api/apply → 422: {"error":"..."}`) instead of guessing. |
 | `staleTimeoutMs` | 20000 | Per-step stale-state timeout |
 | `continueOnError` | false | Don't abort pipeline on step failure |
 | `screenshotAfterEach` | false | Screenshot after every step (expensive) |
@@ -154,6 +166,24 @@ execute: steps: [{ type: "click", selector: "@e3" }, { type: "screenshot" }]
 
 The more interactions (scroll, click tabs, navigate sub-pages), the more API calls get captured. Results feed into the knowledge cache automatically.
 
+### Reverse-engineer inside a browser iframer doesn't control (T3 Code, Playwright MCP, ...)
+
+When you already drive a browser through your own tools (a T3 Code preview tab, Playwright MCP, Chrome DevTools MCP), don't launch iframer's browser: load the in-page kit into that tab. It records the page's own calls on the user's real session and exports shapes, never values. Use the CLI for this, not the MCP tools.
+
+```
+1. iframer kit --for t3          # or --for playwright-mcp / --for any. Prints the exact tool calls.
+2. Follow the printed install steps (upload the kit file, stash it, top-level eval). Never paste the kit's source.
+3. Drive the site with your normal tools. After EVERY full page load, re-run the printed eval step (step 4).
+4. window.__ifk.dump(), then: iframer ingest --latest --typed
+   -> ./<host>-api/api.json, client.js, client.d.ts, plus the knowledge cache. The capture file is deleted.
+```
+
+- Replays run IN the page: `window.__ifk.call('GET','/path')`, or load the generated `client.js` (`iframer kit --for t3 --client ./<host>-api/client.js` prints the steps). HttpOnly cookies never leave the browser, so curl replays fail.
+- Writes need `{approved:true}`: ask the user first. Replays stop for good on 403, 429 or a captcha page. Do not retry or work around it.
+- Sites that detect tampering: `iframer kit --mode passive` (no patching; URLs and form posts only).
+- Report coverage as the kit states it. It is never complete: workers, redirect hops and cross-origin iframes are invisible.
+- `ingest` refuses (exit 2) if anything value-like slipped into the capture. Don't edit the file to force it through; tell the user what was found.
+
 ### Handle captchas
 
 Captcha handling depends on the browser mode:
@@ -168,16 +198,13 @@ Captcha handling depends on the browser mode:
 execute: steps: [{ type: "screenshot" }]  // verify captcha is gone, then continue
 ```
 
-**Docker-headful** (remote browser via noVNC):
-- Use the `solve-captcha` step — it auto-detects and solves reCAPTCHA/hCaptcha via vision AI.
+**Headless, or nobody at the screen**:
+- If you hit a captcha in headless, iframer auto-escalates to binary-headful.
+- If nobody can solve it by hand, try the `solve-captcha` step. It's best effort: it uses vision AI, needs `ANTHROPIC_API_KEY`, and fails on many challenges.
 ```
 execute: steps: [{ type: "solve-captcha" }, { type: "screenshot" }]
 ```
 
-**Headless**:
-- Captchas can't be solved in headless. If you hit one, iframer auto-escalates to binary-headful.
-- If it still fails, switch to docker-headful and use `solve-captcha`.
-
 ### Handle OTP / 2FA codes
 
 If the login step encounters a code field and no TOTP secret is stored:
@@ -198,7 +225,7 @@ When the browser crashes, freezes, or you get connection errors:
 ```
 session: { action: "restart" }
 ```
-This kills ALL running browser instances (local + Docker) and resets state. Credentials and knowledge cache are NOT affected. The next `execute` call launches a fresh browser automatically.
+This kills ALL running browser instances and resets state. Credentials and knowledge cache are NOT affected. The next `execute` call launches a fresh browser automatically.
 
 **The execute tool also auto-recovers:** if it detects a *connection-level* crash (connection closed, ECONNREFUSED/ECONNRESET/EPIPE, socket hang up), it restarts the browser server and retries once before reporting failure. Timeouts are deliberately NOT auto-retried — a slow-but-healthy pipeline that hit the client timeout must not be re-run, or non-idempotent steps (form submits, purchases) would double-execute. If auto-recovery fails, or a step timed out, call `session restart` manually and continue from where you left off (don't blindly re-run the whole pipeline).
 
@@ -229,11 +256,11 @@ knowledge: { action: "clear", domain: "example.com" }
 
 If an `execute` call takes a very long time (30s+), it's probably stuck on one of:
 
-1. **Captcha blocking progress** — the login/navigate step can't proceed because a captcha appeared. In binary-headful mode, tell the user to solve it in the visible window. In docker-headful mode, add a `solve-captcha` step.
+1. **Captcha blocking progress** — the login/navigate step can't proceed because a captcha appeared. In binary-headful mode, tell the user to solve it in the visible window. If nobody can, add a `solve-captcha` step.
 
 2. **Waiting for a page transition that never happens** — the site didn't redirect after form submission. Take a screenshot to see the current state, then decide what to do (wrong button clicked, error message on page, etc.).
 
-3. **Stale-state timeout** — iframer's stale monitor will abort after `staleTimeoutMs` (default 20s). The error will say "stale-state" and include a screenshot. Read the screenshot to understand what the page looks like.
+3. **Stale-state timeout** — iframer's stale monitor will abort after `staleTimeoutMs` (default 20s). The error will say "stale-state" and include a screenshot. Read the screenshot to understand what the page looks like. If the step targeted a selector that no longer exists, the error is reported as `element-not-found` instead ("No element matches …"): the page re-rendered it away. Re-snapshot right before acting and retry — do not change styles or theorise about visibility.
 
 **Recovery pattern for all stalls:**
 ```
@@ -242,7 +269,7 @@ If an `execute` call takes a very long time (30s+), it's probably stuck on one o
 
 2. If captcha visible:
    - binary-headful → ask user to solve it in the visible window
-   - docker-headful → execute: steps: [{ type: "solve-captcha" }]
+   - nobody at the screen → execute: steps: [{ type: "solve-captcha" }]
 
 3. If the page shows an error or unexpected state:
    - Take a snapshot to get element refs
@@ -253,11 +280,9 @@ If an `execute` call takes a very long time (30s+), it's probably stuck on one o
 
 **CRITICAL: Don't restart from scratch.** The browser session persists between execute calls (same daemon, same page context). If you were mid-login and hit a captcha, you don't need to navigate back to the login page — you're still there. Just solve the captcha and continue.
 
-**Timeouts:** Docker-headful is slower than local modes — Xvfb + VNC + Chrome share one container. Heavy SPAs (Slack, Figma) can take 30+ seconds to load. If an execute call times out:
+**Timeouts:** Heavy SPAs (Slack, Figma) can take 30+ seconds to load. If an execute call times out:
 1. Use `staleTimeoutMs: 60000` or higher for heavy sites
-2. Check `status` to confirm Docker is still running
-3. Take a screenshot to see where the browser is — the session is likely still alive
-4. If the container crashed, `docker compose up -d` restarts it; stored sessions/credentials survive since they're in the host's SQLite
+2. Take a screenshot to see where the browser is — the session is likely still alive
 
 ## Multi-step flows (email → code, OAuth, etc.)
 
@@ -292,7 +317,6 @@ Instead:
 **Your progress is NOT lost.** Credentials, sessions, and knowledge cache are in the SQLite database on disk — they survive MCP server restarts. You don't need to re-login or re-store anything.
 
 If the MCP disconnects repeatedly on the same operation, the site is probably too heavy for the current mode. Switch to a lighter approach:
-- If using `docker-headful` → switch to `binary-headful` (less overhead)
 - If doing a complex multi-step pipeline → break it into smaller execute calls
 - If just trying to capture APIs → you probably already have enough from previous runs. Check `knowledge get <domain>` before retrying.
 
@@ -302,7 +326,7 @@ If the MCP disconnects repeatedly on the same operation, the site is probably to
 |---------|-----------------|
 | Guessing CSS selectors like `button[data-qa='submit']` | Use `snapshot` → read refs → use `@e` refs |
 | Passing `usernameSelector`/`passwordSelector` to login step | Omit them — auto-detect handles it |
-| Re-storing credentials when login fails | Retry with stronger mode (`binary-headful` or `docker-headful`) |
+| Re-storing credentials when login fails | Retry with stronger mode (`binary-headful`) |
 | Launching browser without checking knowledge cache | Always call `knowledge get <domain>` first |
 | Using `browse` for a page that needs interaction | Use `execute` with a pipeline |
 | Running a huge pipeline without snapshots | Split into navigate+snapshot, then interact based on refs |
@@ -314,6 +338,56 @@ If the MCP disconnects repeatedly on the same operation, the site is probably to
 | Not passing `mode` consistently across execute calls in the same flow | Always pass the same `options.mode` for all steps in a multi-call flow |
 | Panicking when MCP disconnects (running `claude mcp`, inspecting source, giving up) | Just wait 2-3 seconds, call `status` to verify reconnect, then `session restart` and retry |
 
+## Driving the user's real Chrome (extension mode)
+
+Use this whenever the user says "my browser", "my tabs", "the page I have open", or the task
+needs accounts they are already signed into. A launched browser has a fresh, logged-out profile.
+
+1. `tabs` with `action: "list"` — returns open tabs as `[id <n>] title / url`. If it reports the
+   extension is not connected, tell the user to open Chrome / check the extension; do NOT fall
+   back to launching a browser.
+2. `execute` with `options.mode: "extension"` AND `options.tabId: <n>` from that list. Both are
+   required: `mode` alone is refused and launches nothing.
+3. Need a different page? `tabs` with `action: "open"` returns the new tab id. Do NOT use a
+   navigate-only pipeline in a launched browser to "get there".
+
+Rules while in extension mode:
+
+- **Do not call `session get-cookies` / `get-auth` / `capture-start` without a mode.** They default
+  to launching a separate window. The tab you are driving already has the auth; read it there.
+- **Do not call `session stop`.** There is nothing to stop, and it is the user's own browser.
+- `execute` never escalates in extension mode. If it fails, report why (extension disconnected,
+  tab closed) instead of retrying in a launched mode, which would silently use a logged-out profile.
+
+Driving a live single-page app (most dashboards) in extension mode:
+
+- **Act only on `@e` refs** from a `find`/`snapshot` in the same execute call. Never CSS you
+  guessed, never text selectors.
+- **Give steps that can wedge the tab their own execute call:** `navigate`, anything that opens or
+  closes a modal, and `evaluate`. If one hangs, the steps around it are not lost.
+- **If a step reports the tab stopped responding,** run a lone `read` or `snapshot` to confirm it
+  recovered, then continue. Don't resend the same step unchanged.
+- **Route changes:** `navigate` first. If the page never leaves, use `evaluate` with
+  `location.assign("<url>")`, or click the site's own link.
+- **Custom dropdowns (MUI, react-select, comboboxes):** click the field or its arrow button, `find`
+  the option (`role: "option", name: "Texas"`), click that `@e` ref. To filter first, `fill` the
+  field (it stays focused so the list stays open), then `find` and click the option. Don't type
+  with Cmd+A/Backspace tricks.
+- **Don't scroll with `evaluate` (`scrollIntoView`).** `click` scrolls its target into view itself.
+
+## Cleaning up after yourself
+
+A browser opened by mistake is a real, visible window sitting on the user's desktop. It is your
+job to close it, not theirs.
+
+- Any pipeline that opens a browser must end with `session stop` once the task is done.
+- If you opened a browser you did not need (wrong mode, wrong call, task turned out not to need
+  one), call `session stop` right away, in the same turn you noticed.
+- `status` lists live windows with their `instanceId`. If you see one you do not recognise from
+  your own work in this conversation, say so rather than silently leaving it.
+- A pipeline with no `navigate` step is refused unless a live browser already holds that
+  `instanceId`. If you get that error, you probably meant extension mode — go get a `tabId`.
+
 ## Mode selection guide
 
 | Scenario | Recommended mode |
@@ -321,15 +395,13 @@ If the MCP disconnects repeatedly on the same operation, the site is probably to
 | Simple data extraction, no login | Omit mode (auto-select, starts headless) |
 | Site with login | Omit mode (auto-escalation if headless is blocked) |
 | Known bot-detection (Cloudflare, etc.) | `binary-headful` |
-| Heavy SPA (Slack, Figma, Linear, etc.) | `binary-headful` — Docker often stalls on heavy pages |
+| Heavy SPA (Slack, Figma, Linear, etc.) | `binary-headful` |
 | Interactive session (browsing, clicking around) | `binary-headful` — most stable for long flows |
-| Need automated captcha solving | `docker-headful` (uses vision AI to solve) |
-| Need to watch the browser remotely / headless server | `docker-headful` |
+| Nobody available to solve a captcha | Any mode, with a best-effort `solve-captcha` step (needs `ANTHROPIC_API_KEY`) |
 | Reverse-engineering after login | Same mode that login succeeded in |
+| "Use my browser" / user's own logins / their open tab | `extension` + `tabId` from the `tabs` tool |
 
-**Prefer `binary-headful` over `docker-headful` for most tasks.** Binary-headful runs Chrome directly on the user's machine — it's faster, more stable for long sessions, and the user can solve captchas manually. Docker-headful adds overhead (Xvfb, VNC, container networking) and heavy SPAs frequently stall or timeout inside the container.
-
-Use `docker-headful` only when: (a) automated captcha solving is needed, (b) the user asks for it explicitly, or (c) it's a remote/headless server with no display.
+**Prefer `binary-headful` for sites that block headless.** It runs Chrome directly on the user's machine, is stable for long sessions, and the user can solve captchas manually.
 
 ## Response handling