iframer-toolkit 0.10.5 → 0.10.6

package/README.md

@@ -1,23 +1,62 @@
-# iframer-toolkit
+<div align="center">
 
-Browser access for AI agents when normal fetching fails. Give Claude, Codex, or any MCP-compatible agent a real browser: with session persistence, stealth fingerprinting, encrypted credential storage, and automatic captcha solving, and most of all, API Reverse Engineering.
+<img src="docs/logo.svg" width="104" alt="iframer">
 
-### Is this AI slop?
-No. This is a tool I use almost EVERY SINGLE DAY and I have put a lot of thought and real engineering to actually make this a tool I am proud of.
-Why did I develop this? Because I felt like the other agentic browsers didn't give me enough power and felt very expensive token-wise.
+# iframer
 
-## Example:
-<img width="2962" height="1116" alt="image" src="https://github.com/user-attachments/assets/c937b39c-5dba-4d6a-892d-71243ecf8146" />
-Resulting rev API example:
-<img width="2141" height="1173" alt="image" src="https://github.com/user-attachments/assets/19a27904-ebe0-4e65-8a44-a8cfb3756ed8" />
+**Give an AI agent a real browser, get an API back.**<br>
+Drive a site once and iframer hands the agent the actual request behind it, no devtools, no HAR file.
 
-Ships as:
-- **CLI** (`iframer-toolkit` / `iframer`) — browse, screenshot, credentials, sessions, reverse-engineer APIs
-- **MCP server** — plugs directly into Claude Code or Codex so agents can drive the browser themselves
-- **Browser extension** (optional) — lets the agent drive tabs in **your real Chrome**, on your real logged-in session
-- **Self-hosted Docker server** (optional) — adds live headful browsing over noVNC for remote/multi-user setups
+![npm](https://img.shields.io/npm/v/iframer-toolkit?color=6D5EF0&label=npm)
+![license](https://img.shields.io/badge/license-MIT-6D5EF0)
+![node](https://img.shields.io/badge/node-%E2%89%A518-6D5EF0)
+![CLI](https://img.shields.io/badge/-CLI-6D5EF0)
+![MCP](https://img.shields.io/badge/-MCP-6D5EF0)
+![extension](https://img.shields.io/badge/-extension-6D5EF0)
+
+<img src="docs/example/demo.gif" width="760" alt="left: a demo page being clicked. right: the terminal, running iframer reverse-engineer, showing the captured endpoint">
+
+</div>
+
+Most agent-driven browsers only replay clicks. Every visit to the same page means rendering the whole thing again, screenshot after screenshot, just to read one field off it.
+
+iframer drives a real, stealth-patched Chromium, or your own logged-in Chrome through the extension, and while it does:
+
+- Records the fetch and XHR calls the page makes while it drives it (not static assets), no manual devtools work
+- Hands back the real request behind a login form or a button click: method, headers, auth, a working curl
+- Generates typed functions ready to `import`, when an agent is doing the driving
+- Caches the domain after the first run, so the next visit skips the browser and hits the endpoint directly, seconds down to milliseconds
+
+## Example
+
+The left side of the GIF above is [`docs/example/site/index.html`](docs/example/site/index.html): one button, one hidden API call. The right side is a real terminal, only the typing pace was staged for the recording; the command and its output are exactly what running it produces. Capturing it for real:
+
+```sh
+cd docs/example/site && python3 -m http.server 8934 &
+iframer-toolkit reverse-engineer '[
+  {"type":"navigate","url":"http://localhost:8934/index.html"},
+  {"type":"click","selector":"#load"},
+  {"type":"wait","ms":1000}
+]' --output docs/example/api
+```
+
+That produced [`docs/example/api/captured-api.json`](docs/example/api/captured-api.json): the exact request, headers, response body, and a curl command, all automatic, no agent required.
+
+The rest of `docs/example/api/` (`getTodos.ts`, `transport/rest.ts`, `types.ts`, `README.md`) is what an agent writes from that capture: iframer tells it the endpoint's protocol, verb, and function name, and the agent turns that into a typed function. That's what happened here, driven by Claude through the MCP `reverse-engineer` tool.
+
+The whole round trip, one click to one typed endpoint, cost about 590 tokens back to the agent. This is the whole thing you get: no HAR file, no manual devtools work, a function you can call.
+
+## Token cost
 
-The rest of this description is AI generated, bad for humans, good for LLMs:
+Most of iframer's savings come from the snapshot it hands back: capped and filtered, instead of the full accessibility tree other tools dump into context. Measured with the same tokenizer, same steps, on the same three tasks:
+
+| Task | iframer (MCP) | iframer (CLI) | agent-browser | Playwright MCP |
+|------|---------------|----------------|----------------|-----------------|
+| Wiki link chase (6 hops, Everest → Gojo) | 517 | 721 | 913 | 1,405 |
+| Store purchase (login → sort → cart → checkout) | 552 | 697 | 851 | 2,739 |
+| Heavy-page recon (GitHub: look + pull 4 facts) | 986 | 1,058 | 8,433 | 19,906 |
+
+The gap grows with how much DOM the page has. On the GitHub page, iframer spent about 986 tokens where Playwright MCP spent about 19,900, roughly 20x. That's per-task cost. It doesn't include the roughly 5.1k tokens of MCP tool-definition overhead loaded once per session, not per task.
 
 ## Install
 
@@ -47,11 +86,17 @@ iframer-toolkit update             # update via npm (also refreshes the extensio
 iframer-toolkit update --check     # just report whether a newer version exists
 ```
 
-> **Note:** If you prefer, your agent can run `iframer-toolkit install` for you — it'll figure the rest out.
+Your agent can also just run `iframer-toolkit install` for you; it figures out the rest.
+
+Ships as:
+- **CLI** (`iframer-toolkit` / `iframer`): browse, screenshot, credentials, sessions, reverse-engineer APIs
+- **MCP server**: plugs into Claude Code or Codex so agents drive the browser themselves
+- **Browser extension** (optional): lets the agent drive tabs in your real Chrome, on your real logged-in session
+- **Self-hosted Docker server** (optional): adds live headful browsing over noVNC for remote or multi-user setups
 
 ## Quick start
 
-Once installed, you can either drive the browser via the CLI directly, or ask your agent (Claude Code / Codex) to do it for you via the MCP.
+Once installed, drive the browser via the CLI directly, or ask your agent (Claude Code / Codex) to do it through the MCP.
 
 **CLI:**
 
@@ -60,7 +105,7 @@ iframer-toolkit status                                      # system + browser m
 iframer-toolkit browse https://example.com --extract 'document.title'
 iframer-toolkit screenshot https://news.ycombinator.com -o /tmp/hn.png
 iframer-toolkit credentials add github.com                  # interactive masked prompt
-iframer-toolkit reverse-engineer https://some-spa.com       # capture the APIs it calls
+iframer-toolkit reverse-engineer https://some-spa.com        # capture the APIs it calls
 iframer-toolkit --cache                                     # list cached domains
 ```
 
@@ -68,44 +113,80 @@ iframer-toolkit --cache                                     # list cached domain
 
 > "Log into my account on example.com and extract the latest invoice."
 
-The agent will call `knowledge` first (to see if a direct-API path is cached), fall back to `credentials` + `execute` if not, auto-escalate browser modes if a site blocks headless, and return the result. No copying cookies, no proxies, no manual login.
+The agent calls `knowledge` first to check for a cached direct-API path, falls back to `credentials` + `execute` if there isn't one, auto-escalates browser modes if a site blocks headless, and returns the result. No copying cookies, no proxies, no manual login.
 
 ## How it works
 
 ```
-Claude / Codex (MCP) ──→ iframer MCP server ──→ shared local server (127.0.0.1)
-                                                    ├─ patchright (stealth Chromium)
-                                                    ├─ Chrome for Testing
-                                                    ├─ your real Chrome (optional, via extension)
-                                                    └─ SQLite at ~/.iframer — one file for
-                                                       encrypted credentials, session state,
-                                                       and per-domain knowledge cache
+Claude / Codex (MCP) ──▶ iframer MCP server ──▶ shared local server (127.0.0.1)
+                                                   ├─ patchright (stealth Chromium)
+                                                   ├─ Chrome for Testing
+                                                   ├─ your real Chrome (optional, via extension)
+                                                   └─ SQLite at ~/.iframer: one file for
+                                                      encrypted credentials, session state,
+                                                      and per-domain knowledge cache
 ```
 
-By default, `install mcp` runs in **local mode**: no Docker needed. Both the MCP clients and the `iframer execute` CLI talk to **one shared local server** (discovered via `~/.iframer/server.json`, loopback-only) that keeps a stealth-patched Chromium (via [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright)) warm between calls, tracks every browser it spawns in an on-disk PID registry so nothing leaks, and idle-exits when no one needs it. `install mcp` registers the server in both Claude Code (`~/.claude.json`) and Codex (`~/.codex/config.toml`).
+By default, `install mcp` runs in **local mode**, no Docker needed. Both the MCP clients and the `iframer execute` CLI talk to one shared local server (discovered via `~/.iframer/server.json`, loopback only) that keeps a stealth-patched Chromium warm between calls, tracks every browser it spawns in an on-disk PID registry so nothing leaks, and idle-exits when no one needs it. `install mcp` registers the server in both Claude Code (`~/.claude.json`) and Codex (`~/.codex/config.toml`).
 
-**Warm CLI.** `iframer execute` routes to that shared daemon instead of launching a cold browser per invocation, so after the first call it's as fast as the MCP (pass `--in-process` to force a private throwaway browser). Each agent gets its own browser instance (keyed off `CLAUDE_CODE_SESSION_ID` / `TERM_SESSION_ID`, or set `IFRAMER_INSTANCE`), so concurrent agents never share a window, while all CLI calls share one login session.
+**Warm CLI.** `iframer execute` routes to that shared daemon instead of launching a cold browser per invocation, so after the first call it's as fast as the MCP (pass `--in-process` to force a private throwaway browser). Each agent gets its own browser instance, keyed off `CLAUDE_CODE_SESSION_ID` / `TERM_SESSION_ID` or `IFRAMER_INSTANCE`, so concurrent agents never share a window, while all CLI calls share one login session.
 
-**Resuming a task.** The browser and its page persist in the daemon between calls and survive an interrupt (Ctrl-C). Name a multi-step or human-in-the-loop task's browser with a stable `instanceId`, and to resume just run `execute` again with the same `instanceId` and act on the current page (`snapshot`/`read`/`find`) — don't navigate again, which would reload and lose the state (e.g. an OTP screen). `iframer instances` lists the live windows and which page each is on.
+**Resuming a task.** The browser and its page persist in the daemon between calls and survive an interrupt (Ctrl-C). Name a multi-step or human-in-the-loop task's browser with a stable `instanceId`, and to resume, run `execute` again with the same `instanceId` and act on the current page (`snapshot` / `read` / `find`). Don't navigate again; that reloads the page and loses state like an OTP screen. `iframer instances` lists the live windows and which page each is on.
 
-**One credential store for every browser mode.** Stored credentials, session cookies/localStorage, and the knowledge cache all live in a single SQLite file at `~/.iframer/iframer.db`. Store a password once (via CLI or MCP) and every mode — `headless`, `binary-headful`, `docker-headful` — uses the same row. No split-brain between modes.
+**One credential store for every browser mode.** Stored credentials, session cookies/localStorage, and the knowledge cache all live in one SQLite file at `~/.iframer/iframer.db`. Store a password once, from the CLI or the MCP, and every mode (`headless`, `binary-headful`, `docker-headful`) uses the same row.
 
-**Auto-escalation.** When a pipeline is blocked in `headless` (bot detection, captcha), iframer transparently retries in `binary-headful` and then `docker-headful` without a round-trip to the agent. The mode that worked is recorded in `~/.iframer/domain-modes.json` so the next run on that domain starts at the right mode.
+**Auto-escalation.** When a pipeline is blocked in `headless` (bot detection, captcha), iframer retries in `binary-headful` and then `docker-headful` without a round trip to the agent. The mode that worked is recorded in `~/.iframer/domain-modes.json`, so the next run on that domain starts at the right mode.
 
-**Knowledge cache.** After every successful run, iframer writes a per-domain markdown file at `~/.iframer/knowledge/<domain>.md` recording which cookies / localStorage keys / headers the site uses for auth, plus any API endpoints observed along the way. Next time the agent needs data from that domain, it reads the cache first — if there's a direct-API path, it skips the browser entirely. See [Knowledge cache](#knowledge-cache) below.
+**Knowledge cache.** After every successful run, iframer writes a per-domain markdown file at `~/.iframer/knowledge/<domain>.md` recording which cookies, localStorage keys, and headers the site uses for auth, plus any API endpoints seen along the way. The next time an agent needs data from that domain, it reads the cache first; if there's a direct-API path, it skips the browser entirely. See [Knowledge cache](#knowledge-cache) below.
 
 For live remote viewing, multi-user, or Linux server deployments, see [Self-hosting with Docker](#self-hosting-with-docker) below.
 
-## Extension mode — drive your real Chrome
+## Extension mode: drive your real Chrome
+
+The optional browser extension lets the agent drive a tab you already have open in your real Chrome. No relaunch, no remote-debugging port, your real logged-in session. The extension dials out to iframer's local server and relays the CDP protocol via `chrome.debugger`, so your live tab is driven by the exact same pipeline engine (find, click, snapshot, obstacle handling, API capture) as every other mode, with real trusted input. While a run is active, Chrome shows its yellow "started debugging" bar.
 
-The optional browser extension lets the agent drive **a tab you already have open in your real Chrome** — no relaunch, no remote-debugging port, on your real logged-in session. The extension dials out to iframer's local server and relays the CDP protocol via `chrome.debugger`, so your live tab is driven by the exact same pipeline engine (find / click / snapshot / obstacle handling / API capture) as every other mode, with real trusted input. While a run is active, Chrome shows its yellow "started debugging" bar.
+There's no Chrome Web Store listing. You load it as an unpacked, developer-mode extension, straight from the files npm just installed on your machine.
 
+### Installing it
+
+1. Run:
+   ```sh
+   iframer install extension chrome
+   ```
+   This installs a small pairing host and prints a folder path. That folder is the extension itself.
+
+2. Open `chrome://extensions` in Chrome.
+3. Turn on **Developer mode**, the toggle is top right.
+4. Click **Load unpacked**.
+5. In the file picker, paste the folder path from step 1 and select it.
+
+Chrome loads the extension and pins its icon to the toolbar. That's the whole install, one time only.
+
+### Finding the path again
+
+Lost the path, or setting up a second machine? Run:
 ```sh
-iframer install extension chrome   # installs the pairing host, prints the folder to load
-iframer extension path             # prints it again
+iframer extension path
 ```
+It prints the same folder every time.
 
-Then load it once: `chrome://extensions` → Developer mode → **Load unpacked** → the printed folder. Once paired, the agent lists your open tabs with the `tabs` tool and drives the one you mean. Multiple Chrome profiles/browsers can be paired at once — each identifies itself with a profile name, and iframer routes work to the profile that owns the target tab. See [`extension/README.md`](extension/README.md) for details.
+That folder lives inside wherever npm put `iframer-toolkit` globally, not inside a git checkout of this repo. On most machines that's something like `/usr/local/lib/node_modules/iframer-toolkit/extension` or, with nvm, a path under `~/.nvm/versions/node/<version>/lib/node_modules/iframer-toolkit/extension`. Run the command above instead of guessing, the exact path depends on your Node setup.
+
+Loading it from that exact path matters: it's what lets `iframer update` refresh the extension files in place, then tell Chrome to reload it, no repeating the install steps after every update.
+
+### Pairing (automatic)
+
+Once loaded, the extension pairs itself. The pairing host installed in step 1 hands it the same local token the CLI and MCP already use, straight from `~/.iframer/secret`. Nothing to copy, nothing to type.
+
+If that ever fails (host not installed, or you're on a platform it doesn't support), click the iframer icon in the toolbar and paste the token yourself:
+```sh
+cat ~/.iframer/secret
+```
+then **Save & connect**.
+
+Either way, check the popup: the dot next to "iframer" turns green once it finds the running local server.
+
+You can repeat this whole install on more than one Chrome profile, or in Brave, Edge, or another Chromium-based browser, at the same time. Each one pairs under its own profile name, and iframer routes each task to whichever profile actually has the tab. Once paired, the agent lists your open tabs with the `tabs` tool and drives the one you mean. See [`extension/README.md`](extension/README.md) for how the pipeline and multi-profile routing work under the hood.
 
 ## CLI reference
 
@@ -175,7 +256,7 @@ Setup:
 
 Browser:
   modes                            Show available browser modes
-  status                           Show system status
+  status                            Show system status
 ```
 
 The binary is available as either `iframer-toolkit` (full name) or `iframer` (short alias). `npx iframer-toolkit ...` also works without a global install.
@@ -184,33 +265,33 @@ The binary is available as either `iframer-toolkit` (full name) or `iframer` (sh
 
 Once the MCP is registered, the agent has access to:
 
-- **`status`** — system health, session state, stored credentials, available browser modes
-- **`knowledge`** — read / list / clear the per-domain knowledge cache. Agents are told to check this **before** every `execute` or `browse` — if the cache has a direct-API path for the data, the browser doesn't launch at all.
-- **`execute`** — run a pipeline of browser steps (navigate, click, right-click, fill, select, human-click, human-type, type-code, read, evaluate, extract, wait, wait-for, scroll, keyboard, upload, paste, download, login, solve-captcha, recaptcha, screenshot, snapshot, find). Each step has a 20s stale-state timeout. On failure, returns the exact step, error type, and a screenshot of the page at the point of failure. Auto-escalates browser modes transparently on bot-block.
-- **`browse`** — fast headless fetch with session persistence for pages that don't need a full pipeline
-- **`reverse-engineer`** — capture the APIs a site calls (feeds into the knowledge cache so future runs can skip the browser)
-- **`remember`** — per-domain **element anchors**: save a selector once under a name, then target it as `@a:<name>` in any later pipeline — no re-finding the element on every visit. Anchors self-heal (a failed anchor triggers re-discovery, not blind retries) and the tool also records per-site quirks.
-- **`tabs`** — list and target tabs in your real Chrome (requires the [extension](#extension-mode--drive-your-real-chrome))
-- **`clipboard`** — read/write the machine's clipboard (the same one your Chrome uses): read a code a site just copied, or stage text to paste into a field via the `paste` step
-- **`session`** — `stop` (save state) or `clear` (wipe)
-- **`credentials`** — `store` (secure form via MCP elicitation), `list`. The `store` action will **refuse to overwrite** existing credentials unless explicitly told to (`force: true`) — this blocks the common "login failed, re-ask for password" anti-pattern when the real problem is browser mode / bot detection.
+- **`status`**: system health, session state, stored credentials, available browser modes
+- **`knowledge`**: read/list/clear the per-domain knowledge cache. Agents are told to check this before every `execute` or `browse`; if the cache already has a direct-API path, the browser never launches.
+- **`execute`**: run a pipeline of browser steps (navigate, click, right-click, fill, select, human-click, human-type, type-code, read, evaluate, extract, wait, wait-for, scroll, keyboard, upload, paste, download, login, solve-captcha, recaptcha, screenshot, snapshot, find). Each step has a 20s stale-state timeout. On failure, it returns the exact step, error type, and a screenshot of the page at the point of failure. Auto-escalates browser modes on bot-block.
+- **`browse`**: fast headless fetch with session persistence for pages that don't need a full pipeline
+- **`reverse-engineer`**: capture the APIs a site calls (feeds the knowledge cache so future runs can skip the browser)
+- **`remember`**: per-domain element anchors. Save a selector once under a name, then target it as `@a:<name>` in any later pipeline, no re-finding the element on every visit. Anchors self-heal (a failed anchor triggers re-discovery, not blind retries) and the tool also records per-site quirks.
+- **`tabs`**: list and target tabs in your real Chrome (requires the [extension](#extension-mode-drive-your-real-chrome))
+- **`clipboard`**: read/write the machine's clipboard, the same one your Chrome uses: read a code a site just copied, or stage text to paste into a field via the `paste` step
+- **`session`**: `stop` (save state) or `clear` (wipe)
+- **`credentials`**: `store` (secure form via MCP elicitation), `list`. `store` refuses to overwrite existing credentials unless told to (`force: true`), which blocks the common "login failed, re-ask for password" mistake when the real problem is browser mode or bot detection.
 
 ## Session persistence
 
-Session data (cookies + localStorage) and credentials are stored in SQLite at `~/.iframer/iframer.db` and encrypted with AES-256-GCM. Data is automatically re-injected on the next `execute` or `browse` so agents stay logged in across restarts — and across browser modes: a session captured in `binary-headful` will transparently load into `headless` on the next run.
+Session data (cookies + localStorage) and credentials are stored in SQLite at `~/.iframer/iframer.db` and encrypted with AES-256-GCM. Data is automatically re-injected on the next `execute` or `browse` so agents stay logged in across restarts, and across browser modes: a session captured in `binary-headful` will load into `headless` on the next run.
 
-The encryption key lives at `~/.iframer/secret` (0600 permissions), generated on first `install mcp`. Set `IFRAMER_SECRET` in your environment to override it — useful if you want to synchronize the key across multiple machines.
+The encryption key lives at `~/.iframer/secret` (0600 permissions), generated on first `install mcp`. Set `IFRAMER_SECRET` in your environment to override it, useful if you want to sync the key across machines.
 
 ## Knowledge cache
 
-Every successful `execute` run updates a plain markdown file at `~/.iframer/knowledge/<domain>.md` — human-readable, grep-able, and editable. Each file captures:
+Every successful `execute` run updates a plain markdown file at `~/.iframer/knowledge/<domain>.md`, human-readable, grep-able, editable. Each file captures:
 
-- Which cookies, localStorage keys, and headers are load-bearing for authentication (names only — values stay encrypted in the session store)
-- Which API endpoints the site called during the run (with method, path, example curl, and status code)
+- Which cookies, localStorage keys, and headers are load-bearing for authentication (names only; values stay encrypted in the session store)
+- Which API endpoints the site called during the run, with method, path, example curl, and status code
 - Which browser mode last worked for the domain
-- Notes about captcha/bot-detection behavior
+- Notes about captcha or bot-detection behavior
 
-The MCP `knowledge` tool exposes this to the agent with `get`, `list`, and `clear` actions. Agents are instructed to call it **before** every browser-touching tool — if the cache already knows a direct-API path for the data the user is asking about, the agent hits the API directly and never launches a browser. First request takes seconds; repeat requests on the same domain take milliseconds.
+The MCP `knowledge` tool exposes this with `get`, `list`, and `clear` actions. Agents are instructed to call it before every browser-touching tool; if the cache already has a direct-API path for the data, the agent hits the API directly and never launches a browser. First request takes seconds, repeat requests on the same domain take milliseconds.
 
 Inspect the cache yourself:
 
@@ -223,7 +304,7 @@ iframer-toolkit --clear-cache figma.com     # wipe one domain
 
 ## Captcha solving
 
-iframer auto-detects and solves reCAPTCHA and hCaptcha using Claude's vision API. Use the `solve-captcha` step in a pipeline:
+iframer detects and solves reCAPTCHA and hCaptcha using Claude's vision API. Use the `solve-captcha` step in a pipeline:
 
 ```json
 { "type": "solve-captcha" }
@@ -245,9 +326,9 @@ Requires `ANTHROPIC_API_KEY` in your environment.
 
 ## Self-hosting with Docker
 
-The Docker server adds a live headful browsing mode over noVNC (watch the agent drive the browser in real time) and lets multiple clients share one browser pool. The Docker path is only used when the pipeline explicitly requests `mode: "docker-headful"` — `headless` and `binary-headful` always run directly on the host even when Docker is up, so host-stored credentials are always visible.
+The Docker server adds a live headful browsing mode over noVNC (watch the agent drive the browser in real time) and lets multiple clients share one browser pool. The Docker path only runs when a pipeline explicitly requests `mode: "docker-headful"`; `headless` and `binary-headful` always run directly on the host even when Docker is up, so host-stored credentials stay visible.
 
-The `docker-compose.yml` bind-mounts `~/.iframer` from the host into the container as `/iframer-data` and sets `IFRAMER_DATA_DIR=/iframer-data`, so **container and host share the same SQLite file**. Credentials stored via the CLI or MCP on the host are immediately visible to `docker-headful` without any copy step.
+`docker-compose.yml` bind-mounts `~/.iframer` from the host into the container as `/iframer-data` and sets `IFRAMER_DATA_DIR=/iframer-data`, so container and host share the same SQLite file. Credentials stored via the CLI or MCP on the host are immediately visible to `docker-headful`, no copy step.
 
 **1. Clone and configure**
 
@@ -255,7 +336,7 @@ The `docker-compose.yml` bind-mounts `~/.iframer` from the host into the contain
 git clone https://github.com/EduardoFazolo/iframer-toolkit.git
 cd iframer-toolkit
 cp .env.example .env
-# Edit .env — set ANTHROPIC_API_KEY (for captcha) and IFRAMER_SECRET (for auth)
+# Edit .env: set ANTHROPIC_API_KEY (for captcha) and IFRAMER_SECRET (for auth)
 ```
 
 **2. Start**
@@ -268,13 +349,13 @@ bun run stop:docker    # stop containers
 
 **3. Point the MCP at it (remote host only)**
 
-If you're running Docker on a remote machine and the MCP on a different machine, install the MCP with the remote URL:
+If Docker runs on a remote machine and the MCP on a different one, install the MCP with the remote URL:
 
 ```sh
 IFRAMER_URL=https://your-host:3021 iframer-toolkit install mcp --dev
 ```
 
-When Docker is on the same host as the MCP, no extra setup is needed — the local MCP server will auto-detect the Docker API on `localhost:3021` and route only `docker-headful` requests through it.
+When Docker is on the same host as the MCP, no extra setup is needed; the local MCP server auto-detects the Docker API on `localhost:3021` and routes only `docker-headful` requests through it.
 
 **4. Watch the browser live**
 
@@ -292,17 +373,36 @@ Or run `iframer-toolkit watch` to auto-open it.
 |--------------------|------------|
 | Browser engine     | [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) (stealth-patched Playwright fork) |
 | Browser binary     | Chrome for Testing (downloaded to `~/.iframer/chrome/`) |
-| Local server       | One shared warm server per machine (loopback-only, discovered via `~/.iframer/server.json`), on-disk browser PID registry, idle auto-exit |
-| Extension mode     | Chrome extension relaying CDP over WebSocket (`chrome.debugger` → `connectOverCDP`) — drives tabs in your real Chrome |
+| Local server       | One shared warm server per machine (loopback only, discovered via `~/.iframer/server.json`), on-disk browser PID registry, idle auto-exit |
+| Extension mode     | Chrome extension relaying CDP over WebSocket (`chrome.debugger` → `connectOverCDP`), drives tabs in your real Chrome |
 | Stealth            | Fingerprint injection, WebRTC leak prevention, worker patching |
 | Credential store   | SQLite at `~/.iframer/iframer.db`, AES-256-GCM encrypted, shared by every browser mode |
-| Session persistence| Same SQLite file — cookies + localStorage re-injected across runs and modes |
+| Session persistence| Same SQLite file, cookies + localStorage re-injected across runs and modes |
 | Knowledge cache    | Plain markdown at `~/.iframer/knowledge/<domain>.md` |
 | Captcha solving    | Anthropic vision API (`@anthropic-ai/sdk`) |
 | Live viewing       | Xvfb + x11vnc + noVNC + websockify (Docker mode only) |
 | MCP server         | `@modelcontextprotocol/sdk` |
 | Runtime            | Node.js ≥18 (Bun for development) |
 
+## Repo layout
+
+```
+src/lib/            core: pipeline executor, actions, auth, screenshot, knowledge cache
+src/mcp/             MCP server + tool definitions
+bin/cli.js           CLI entry point
+extension/           Chrome extension (manifest v3, CDP relay)
+docs/example/        the demo in this README: site/ + captured api/
+benchmark/           token-cost benchmark harness vs agent-browser and Playwright MCP (gitignored, local only)
+```
+
+## What's not done yet
+
+- Stealth fingerprinting is injected right after the page loads, not at browser-context creation, so a site checking for it in the first tick of navigation can still catch it. This is the main remaining cause of false bot-blocks.
+- The knowledge cache's markdown merge can drop an endpoint's description or example fields when the same domain is captured a second time.
+- Consent banners rendered inside a cross-origin iframe (common with some CMPs) aren't recognized as an obstacle yet.
+- Element anchors (`remember`) still have to be saved by hand after a successful run; auto-saving the selectors that worked isn't wired up.
+- A few internal modules (the SQLite adapter, the stealth/humanize helpers) still carry loose `any` types.
+
 ## Development
 
 ```sh
@@ -310,7 +410,7 @@ git clone https://github.com/EduardoFazolo/iframer-toolkit.git
 cd iframer-toolkit
 bun install
 
-# Run the CLI from source (no build needed — bun runs .ts directly)
+# Run the CLI from source (no build needed, bun runs .ts directly)
 bun run bin/cli.js status
 
 # Run the MCP server from source

package/dist/cli.js

@@ -902,6 +902,10 @@ function findChromeExecutable() {
   if (import_fs3.default.existsSync("/usr/bin/google-chrome-stable")) return "/usr/bin/google-chrome-stable";
   return void 0;
 }
+async function getHeadedBrowser() {
+  if (!headedBrowser?.isConnected()) headedBrowser = await import_patchright.chromium.launch({ headless: false, args: STEALTH_ARGS });
+  return headedBrowser;
+}
 async function getBrowser(_name = "chromium") {
   if (cachedBrowser) {
     if (cachedBrowser.isConnected()) return cachedBrowser;
@@ -919,6 +923,11 @@ async function getBrowser(_name = "chromium") {
   return cachedBrowser;
 }
 async function closeBrowser() {
+  if (headedBrowser) {
+    await headedBrowser.close().catch(() => {
+    });
+    headedBrowser = null;
+  }
   if (!cachedBrowser) return;
   try {
     await cachedBrowser.close();
@@ -955,7 +964,7 @@ async function launchHeadful(displayNum) {
   log2.debug(`headful: ${executablePath || "patchright chromium"}, extensions: ${hasExtensions}`);
   return import_patchright.chromium.launch(launchOpts);
 }
-var import_fs3, import_patchright, log2, UBLOCK_PATH, cachedBrowser;
+var import_fs3, import_patchright, log2, UBLOCK_PATH, cachedBrowser, headedBrowser;
 var init_launcher = __esm({
   "src/lib/browser/launcher.ts"() {
     "use strict";
@@ -966,6 +975,7 @@ var init_launcher = __esm({
     log2 = createLogger("launcher");
     UBLOCK_PATH = "/extensions/uBlock0.chromium";
     cachedBrowser = null;
+    headedBrowser = null;
   }
 });
 
@@ -2273,6 +2283,59 @@ var init_ref_store = __esm({
   }
 });
 
+// src/lib/site-protection.ts
+async function paceProtectedDomain(domain) {
+  const now = Date.now();
+  const slot = Math.max(now, nextRequest.get(domain) || 0);
+  nextRequest.set(domain, slot + PROTECTED_INTERVAL_MS);
+  if (slot > now) await new Promise((resolve) => setTimeout(resolve, slot - now));
+}
+async function detectSiteProtection(page) {
+  const cookies = await page.context().cookies(page.url()).catch(() => []);
+  if (cookies.some((c) => /^(incap_ses_|visid_incap_|nlbi_)/i.test(c.name))) return "imperva";
+  if (cookies.some((c) => /^(?:_abck|bm_sz|ak_bmsc|bm_sv)$/.test(c.name))) return "akamai";
+  const html = await page.content().catch(() => "");
+  if (/\/[_]Incapsula_Resource|incapsula incident id|powered by imperva/i.test(html)) return "imperva";
+  if (/errors\.edgesuite\.net|akamai bot manager/i.test(html)) return "akamai";
+  return null;
+}
+function protectionHooks(headless, respectful = false) {
+  return {
+    before: async (page, step) => {
+      const protection = await detectSiteProtection(page);
+      if (protection) {
+        respectful = true;
+        if (headless) throw new HeadedRequiredError(page.url());
+      }
+      if (respectful) await paceProtectedDomain(new URL(step.url || page.url()).hostname);
+    },
+    after: async (page) => {
+      const protection = await detectSiteProtection(page);
+      if (!protection) return;
+      respectful = true;
+      if (headless) throw new HeadedRequiredError(page.url());
+    }
+  };
+}
+var PROTECTED_INTERVAL_MS, nextRequest, HeadedRequiredError;
+var init_site_protection = __esm({
+  "src/lib/site-protection.ts"() {
+    "use strict";
+    PROTECTED_INTERVAL_MS = 3e3;
+    nextRequest = /* @__PURE__ */ new Map();
+    HeadedRequiredError = class extends Error {
+      constructor(url) {
+        super("This site requires a headed browser.");
+        this.url = url;
+      }
+      url;
+      stepIndex = 0;
+      results = [];
+      obstacles = [];
+    };
+  }
+});
+
 // src/lib/actions/types.ts
 function failedStepResult(step, error, durationMs, stepIndex = -1) {
   return { stepIndex, step, ok: false, error, durationMs };
@@ -5561,6 +5624,7 @@ var DEFAULT_STALE_TIMEOUT_MS2, PipelineRunner;
 var init_pipeline = __esm({
   "src/lib/pipeline.ts"() {
     "use strict";
+    init_site_protection();
     init_registry2();
     init_types();
     init_stale_monitor();
@@ -5574,10 +5638,12 @@ var init_pipeline = __esm({
     init_resolve_selector();
     DEFAULT_STALE_TIMEOUT_MS2 = 2e4;
     PipelineRunner = class {
-      constructor(ctx) {
+      constructor(ctx, hooks) {
         this.ctx = ctx;
+        this.hooks = hooks;
       }
       ctx;
+      hooks;
       async run(initialPage, pipeline) {
         const tracker = new TabTracker(initialPage.context(), initialPage);
         try {
@@ -5623,12 +5689,19 @@ var init_pipeline = __esm({
           const monitor = new StaleStateMonitor(page, staleTimeoutMs);
           let stepResult;
           try {
+            await this.hooks?.before(page, step);
             stepResult = await monitor.withMonitoring(async () => {
               const r = await executeAction(page, step, this.ctx, monitor);
+              if (step.type === "navigate") await this.hooks?.after(page, step.url);
               r.stepIndex = i;
               return r;
             });
           } catch (err) {
+            if (err instanceof HeadedRequiredError) {
+              Object.assign(err, { stepIndex: i, results, obstacles });
+              await finishCapture();
+              throw err;
+            }
             const asError = err instanceof Error ? err : new Error(String(err));
             const errorType = classifyError(asError, step);
             const pageState = await capturePageState(tracker.active(), this.ctx, { screenshot: true });
@@ -10119,7 +10192,7 @@ var init_cdp_relay = __esm({
             return {};
           case "Browser.close":
             return {};
-          // patchright disconnect — don't kill the user's Chrome
+          // Disconnect without closing the user's Chrome.
           case "Target.setDiscoverTargets":
             return {};
           case "Target.getTargets":
@@ -10226,6 +10299,7 @@ var import_playwright_core, log18, PipelineExecutor;
 var init_pipeline_executor = __esm({
   "src/lib/execution/pipeline-executor.ts"() {
     "use strict";
+    init_site_protection();
     import_playwright_core = require("playwright-core");
     init_daemon();
     init_pipeline();
@@ -10245,8 +10319,8 @@ var init_pipeline_executor = __esm({
         this.deps = deps;
       }
       deps;
-      /** Runtime elicitation hook, set per-call via execute(). Consumed once by
-       *  executeLocal and immediately cleared. */
+      /** Shared runtime hook read by local and extension execution;
+       *  execute() clears it when the call finishes. */
       pendingElicitOtp;
       /** One pipeline per real tab at a time: chrome.debugger and the CDP relay
        *  can't share a tab, so concurrent executes on the same tab queue up here. */
@@ -10292,6 +10366,7 @@ var init_pipeline_executor = __esm({
           mode = availableModes[0] || "headless";
         }
         let result = await this.executeWithMode(userId, token, pipeline, mode, instanceId);
+        mode = result.modeUsed || mode;
         if (!result.ok && autoEscalate && domain && result.error?.errorType === "bot-blocked") {
           const failedMode = mode;
           if (domain) this.deps.domainModes.recordFailure(domain, failedMode, result.error?.message || "blocked");
@@ -10328,11 +10403,11 @@ var init_pipeline_executor = __esm({
         }
         return result;
       }
-      async executeWithMode(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE) {
+      async executeWithMode(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE, respectful = false) {
         if (mode === "docker-headful") {
-          return this.executeDocker(userId, token, pipeline);
+          return this.executeDocker(userId, token, pipeline, respectful);
         }
-        return this.executeLocal(userId, token, pipeline, mode, instanceId);
+        return this.executeLocal(userId, token, pipeline, mode, instanceId, respectful);
       }
       /** Execute against the user's real Chrome tab via the extension CDP relay,
        *  using the SAME PipelineRunner as every other mode. No session inject/extract
@@ -10425,7 +10500,7 @@ var init_pipeline_executor = __esm({
         }
       }
       /** Execute via Docker session-manager. */
-      async executeDocker(userId, token, pipeline) {
+      async executeDocker(userId, token, pipeline, respectful = false) {
         let session = getSession(userId);
         if (!session) {
           const firstNav = pipeline.steps.find((s) => s.type === "navigate");
@@ -10434,7 +10509,7 @@ var init_pipeline_executor = __esm({
         }
         resetTimeout(userId);
         const ctx = this.deps.refStore.makeContext(userId, token);
-        const runner = new PipelineRunner(ctx);
+        const runner = new PipelineRunner(ctx, protectionHooks(false, respectful));
         const result = await runner.run(session.page, pipeline);
         if (result.ok) {
           const blockResult = await detectBlock(session.page);
@@ -10461,7 +10536,7 @@ var init_pipeline_executor = __esm({
         return result;
       }
       /** Execute via local Chrome daemon (headless + binary-headful). */
-      async executeLocal(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE) {
+      async executeLocal(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE, respectful = false) {
         const startTime = Date.now();
         let acquired = false;
         try {
@@ -10483,8 +10558,38 @@ var init_pipeline_executor = __esm({
           const ctx = this.deps.refStore.makeContext(userId, token);
           if (sessionData) ctx.sessionData = sessionData;
           if (this.pendingElicitOtp) ctx.elicitOtp = this.pendingElicitOtp;
-          const runner = new PipelineRunner(ctx);
-          const result = await runner.run(page, pipeline);
+          const runner = new PipelineRunner(ctx, protectionHooks(mode === "headless", respectful));
+          let result;
+          try {
+            result = await runner.run(page, pipeline);
+          } catch (err) {
+            if (!(err instanceof HeadedRequiredError)) throw err;
+            this.deps.daemon.release(mode, instanceId);
+            acquired = false;
+            await this.deps.daemon.stopMode(mode, instanceId);
+            const available = this.deps.availableModes();
+            const headed = available.includes("binary-headful") ? "binary-headful" : "docker-headful";
+            if (!available.includes(headed)) throw new Error("A headed browser is required but unavailable.");
+            await new Promise((resolve) => setTimeout(resolve, PROTECTED_INTERVAL_MS));
+            const index = err.stepIndex;
+            const remaining = pipeline.steps.slice(index);
+            const insertedNavigation = remaining[0]?.type !== "navigate";
+            if (insertedNavigation) remaining.unshift({ type: "navigate", url: err.url });
+            const resumed = await this.executeWithMode(userId, token, { ...pipeline, steps: remaining }, headed, instanceId, true);
+            const offset = index - Number(insertedNavigation);
+            resumed.results = [...err.results, ...resumed.results.filter((r) => !insertedNavigation || r.stepIndex > 0).map((r) => ({ ...r, stepIndex: r.stepIndex + offset }))];
+            resumed.completedSteps = index + Math.max(0, resumed.completedSteps - Number(insertedNavigation));
+            resumed.totalSteps = pipeline.steps.length;
+            resumed.obstacles = [...err.obstacles, ...resumed.obstacles];
+            if (resumed.error) {
+              resumed.error.failedAtStep = Math.max(index, resumed.error.failedAtStep + offset);
+              resumed.error.failedStep = pipeline.steps[resumed.error.failedAtStep];
+            }
+            resumed.durationMs = Date.now() - startTime;
+            resumed.modeEscalated = true;
+            resumed.modeUsed = headed;
+            return resumed;
+          }
           if (result.ok) {
             const blockResult = await detectBlock(page);
             if (blockResult.blocked) {
@@ -10557,6 +10662,8 @@ var FetchService;
 var init_fetch_service = __esm({
   "src/lib/execution/fetch-service.ts"() {
     "use strict";
+    init_site_protection();
+    init_launcher();
     init_launcher();
     init_stealth();
     init_humanize();
@@ -10587,12 +10694,30 @@ var init_fetch_service = __esm({
           const { browser, name: browserName } = await getBrowserWithFallback(preferredBrowser);
           context = await browser.newContext(stealthContextOptions({ locale, extraHTTPHeaders: { ...headers } }, userId ?? void 0));
           if (sessionData) await injectCookies(context, sessionData);
-          const page = await context.newPage();
+          let page = await context.newPage();
           await applyStealthToPage(page);
           await page.goto(url, { waitUntil: waitUntil || "domcontentloaded", timeout: TIMEOUTS.NAVIGATION });
+          let protection = false;
+          const pivotIfDetected = async () => {
+            if (protection || !await detectSiteProtection(page)) return;
+            protection = true;
+            const targetUrl = page.url();
+            const cookies = await context.cookies();
+            await context.close();
+            context = await (await getHeadedBrowser()).newContext(stealthContextOptions({ locale, extraHTTPHeaders: { ...headers } }, userId ?? void 0));
+            await context.addCookies(cookies);
+            page = await context.newPage();
+            await applyStealthToPage(page);
+            await new Promise((resolve) => setTimeout(resolve, PROTECTED_INTERVAL_MS));
+            await paceProtectedDomain(new URL(targetUrl).hostname);
+            await page.goto(targetUrl, { waitUntil, timeout: TIMEOUTS.NAVIGATION });
+          };
+          await pivotIfDetected();
           if (sessionData) await injectStorage(page, sessionData);
           if (waitForSelector) await page.waitForSelector(waitForSelector, { timeout: TIMEOUTS.SELECTOR_WAIT });
           for (const action of actions) {
+            await pivotIfDetected();
+            if (protection) await paceProtectedDomain(new URL(page.url()).hostname);
             switch (action.type) {
               case "click":
                 await page.click(action.selector);
@@ -10623,6 +10748,7 @@ var init_fetch_service = __esm({
                 break;
             }
           }
+          await pivotIfDetected();
           const finalUrl = page.url();
           const html = returnHtml ? await page.content() : void 0;
           const result = extract2 ? await page.evaluate(extract2) : void 0;

package/dist/local-server.cjs

@@ -551,6 +551,11 @@ function findChromeExecutable() {
   return void 0;
 }
 var cachedBrowser = null;
+var headedBrowser = null;
+async function getHeadedBrowser() {
+  if (!headedBrowser?.isConnected()) headedBrowser = await import_patchright.chromium.launch({ headless: false, args: STEALTH_ARGS });
+  return headedBrowser;
+}
 async function getBrowser(_name = "chromium") {
   if (cachedBrowser) {
     if (cachedBrowser.isConnected()) return cachedBrowser;
@@ -568,6 +573,11 @@ async function getBrowser(_name = "chromium") {
   return cachedBrowser;
 }
 async function closeBrowser() {
+  if (headedBrowser) {
+    await headedBrowser.close().catch(() => {
+    });
+    headedBrowser = null;
+  }
   if (!cachedBrowser) return;
   try {
     await cachedBrowser.close();
@@ -1880,6 +1890,53 @@ var RefStore = class {
   }
 };
 
+// src/lib/site-protection.ts
+var PROTECTED_INTERVAL_MS = 3e3;
+var nextRequest = /* @__PURE__ */ new Map();
+async function paceProtectedDomain(domain) {
+  const now = Date.now();
+  const slot = Math.max(now, nextRequest.get(domain) || 0);
+  nextRequest.set(domain, slot + PROTECTED_INTERVAL_MS);
+  if (slot > now) await new Promise((resolve) => setTimeout(resolve, slot - now));
+}
+async function detectSiteProtection(page) {
+  const cookies = await page.context().cookies(page.url()).catch(() => []);
+  if (cookies.some((c) => /^(incap_ses_|visid_incap_|nlbi_)/i.test(c.name))) return "imperva";
+  if (cookies.some((c) => /^(?:_abck|bm_sz|ak_bmsc|bm_sv)$/.test(c.name))) return "akamai";
+  const html = await page.content().catch(() => "");
+  if (/\/[_]Incapsula_Resource|incapsula incident id|powered by imperva/i.test(html)) return "imperva";
+  if (/errors\.edgesuite\.net|akamai bot manager/i.test(html)) return "akamai";
+  return null;
+}
+var HeadedRequiredError = class extends Error {
+  constructor(url) {
+    super("This site requires a headed browser.");
+    this.url = url;
+  }
+  url;
+  stepIndex = 0;
+  results = [];
+  obstacles = [];
+};
+function protectionHooks(headless, respectful = false) {
+  return {
+    before: async (page, step) => {
+      const protection = await detectSiteProtection(page);
+      if (protection) {
+        respectful = true;
+        if (headless) throw new HeadedRequiredError(page.url());
+      }
+      if (respectful) await paceProtectedDomain(new URL(step.url || page.url()).hostname);
+    },
+    after: async (page) => {
+      const protection = await detectSiteProtection(page);
+      if (!protection) return;
+      respectful = true;
+      if (headless) throw new HeadedRequiredError(page.url());
+    }
+  };
+}
+
 // src/lib/execution/pipeline-executor.ts
 var import_playwright_core = require("playwright-core");
 
@@ -4912,10 +4969,12 @@ function getSuggestion(errorType, step) {
   }
 }
 var PipelineRunner = class {
-  constructor(ctx) {
+  constructor(ctx, hooks) {
     this.ctx = ctx;
+    this.hooks = hooks;
   }
   ctx;
+  hooks;
   async run(initialPage, pipeline) {
     const tracker = new TabTracker(initialPage.context(), initialPage);
     try {
@@ -4961,12 +5020,19 @@ var PipelineRunner = class {
       const monitor = new StaleStateMonitor(page, staleTimeoutMs);
       let stepResult;
       try {
+        await this.hooks?.before(page, step);
         stepResult = await monitor.withMonitoring(async () => {
           const r = await executeAction(page, step, this.ctx, monitor);
+          if (step.type === "navigate") await this.hooks?.after(page, step.url);
           r.stepIndex = i;
           return r;
         });
       } catch (err) {
+        if (err instanceof HeadedRequiredError) {
+          Object.assign(err, { stepIndex: i, results, obstacles });
+          await finishCapture();
+          throw err;
+        }
         const asError = err instanceof Error ? err : new Error(String(err));
         const errorType = classifyError(asError, step);
         const pageState = await capturePageState(tracker.active(), this.ctx, { screenshot: true });
@@ -5743,7 +5809,7 @@ var CdpRelay = class {
         return {};
       case "Browser.close":
         return {};
-      // patchright disconnect — don't kill the user's Chrome
+      // Disconnect without closing the user's Chrome.
       case "Target.setDiscoverTargets":
         return {};
       case "Target.getTargets":
@@ -5850,8 +5916,8 @@ var PipelineExecutor = class {
     this.deps = deps;
   }
   deps;
-  /** Runtime elicitation hook, set per-call via execute(). Consumed once by
-   *  executeLocal and immediately cleared. */
+  /** Shared runtime hook read by local and extension execution;
+   *  execute() clears it when the call finishes. */
   pendingElicitOtp;
   /** One pipeline per real tab at a time: chrome.debugger and the CDP relay
    *  can't share a tab, so concurrent executes on the same tab queue up here. */
@@ -5897,6 +5963,7 @@ var PipelineExecutor = class {
       mode = availableModes[0] || "headless";
     }
     let result = await this.executeWithMode(userId, token, pipeline, mode, instanceId);
+    mode = result.modeUsed || mode;
     if (!result.ok && autoEscalate && domain && result.error?.errorType === "bot-blocked") {
       const failedMode = mode;
       if (domain) this.deps.domainModes.recordFailure(domain, failedMode, result.error?.message || "blocked");
@@ -5933,11 +6000,11 @@ var PipelineExecutor = class {
     }
     return result;
   }
-  async executeWithMode(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE) {
+  async executeWithMode(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE, respectful = false) {
     if (mode === "docker-headful") {
-      return this.executeDocker(userId, token, pipeline);
+      return this.executeDocker(userId, token, pipeline, respectful);
     }
-    return this.executeLocal(userId, token, pipeline, mode, instanceId);
+    return this.executeLocal(userId, token, pipeline, mode, instanceId, respectful);
   }
   /** Execute against the user's real Chrome tab via the extension CDP relay,
    *  using the SAME PipelineRunner as every other mode. No session inject/extract
@@ -6030,7 +6097,7 @@ var PipelineExecutor = class {
     }
   }
   /** Execute via Docker session-manager. */
-  async executeDocker(userId, token, pipeline) {
+  async executeDocker(userId, token, pipeline, respectful = false) {
     let session = getSession(userId);
     if (!session) {
       const firstNav = pipeline.steps.find((s) => s.type === "navigate");
@@ -6039,7 +6106,7 @@ var PipelineExecutor = class {
     }
     resetTimeout(userId);
     const ctx = this.deps.refStore.makeContext(userId, token);
-    const runner = new PipelineRunner(ctx);
+    const runner = new PipelineRunner(ctx, protectionHooks(false, respectful));
     const result = await runner.run(session.page, pipeline);
     if (result.ok) {
       const blockResult = await detectBlock(session.page);
@@ -6066,7 +6133,7 @@ var PipelineExecutor = class {
     return result;
   }
   /** Execute via local Chrome daemon (headless + binary-headful). */
-  async executeLocal(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE) {
+  async executeLocal(userId, token, pipeline, mode, instanceId = DEFAULT_INSTANCE, respectful = false) {
     const startTime = Date.now();
     let acquired = false;
     try {
@@ -6088,8 +6155,38 @@ var PipelineExecutor = class {
       const ctx = this.deps.refStore.makeContext(userId, token);
       if (sessionData) ctx.sessionData = sessionData;
       if (this.pendingElicitOtp) ctx.elicitOtp = this.pendingElicitOtp;
-      const runner = new PipelineRunner(ctx);
-      const result = await runner.run(page, pipeline);
+      const runner = new PipelineRunner(ctx, protectionHooks(mode === "headless", respectful));
+      let result;
+      try {
+        result = await runner.run(page, pipeline);
+      } catch (err) {
+        if (!(err instanceof HeadedRequiredError)) throw err;
+        this.deps.daemon.release(mode, instanceId);
+        acquired = false;
+        await this.deps.daemon.stopMode(mode, instanceId);
+        const available = this.deps.availableModes();
+        const headed = available.includes("binary-headful") ? "binary-headful" : "docker-headful";
+        if (!available.includes(headed)) throw new Error("A headed browser is required but unavailable.");
+        await new Promise((resolve) => setTimeout(resolve, PROTECTED_INTERVAL_MS));
+        const index = err.stepIndex;
+        const remaining = pipeline.steps.slice(index);
+        const insertedNavigation = remaining[0]?.type !== "navigate";
+        if (insertedNavigation) remaining.unshift({ type: "navigate", url: err.url });
+        const resumed = await this.executeWithMode(userId, token, { ...pipeline, steps: remaining }, headed, instanceId, true);
+        const offset = index - Number(insertedNavigation);
+        resumed.results = [...err.results, ...resumed.results.filter((r) => !insertedNavigation || r.stepIndex > 0).map((r) => ({ ...r, stepIndex: r.stepIndex + offset }))];
+        resumed.completedSteps = index + Math.max(0, resumed.completedSteps - Number(insertedNavigation));
+        resumed.totalSteps = pipeline.steps.length;
+        resumed.obstacles = [...err.obstacles, ...resumed.obstacles];
+        if (resumed.error) {
+          resumed.error.failedAtStep = Math.max(index, resumed.error.failedAtStep + offset);
+          resumed.error.failedStep = pipeline.steps[resumed.error.failedAtStep];
+        }
+        resumed.durationMs = Date.now() - startTime;
+        resumed.modeEscalated = true;
+        resumed.modeUsed = headed;
+        return resumed;
+      }
       if (result.ok) {
         const blockResult = await detectBlock(page);
         if (blockResult.blocked) {
@@ -6180,12 +6277,30 @@ var FetchService = class {
       const { browser, name: browserName } = await getBrowserWithFallback(preferredBrowser);
       context = await browser.newContext(stealthContextOptions({ locale, extraHTTPHeaders: { ...headers } }, userId ?? void 0));
       if (sessionData) await injectCookies(context, sessionData);
-      const page = await context.newPage();
+      let page = await context.newPage();
       await applyStealthToPage(page);
       await page.goto(url, { waitUntil: waitUntil || "domcontentloaded", timeout: TIMEOUTS.NAVIGATION });
+      let protection = false;
+      const pivotIfDetected = async () => {
+        if (protection || !await detectSiteProtection(page)) return;
+        protection = true;
+        const targetUrl = page.url();
+        const cookies = await context.cookies();
+        await context.close();
+        context = await (await getHeadedBrowser()).newContext(stealthContextOptions({ locale, extraHTTPHeaders: { ...headers } }, userId ?? void 0));
+        await context.addCookies(cookies);
+        page = await context.newPage();
+        await applyStealthToPage(page);
+        await new Promise((resolve) => setTimeout(resolve, PROTECTED_INTERVAL_MS));
+        await paceProtectedDomain(new URL(targetUrl).hostname);
+        await page.goto(targetUrl, { waitUntil, timeout: TIMEOUTS.NAVIGATION });
+      };
+      await pivotIfDetected();
       if (sessionData) await injectStorage(page, sessionData);
       if (waitForSelector) await page.waitForSelector(waitForSelector, { timeout: TIMEOUTS.SELECTOR_WAIT });
       for (const action of actions) {
+        await pivotIfDetected();
+        if (protection) await paceProtectedDomain(new URL(page.url()).hostname);
         switch (action.type) {
           case "click":
             await page.click(action.selector);
@@ -6216,6 +6331,7 @@ var FetchService = class {
             break;
         }
       }
+      await pivotIfDetected();
       const finalUrl = page.url();
       const html = returnHtml ? await page.content() : void 0;
       const result = extract2 ? await page.evaluate(extract2) : void 0;

package/dist/mcp-server.cjs

@@ -2087,6 +2087,8 @@ CRITICAL RULES:
 7. If the browser crashes, call "session restart" and retry. Don't panic.
 8. ALWAYS call "session" action=stop when you are done with browser work. It saves session state and frees the browser. Idle browsers are auto-reclaimed, but don't rely on that.
 
+If headless detects Imperva or Akamai, immediately stop headless and pivot to headed in the same run; do not wait for a block, retry headless, or defer the switch to a future request. Use headed from the outset when the provider is already known in the current task. On detection, switch to headed and use respectful pacing (at least 3 seconds between top-level actions; no parallel bursts or repeated blocked retries). Do not announce the provider or explain the switch unless the user asks why. Only when asked, explain that the site is covered by Imperva or Akamai and iframer adjusts its browser mode and pacing to respect it.
+
 BROWSER MODES: Don't specify options.mode \u2014 iframer auto-selects and auto-escalates (headless \u2192 binary-headful). Only set a mode if the user explicitly asks.
 
 CAPTCHA: In binary-headful mode, ask the user to solve it in the visible window. In docker-headful mode, use the "solve-captcha" step.

package/extension/manifest.json

@@ -1,7 +1,7 @@
 {
   "manifest_version": 3,
   "name": "iframer",
-  "version": "0.10.5",
+  "version": "0.10.6",
   "description": "Let iframer drive a Chrome tab you choose, on your real logged-in session.",
   "key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp2wbce98k+xDnEgtuzLyNXYBLEDY0nqChuvolb94B3VV1HsHFUulUyNe92eN6MjQuadDohQ2j5mj7zWs4yQoixgTQPe7Gm/+GHWdgxwM2s9t5opnLHwF90LPiJm6q1NGWLM4aK6JT+2I6FnUGmj2bzB4WIlf55C3aMEBoXTqIzG2U8s2cxkXGebWWqbPlMK/Phsmsvo4POWnidbGHPhBOq0iyPTu7RdSDlNdJdJ+dtdNXkMmltKUS881MQI9T6GcCCKhIKcZrJGGnk0xphlWyl+ublTAO0tbQSVB/K7D3JmCxMv690TPtYUPPhGi+C6VCv3IeutLmNo60mvux3wbGQIDAQAB",
   "permissions": [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "iframer-toolkit",
-  "version": "0.10.5",
+  "version": "0.10.6",
   "description": "Browser access toolkit for AI agents — CLI and MCP server for headless/headful browsing, login flows, and pipeline execution.",
   "keywords": [
     "iframer",

package/skills/iframer.md

@@ -20,7 +20,8 @@ You are an expert at using the iframer browser automation toolkit. The user want
 2. **NEVER guess CSS selectors.** Use `snapshot` to see what's on the page, then use `@e` refs from the snapshot for click/fill/etc.
 3. **NEVER re-store credentials when login fails.** If credentials exist in `credentials list`, they are valid. Login failures are browser-mode or bot-detection problems, not credential problems. Retry with a stronger mode.
 4. **NEVER pass explicit selectors to the login step** unless the auto-detect failed AND you took a snapshot to find the right ones. The login step auto-detects email, password, submit, and OTP fields.
-5. **Let iframer handle mode escalation.** Don't hardcode modes. If you must pick one, prefer no mode (auto-select) or `binary-headful` for sites known to block headless.
+5. If headless detects Imperva or Akamai, immediately stop headless and pivot to headed in the same run; do not wait for a block, retry headless, or defer the switch to a future request. Use headed from the outset when the provider is already known in the current task. On detection, switch to headed and use respectful pacing (at least 3 seconds between top-level actions; no parallel bursts or repeated blocked retries). Do not announce the provider or explain the switch unless the user asks why. Only when asked, explain that the site is covered by Imperva or Akamai and iframer adjusts its browser mode and pacing to respect it.
+6. **Let iframer handle mode escalation.** Don't hardcode modes. If you must pick one, prefer no mode (auto-select) or `binary-headful` for sites known to block headless.
 
 ## How to build pipelines
 
@@ -199,7 +200,7 @@ session: { action: "restart" }
 ```
 This kills ALL running browser instances (local + Docker) and resets state. Credentials and knowledge cache are NOT affected. The next `execute` call launches a fresh browser automatically.
 
-**The execute tool also auto-recovers:** if it detects a crash (connection closed, timeout, ECONNREFUSED), it automatically restarts the browser and retries once before reporting failure. If auto-recovery also fails, the agent should call `session restart` manually and try again.
+**The execute tool also auto-recovers:** if it detects a *connection-level* crash (connection closed, ECONNREFUSED/ECONNRESET/EPIPE, socket hang up), it restarts the browser server and retries once before reporting failure. Timeouts are deliberately NOT auto-retried — a slow-but-healthy pipeline that hit the client timeout must not be re-run, or non-idempotent steps (form submits, purchases) would double-execute. If auto-recovery fails, or a step timed out, call `session restart` manually and continue from where you left off (don't blindly re-run the whole pipeline).
 
 ### Manage credentials
 
@@ -274,7 +275,7 @@ Many modern sites don't have email+password on one page. The login step handles
 
 ## Recovering from MCP disconnect
 
-The iframer MCP server runs Chrome in-process. If Chrome crashes hard enough (native segfault, out of memory), the entire MCP server process dies and Claude Code shows "MCP disconnected."
+The iframer MCP server does NOT run Chrome — Chrome lives in a separate, shared local background server (the daemon on 127.0.0.1). The MCP process is just a thin HTTP/stdio client. So a browser crash can't take the MCP down with it: if Chrome dies, the daemon's own recovery handles it and the MCP keeps running. A real "MCP disconnected" in Claude Code means the MCP client process itself dropped (session killed, pipe closed), not a browser crash — the daemon and its warm browsers survive that too.
 
 **Recovery is simple — Claude Code auto-reconnects MCP servers.** Just wait 2-3 seconds and retry the tool call. Don't:
 - Try to run `claude mcp` commands