@imenam/simple-scraper 1.0.23 → 1.0.27

package/README.md

@@ -1,398 +1,438 @@
-# simple-scraper-mcp
-
-A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
-
-## Features
-
-- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
-- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
-- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
-- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
-- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
-- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
-- **Cookie support** — Load Netscape-format cookie files automatically before each request
-- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
-
-## Requirements
-
-- Node.js >= 18
-- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
-
-## Installation
-
-```bash
-npx @imenam/simple-scraper
-```
-
-Or install globally:
-
-```bash
-npm install -g @imenam/simple-scraper
-simple-scraper
-```
-
-## MCP Configuration
-
-### Quick setup
-
-From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
-
-```bash
-npx @imenam/simple-scraper --claude-setup-mcp
-```
-
-It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
-folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
-viewport, and prints the remaining optional variables. An empty variable is simply inactive
-(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
-other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
-`--force` is passed.
-
-## Environment Variables
-
-| Variable | Required | Default | Description |
-|----------|----------|---------|-------------|
-| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
-| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
-| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
-| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
-| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
-| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. The folder is created automatically if it does not exist. All files are loaded and merged automatically before each request. |
-| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
-| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
-| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
-| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
-| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
-| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
-| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
-
-## Configuration
-
-Copy `.env.example` to `.env` and configure the variables:
-
-```env
-# Puppeteer options (optional)
-PUPPETEER_HEADLESS=true
-PUPPETEER_TIMEOUT=30000
-PUPPETEER_VIEWPORT_WIDTH=1280
-PUPPETEER_VIEWPORT_HEIGHT=800
-PUPPETEER_DEVICE_SCALE_FACTOR=1
-
-# Optional: path to a folder containing Netscape-format cookie files (.txt)
-# All files in this folder will be loaded automatically before each request.
-# COOKIES_DIR=/path/to/cookies
-
-# GUI (optional) — required to enable the cookie manager interface
-# PROXY_URL=http://localhost:3000
-# PROXY_APP_PATH=/simple-scraper-mcp
-# PROXY_APP_NAME=Simple Scraper MCP
-# Optional: groups this app under a collapsible section of the proxy dashboard
-# APP_GROUP=Scraping
-```
-
-### The process environment wins over `.env`
-
-Any variable already set in the environment of the process that starts this MCP
-(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
-such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
-this repository's `.env`**. This is dotenv's default behaviour, and the usual
-Unix convention: the caller wins.
-
-The `.env` is still reloaded on the fly — the GUI writes to it and the change is
-picked up without restarting the server — but that reload can no longer clobber
-what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
-`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
-applies when nobody passed one.
-
-To change a variable that the caller provides, change it where the caller sets
-it; editing `.env` will have no effect on that key.
-
-## Usage with Claude Desktop
-
-Add the following to your `claude_desktop_config.json`. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "COOKIES_DIR": "/path/to/your/cookies"
-      }
-    }
-  }
-}
-```
-
-## Usage with Cursor
-
-In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["-y", "@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
-
-## MCP Tools
-
-### `scrape_page`
-
-Navigate to a URL and return the full rendered HTML.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page to scrape |
-| `wait_for` | string | | CSS selector to wait for before capturing HTML |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `execute_js`
-
-Navigate to a URL and execute custom JavaScript in the page context.
-
-The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
-
-```js
-new Function(script)();
-```
-
-To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
-
-Example:
-
-```js
-return {
-  title: document.title,
-  url: window.location.href,
-  text: document.body.innerText.slice(0, 500)
-};
-```
-
-For asynchronous work, return a promise, for example with an async IIFE:
-
-```js
-return (async () => {
-  const response = await fetch('/api/data');
-  return await response.json();
-})();
-```
-
-Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_page_inputs`
-
-Extract all form inputs from a page as a structured JSON object.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
-| `wait_for` | string | | CSS selector to wait for before extracting |
-| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_show_page`
-
-Parse a detail page and extract key-value blocks and tables as structured JSON.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `keys_map` | object | | Map of HTML label → JS key for field name translation |
-| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
-| `tables_max_items` | number | | Max rows per table (default: 2) |
-| `wait_for` | string | | CSS selector to wait for before extraction |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
----
-
-## Interactive Sessions
-
-Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
-
-### Typical workflow
-
-```
-open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
-```
-
-### `open_session`
-
-Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
-
-### `close_session`
-
-Close a session and free its resources. Always call this when you are done.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID returned by `open_session` |
-
-### `list_sessions`
-
-List all currently active sessions with their IDs and timestamps. No parameters.
-
-### `session_goto`
-
-Navigate the session to a new URL without closing it.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for after navigation |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_click`
-
-Click an element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the element to click |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_type`
-
-Type text into an input element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the input element |
-| `text` | string | ✅ | Text to type |
-| `clear` | boolean | | Clear the field before typing (default: false) |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_wait_for`
-
-Wait for a CSS selector to appear in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector to wait for |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_evaluate`
-
-Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_html`
-
-Return the current full rendered HTML of the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-
-### `screenshot`
-
-Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
-
-The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
-| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
-| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `selector` | string | | CSS selector of a specific element to capture |
-| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
-| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
-| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
-| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
-| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` writes it on the agent machine, `both` does both |
-| `destination_path` | string | ⚠️ | **Absolute directory** on the agent machine where the image is written. Created if missing. Required when `output` is `file` or `both` |
-| `filename` | string | ⚠️ | Name the image gets there, e.g. `home.png`. No path separators. Required when `output` is `file` or `both` |
-| `destination_overwrite` | boolean | | Replace the destination file if it exists (default: `false`) |
-| `destination_relay` | boolean | | Filled in by the mcp-http-gateway relay, never by the agent |
-
-> **Where the file lands.** There is no default destination: one would resolve on *this
-> server's* disk, which is not necessarily the agent's — a file nobody finds rather than an
-> error. In stdio the server writes it itself; behind
-> [`@imenam/mcp-http-gateway`](https://www.npmjs.com/package/@imenam/mcp-http-gateway) it
-> returns the bytes and the relay, which runs on the agent machine, writes them at
-> `destination_path`. That path requires `GATEWAY_INLINE_WRITE_ROOTS` in the relay's `env`
-> block. The image never passes through the conversation unless `output` includes `inline`.
-
----
-
-## Cookie Files
-
-Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request. If the folder does not exist yet, it is created automatically (including any missing parent directories), so a fresh `COOKIES_DIR` no longer raises an error.
-
-## GUI (Optional)
-
-When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
-
-### Network exposure
-
-The GUI listens on `127.0.0.1` only: other machines reach it through the proxy or the gateway manager. A request guard from [`@imenam/mcp-gui-interface`](https://www.npmjs.com/package/@imenam/mcp-gui-interface) answers `403` to any request whose `Host` header is not `localhost`, `127.0.0.1` or `::1` (DNS rebinding protection), and to modifying requests (`POST`, `DELETE`…) sent from another site (foreign `Sec-Fetch-Site` or `Origin`).
-
-## License
-
-ISC
+# simple-scraper-mcp
+
+A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Logged-in sites keep their session: cookies are saved per site, renewed automatically, kept alive on a schedule, and restored through an assisted login when the site ends the session. An optional GUI manages sites and shows live sessions.
+
+## Features
+
+- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
+- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
+- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
+- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
+- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
+- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
+- **Sites** — Per-site cookies loaded into the browser, written back as the site renews them, kept alive on a schedule; assisted login in a visible window when a session is lost (see [Sites](#sites-connected-accounts))
+- **Optional GUI** — Sites, live sessions and activity, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
+
+## Requirements
+
+- Node.js >= 18
+- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
+
+## Installation
+
+```bash
+npx @imenam/simple-scraper
+```
+
+Or install globally:
+
+```bash
+npm install -g @imenam/simple-scraper
+simple-scraper
+```
+
+## MCP Configuration
+
+### Quick setup
+
+From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
+
+```bash
+npx @imenam/simple-scraper --claude-setup-mcp
+```
+
+It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
+folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
+viewport, and prints the remaining optional variables. An empty variable is simply inactive
+(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
+other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
+`--force` is passed.
+
+## Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
+| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
+| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
+| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
+| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
+| `SCRAPER_DATA_DIR` | No | - | Absolute path to the folder where sites are stored (settings, cookies, activity). Created if missing. Without it, sites are disabled. See [Sites](#sites-connected-accounts). |
+| `COOKIES_DIR` | No | - | Other name for `SCRAPER_DATA_DIR`, so existing configurations keep working. If both are set, they must point to the same folder, otherwise the server refuses to start. |
+| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
+| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
+| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
+| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
+| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
+| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
+| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
+
+## Configuration
+
+Copy `.env.example` to `.env` and configure the variables:
+
+```env
+# Puppeteer options (optional)
+PUPPETEER_HEADLESS=true
+PUPPETEER_TIMEOUT=30000
+PUPPETEER_VIEWPORT_WIDTH=1280
+PUPPETEER_VIEWPORT_HEIGHT=800
+PUPPETEER_DEVICE_SCALE_FACTOR=1
+
+# Optional: folder where sites are stored (settings, cookies, activity).
+# COOKIES_DIR is accepted as another name for the same setting.
+# SCRAPER_DATA_DIR=/path/to/scraper-data
+
+# GUI (optional) — required to enable the web interface
+# PROXY_URL=http://localhost:3000
+# PROXY_APP_PATH=/simple-scraper-mcp
+# PROXY_APP_NAME=Simple Scraper MCP
+# Optional: groups this app under a collapsible section of the proxy dashboard
+# APP_GROUP=Scraping
+```
+
+### The process environment wins over `.env`
+
+Any variable already set in the environment of the process that starts this MCP
+(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
+such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
+this repository's `.env`**. This is dotenv's default behaviour, and the usual
+Unix convention: the caller wins.
+
+The `.env` is still reloaded on the fly — the GUI writes to it and the change is
+picked up without restarting the server — but that reload can no longer clobber
+what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
+`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
+applies when nobody passed one.
+
+To change a variable that the caller provides, change it where the caller sets
+it; editing `.env` will have no effect on that key.
+
+## Usage with Claude Desktop
+
+Add the following to your `claude_desktop_config.json`. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+To use logged-in sites, add `SCRAPER_DATA_DIR` (see [Sites](#sites-connected-accounts)):
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data"
+      }
+    }
+  }
+}
+```
+
+## Usage with Cursor
+
+In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["-y", "@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
+
+## MCP Tools
+
+### `scrape_page`
+
+Navigate to a URL and return the full rendered HTML.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page to scrape |
+| `wait_for` | string | | CSS selector to wait for before capturing HTML |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `execute_js`
+
+Navigate to a URL and execute custom JavaScript in the page context.
+
+The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
+
+```js
+new Function(script)();
+```
+
+To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
+
+Example:
+
+```js
+return {
+  title: document.title,
+  url: window.location.href,
+  text: document.body.innerText.slice(0, 500)
+};
+```
+
+For asynchronous work, return a promise, for example with an async IIFE:
+
+```js
+return (async () => {
+  const response = await fetch('/api/data');
+  return await response.json();
+})();
+```
+
+Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_page_inputs`
+
+Extract all form inputs from a page as a structured JSON object.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
+| `wait_for` | string | | CSS selector to wait for before extracting |
+| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_show_page`
+
+Parse a detail page and extract key-value blocks and tables as structured JSON.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `keys_map` | object | | Map of HTML label → JS key for field name translation |
+| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
+| `tables_max_items` | number | | Max rows per table (default: 2) |
+| `wait_for` | string | | CSS selector to wait for before extraction |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+---
+
+## Interactive Sessions
+
+Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
+
+### Typical workflow
+
+```
+open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
+```
+
+### `open_session`
+
+Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
+
+### `close_session`
+
+Close a session and free its resources. Always call this when you are done.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID returned by `open_session` |
+
+### `list_sessions`
+
+List all currently active sessions with their IDs and timestamps. No parameters.
+
+### `session_goto`
+
+Navigate the session to a new URL without closing it.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for after navigation |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_click`
+
+Click an element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the element to click |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_type`
+
+Type text into an input element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the input element |
+| `text` | string | ✅ | Text to type |
+| `clear` | boolean | | Clear the field before typing (default: false) |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_wait_for`
+
+Wait for a CSS selector to appear in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector to wait for |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_evaluate`
+
+Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_html`
+
+Return the current full rendered HTML of the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+
+### `screenshot`
+
+Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
+
+The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
+| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
+| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `selector` | string | | CSS selector of a specific element to capture |
+| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
+| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
+| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
+| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
+| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` writes it on the agent machine, `both` does both |
+| `destination_path` | string | ⚠️ | **Absolute directory** on the agent machine where the image is written. Created if missing. Required when `output` is `file` or `both`, except behind the relay, which keeps it on the agent side and only forwards `filename` |
+| `filename` | string | ⚠️ | Name the image gets there, e.g. `home.png`. No path separators. Required when `output` is `file` or `both` |
+| `destination_overwrite` | boolean | | Replace the destination file if it exists (default: `false`) |
+| `destination_relay` | boolean | | Filled in by the mcp-http-gateway relay, never by the agent |
+
+> **Where the file lands.** There is no default destination: one would resolve on *this
+> server's* disk, which is not necessarily the agent's — a file nobody finds rather than an
+> error. In stdio the server writes it itself; behind
+> [`@imenam/mcp-http-gateway`](https://www.npmjs.com/package/@imenam/mcp-http-gateway) it
+> returns the bytes and the relay, which runs on the agent machine, writes them at
+> `destination_path`. That path requires `GATEWAY_INLINE_WRITE_ROOTS` in the relay's `env`
+> block. The image never passes through the conversation unless `output` includes `inline`.
+
+---
+
+## Sites (connected accounts)
+
+A **site** is an account the scraper uses while logged in: a name, the domains it covers, its login page and the sign that proves a page is seen logged in. Sites are stored in the data folder (`SCRAPER_DATA_DIR`, or `COOKIES_DIR` under its other name) and managed from the GUI (its labels are in French).
+
+### How cookies are used
+
+- When the shared browser starts, it loads the cookies of every site. Any tool (`scrape_page`, `execute_js`, `open_session`, `screenshot`…) visiting one of a site's domains is therefore logged in, with nothing to pass to the tool.
+- After each page closes, and every 5 minutes, the cookies the browser holds for each site's domains are written back to that site: cookies renewed by the site are kept, cookies it deleted are removed. Session cookies (no expiry date) are kept too, so a login survives a restart of the MCP.
+
+### Login sign
+
+The sign tells the scraper whether a page is seen logged in:
+
+- **L'adresse contient** (the address contains): the page address (host and path, without query or anchor) contains the value, e.g. `crm.example.com/app`;
+- **Un élément est présent** (an element is present): a CSS selector that only exists once logged in, e.g. `#user-menu`.
+
+### Assisted login
+
+**Se reconnecter** opens a visible Chromium window, on a blank temporary profile, at the site's login page, on the machine running the MCP. You log in there yourself (credentials, two-factor authentication, SSO). As soon as the login sign appears, the window's cookies are saved for the site and the window closes; **Je suis connecté** saves them without waiting for the sign. Credentials never go through the MCP. **Tout reconnecter** chains every site to reconnect, one after the other.
+
+### Keep-alive
+
+When enabled, the keep-alive visits the site's **Page visitée** at a fixed interval (5 min to 1 h), day and night, so that the session does not expire from inactivity. Each visit is also a check: if the login sign no longer appears, the site is marked **À reconnecter**, shown in the banner of the Sites page (unless disabled for that site), and the keep-alive stops visiting it until the next login. The visited page is also used by **Vérifier maintenant**.
+
+The keep-alive cannot prevent a logout the site imposes on its own schedule (e.g. every night): it only avoids inactivity timeouts. After such a logout, the assisted login takes a few seconds per site.
+
+### Data folder layout
+
+| Path | Content |
+|------|---------|
+| `sites.json` | Settings and state of every site |
+| `events.json` | Activity: checks, lost sessions, logins, imports (last 1000 events) |
+| `cookies/<site id>.json` | Cookies of each site |
+
+Netscape cookie files (`.txt`) placed in this folder are not loaded. To start from an exported file, open the site and use **Importer cookies**.
+
+## GUI (Optional)
+
+When `PROXY_URL` is set, a web interface is registered with the MCP proxy:
+
+- **Sites** — every site with its state (logged in, to reconnect, not checked), its session and its keep-alive; banner and **Tout reconnecter** when sessions are lost; assisted login with a live preview of the login window.
+- **Site page** — domains, login page, login sign, keep-alive (interval, visited page), recent activity, cookie import and **Effacer la session**.
+- **Sessions** — the sessions opened by the agent, with a live view (refreshed every 2.5 s, can be paused), the last action, the matching site and the console messages.
+- **Activity** — the history of every site, filterable by site.
+
+The interface refreshes by polling: no connection stays open.
+
+### Network exposure
+
+The GUI listens on `127.0.0.1` only: other machines reach it through the proxy or the gateway manager. A request guard from [`@imenam/mcp-gui-interface`](https://www.npmjs.com/package/@imenam/mcp-gui-interface) answers `403` to any request whose `Host` header is not `localhost`, `127.0.0.1` or `::1` (DNS rebinding protection), and to modifying requests (`POST`, `DELETE`…) sent from another site (foreign `Sec-Fetch-Site` or `Origin`).
+
+## License
+
+ISC