@imenam/simple-scraper 1.0.24 → 1.0.28

package/README.md

@@ -1,398 +1,444 @@
-# simple-scraper-mcp
-
-A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
-
-## Features
-
-- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
-- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
-- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
-- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
-- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
-- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
-- **Cookie support** — Load Netscape-format cookie files automatically before each request
-- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
-
-## Requirements
-
-- Node.js >= 18
-- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
-
-## Installation
-
-```bash
-npx @imenam/simple-scraper
-```
-
-Or install globally:
-
-```bash
-npm install -g @imenam/simple-scraper
-simple-scraper
-```
-
-## MCP Configuration
-
-### Quick setup
-
-From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
-
-```bash
-npx @imenam/simple-scraper --claude-setup-mcp
-```
-
-It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
-folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
-viewport, and prints the remaining optional variables. An empty variable is simply inactive
-(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
-other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
-`--force` is passed.
-
-## Environment Variables
-
-| Variable | Required | Default | Description |
-|----------|----------|---------|-------------|
-| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
-| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
-| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
-| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
-| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
-| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. The folder is created automatically if it does not exist. All files are loaded and merged automatically before each request. |
-| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
-| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
-| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
-| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
-| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
-| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
-| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
-
-## Configuration
-
-Copy `.env.example` to `.env` and configure the variables:
-
-```env
-# Puppeteer options (optional)
-PUPPETEER_HEADLESS=true
-PUPPETEER_TIMEOUT=30000
-PUPPETEER_VIEWPORT_WIDTH=1280
-PUPPETEER_VIEWPORT_HEIGHT=800
-PUPPETEER_DEVICE_SCALE_FACTOR=1
-
-# Optional: path to a folder containing Netscape-format cookie files (.txt)
-# All files in this folder will be loaded automatically before each request.
-# COOKIES_DIR=/path/to/cookies
-
-# GUI (optional) — required to enable the cookie manager interface
-# PROXY_URL=http://localhost:3000
-# PROXY_APP_PATH=/simple-scraper-mcp
-# PROXY_APP_NAME=Simple Scraper MCP
-# Optional: groups this app under a collapsible section of the proxy dashboard
-# APP_GROUP=Scraping
-```
-
-### The process environment wins over `.env`
-
-Any variable already set in the environment of the process that starts this MCP
-(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
-such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
-this repository's `.env`**. This is dotenv's default behaviour, and the usual
-Unix convention: the caller wins.
-
-The `.env` is still reloaded on the fly — the GUI writes to it and the change is
-picked up without restarting the server — but that reload can no longer clobber
-what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
-`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
-applies when nobody passed one.
-
-To change a variable that the caller provides, change it where the caller sets
-it; editing `.env` will have no effect on that key.
-
-## Usage with Claude Desktop
-
-Add the following to your `claude_desktop_config.json`. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "COOKIES_DIR": "/path/to/your/cookies"
-      }
-    }
-  }
-}
-```
-
-## Usage with Cursor
-
-In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["-y", "@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
-
-## MCP Tools
-
-### `scrape_page`
-
-Navigate to a URL and return the full rendered HTML.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page to scrape |
-| `wait_for` | string | | CSS selector to wait for before capturing HTML |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `execute_js`
-
-Navigate to a URL and execute custom JavaScript in the page context.
-
-The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
-
-```js
-new Function(script)();
-```
-
-To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
-
-Example:
-
-```js
-return {
-  title: document.title,
-  url: window.location.href,
-  text: document.body.innerText.slice(0, 500)
-};
-```
-
-For asynchronous work, return a promise, for example with an async IIFE:
-
-```js
-return (async () => {
-  const response = await fetch('/api/data');
-  return await response.json();
-})();
-```
-
-Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_page_inputs`
-
-Extract all form inputs from a page as a structured JSON object.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
-| `wait_for` | string | | CSS selector to wait for before extracting |
-| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_show_page`
-
-Parse a detail page and extract key-value blocks and tables as structured JSON.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `keys_map` | object | | Map of HTML label → JS key for field name translation |
-| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
-| `tables_max_items` | number | | Max rows per table (default: 2) |
-| `wait_for` | string | | CSS selector to wait for before extraction |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
----
-
-## Interactive Sessions
-
-Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
-
-### Typical workflow
-
-```
-open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
-```
-
-### `open_session`
-
-Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
-
-### `close_session`
-
-Close a session and free its resources. Always call this when you are done.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID returned by `open_session` |
-
-### `list_sessions`
-
-List all currently active sessions with their IDs and timestamps. No parameters.
-
-### `session_goto`
-
-Navigate the session to a new URL without closing it.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for after navigation |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_click`
-
-Click an element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the element to click |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_type`
-
-Type text into an input element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the input element |
-| `text` | string | ✅ | Text to type |
-| `clear` | boolean | | Clear the field before typing (default: false) |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_wait_for`
-
-Wait for a CSS selector to appear in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector to wait for |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_evaluate`
-
-Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_html`
-
-Return the current full rendered HTML of the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-
-### `screenshot`
-
-Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
-
-The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
-| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
-| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `selector` | string | | CSS selector of a specific element to capture |
-| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
-| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
-| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
-| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
-| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` writes it on the agent machine, `both` does both |
-| `destination_path` | string | ⚠️ | **Absolute directory** on the agent machine where the image is written. Created if missing. Required when `output` is `file` or `both` |
-| `filename` | string | ⚠️ | Name the image gets there, e.g. `home.png`. No path separators. Required when `output` is `file` or `both` |
-| `destination_overwrite` | boolean | | Replace the destination file if it exists (default: `false`) |
-| `destination_relay` | boolean | | Filled in by the mcp-http-gateway relay, never by the agent |
-
-> **Where the file lands.** There is no default destination: one would resolve on *this
-> server's* disk, which is not necessarily the agent's — a file nobody finds rather than an
-> error. In stdio the server writes it itself; behind
-> [`@imenam/mcp-http-gateway`](https://www.npmjs.com/package/@imenam/mcp-http-gateway) it
-> returns the bytes and the relay, which runs on the agent machine, writes them at
-> `destination_path`. That path requires `GATEWAY_INLINE_WRITE_ROOTS` in the relay's `env`
-> block. The image never passes through the conversation unless `output` includes `inline`.
-
----
-
-## Cookie Files
-
-Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request. If the folder does not exist yet, it is created automatically (including any missing parent directories), so a fresh `COOKIES_DIR` no longer raises an error.
-
-## GUI (Optional)
-
-When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
-
-### Network exposure
-
-The GUI listens on `127.0.0.1` only: other machines reach it through the proxy or the gateway manager. A request guard from [`@imenam/mcp-gui-interface`](https://www.npmjs.com/package/@imenam/mcp-gui-interface) answers `403` to any request whose `Host` header is not `localhost`, `127.0.0.1` or `::1` (DNS rebinding protection), and to modifying requests (`POST`, `DELETE`…) sent from another site (foreign `Sec-Fetch-Site` or `Origin`).
-
-## License
-
-ISC
+# simple-scraper-mcp
+
+A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Logged-in sites keep their session: cookies are saved per site, renewed automatically, kept alive on a schedule, and restored through an assisted login when the site ends the session. An optional GUI manages sites and shows live sessions.
+
+## Features
+
+- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
+- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
+- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
+- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
+- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
+- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
+- **Sites** — Per-site cookies loaded into the browser, written back as the site renews them, kept alive on a schedule; assisted login in a visible window when a session is lost (see [Sites](#sites-connected-accounts))
+- **Optional GUI** — Sites, live sessions and activity, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
+
+## Requirements
+
+- Node.js >= 18
+- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
+
+## Installation
+
+```bash
+npx @imenam/simple-scraper
+```
+
+Or install globally:
+
+```bash
+npm install -g @imenam/simple-scraper
+simple-scraper
+```
+
+## MCP Configuration
+
+### Quick setup
+
+From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
+
+```bash
+npx @imenam/simple-scraper --claude-setup-mcp
+```
+
+It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
+folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
+viewport, and prints the remaining optional variables. An empty variable is simply inactive
+(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
+other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
+`--force` is passed.
+
+## Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
+| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
+| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
+| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
+| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
+| `SCRAPER_DATA_DIR` | No | - | Absolute path to the folder where sites are stored (settings, cookies, activity). Created if missing. Without it, sites are disabled. See [Sites](#sites-connected-accounts). |
+| `COOKIES_DIR` | No | - | Other name for `SCRAPER_DATA_DIR`, so existing configurations keep working. If both are set, they must point to the same folder, otherwise the server refuses to start. |
+| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
+| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
+| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
+| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
+| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
+| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
+| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
+
+## Configuration
+
+Copy `.env.example` to `.env` and configure the variables:
+
+```env
+# Puppeteer options (optional)
+PUPPETEER_HEADLESS=true
+PUPPETEER_TIMEOUT=30000
+PUPPETEER_VIEWPORT_WIDTH=1280
+PUPPETEER_VIEWPORT_HEIGHT=800
+PUPPETEER_DEVICE_SCALE_FACTOR=1
+
+# Optional: folder where sites are stored (settings, cookies, activity).
+# COOKIES_DIR is accepted as another name for the same setting.
+# SCRAPER_DATA_DIR=/path/to/scraper-data
+
+# GUI (optional) — required to enable the web interface
+# PROXY_URL=http://localhost:3000
+# PROXY_APP_PATH=/simple-scraper-mcp
+# PROXY_APP_NAME=Simple Scraper MCP
+# Optional: groups this app under a collapsible section of the proxy dashboard
+# APP_GROUP=Scraping
+```
+
+### The process environment wins over `.env`
+
+Any variable already set in the environment of the process that starts this MCP
+(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
+such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
+this repository's `.env`**. This is dotenv's default behaviour, and the usual
+Unix convention: the caller wins.
+
+The `.env` is still reloaded on the fly — the GUI writes to it and the change is
+picked up without restarting the server — but that reload can no longer clobber
+what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
+`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
+applies when nobody passed one.
+
+To change a variable that the caller provides, change it where the caller sets
+it; editing `.env` will have no effect on that key.
+
+## Usage with Claude Desktop
+
+Add the following to your `claude_desktop_config.json`. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+To use logged-in sites, add `SCRAPER_DATA_DIR` (see [Sites](#sites-connected-accounts)):
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data"
+      }
+    }
+  }
+}
+```
+
+## Usage with Cursor
+
+In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["-y", "@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "SCRAPER_DATA_DIR": "/path/to/your/scraper-data",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
+
+## MCP Tools
+
+### `scrape_page`
+
+Navigate to a URL and return the full rendered HTML.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page to scrape |
+| `wait_for` | string | | CSS selector to wait for before capturing HTML |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `execute_js`
+
+Navigate to a URL and execute custom JavaScript in the page context.
+
+The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
+
+```js
+new Function(script)();
+```
+
+To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
+
+Example:
+
+```js
+return {
+  title: document.title,
+  url: window.location.href,
+  text: document.body.innerText.slice(0, 500)
+};
+```
+
+For asynchronous work, return a promise, for example with an async IIFE:
+
+```js
+return (async () => {
+  const response = await fetch('/api/data');
+  return await response.json();
+})();
+```
+
+Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_page_inputs`
+
+Extract all form inputs from a page as a structured JSON object.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
+| `wait_for` | string | | CSS selector to wait for before extracting |
+| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_show_page`
+
+Parse a detail page and extract key-value blocks and tables as structured JSON.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `keys_map` | object | | Map of HTML label → JS key for field name translation |
+| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
+| `tables_max_items` | number | | Max rows per table (default: 2) |
+| `wait_for` | string | | CSS selector to wait for before extraction |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+---
+
+## Interactive Sessions
+
+Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
+
+### Typical workflow
+
+```
+open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
+```
+
+### `open_session`
+
+Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
+
+### `close_session`
+
+Close a session and free its resources. Always call this when you are done.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID returned by `open_session` |
+
+### `list_sessions`
+
+List all currently active sessions with their IDs and timestamps. No parameters.
+
+### `session_goto`
+
+Navigate the session to a new URL without closing it.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for after navigation |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_click`
+
+Click an element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the element to click |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_type`
+
+Type text into an input element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the input element |
+| `text` | string | ✅ | Text to type |
+| `clear` | boolean | | Clear the field before typing (default: false) |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_wait_for`
+
+Wait for a CSS selector to appear in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector to wait for |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_evaluate`
+
+Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_html`
+
+Return the current full rendered HTML of the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+
+### `screenshot`
+
+Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
+
+The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
+| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
+| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `selector` | string | | CSS selector of a specific element to capture |
+| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
+| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
+| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
+| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
+| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` writes it on the agent machine, `both` does both |
+| `destination_path` | string | ⚠️ | **Absolute directory** on the agent machine where the image is written. Created if missing. Required when `output` is `file` or `both`, except behind the relay, which keeps it on the agent side and only forwards `filename` |
+| `filename` | string | ⚠️ | Name the image gets there, e.g. `home.png`. No path separators. Required when `output` is `file` or `both` |
+| `destination_overwrite` | boolean | | Replace the destination file if it exists (default: `false`) |
+| `destination_relay` | boolean | | Filled in by the mcp-http-gateway relay, never by the agent |
+
+> **Where the file lands.** There is no default destination: one would resolve on *this
+> server's* disk, which is not necessarily the agent's — a file nobody finds rather than an
+> error. In stdio the server writes it itself; behind
+> [`@imenam/mcp-http-gateway`](https://www.npmjs.com/package/@imenam/mcp-http-gateway) it
+> returns the bytes and the relay, which runs on the agent machine, writes them at
+> `destination_path`. That path requires `GATEWAY_INLINE_WRITE_ROOTS` in the relay's `env`
+> block. The image never passes through the conversation unless `output` includes `inline`.
+
+---
+
+## Sites (connected accounts)
+
+A **site** is an account the scraper uses while logged in: a name, the domains it covers, its login page and the sign that proves a page is seen logged in. Sites are stored in the data folder (`SCRAPER_DATA_DIR`, or `COOKIES_DIR` under its other name) and managed from the GUI (its labels are in French).
+
+### How cookies are used
+
+- When the shared browser starts, it loads the cookies of every site. Any tool (`scrape_page`, `execute_js`, `open_session`, `screenshot`…) visiting one of a site's domains is therefore logged in, with nothing to pass to the tool.
+- After each page closes, and every 5 minutes, the cookies the browser holds for each site's domains are written back to that site: cookies renewed by the site are kept, cookies it deleted are removed. Session cookies (no expiry date) are kept too, so a login survives a restart of the MCP.
+
+### Login sign
+
+The sign tells the scraper whether a page is seen logged in:
+
+- **L'adresse contient** (the address contains): the page address (host and path, without query or anchor) contains the value, e.g. `crm.example.com/app`;
+- **Un élément est présent** (an element is present): a selector that only matches once logged in, e.g. `.user-menu`. Puppeteer's extended selectors are accepted, so an element can be matched by its text: `.user-menu span::-p-text(Galadrim)`.
+
+**Choisir sur la page** opens the visited page as the scraper sees it logged in; clicking an element (your name, for instance) proposes selectors for it, from the most precise (its text inside its container) to the most general. **Tester** checks the sign as it is typed. Both check each selector twice: it must be found on the visited page with the site's cookies, and absent from the same page without cookies; a selector passing both is a reliable login sign.
+
+### Assisted login
+
+**Se reconnecter** opens the site's login page in a headless Chromium on a blank temporary profile, on the machine running the MCP, and shows it in the GUI. You log in there yourself (credentials, two-factor authentication, SSO): clicks, typing, pasting and scrolling on the capture are replayed in the page, and the address bar allows going back or opening another address. No display is needed, so this works the same on a remote server. As soon as the login sign appears, the browser's cookies are saved for the site and it closes; **Je suis connecté** saves them without waiting for the sign. **Tout reconnecter** chains every site to reconnect, one after the other.
+
+What you type travels from the GUI to the MCP (through the proxy when the GUI is reached remotely, which should then be served over HTTPS); it is never stored. Some sites detect headless browsers and may show a captcha or refuse the login.
+
+Every browser of the scraper starts with Chromium's sandbox disabled (`--no-sandbox`): under a service account, on a Linux that restricts unprivileged user namespaces (Ubuntu 23.10+ with AppArmor), the sandbox cannot start.
+
+### Keep-alive
+
+When enabled, the keep-alive visits the site's **Page visitée** at a fixed interval (5 min to 1 h), day and night, so that the session does not expire from inactivity. Each visit is also a check: if the login sign no longer appears, the site is marked **À reconnecter**, shown in the banner of the Sites page (unless disabled for that site), and the keep-alive stops visiting it until the next login. The visited page is also used by **Vérifier maintenant**.
+
+The keep-alive cannot prevent a logout the site imposes on its own schedule (e.g. every night): it only avoids inactivity timeouts. After such a logout, the assisted login takes a few seconds per site.
+
+### Data folder layout
+
+| Path | Content |
+|------|---------|
+| `sites.json` | Settings and state of every site |
+| `events.json` | Activity: checks, lost sessions, logins, imports (last 1000 events) |
+| `cookies/<site id>.json` | Cookies of each site |
+
+Netscape cookie files (`.txt`) placed in this folder are not loaded. To start from an exported file, open the site and use **Importer cookies**.
+
+## GUI (Optional)
+
+When `PROXY_URL` is set, a web interface is registered with the MCP proxy:
+
+- **Sites** — every site with its state (logged in, to reconnect, not checked), its session and its keep-alive; banner and **Tout reconnecter** when sessions are lost; assisted login in a remotely controlled view of the login page, which can be shown full screen.
+- **Site page** — domains, login page, login sign (picked on the page and tested), keep-alive (interval, visited page), recent activity, cookie import and **Effacer la session**.
+- **Sessions** — the sessions opened by the agent, with a live view (refresh rate selectable from 0.5 s to 10 s, 2.5 s by default; can be paused or shown full screen), the last action, the matching site and the console messages.
+- **Activity** — the history of every site, filterable by site.
+
+The interface refreshes by polling: no connection stays open.
+
+### Network exposure
+
+The GUI listens on `127.0.0.1` only: other machines reach it through the proxy or the gateway manager. A request guard from [`@imenam/mcp-gui-interface`](https://www.npmjs.com/package/@imenam/mcp-gui-interface) answers `403` to any request whose `Host` header is not `localhost`, `127.0.0.1` or `::1` (DNS rebinding protection), and to modifying requests (`POST`, `DELETE`…) sent from another site (foreign `Sec-Fetch-Site` or `Origin`).
+
+## License
+
+ISC