@imenam/simple-scraper 1.0.21 → 1.0.23

package/README.md

@@ -1,383 +1,398 @@
-# simple-scraper-mcp
-
-A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
-
-## Features
-
-- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
-- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
-- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
-- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
-- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
-- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
-- **Cookie support** — Load Netscape-format cookie files automatically before each request
-- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
-
-## Requirements
-
-- Node.js >= 18
-- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
-
-## Installation
-
-```bash
-npx @imenam/simple-scraper
-```
-
-Or install globally:
-
-```bash
-npm install -g @imenam/simple-scraper
-simple-scraper
-```
-
-## MCP Configuration
-
-### Quick setup
-
-From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
-
-```bash
-npx @imenam/simple-scraper --claude-setup-mcp
-```
-
-It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
-folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
-viewport, and prints the remaining optional variables. An empty variable is simply inactive
-(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
-other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
-`--force` is passed.
-
-## Environment Variables
-
-| Variable | Required | Default | Description |
-|----------|----------|---------|-------------|
-| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
-| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
-| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
-| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
-| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
-| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. The folder is created automatically if it does not exist. All files are loaded and merged automatically before each request. |
-| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
-| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
-| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
-| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
-| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
-| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
-| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
-
-## Configuration
-
-Copy `.env.example` to `.env` and configure the variables:
-
-```env
-# Puppeteer options (optional)
-PUPPETEER_HEADLESS=true
-PUPPETEER_TIMEOUT=30000
-PUPPETEER_VIEWPORT_WIDTH=1280
-PUPPETEER_VIEWPORT_HEIGHT=800
-PUPPETEER_DEVICE_SCALE_FACTOR=1
-
-# Optional: path to a folder containing Netscape-format cookie files (.txt)
-# All files in this folder will be loaded automatically before each request.
-# COOKIES_DIR=/path/to/cookies
-
-# GUI (optional) — required to enable the cookie manager interface
-# PROXY_URL=http://localhost:3000
-# PROXY_APP_PATH=/simple-scraper-mcp
-# PROXY_APP_NAME=Simple Scraper MCP
-# Optional: groups this app under a collapsible section of the proxy dashboard
-# APP_GROUP=Scraping
-```
-
-### The process environment wins over `.env`
-
-Any variable already set in the environment of the process that starts this MCP
-(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
-such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
-this repository's `.env`**. This is dotenv's default behaviour, and the usual
-Unix convention: the caller wins.
-
-The `.env` is still reloaded on the fly — the GUI writes to it and the change is
-picked up without restarting the server — but that reload can no longer clobber
-what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
-`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
-applies when nobody passed one.
-
-To change a variable that the caller provides, change it where the caller sets
-it; editing `.env` will have no effect on that key.
-
-## Usage with Claude Desktop
-
-Add the following to your `claude_desktop_config.json`. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "COOKIES_DIR": "/path/to/your/cookies"
-      }
-    }
-  }
-}
-```
-
-## Usage with Cursor
-
-In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
-
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["-y", "@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-
-> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
-
-## MCP Tools
-
-### `scrape_page`
-
-Navigate to a URL and return the full rendered HTML.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page to scrape |
-| `wait_for` | string | | CSS selector to wait for before capturing HTML |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `execute_js`
-
-Navigate to a URL and execute custom JavaScript in the page context.
-
-The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
-
-```js
-new Function(script)();
-```
-
-To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
-
-Example:
-
-```js
-return {
-  title: document.title,
-  url: window.location.href,
-  text: document.body.innerText.slice(0, 500)
-};
-```
-
-For asynchronous work, return a promise, for example with an async IIFE:
-
-```js
-return (async () => {
-  const response = await fetch('/api/data');
-  return await response.json();
-})();
-```
-
-Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_page_inputs`
-
-Extract all form inputs from a page as a structured JSON object.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
-| `wait_for` | string | | CSS selector to wait for before extracting |
-| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `get_show_page`
-
-Parse a detail page and extract key-value blocks and tables as structured JSON.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `keys_map` | object | | Map of HTML label → JS key for field name translation |
-| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
-| `tables_max_items` | number | | Max rows per table (default: 2) |
-| `wait_for` | string | | CSS selector to wait for before extraction |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
----
-
-## Interactive Sessions
-
-Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
-
-### Typical workflow
-
-```
-open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
-```
-
-### `open_session`
-
-Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
-
-### `close_session`
-
-Close a session and free its resources. Always call this when you are done.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID returned by `open_session` |
-
-### `list_sessions`
-
-List all currently active sessions with their IDs and timestamps. No parameters.
-
-### `session_goto`
-
-Navigate the session to a new URL without closing it.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `url` | string | ✅ | URL to navigate to |
-| `wait_for` | string | | CSS selector to wait for after navigation |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_click`
-
-Click an element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the element to click |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_type`
-
-Type text into an input element in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector of the input element |
-| `text` | string | ✅ | Text to type |
-| `clear` | boolean | | Clear the field before typing (default: false) |
-| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
-
-### `session_wait_for`
-
-Wait for a CSS selector to appear in the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `selector` | string | ✅ | CSS selector to wait for |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_evaluate`
-
-Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-
-### `session_html`
-
-Return the current full rendered HTML of the session page.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | ✅ | Session ID |
-
-### `screenshot`
-
-Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
-
-The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
-
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
-| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
-| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-| `selector` | string | | CSS selector of a specific element to capture |
-| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
-| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
-| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
-| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
-| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` saves to disk and returns the path, `both` does both |
-| `path` | string | | **Absolute** path for the saved file — relative paths are rejected, since they would resolve against the server's working directory, which the client sets unpredictably. Missing parent directories are created. Defaults to `<server_root>/screenshots/screenshot-<timestamp>.<format>` |
-
----
-
-## Cookie Files
-
-Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request. If the folder does not exist yet, it is created automatically (including any missing parent directories), so a fresh `COOKIES_DIR` no longer raises an error.
-
-## GUI (Optional)
-
-When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
-
-## License
-
-ISC
+# simple-scraper-mcp
+
+A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
+
+## Features
+
+- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
+- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
+- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
+- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
+- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
+- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
+- **Cookie support** — Load Netscape-format cookie files automatically before each request
+- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
+
+## Requirements
+
+- Node.js >= 18
+- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
+
+## Installation
+
+```bash
+npx @imenam/simple-scraper
+```
+
+Or install globally:
+
+```bash
+npm install -g @imenam/simple-scraper
+simple-scraper
+```
+
+## MCP Configuration
+
+### Quick setup
+
+From the repo you want to scrape from, generate a ready-to-fill `.mcp.json` in the current directory:
+
+```bash
+npx @imenam/simple-scraper --claude-setup-mcp
+```
+
+It creates the `simple-scraper-mcp` entry with the proxy variables pre-filled from the current
+folder name (`PROXY_URL`, `APP_GROUP`, `PROXY_APP_NAME`, `PROXY_APP_PATH`) plus a 1920x1080
+viewport, and prints the remaining optional variables. An empty variable is simply inactive
+(no `PROXY_URL` = no GUI), so the generated config runs as-is. An existing `.mcp.json` is merged,
+other servers are preserved, and an existing `simple-scraper-mcp` entry is only replaced when
+`--force` is passed.
+
+## Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
+| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
+| `PUPPETEER_VIEWPORT_WIDTH` | No | `1280` | Viewport width in CSS pixels, applied to every page. Affects responsive layouts and lazy-loading, not just screenshots. |
+| `PUPPETEER_VIEWPORT_HEIGHT` | No | `800` | Viewport height in CSS pixels, applied to every page. |
+| `PUPPETEER_DEVICE_SCALE_FACTOR` | No | `1` | Physical pixels per CSS pixel. Set to `2` for retina-density screenshots (same framing, twice the detail, ~4× the file size). |
+| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. The folder is created automatically if it does not exist. All files are loaded and merged automatically before each request. |
+| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
+| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
+| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
+| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
+| `APP_GROUP` | No | - | Groups the app under a collapsible section (folded by default) in the proxy dashboard. Apps sharing the same value appear together; if unset, the app lands in the default "ungrouped" section. |
+| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
+| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
+
+## Configuration
+
+Copy `.env.example` to `.env` and configure the variables:
+
+```env
+# Puppeteer options (optional)
+PUPPETEER_HEADLESS=true
+PUPPETEER_TIMEOUT=30000
+PUPPETEER_VIEWPORT_WIDTH=1280
+PUPPETEER_VIEWPORT_HEIGHT=800
+PUPPETEER_DEVICE_SCALE_FACTOR=1
+
+# Optional: path to a folder containing Netscape-format cookie files (.txt)
+# All files in this folder will be loaded automatically before each request.
+# COOKIES_DIR=/path/to/cookies
+
+# GUI (optional) — required to enable the cookie manager interface
+# PROXY_URL=http://localhost:3000
+# PROXY_APP_PATH=/simple-scraper-mcp
+# PROXY_APP_NAME=Simple Scraper MCP
+# Optional: groups this app under a collapsible section of the proxy dashboard
+# APP_GROUP=Scraping
+```
+
+### The process environment wins over `.env`
+
+Any variable already set in the environment of the process that starts this MCP
+(a shell `export`, a `claude_desktop_config.json` `env` block, or a supervisor
+such as `@imenam/mcp-gateway-manager`) **takes precedence over the same key in
+this repository's `.env`**. This is dotenv's default behaviour, and the usual
+Unix convention: the caller wins.
+
+The `.env` is still reloaded on the fly — the GUI writes to it and the change is
+picked up without restarting the server — but that reload can no longer clobber
+what the caller injected. Concretely, an `APP_GROUP=Scraping` line in this
+`.env` no longer overrides an `APP_GROUP` passed in by a supervisor; it only
+applies when nobody passed one.
+
+To change a variable that the caller provides, change it where the caller sets
+it; editing `.env` will have no effect on that key.
+
+## Usage with Claude Desktop
+
+Add the following to your `claude_desktop_config.json`. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "COOKIES_DIR": "/path/to/your/cookies",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "COOKIES_DIR": "/path/to/your/cookies"
+      }
+    }
+  }
+}
+```
+
+## Usage with Cursor
+
+In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
+
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["-y", "@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "COOKIES_DIR": "/path/to/your/cookies",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+
+> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
+
+## MCP Tools
+
+### `scrape_page`
+
+Navigate to a URL and return the full rendered HTML.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page to scrape |
+| `wait_for` | string | | CSS selector to wait for before capturing HTML |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `execute_js`
+
+Navigate to a URL and execute custom JavaScript in the page context.
+
+The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
+
+```js
+new Function(script)();
+```
+
+To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
+
+Example:
+
+```js
+return {
+  title: document.title,
+  url: window.location.href,
+  text: document.body.innerText.slice(0, 500)
+};
+```
+
+For asynchronous work, return a promise, for example with an async IIFE:
+
+```js
+return (async () => {
+  const response = await fetch('/api/data');
+  return await response.json();
+})();
+```
+
+Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_page_inputs`
+
+Extract all form inputs from a page as a structured JSON object.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
+| `wait_for` | string | | CSS selector to wait for before extracting |
+| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `get_show_page`
+
+Parse a detail page and extract key-value blocks and tables as structured JSON.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `keys_map` | object | | Map of HTML label → JS key for field name translation |
+| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
+| `tables_max_items` | number | | Max rows per table (default: 2) |
+| `wait_for` | string | | CSS selector to wait for before extraction |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+---
+
+## Interactive Sessions
+
+Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
+
+### Typical workflow
+
+```
+open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
+```
+
+### `open_session`
+
+Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `viewport` | object | | Viewport override for this session, applied before navigation: `{ width, height, device_scale_factor }`. Omitted fields fall back to the server defaults. |
+
+### `close_session`
+
+Close a session and free its resources. Always call this when you are done.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID returned by `open_session` |
+
+### `list_sessions`
+
+List all currently active sessions with their IDs and timestamps. No parameters.
+
+### `session_goto`
+
+Navigate the session to a new URL without closing it.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for after navigation |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_click`
+
+Click an element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the element to click |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_type`
+
+Type text into an input element in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the input element |
+| `text` | string | ✅ | Text to type |
+| `clear` | boolean | | Clear the field before typing (default: false) |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+
+### `session_wait_for`
+
+Wait for a CSS selector to appear in the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector to wait for |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_evaluate`
+
+Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+
+### `session_html`
+
+Return the current full rendered HTML of the session page.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+
+### `screenshot`
+
+Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
+
+The capture area is resolved by precedence: `selector` → `clip` → `full_page` → the visible viewport.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
+| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
+| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `selector` | string | | CSS selector of a specific element to capture |
+| `clip` | object | | Arbitrary pixel region `{ x, y, width, height }` in CSS pixels, measured from the top-left of the page. May extend below the visible viewport. Ignored when `selector` is set; cannot be combined with `full_page`. |
+| `viewport` | object | | Viewport override for this capture: `{ width, height, device_scale_factor }`. For a session, the previous viewport is restored afterwards. |
+| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` or `clip` is provided) |
+| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
+| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` writes it on the agent machine, `both` does both |
+| `destination_path` | string | ⚠️ | **Absolute directory** on the agent machine where the image is written. Created if missing. Required when `output` is `file` or `both` |
+| `filename` | string | ⚠️ | Name the image gets there, e.g. `home.png`. No path separators. Required when `output` is `file` or `both` |
+| `destination_overwrite` | boolean | | Replace the destination file if it exists (default: `false`) |
+| `destination_relay` | boolean | | Filled in by the mcp-http-gateway relay, never by the agent |
+
+> **Where the file lands.** There is no default destination: one would resolve on *this
+> server's* disk, which is not necessarily the agent's — a file nobody finds rather than an
+> error. In stdio the server writes it itself; behind
+> [`@imenam/mcp-http-gateway`](https://www.npmjs.com/package/@imenam/mcp-http-gateway) it
+> returns the bytes and the relay, which runs on the agent machine, writes them at
+> `destination_path`. That path requires `GATEWAY_INLINE_WRITE_ROOTS` in the relay's `env`
+> block. The image never passes through the conversation unless `output` includes `inline`.
+
+---
+
+## Cookie Files
+
+Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request. If the folder does not exist yet, it is created automatically (including any missing parent directories), so a fresh `COOKIES_DIR` no longer raises an error.
+
+## GUI (Optional)
+
+When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
+
+### Network exposure
+
+The GUI listens on `127.0.0.1` only: other machines reach it through the proxy or the gateway manager. A request guard from [`@imenam/mcp-gui-interface`](https://www.npmjs.com/package/@imenam/mcp-gui-interface) answers `403` to any request whose `Host` header is not `localhost`, `127.0.0.1` or `::1` (DNS rebinding protection), and to modifying requests (`POST`, `DELETE`…) sent from another site (foreign `Sec-Fetch-Site` or `Origin`).
+
+## License
+
+ISC

package/dist/gui.js

@@ -3,7 +3,7 @@ import bodyParser from 'body-parser';
 import path from 'path';
 import fs from 'fs';
 import { fileURLToPath } from 'url';
-import { setupLogging, ProxyClient, IpcHub } from '@imenam/mcp-gui-interface';
+import { setupLogging, ProxyClient, IpcHub, GUI_LISTEN_HOST, expressGuiGuard } from '@imenam/mcp-gui-interface';
 // eslint-disable-next-line @typescript-eslint/no-explicit-any
 const ipc = new IpcHub(process);
 async function callMaster(type, data, timeoutMs = 5000) {
@@ -70,6 +70,7 @@ const getCookiesDir = () => {
     return cookiesDir;
 };
 const app = express();
+app.use(expressGuiGuard());
 app.use(bodyParser.json());
 app.use(express.static(path.join(rootDir, 'public/dist')));
 app.get(/^(?!\/api\/|\/proxy\/)/, (_req, res) => {
@@ -223,7 +224,7 @@ async function startServer() {
             }
         }, 500);
     });
-    const serverListener = app.listen(port, () => {
+    const serverListener = app.listen(port, GUI_LISTEN_HOST, () => {
         console.error(`[GUI] Server running at http://localhost:${port}`);
     });
     serverListener.on('error', (err) => {

package/dist/index.js

@@ -4,7 +4,6 @@ import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'
 // Importé AVANT tout appel à dotenv : ce module prend l'instantané de
 // l'environnement hérité du process, qui doit primer sur le `.env` du dépôt.
 import { loadEnvFile } from './env-precedence.js';
-import fs from 'fs';
 import path from 'path';
 import { fileURLToPath } from 'url';
 import { z } from 'zod';
@@ -14,6 +13,7 @@ import { loadAllCookies } from './cookies.js';
 import { GuiLauncher, reconnectGuiToolDefinition } from '@imenam/mcp-gui-interface';
 import { createSession, getSession, closeSession, closeAllSessions, listSessions, getSessionLogs, clearSessionLogs, resolveViewport } from './sessions.js';
 import { registerSessionRpc } from './session-rpc.js';
+import { deliverFile, RELAY_SHAPE_DESCRIPTIONS } from './relay.js';
 const __dirname = path.dirname(fileURLToPath(import.meta.url));
 const rootDir = path.resolve(__dirname, '..');
 const envPath = path.join(rootDir, '.env');
@@ -602,21 +602,34 @@ server.tool('screenshot', 'Take a screenshot of a page. Provide either session_i
     viewport: viewportSchema.describe('Viewport override for this capture. For a session, the previous viewport is restored afterwards.'),
     full_page: z.boolean().optional().describe('Capture the full scrollable page height (default: false, ignored when selector or clip is provided)'),
     format: z.enum(['png', 'jpeg']).optional().describe('Image format (default: png)'),
-    output: z.enum(['inline', 'file', 'both']).describe('Return format: "inline" embeds the image in the response, "file" saves to disk and returns the path, "both" does both'),
-    path: z
+    output: z.enum(['inline', 'file', 'both']).describe('Return format: "inline" embeds the image in the response, "file" saves the image on the agent machine, "both" does both'),
+    destination_path: z
+        .string()
+        .optional()
+        .describe(`${RELAY_SHAPE_DESCRIPTIONS.destination_path}. Required when output is "file" or "both"`),
+    filename: z
         .string()
-        .refine(p => path.isAbsolute(p), {
-        message: 'path must be absolute — a relative path would resolve against the MCP server\'s working directory, which is set by the client and not predictable.',
-    })
         .optional()
-        .describe('Absolute file path for the saved image (used when output is "file" or "both"). Must be absolute; missing parent directories are created. Defaults to <server_root>/screenshots/screenshot-<timestamp>.<format>'),
-}, async ({ session_id, url, wait_for, timeout, selector, clip, viewport, full_page, format, output, path: filePath }) => {
+        .describe(`${RELAY_SHAPE_DESCRIPTIONS.filename}. Required when output is "file" or "both"`),
+    destination_overwrite: z.boolean().optional().describe(RELAY_SHAPE_DESCRIPTIONS.destination_overwrite),
+    destination_relay: z.boolean().optional().describe(RELAY_SHAPE_DESCRIPTIONS.destination_relay),
+}, async ({ session_id, url, wait_for, timeout, selector, clip, viewport, full_page, format, output, destination_path, filename, destination_overwrite, destination_relay }) => {
     const effectiveTimeout = timeout ?? getDefaultTimeout();
     const fmt = format ?? 'png';
     const mimeType = fmt === 'jpeg' ? 'image/jpeg' : 'image/png';
     if (clip && full_page) {
         return { content: [{ type: 'text', text: 'Error: clip and full_page cannot be combined — clip already defines the region to capture.' }], isError: true };
     }
+    // Écrire un fichier suppose de savoir où, et sur quelle machine. Il n'y a pas de
+    // destination par défaut : elle atterrirait sur le disque de ce serveur, qui n'est
+    // pas forcément celui de l'agent — un fichier introuvable plutôt qu'une erreur.
+    const wantsFile = output === 'file' || output === 'both';
+    if (wantsFile && (!destination_path || !filename)) {
+        return {
+            content: [{ type: 'text', text: 'Error: destination_path and filename are required when output is "file" or "both".' }],
+            isError: true,
+        };
+    }
     let page = null;
     let ownedPage = false;
     // Only set for session pages, whose viewport must survive this call unchanged.
@@ -662,25 +675,17 @@ server.tool('screenshot', 'Take a screenshot of a page. Provide either session_i
         else {
             imageBuffer = Buffer.from(await page.screenshot({ type: fmt, fullPage: full_page ?? false }));
         }
-        const base64 = imageBuffer.toString('base64');
-        // Resolve file path. filePath is guaranteed absolute by the schema; resolve()
-        // is kept only to normalise separators and any ".." segments.
-        const resolvedPath = (() => {
-            if (filePath)
-                return path.resolve(filePath);
-            const ts = Date.now();
-            const dir = path.join(rootDir, 'screenshots');
-            return path.join(dir, `screenshot-${ts}.${fmt}`);
-        })();
         const resultContent = [];
         if (output === 'inline' || output === 'both') {
-            resultContent.push({ type: 'image', data: base64, mimeType });
+            resultContent.push({ type: 'image', data: imageBuffer.toString('base64'), mimeType });
         }
-        if (output === 'file' || output === 'both') {
-            const dir = path.dirname(resolvedPath);
-            fs.mkdirSync(dir, { recursive: true });
-            fs.writeFileSync(resolvedPath, imageBuffer);
-            resultContent.push({ type: 'text', text: `Screenshot saved to: ${resolvedPath}` });
+        if (wantsFile) {
+            resultContent.push(...deliverFile(imageBuffer, mimeType, {
+                destination_path: destination_path,
+                filename: filename,
+                destination_overwrite,
+                destination_relay,
+            }));
         }
         return { content: resultContent };
     }

package/dist/relay.js

@@ -0,0 +1,67 @@
+import fs from "fs";
+import path from "path";
+// Un fichier produit par ce MCP n'a aucun moyen d'atteindre la machine de l'agent quand
+// le MCP tourne derrière mcp-http-gateway : le protocole n'offre au serveur aucune
+// primitive pour y écrire. La convention du relais règle ça — dès qu'un outil déclare la
+// paire `destination_path` / `destination_relay`, le relais retient la destination, pose
+// `destination_relay`, et écrit lui-même les octets qu'on lui renvoie.
+//
+// Ici, on ne sait pas si un relais est dans la chaîne : `destination_relay` le dit. Posé,
+// on renvoie le contenu ; absent, les deux machines n'en font qu'une et on écrit nous-même.
+/** Plafond de ce qui peut être encodé dans une réponse d'outil. Le relais applique sa
+ *  propre limite, configurable ; celle-ci n'existe que pour qu'un fichier démesuré
+ *  échoue proprement ici plutôt qu'en épuisant la mémoire pendant l'encodage base64. */
+const RELAY_MAX_BYTES = 32 * 1024 * 1024;
+export const RELAY_SHAPE_DESCRIPTIONS = {
+    destination_path: "ABSOLUTE path of the destination DIRECTORY on the agent machine. It is created if missing, and the file keeps the name given in filename",
+    filename: "Name of the file to create, extension included (e.g. 'screenshot.png'). Must not contain path separators",
+    destination_overwrite: "Replace the destination file if it already exists. Defaults to false",
+    destination_relay: "Filled in by the mcp-http-gateway relay, never by the agent. Signals that the destination lives on another machine and that the file content must travel in the response",
+};
+/**
+ * Livre un fichier produit par le MCP, du bon côté de la chaîne.
+ *
+ * Renvoie les blocs de contenu à faire figurer dans le résultat de l'outil : un bloc
+ * `resource` que le relais interceptera et retirera, ou une simple confirmation quand
+ * l'écriture a eu lieu ici. Toute impossibilité est une erreur franche : mieux vaut un
+ * fichier perdu qu'un fichier écrit sur la mauvaise machine.
+ */
+export function deliverFile(data, mimeType, target) {
+    const name = path.basename(target.filename);
+    if (name !== target.filename || name === "." || name === "..") {
+        throw new Error(`filename must be a bare file name, without path separators: ${target.filename}`);
+    }
+    if (data.length > RELAY_MAX_BYTES) {
+        throw new Error(`File too large to transfer: ${data.length} bytes, the limit is ${RELAY_MAX_BYTES}.`);
+    }
+    // Le relais tourne sur la machine de l'agent et a retenu la destination : il attend les
+    // octets, qu'il écrira lui-même. Écrire ici viserait le mauvais disque.
+    if (target.destination_relay) {
+        return [
+            {
+                type: "resource",
+                resource: {
+                    uri: `file:///${name}`,
+                    name,
+                    mimeType,
+                    blob: data.toString("base64"),
+                },
+            },
+        ];
+    }
+    // Pas de relais : le MCP tourne à côté de l'agent, son disque est le bon.
+    if (!path.isAbsolute(target.destination_path)) {
+        throw new Error(`destination_path must be an absolute directory path: ${target.destination_path}`);
+    }
+    if (fs.existsSync(target.destination_path) &&
+        !fs.statSync(target.destination_path).isDirectory()) {
+        throw new Error(`destination_path must be a directory, but a file already exists there: ${target.destination_path}`);
+    }
+    const written = path.join(target.destination_path, name);
+    if (!target.destination_overwrite && fs.existsSync(written)) {
+        throw new Error(`Destination already exists: ${written}. Pass destination_overwrite to replace it.`);
+    }
+    fs.mkdirSync(target.destination_path, { recursive: true });
+    fs.writeFileSync(written, data);
+    return [{ type: "text", text: `File written to ${written} (${data.length} bytes).` }];
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@imenam/simple-scraper",
-  "version": "1.0.21",
+  "version": "1.0.23",
   "description": "MCP server for web scraping and JavaScript execution using Puppeteer",
   "type": "module",
   "main": "dist/index.js",
@@ -35,7 +35,7 @@
     "node": ">=18.0.0"
   },
   "dependencies": {
-    "@imenam/mcp-gui-interface": "^1.0.12",
+    "@imenam/mcp-gui-interface": "^1.0.14",
     "@modelcontextprotocol/sdk": "^1.28.0",
     "body-parser": "^2.2.2",
     "dotenv": "^17.3.1",