npm - @imenam/simple-scraper - Versions diffs - 1.0.7 → 1.0.8 - Mend

@imenam/simple-scraper 1.0.7 → 1.0.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (5) hide show

package/README.md CHANGED Viewed

@@ -1,188 +1,335 @@
-# simple-scraper-mcp
-A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
-## Features
-- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
-- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
-- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
-- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
-- **Cookie support** — Load Netscape-format cookie files automatically before each request
-- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
-## Requirements
-- Node.js >= 18
-- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
-## Installation
-```bash
-npx @imenam/simple-scraper
-```
-Or install globally:
-```bash
-npm install -g @imenam/simple-scraper
-simple-scraper
-```
-## Environment Variables
-| Variable | Required | Default | Description |
-|----------|----------|---------|-------------|
-| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
-| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
-| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. All files are loaded and merged automatically before each request. |
-| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
-| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
-| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
-| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
-## Configuration
-Copy `.env.example` to `.env` and configure the variables:
-```env
-# Puppeteer options (optional)
-PUPPETEER_HEADLESS=true
-PUPPETEER_TIMEOUT=30000
-# Optional: path to a folder containing Netscape-format cookie files (.txt)
-# All files in this folder will be loaded automatically before each request.
-# COOKIES_DIR=/path/to/cookies
-# GUI (optional) — required to enable the cookie manager interface
-# PROXY_URL=http://localhost:3000
-# PROXY_APP_PATH=/simple-scraper-mcp
-# PROXY_APP_NAME=Simple Scraper MCP
-```
-## Usage with Claude Desktop
-Add the following to your `claude_desktop_config.json`. Full example with all available options:
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "COOKIES_DIR": "/path/to/your/cookies"
-      }
-    }
-  }
-}
-```
-## Usage with Cursor
-In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
-```json
-{
-  "mcpServers": {
-    "simple-scraper": {
-      "command": "npx",
-      "args": ["-y", "@imenam/simple-scraper"],
-      "env": {
-        "PUPPETEER_HEADLESS": "true",
-        "PUPPETEER_TIMEOUT": "30000",
-        "COOKIES_DIR": "/path/to/your/cookies",
-        "MCP_LOG_DIR": "/path/to/your/logs",
-        "PROXY_URL": "http://localhost:4500",
-        "PROXY_APP_PATH": "/simple-scraper",
-        "PROXY_APP_NAME": "Simple Scraper"
-      }
-    }
-  }
-}
-```
-> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
-## MCP Tools
-### `scrape_page`
-Navigate to a URL and return the full rendered HTML.
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page to scrape |
-| `wait_for` | string | | CSS selector to wait for before capturing HTML |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-### `execute_js`
-Navigate to a URL and execute custom JavaScript in the page context.
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `script` | string | ✅ | JavaScript code to execute |
-| `wait_for` | string | | CSS selector to wait for before executing |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-### `get_page_inputs`
-Extract all form inputs from a page as a structured JSON object.
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
-| `wait_for` | string | | CSS selector to wait for before extracting |
-| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-### `get_show_page`
-Parse a detail page and extract key-value blocks and tables as structured JSON.
-| Parameter | Type | Required | Description |
-|-----------|------|----------|-------------|
-| `url` | string | ✅ | URL of the page |
-| `keys_map` | object | | Map of HTML label → JS key for field name translation |
-| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
-| `tables_max_items` | number | | Max rows per table (default: 2) |
-| `wait_for` | string | | CSS selector to wait for before extraction |
-| `timeout` | number | | Timeout in ms (default: 30000) |
-## Cookie Files
-Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request.
-## GUI (Optional)
-When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
-## License
-ISC
+# simple-scraper-mcp
+A Model Context Protocol (MCP) server for web scraping and JavaScript execution using a headless browser (Puppeteer). Includes an optional GUI for cookie management.
+## Features
+- **`scrape_page`** — Navigate to a URL and return the full rendered HTML of the page
+- **`execute_js`** — Navigate to a URL and execute custom JavaScript in the page context
+- **`get_page_inputs`** — Extract all form inputs from a page as a structured JSON object
+- **`get_show_page`** — Parse a detail/show page and extract key-value blocks and tables as structured JSON
+- **Interactive sessions** — Keep a browser page alive across multiple tool calls to navigate, click, type, run JS, and screenshot the page in any desired state
+- **`screenshot`** — Capture a screenshot of any page or active session, with inline or file output
+- **Cookie support** — Load Netscape-format cookie files automatically before each request
+- **Optional GUI** — Cookie manager interface, available when integrated with the [MCP proxy](https://www.npmjs.com/package/@imenam/mcp-http-gateway)
+## Requirements
+- Node.js >= 18
+- Puppeteer will automatically download a compatible Chromium browser (~300 MB) on first install
+## Installation
+```bash
+npx @imenam/simple-scraper
+```
+Or install globally:
+```bash
+npm install -g @imenam/simple-scraper
+simple-scraper
+```
+## Environment Variables
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `PUPPETEER_HEADLESS` | No | `true` | Run Chromium in headless mode. Set to `false` to display the browser window. |
+| `PUPPETEER_TIMEOUT` | No | `30000` | Default timeout in milliseconds for page navigation and waits. |
+| `COOKIES_DIR` | No | - | Absolute path to a folder containing Netscape-format `.txt` cookie files. All files are loaded and merged automatically before each request. |
+| `MCP_LOG_DIR` | No | `.mcp-gui/logs` | Absolute path to the directory where log files are written. |
+| `PROXY_URL` | No | - | Base URL of the [MCP HTTP Gateway](https://www.npmjs.com/package/@imenam/mcp-http-gateway). Required to enable the GUI. |
+| `PROXY_APP_PATH` | No | `/simple-scraper-mcp` | URL path under which the GUI is registered on the proxy. |
+| `PROXY_APP_NAME` | No | `Simple Scraper MCP` | Display name shown in the proxy's app list. |
+| `SCRAPER_MAX_SESSIONS` | No | `5` | Maximum number of concurrent interactive sessions. |
+| `SCRAPER_SESSION_TTL_MS` | No | `600000` | Inactivity TTL for sessions in milliseconds (default: 10 minutes). Sessions unused beyond this duration are closed automatically. |
+## Configuration
+Copy `.env.example` to `.env` and configure the variables:
+```env
+# Puppeteer options (optional)
+PUPPETEER_HEADLESS=true
+PUPPETEER_TIMEOUT=30000
+# Optional: path to a folder containing Netscape-format cookie files (.txt)
+# All files in this folder will be loaded automatically before each request.
+# COOKIES_DIR=/path/to/cookies
+# GUI (optional) — required to enable the cookie manager interface
+# PROXY_URL=http://localhost:3000
+# PROXY_APP_PATH=/simple-scraper-mcp
+# PROXY_APP_NAME=Simple Scraper MCP
+```
+## Usage with Claude Desktop
+Add the following to your `claude_desktop_config.json`. Full example with all available options:
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "COOKIES_DIR": "/path/to/your/cookies",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+To load cookies automatically, add `COOKIES_DIR` pointing to a folder containing `.txt` files in [Netscape cookie format](http://www.cookiecentral.com/faq/#3.5):
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "COOKIES_DIR": "/path/to/your/cookies"
+      }
+    }
+  }
+}
+```
+## Usage with Cursor
+In Cursor, MCP servers are configured in `.cursor/mcp.json`. You can pass environment variables directly in the config. Full example with all available options:
+```json
+{
+  "mcpServers": {
+    "simple-scraper": {
+      "command": "npx",
+      "args": ["-y", "@imenam/simple-scraper"],
+      "env": {
+        "PUPPETEER_HEADLESS": "true",
+        "PUPPETEER_TIMEOUT": "30000",
+        "COOKIES_DIR": "/path/to/your/cookies",
+        "MCP_LOG_DIR": "/path/to/your/logs",
+        "PROXY_URL": "http://localhost:4500",
+        "PROXY_APP_PATH": "/simple-scraper",
+        "PROXY_APP_NAME": "Simple Scraper"
+      }
+    }
+  }
+}
+```
+> **Note:** The `-y` flag in `args` avoids the interactive confirmation prompt when using `npx`.
+## MCP Tools
+### `scrape_page`
+Navigate to a URL and return the full rendered HTML.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page to scrape |
+| `wait_for` | string | | CSS selector to wait for before capturing HTML |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `execute_js`
+Navigate to a URL and execute custom JavaScript in the page context.
+The `script` parameter is executed as the body of a JavaScript function in the browser page context, equivalent to:
+```js
+new Function(script)();
+```
+To return data from the tool, the script must use an explicit `return`. A bare expression such as `document.title` will evaluate but the tool will receive `undefined`.
+Example:
+```js
+return {
+  title: document.title,
+  url: window.location.href,
+  text: document.body.innerText.slice(0, 500)
+};
+```
+For asynchronous work, return a promise, for example with an async IIFE:
+```js
+return (async () => {
+  const response = await fetch('/api/data');
+  return await response.json();
+})();
+```
+Returned objects and arrays are serialized as formatted JSON. Primitive values are returned as text.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `script` | string | ✅ | JavaScript function body to execute in the page context. Use `return` to send a result back to the tool. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `get_page_inputs`
+Extract all form inputs from a page as a structured JSON object.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `selector` | string | | CSS selector to scope the search (e.g. `#my-form`) |
+| `wait_for` | string | | CSS selector to wait for before extracting |
+| `show_hidden` | boolean | | Include `input[type=hidden]` fields (default: false) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `get_show_page`
+Parse a detail page and extract key-value blocks and tables as structured JSON.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL of the page |
+| `keys_map` | object | | Map of HTML label → JS key for field name translation |
+| `box_selector` | string | | CSS selector for section containers (default: `.box.box-primary`) |
+| `tables_max_items` | number | | Max rows per table (default: 2) |
+| `wait_for` | string | | CSS selector to wait for before extraction |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+---
+## Interactive Sessions
+Interactive sessions let you keep a browser page alive across multiple tool calls, so you can bring the page into the exact state you need before extracting data or taking a screenshot.
+### Typical workflow
+```
+open_session → session_click / session_type / session_evaluate → screenshot / session_html → close_session
+```
+### `open_session`
+Open a persistent browser session. Returns a `session_id` to use with all `session_*` tools and `screenshot`.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for before the session is considered ready |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `close_session`
+Close a session and free its resources. Always call this when you are done.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID returned by `open_session` |
+### `list_sessions`
+List all currently active sessions with their IDs and timestamps. No parameters.
+### `session_goto`
+Navigate the session to a new URL without closing it.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `url` | string | ✅ | URL to navigate to |
+| `wait_for` | string | | CSS selector to wait for after navigation |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `session_click`
+Click an element in the session page.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the element to click |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+### `session_type`
+Type text into an input element in the session page.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector of the input element |
+| `text` | string | ✅ | Text to type |
+| `clear` | boolean | | Clear the field before typing (default: false) |
+| `timeout` | number | | Timeout in ms to wait for the element (default: 30000) |
+### `session_wait_for`
+Wait for a CSS selector to appear in the session page.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `selector` | string | ✅ | CSS selector to wait for |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `session_evaluate`
+Execute JavaScript in the context of the session page. Same conventions as `execute_js`.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+| `script` | string | ✅ | JavaScript function body. Use `return` to get a result back. |
+| `wait_for` | string | | CSS selector to wait for before executing |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+### `session_html`
+Return the current full rendered HTML of the session page.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | ✅ | Session ID |
+### `screenshot`
+Capture a screenshot of a page. Use `session_id` to capture an active session in its current state, or `url` for a one-shot capture.
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `session_id` | string | | Session ID. If provided, `url` is ignored and the current page state is captured. |
+| `url` | string | | URL for a one-shot screenshot. Required if `session_id` is not provided. |
+| `wait_for` | string | | CSS selector to wait for (one-shot mode only) |
+| `timeout` | number | | Timeout in ms (default: 30000) |
+| `selector` | string | | CSS selector of a specific element to capture |
+| `full_page` | boolean | | Capture the full scrollable page height (default: false, ignored when `selector` is provided) |
+| `format` | `png` \| `jpeg` | | Image format (default: `png`) |
+| `output` | `inline` \| `file` \| `both` | ✅ | `inline` embeds the image in the response, `file` saves to disk and returns the path, `both` does both |
+| `path` | string | | Absolute or relative path for the saved file. Defaults to `./screenshots/screenshot-<timestamp>.<format>` |
+---
+## Cookie Files
+Cookies are loaded from `.txt` files in [Netscape format](http://www.cookiecentral.com/faq/#3.5). Place them in the folder specified by `COOKIES_DIR`. All files in the folder are loaded and merged automatically before each request.
+## GUI (Optional)
+When `PROXY_URL` is set, a cookie manager web interface is registered with the MCP proxy. It allows you to list, upload, and delete cookie files through a browser UI.
+## License
+ISC

package/dist/browser.js CHANGED Viewed

@@ -30,6 +30,11 @@ export async function closeBrowser() {
 }
 for (const sig of ['SIGTERM', 'SIGINT']) {
     process.on(sig, async () => {
+        // Sessions must be closed before the browser to avoid dangling page handles.
+        // closeAllSessions is imported dynamically to avoid a circular dependency
+        // (sessions.ts → browser.ts → sessions.ts).
+        const { closeAllSessions } = await import('./sessions.js');
+        await closeAllSessions();
         await closeBrowser();
         process.exit(0);
     });