searchfetch 1.0.2 → 3.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Max
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,81 +1,96 @@
-# SearchFetch
+# SearchFetch (MCP Server)
 
-A fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built specifically for AI Agents, it bypasses Google's GDPR consent screens, Cloudflare Turnstile, and converts heavy HTML into clean, token-optimized Markdown.
+A maximum fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built specifically for AI Agents (Cursor, Claude Code, OpenCode), it completely bypasses bot detection (Cloudflare Turnstile, Datadome), dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.
 
 ## Features
-* **Multi-Engine Search:** Natively supports DuckDuckGo and Google parsing out of the box. DuckDuckGo is set as the preferred default.
-* **Aggressive Base64 / Image Scrubber:** Implements "nuclear" DOM scrubbing prior to parsing. Guaranteed to NEVER pollute your LLM's context window with giant base64 image strings (`data:image/...`).
-* **Stealth CloakBrowser:** Avoids FingerprintJS, reCAPTCHA, and Cloudflare using Chromium C++ patches and humanized mouse movements natively. 
-* **SPA & React Support:** Waits for network idle to ensure modern Single Page Applications fully execute JavaScript and render before extracting content.
-* **Fault Tolerant:** Extracts whatever DOM was successfully loaded even if a massive, clunky page times out mid-render.
-* **Pagination Support:** Fetches massive webpages iteratively via `start_index` and `max_length` without blowing out AI context tokens limits.
-
-## Installation
-
-1. Clone or copy the directory.
-   ```bash
-   git clone https://github.com/maxylev/searchfetch
-   ```
-2. Install dependencies:
-   ```bash
-   npm install
-   ```
-3. Make the main script executable:
-   ```bash
-   chmod +x index.js
-   ```
-4. Link it globally to your system:
-   ```bash
-   npm link
-   ```
-
-## Configuration
-
-Configure your AI tool/IDE (Cursor, Claude Desktop, Opencode, etc.) to point to this server.
-
-### Example `config.json` (Opencode, Cursor):
+* **Maximum Fault Tolerance:** Implements auto-healing browser sessions, grace-period timeouts for clunky SPAs, and network-level aborting of tracking scripts and media.
+* **Stealth Engine:** Powered by CloakBrowser C++ patches + `humanize` logic. Antibot systems score it as a normal browser because it mathematically moves and renders exactly like one.
+* **Nuclear Token Scrubber:** Strips Base64 images, SVGs, scripts, and inline styles out of the DOM *before* Markdown conversion, guaranteeing your LLM context window won't blow out.
+* **Dual Execution Paths:** Natively supports zero-install execution via both Python (`uvx`) and Node.js (`npx`).
+
+---
+
+## Usage & Installation
+
+You do not need to install this repository manually. Configure your agent to use the zero-install commands `npx` or `uvx` depending on your environment.
+
+### Claude Desktop Configuration
+Add the following to your config:
+
+**Option A: Using Python (`uvx` - Recommended)**
 ```json
 {
-  "mcp": {
+  "mcpServers": {
     "searchfetch": {
-      "type": "local",
-      "command":["npx", "searchfetch"],
-      "enabled": true
+      "command": "uvx",
+      "args": ["searchfetch"]
     }
   }
 }
 ```
 
-### Example `claude_desktop_config.json`:
+**Option B: Using Node.js (`npx`)**
 ```json
 {
   "mcpServers": {
     "searchfetch": {
       "command": "npx",
-      "args": ["searchfetch"]
+      "args": ["-y", "searchfetch"]
     }
   }
 }
 ```
 
+### Cursor / IDE Configuration
+Add it via the **MCP panel** in Cursor settings:
+* **Type:** `command`
+* **Command:** `uvx searchfetch` (or `npx -y searchfetch`)
+
+---
+
 ## Available Tools
 
 ### 1. `websearch`
-Searches the web via Google or DuckDuckGo and returns structured snippets.
-* **`query`** (string): Your search query.
-* **`engine`** (string): `"google"` or `"duckduckgo"` (default `"duckduckgo"`).
-* **`max_results`** (number): Number of results to return (default `10`).
+Searches the web through the v3 template pipeline. DuckDuckGo and Google are built-in templates, and custom search templates can be selected by name.
+
+**Parameters:**
+* **`query`** *(string, required)*: The search query string.
+* **`engine`** *(string, optional)*: Search engine/template to use. Can be `"duckduckgo"`, `"google"`, or a custom search template name. Default is `"duckduckgo"`.
+* **`max_results`** *(number, optional)*: Maximum number of results to return. Default is `10`.
+* **`region`** *(string/null, optional)*: Region and language code to localize search results. 
+  * Examples: `"us-en"`, `"uk-en"`, `"de-de"`. 
+  * For DuckDuckGo, it maps directly. 
+  * For Google, it maps to the `gl` (country) and `hl` (language) query parameters automatically.
+  * `null` uses the template default.
+* **`safe_search`** *(boolean/null, optional)*: Enable safe search. Maps to DuckDuckGo/Google parameters automatically; `null` uses the template default.
+* **`block_media`** *(boolean, optional)*: Block images, media, and fonts at the network layer. Default is `true`.
 
 ### 2. `webfetch`
-Visits a URL as a stealthy human, waits for the JS to render, completely scrubs visual assets/inline styles to save tokens, and returns the markdown content.
-* **`url`** (string): Full HTTP/HTTPS link.
-* **`format`** (string): Set to `"markdown"` (default), `"clean_html"`, or `"raw_html"`.
-* **`start_index`** (number): Pagination offset.
-* **`max_length`** (number): Maximum character length to return per call (default `10000`).
-* **`block_media`** (boolean): Speeds up page loads by ignoring images, videos, and fonts entirely at the network layer (default `true`).
-
-## Debugging
-If you want to debug JSON-RPC shapes locally:
+Fetches a page with CloakBrowser and extracts structured Markdown using a named, inline, or auto-detected template. Built-ins include GitHub repositories/issues, npm, PyPI, crates.io, and ReadTheDocs-style docs pages. Unknown pages fall back to generic Markdown extraction.
+
+**Parameters:**
+* **`url`** *(string, required)*: The full URL of the webpage to fetch (must start with http/https).
+* **`template`** *(string, optional)*: `"auto"`, a built-in template name, or inline JSON template. Default is `"auto"`.
+* **`start_index`** *(number, optional)*: Character offset to start reading from for pagination. Use this if a document is too large to fit in the context window. Default is `0`.
+* **`max_length`** *(number, optional)*: Maximum characters to return per request. Default is `10000`.
+* **`block_media`** *(boolean, optional)*: Block images, videos, and fonts entirely at the network layer to drastically speed up page loads and dodge tracking pixels. Default is `true`.
+
+Template extraction supports `text`, `markdown`, `attribute`, and `html` sections; nested children; repeated sections; URL decoding transforms; per-template cookies; and per-template resource blocking.
+
+Built-in templates live in `templates/*.json` and are shared by the Node.js and Python implementations. Each JSON file defines exactly one template — no duplication between languages.
+
+---
+
+## Architecture & Contributions
+This repository utilizes a flat dual-manifest file structure (`package.json` and `pyproject.toml` in the root). When committing changes, ensure parity between `index.js` and `server.py` logic.
+
+### Local Development
 ```bash
-npm run inspector
+# Node.js Testing
+npm i
+npm run inspector-js
+
+# Python Testing
+pip install -e .
+npm run inspector-py
 ```