wildweb-mcp 0.1.0__tar.gz

wildweb_mcp-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Carel Wentzel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wildweb_mcp-0.1.0/PKG-INFO

@@ -0,0 +1,429 @@
+Metadata-Version: 2.4
+Name: wildweb-mcp
+Version: 0.1.0
+Summary: An extensible MCP server that exposes web scraping as an MCP tool.
+Author-email: Carel Wentzel <carel.wentzel@gmail.com>
+License-Expression: MIT
+Keywords: mcp,model-context-protocol,web-scraping,playwright,llm-tools
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp==1.13.1
+Requires-Dist: playwright==1.65.0
+Requires-Dist: trafilatura==2.3.1
+Requires-Dist: tiktoken==0.14.0
+Requires-Dist: httpx==0.28.1
+Requires-Dist: lxml==6.1.3
+Requires-Dist: pydantic==2.13.5
+Provides-Extra: dev
+Requires-Dist: pytest==7.4.4; extra == "dev"
+Requires-Dist: hypothesis==6.154.1; extra == "dev"
+Dynamic: license-file
+
+# wildweb-mcp
+
+An extensible [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, written
+in Python, that exposes web scraping as an MCP tool. It fetches a web page with a headless
+Chromium browser (so client-rendered pages are captured), strips boilerplate (navigation, ads,
+scripts, styles) with `trafilatura`, and returns token-efficient clean markdown to MCP clients.
+No API key or third-party search/scrape API is required — it talks to the open web directly.
+
+## Requirements
+
+- Python **3.10+**
+- Chromium browser binary — installed automatically on first run, see below
+
+## Installation
+
+Install into a virtualenv rather than system/global Python — it keeps `wildweb-mcp`'s pinned
+dependencies isolated, and (as covered below) MCP clients need this venv's path anyway since they
+launch the server directly, bypassing your shell:
+
+```bash
+python3 -m venv ~/.venvs/wildweb
+~/.venvs/wildweb/bin/pip install wildweb-mcp
+```
+
+(`.venvs\wildweb\Scripts\pip.exe install wildweb-mcp` on Windows.)
+
+### The Chromium browser
+
+`pip install` installs the `playwright` Python package (it's a pinned dependency of
+`wildweb-mcp`) but not the browser binary that Playwright drives.
+
+**You don't need to install it yourself.** If Chromium isn't found when the server starts, it
+runs `playwright install chromium` automatically — a one-time download (~300MB) into the shared
+Playwright cache (`~/.cache/ms-playwright` on Linux/macOS,
+`%USERPROFILE%\AppData\Local\ms-playwright` on Windows) — then retries. Every startup after that
+is unaffected, since the binary is already there. This first-run install can take a few minutes
+and needs network access; if it can't finish within 5 minutes (e.g. no network) or fails for
+another reason, the server falls back to the existing clear startup error instead of hanging.
+Progress and any install errors are logged — see [Where logs go](#where-logs-go) below.
+
+If you'd rather install it ahead of time (e.g. to avoid the first-run delay, or in an offline/
+sandboxed environment where the server shouldn't attempt a network call on its own), run it
+yourself before starting the server — using the venv path from Installation above:
+
+```bash
+~/.venvs/wildweb/bin/playwright install chromium
+```
+
+(`~/.venvs/wildweb/Scripts/playwright.exe install chromium` on Windows.)
+
+**A bare `playwright install chromium` will not work as a copy-paste pre-setup step** unless you
+first activate that venv (`source ~/.venvs/wildweb/bin/activate`, or on Windows
+`~/.venvs/wildweb/Scripts/activate`) — without either activation or the full path, your shell has
+no `playwright` command on `PATH` at all (it only exists inside the venv you installed
+`wildweb-mcp` into) and the command fails with "command not found." The full-path form above
+works unconditionally, with nothing to activate first.
+
+This has to be the `playwright` from the **same environment** `wildweb-mcp` was installed into —
+if you installed globally/user-wide instead (`pip install --user` or system pip) rather than into
+a venv, it lands in pip's script directory instead (e.g. `~/.local/bin` on Linux, your Python
+install's `Scripts`/`bin` folder on macOS/Windows), and a bare `playwright install chromium` works
+directly only if that directory is on `PATH`.
+
+On Linux, Chromium may also need OS-level shared libraries. **This is never installed
+automatically** — it requires root/sudo to run a system package manager, which the server will
+not do on its own. If the server's startup error (or a manual `playwright install chromium`)
+warns about missing dependencies, install them yourself:
+
+```bash
+playwright install-deps chromium
+```
+
+## Configuration
+
+The server reads a single JSON configuration file. Copy the sample and edit it as needed:
+
+```bash
+cp config.example.json config.json
+```
+
+```json
+{
+  "request_timeout_ms": 30000,
+  "token_budget": 4000,
+  "log_level": "INFO",
+  "continuation_ttl_seconds": 300,
+  "browser": {
+    "headless": true,
+    "navigation_timeout_ms": 30000,
+    "wait_until": "load"
+  }
+}
+```
+
+All fields are optional and fall back to documented defaults:
+
+| Field                        | Default   | Description                                                        |
+| ---------------------------- | --------- | ------------------------------------------------------------------ |
+| `request_timeout_ms`         | `30000`   | Logical request timeout in milliseconds.                           |
+| `token_budget`               | `null`    | Max tokens per tool output; `null` disables budgeting/truncation.  |
+| `log_level`                  | `"INFO"`  | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.            |
+| `continuation_ttl_seconds`   | `300`     | Time-to-live for continuation tokens, in seconds.                  |
+| `browser.headless`           | `true`    | Launch Chromium headless.                                          |
+| `browser.navigation_timeout_ms` | `30000` | Playwright navigation timeout; defaults to `request_timeout_ms`.  |
+| `browser.wait_until`         | `"load"`  | One of `load`, `domcontentloaded`, `networkidle`, `commit`.        |
+| `search_result_count`        | `10`      | Number of results the `search` tool returns per query.             |
+| `search_timeout_ms`          | `10000`   | Timeout in milliseconds for a `search` request.                    |
+| `user_agents`                | `null`    | Array of User-Agent strings to rotate through; `null` uses the built-in default pool. |
+| `log_file`                   | `null`    | Path to also write logs to, capped at 10MB (see [Where logs go](#where-logs-go)); `null` logs to stderr only. |
+
+### Simulating a modern browser
+
+Every outgoing request (Playwright page navigation and the `search` tool's DuckDuckGo
+request) identifies as an ordinary modern desktop browser rather than a generic
+client/library default, since some sites are more likely to serve their normal response
+to a request that looks like one. Requests draw from a rotating pool of real browser
+User-Agent strings (Chrome, Firefox, Safari, Edge) round-robin, one per request, rather
+than reusing a single fixed value.
+
+Set `user_agents` to replace the default pool with your own:
+
+```json
+{
+  "user_agents": [
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
+  ]
+}
+```
+
+The array must be non-empty; omit the field or set it to `null` to use the default pool.
+
+## Where logs go
+
+All server logging — startup messages (including the automatic Chromium install above),
+per-request success/failure entries, and errors — is written to **stderr**, never stdout.
+This is deliberate: stdout is reserved exclusively for the MCP JSON-RPC protocol stream between
+the server and its client, so anything else written there would corrupt that stream.
+
+What you see depends on how your MCP client surfaces a subprocess's stderr. Most already capture
+it to their own log file, so check there first:
+
+- **Claude Desktop**: `~/Library/Logs/Claude/mcp-server-<name>.log` (macOS),
+  `%APPDATA%\Claude\logs\mcp-server-<name>.log` (Windows)
+- **Claude Code**: `claude mcp logs <name>`
+- **Kiro CLI / opencode**: each client's own MCP connection log/output panel — check its docs for
+  the exact location
+
+Or run `wildweb` directly in a terminal to see log lines interleaved with your own shell.
+`log_level` (see the [Configuration](#configuration) table above) controls verbosity: `DEBUG`
+through `CRITICAL`, defaulting to `INFO`.
+
+### Writing logs to a file too
+
+If your client doesn't capture stderr usefully, or you just want one log location regardless of
+which client launched the server, set `log_file` to a path:
+
+```json
+{
+  "log_file": "/absolute/path/to/wildweb.log"
+}
+```
+
+This writes the same entries to that file *in addition to* stderr — nothing stops going to
+stderr. The file is capped at 10MB: once it hits that size, it's truncated (overwritten from
+empty) rather than rotated to `.1`/`.2` backups, so it's always exactly one bounded file, never
+growing unbounded across a long-running server process and never leaving old backup files to
+clean up. If the path can't be opened (e.g. its parent directory doesn't exist), the server logs
+a warning and continues with stderr-only logging rather than failing to start.
+
+## Usage
+
+A config file is optional. The config file path is resolved with the following
+precedence:
+
+1. `--config <path>` command-line argument (highest precedence)
+2. `WILDWEB_CONFIG` environment variable
+3. If neither is provided, the server runs with the built-in defaults listed
+   under [Configuration](#configuration) — no config file is required.
+
+```bash
+# with no config at all — runs with built-in defaults
+wildweb
+
+# via command-line argument
+wildweb --config config.json
+
+# or via environment variable
+WILDWEB_CONFIG=config.json wildweb
+```
+
+MCP clients connect over stdio and discover three tools:
+
+- **`scrape`** — fetch a web page by URL and return its main textual content as clean markdown.
+  When the content exceeds the configured `token_budget`, the result is truncated and a
+  continuation token is returned.
+- **`scrape_continue`** — fetch the next portion of a previously scraped page using a
+  continuation token returned by `scrape`.
+- **`search`** — search the web for a query and return a list of results (title, URL, snippet).
+  Input: a non-empty text query. Output: up to `search_result_count` ordered results, or an
+  error. The `search` tool does **not** fetch or scrape any result's page content — pipe a
+  result's `url` into `scrape` for that.
+
+### Search provider caveat
+
+The `search` tool is backed by a free, key-free `SearchProvider` implementation
+(`DuckDuckGoSearchProvider`) that fetches and parses DuckDuckGo's public HTML results page
+(`https://html.duckduckgo.com/html/`). This is an **unofficial surface**, not a documented or
+supported API:
+
+- DuckDuckGo can change its HTML markup, rate-limit, or block automated requests at any time
+  without notice, which will surface as a `search_unavailable` error from the `search` tool.
+- Using it at any significant volume may be subject to DuckDuckGo's Terms of Service; this
+  provider is intended for light, personal-scale use.
+- The `SearchProvider` interface (`src/wildweb/search/base.py`) exists specifically so a
+  different, officially-supported provider (e.g. a paid search API) can be registered later
+  without changing the `search` tool or server wiring — see `src/wildweb/search/registry.py`.
+
+## Adding it to an MCP client
+
+Once installed (see [Installation](#installation) above), the `wildweb` console script is what
+MCP clients launch over stdio.
+
+### Point your client at the venv's executable, not a bare command
+
+An MCP client launches the server as a plain subprocess using exactly the `command` string in
+its own config — that launch does not go through your shell, so it never activates the virtualenv
+you installed into and often doesn't even inherit your normal shell `PATH`. A bare
+`"command": "wildweb"` only works if `wildweb` happens to resolve on whatever `PATH` that
+client's process has, which is unreliable. The robust fix: give the client the venv's own
+executable path directly, instead of relying on `PATH` resolution — use the venv path from
+Installation in place of the bare `wildweb` shown in the client configs below:
+
+```json
+{
+  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
+}
+```
+
+(`...\.venvs\wildweb\Scripts\wildweb.exe` on Windows.) No `args` needed, no `PATH` dependency,
+nothing to activate.
+
+### Config: none, a file, or an environment variable
+
+The simplest entry needs no config at all — the server runs with its built-in defaults:
+
+```json
+{
+  "command": "wildweb"
+}
+```
+
+To use a config file instead, add `--config` with an absolute path (the client usually
+launches the server from its own working directory, not this project's):
+
+```json
+{
+  "command": "wildweb",
+  "args": ["--config", "/absolute/path/to/config.json"]
+}
+```
+
+Alternatively, set `WILDWEB_CONFIG` as an environment variable on the entry instead of passing
+`--config`.
+
+### Claude Desktop
+
+Add a `wildweb` entry under `mcpServers` in Claude Desktop's config file
+(`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS,
+`%APPDATA%\Claude\claude_desktop_config.json` on Windows). No config (built-in defaults):
+
+```json
+{
+  "mcpServers": {
+    "wildweb": {
+      "command": "wildweb"
+    }
+  }
+}
+```
+
+With a config file:
+
+```json
+{
+  "mcpServers": {
+    "wildweb": {
+      "command": "wildweb",
+      "args": ["--config", "/absolute/path/to/config.json"]
+    }
+  }
+}
+```
+
+Restart Claude Desktop for the change to take effect.
+
+### Claude Code
+
+Add it as a user- or project-scoped server with the `claude mcp add` CLI. The first `wildweb` is
+the server's name within Claude Code; the second is the command it runs. No config:
+
+```bash
+claude mcp add wildweb wildweb
+```
+
+With a config file:
+
+```bash
+claude mcp add wildweb wildweb -- --config /absolute/path/to/config.json
+```
+
+Or add the equivalent `mcpServers` block shown above directly to the project's `.mcp.json`.
+
+### Kiro CLI
+
+Add a `wildweb` entry under `mcpServers` in Kiro's MCP config file — globally at
+`~/.kiro/settings/mcp.json`, or per-project at `.kiro/settings/mcp.json` (workspace settings
+take precedence when both exist). No config:
+
+```json
+{
+  "mcpServers": {
+    "wildweb": {
+      "command": "wildweb",
+      "disabled": false,
+      "autoApprove": []
+    }
+  }
+}
+```
+
+With a config file:
+
+```json
+{
+  "mcpServers": {
+    "wildweb": {
+      "command": "wildweb",
+      "args": ["--config", "/absolute/path/to/config.json"],
+      "disabled": false,
+      "autoApprove": []
+    }
+  }
+}
+```
+
+Kiro does not inherit your shell's `PATH`, so if `wildweb` isn't resolvable as a bare command,
+use its absolute path instead (e.g. `/path/to/venv/bin/wildweb`). Run `/mcp` inside Kiro CLI to
+confirm the server loaded.
+
+### opencode
+
+Add a local server under `mcp` in `opencode.jsonc` (project root) or the global
+`~/.config/opencode/opencode.jsonc`. No config:
+
+```jsonc
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "wildweb": {
+      "type": "local",
+      "command": ["wildweb"]
+    }
+  }
+}
+```
+
+With a config file:
+
+```jsonc
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "wildweb": {
+      "type": "local",
+      "command": ["wildweb", "--config", "/absolute/path/to/config.json"]
+    }
+  }
+}
+```
+
+Or add it from the CLI. No config:
+
+```bash
+opencode mcp add wildweb -- wildweb
+```
+
+With a config file:
+
+```bash
+opencode mcp add wildweb -- wildweb --config /absolute/path/to/config.json
+```
+
+### Other clients
+
+Any MCP client that launches servers over stdio (Cursor, Windsurf, etc.) accepts the same
+`command`/`args` shape under whatever key its own config format uses for MCP servers — consult
+that client's docs for the exact file location.