multi-websearch-mcp 0.3.0rc2__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,191 @@
1
+ Metadata-Version: 2.5
2
+ Name: multi-websearch-mcp
3
+ Version: 0.3.0rc2
4
+ Summary: Parallel, captcha-avoiding web search over free sources (SearXNG, Bing RSS, Google News, GDELT, ...) with CLI, MCP server and monitoring
5
+ Project-URL: Homepage, https://github.com/c0decave/multi-websearch-mcp
6
+ Project-URL: Documentation, https://github.com/c0decave/multi-websearch-mcp/blob/main/docs/USAGE.md
7
+ Project-URL: Repository, https://github.com/c0decave/multi-websearch-mcp
8
+ Project-URL: Issues, https://github.com/c0decave/multi-websearch-mcp/issues
9
+ Project-URL: Changelog, https://github.com/c0decave/multi-websearch-mcp/blob/main/CHANGELOG.md
10
+ Author: websearch contributors
11
+ License-Expression: MIT
12
+ License-File: LICENSE
13
+ Keywords: ai agents,llm,mcp,metasearch,model context protocol,news monitoring,search,searxng,web search
14
+ Classifier: Development Status :: 4 - Beta
15
+ Classifier: Environment :: Console
16
+ Classifier: Environment :: Web Environment
17
+ Classifier: Framework :: AsyncIO
18
+ Classifier: Intended Audience :: Developers
19
+ Classifier: Intended Audience :: Information Technology
20
+ Classifier: Natural Language :: English
21
+ Classifier: Natural Language :: German
22
+ Classifier: Operating System :: OS Independent
23
+ Classifier: Programming Language :: Python :: 3
24
+ Classifier: Programming Language :: Python :: 3 :: Only
25
+ Classifier: Programming Language :: Python :: 3.10
26
+ Classifier: Programming Language :: Python :: 3.11
27
+ Classifier: Programming Language :: Python :: 3.12
28
+ Classifier: Programming Language :: Python :: 3.13
29
+ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
30
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
31
+ Classifier: Typing :: Typed
32
+ Requires-Python: >=3.10
33
+ Requires-Dist: ddgs<10,>=9.16
34
+ Requires-Dist: httpx>=0.27
35
+ Requires-Dist: lxml-html-clean>=0.4
36
+ Requires-Dist: mcp<3,>=2.3
37
+ Requires-Dist: pydantic>=2.7
38
+ Requires-Dist: starlette>=0.40
39
+ Requires-Dist: trafilatura<3,>=2.0
40
+ Requires-Dist: typing-extensions>=4.12
41
+ Requires-Dist: uvicorn>=0.30
42
+ Provides-Extra: dev
43
+ Requires-Dist: mypy>=1.10; extra == 'dev'
44
+ Requires-Dist: pytest>=8; extra == 'dev'
45
+ Description-Content-Type: text/markdown
46
+
47
+ # websearch – web search MCP server that avoids captchas
48
+
49
+ [![CI](https://github.com/c0decave/multi-websearch-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/c0decave/multi-websearch-mcp/actions/workflows/ci.yml)
50
+ [![PyPI](https://img.shields.io/pypi/v/multi-websearch-mcp?include_prereleases&label=PyPI)](https://pypi.org/project/multi-websearch-mcp/)
51
+ [![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue)](pyproject.toml)
52
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
53
+
54
+ <!-- mcp-name: io.github.c0decave/multi-websearch-mcp -->
55
+
56
+ **English** · [Deutsch](README.de.md) · [Quickstart](docs/QUICKSTART.md) · [Usage](docs/USAGE.md) · [Changelog](CHANGELOG.md)
57
+
58
+ Parallel web, news, image and video search **plus page fetching** for AI agents, built on free sources:
59
+ a self-hosted [SearXNG](https://github.com/searxng/searxng) (Google, Bing, Brave, DuckDuckGo, Qwant …) and
60
+ official feeds and APIs that never show captchas (Bing RSS, Google News, GDELT, Hacker News, Wikipedia,
61
+ Marginalia, Reddit). Cheap paid APIs are an optional last resort with hard spending limits. Comes as an
62
+ **MCP server** (stdio and HTTP), a CLI, a Python API and a **monitoring service** that reports only new results to
63
+ Slack, Discord, Teams, Telegram, ntfy, e-mail or a webhook.
64
+
65
+ > **Pre-release 0.3.0rc2.** Feature-complete and tested (≈680 offline tests), but still in its pilot phase.
66
+ > Please report problems as [issues](https://github.com/c0decave/multi-websearch-mcp/issues).
67
+
68
+ ## Why
69
+
70
+ Free official web search APIs are basically gone: the Bing Search API was shut down in August 2025, Google's
71
+ Custom Search JSON API takes no new customers and ends on 1 January 2027, and more search engines put
72
+ proof-of-work captchas in front of automated clients. Scraping naively gets you blocked within minutes.
73
+
74
+ This server is built to **avoid** captchas instead of solving them:
75
+
76
+ - **Rate limits per source that hold across processes**, so the CLI, the monitor and the MCP server share one budget.
77
+ - **Each engine is asked only once per query**, and results are cached.
78
+ - **Blocked sources are paused automatically** (the pause doubles with each block, capped at 6 h), and *silent*
79
+ blocks (empty answers instead of errors) are detected.
80
+ - **Captcha-free sources come first:** official feeds and APIs, with search engines as fallback in tiers.
81
+ - **Paid APIs only as a last resort** (Exa, Linkup, Tavily, Brave, Perplexity, Serper), only when the free sources
82
+ failed because of blocks, always in the cheapest mode and within a daily and monthly limit.
83
+
84
+ The background (engine comparison, prices, captcha strategies, proxies) is in [ANALYSE.md](ANALYSE.md) (German).
85
+
86
+ ## MCP tools
87
+
88
+ | Tool | Purpose |
89
+ |---|---|
90
+ | `web_search(query, max_results=10, language="auto", days=None)` | web search across all tiers |
91
+ | `news_search(query, max_results=15, language="auto", days=None)` | current news, newest day first |
92
+ | `search_many(queries[≤20], max_results=8, kind="web"\|"news"\|"ref", language, days)` | several sub-questions in parallel |
93
+ | `image_search(query, max_results=10, language="auto", days=None)` | images: page, image and thumbnail URL |
94
+ | `video_search(query, max_results=10, language="auto", days=None)` | videos with thumbnail, duration and date |
95
+ | `fetch_url(url, max_chars=20000)` | read a page as Markdown (also resolves Google News links) |
96
+ | `source_status()` | which sources are currently paused, and why |
97
+
98
+ Results from all sources are merged, de-duplicated and ranked (Reciprocal Rank Fusion); `engines` shows which
99
+ sources found each hit. All tools have input and output schemas, and errors reach the model as readable tool errors.
100
+
101
+ ## Quickstart
102
+
103
+ **Full setup with Docker** (all sources, monitoring, HTTP MCP server with a token):
104
+
105
+ ```bash
106
+ git clone https://github.com/c0decave/multi-websearch-mcp.git && cd multi-websearch-mcp
107
+ python3 -m venv .venv && . .venv/bin/activate && pip install -e .
108
+ websearch init --contact you@your-company.com --lang en # .env with secrets and your contact
109
+ docker compose up -d # SearXNG, MCP server on 127.0.0.1:8765, monitor
110
+ docker compose exec websearch-mcp websearch doctor # checks everything, tells you what to fix
111
+ T=$(.venv/bin/python -m websearch.envfile --get MCP_AUTH_TOKEN .env) && \
112
+ claude mcp add -s user --transport http websearch http://127.0.0.1:8765/mcp --header "Authorization: Bearer $T"
113
+ ```
114
+
115
+ **Try it in one minute without Docker** (stdio; no SearXNG, the feeds and APIs still answer):
116
+
117
+ ```json
118
+ {
119
+ "mcpServers": {
120
+ "websearch": {
121
+ "command": "uvx",
122
+ "args": ["multi-websearch-mcp@0.3.0rc2"],
123
+ "env": { "WEBSEARCH_UA": "websearch/0.3 (you@your-company.com)" }
124
+ }
125
+ }
126
+ }
127
+ ```
128
+
129
+ Use a real contact (e-mail or URL) instead of `you@your-company.com`: Wikimedia, GDELT and Hacker News ask
130
+ automated clients to identify themselves. Configs for Claude Code, Claude Desktop, Cursor, VS Code and Windsurf,
131
+ plus Podman, systemd and bubblewrap setups: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**.
132
+
133
+ Run it from an office or home connection if you can: data-center IPs get far more captchas.
134
+
135
+ ## Command line
136
+
137
+ ```bash
138
+ websearch search "heat pump subsidies" -l en # web search
139
+ websearch search "EU AI Act" -k news --days 7 # news of the last 7 days
140
+ websearch search "rust ownership" -k videos # images / videos: -k images, -k videos
141
+ websearch many queries.txt -n 5 # one query per line, in parallel
142
+ websearch fetch https://example.org/article # page as Markdown
143
+ websearch monitor watches/watches.json --loop # report only new results
144
+ websearch status # sources and current pauses
145
+ websearch doctor # check the setup, with fixes
146
+ ```
147
+
148
+ Everything else (all options, JSON format, Python API, monitoring rules, notifications, configuration,
149
+ Prometheus metrics, troubleshooting): **[docs/USAGE.md](docs/USAGE.md)**.
150
+
151
+ ## How it works
152
+
153
+ ```
154
+ MCP client / CLI / monitor
155
+ │
156
+ ▼
157
+ websearch engine ── cache, host-wide rate limits, block detection and pauses, budget for paid APIs
158
+ │
159
+ ├─ tier 1 (free): own SearXNG · Google News RSS · Bing RSS · GDELT · Hacker News · Wikipedia · Reddit
160
+ ├─ tier 2 (fallback): Bing RSS · DuckDuckGo / Yahoo / Brave via ddgs · Marginalia (only if tier 1 is thin)
161
+ └─ tier 3 (optional): Exa · Linkup · Tavily · Brave API · Perplexity · Serper (only after blocks)
162
+ │
163
+ ▼
164
+ merge, de-duplicate, rank (RRF), date filter ──► results + per-source status
165
+ ```
166
+
167
+ ## Security
168
+
169
+ - **HTTP server:** listens on `127.0.0.1` only, requires a bearer token (`websearch init` creates one) and checks
170
+ `Host`/`Origin` against DNS rebinding. It refuses to start when it is reachable from outside without a token.
171
+ - **`fetch_url`:** http(s) only. It refuses internal targets (localhost, private and link-local networks), also
172
+ after redirects and with DNS pinning, respects `robots.txt` and limits size and rate.
173
+ - **Containers:** no capabilities, `no-new-privileges`, read-only file systems, SearXNG not running as root.
174
+ - **Without containers:** a [bubblewrap](https://github.com/containers/bubblewrap) sandbox with seccomp, an empty
175
+ `$HOME` and only the paths it needs ([Usage → bubblewrap](docs/USAGE.md#bubblewrap-without-containers)).
176
+ - **Secrets:** tokens and webhook URLs never appear in logs, and paid APIs can be forbidden for agents
177
+ (`WEBSEARCH_MCP_PAID=0`).
178
+
179
+ ## Development
180
+
181
+ ```bash
182
+ pip install -e '.[dev]'
183
+ python -m pytest # offline tests: all sources (recorded responses), monitoring, fetch, MCP, CLI, deploy
184
+ mypy websearch
185
+ ```
186
+
187
+ The release process is in [Usage → Development](docs/USAGE.md#development-tests-and-release).
188
+
189
+ ## License
190
+
191
+ [MIT](LICENSE)
@@ -0,0 +1,22 @@
1
+ websearch/__init__.py,sha256=am3t2UFbtg_gPq03sDmIRE8bPREAU0hSm0vmBC54QcQ,729
2
+ websearch/__main__.py,sha256=Sx1hCvSiZyZFgHclQpKty7UUP2mS8DqnsgEZzwgOv60,29243
3
+ websearch/core.py,sha256=5wKf-thnXKQu65npDCljEKTbDdYx4Tpdn4J2DG0e3PM,40368
4
+ websearch/doctor.py,sha256=8TICm8myHK0Y0_TVZmGyoRDEw2KPFrk8U-6DgkDswCE,29857
5
+ websearch/engine.py,sha256=ymOsmGD4jKWj6wKqJ8_QKVPM-hm14v9jzr-CGGNAibo,48783
6
+ websearch/envfile.py,sha256=LNfQ3R5dIqY_AbL5362te9jNrI_3fF1pl6Yrk8kT_1c,6647
7
+ websearch/fetch.py,sha256=7KIO4bHtamHqFKxkuEiifPOjDmGikVSwe6ujEgBYfjw,28078
8
+ websearch/gnews.py,sha256=O4RkXkKmwC4jcGXXtgISZdp6rSQXsK44YIOIPz8IlS8,19516
9
+ websearch/i18n.py,sha256=eI5mf9fF1RnPLXYyh2xmfAJYoZmnwUGkwIZlMs_SKuA,1823
10
+ websearch/mcp_server.py,sha256=xPMsIczvHfzE1F6YKDoDoJSjCZgrL8iXSOaHFE7O-lE,22785
11
+ websearch/metrics.py,sha256=y05-zVWYKKZbXO1rgGG_nlj4XCeV0kQwCqYXwajECX4,4011
12
+ websearch/monitor.py,sha256=2ihgvB6In402kH9GtWZjlAnQH0Wn_JnITp6GMPOe-30,48162
13
+ websearch/notify.py,sha256=W4j5fQfTmTVRE9ppkq4txjIvu_kSLm4oAvzTpSWISDw,32116
14
+ websearch/providers.py,sha256=PwoWadALyBzlzNND8Mk54A07Igp7hPfGo-KY6DbSq9A,53763
15
+ websearch/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
16
+ websearch/reddit.py,sha256=k9zar2kQERuJACcc_4txEeUxOEdAbsUbUuLP3W7tnic,12039
17
+ websearch/wizard.py,sha256=stWSNO_obdq4UMNmTeLgcqbWJoLWDqqPV3-kRNuqINw,27723
18
+ multi_websearch_mcp-0.3.0rc2.dist-info/METADATA,sha256=TPx7GfmGG0PQX7--Y-gBksW8bC6oZAc494dM4NxGpHQ,10050
19
+ multi_websearch_mcp-0.3.0rc2.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
20
+ multi_websearch_mcp-0.3.0rc2.dist-info/entry_points.txt,sha256=wgvAWnDrnkGi1P0SnGVJ3zSH-WXlxfqHxid_Iib9tAc,102
21
+ multi_websearch_mcp-0.3.0rc2.dist-info/licenses/LICENSE,sha256=9t2Vz4pxGUGr5IVSEis9MVD24_2uxHhxJVXB_6jDQSE,1079
22
+ multi_websearch_mcp-0.3.0rc2.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.33.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,3 @@
1
+ [console_scripts]
2
+ multi-websearch-mcp = websearch.__main__:run_mcp
3
+ websearch = websearch.__main__:run
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 websearch contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
websearch/__init__.py ADDED
@@ -0,0 +1,21 @@
1
+ """websearch - parallele, captcha-arme Websuche über freie Quellen (mit optionaler Bezahl-Reserve).
2
+
3
+ import asyncio
4
+ from websearch import SearchEngine
5
+
6
+ async def main():
7
+ async with SearchEngine() as eng:
8
+ res = await eng.search("open source metasearch", n=5, lang="de")
9
+ for r in res["results"]:
10
+ print(r["title"], r["url"])
11
+
12
+ asyncio.run(main())
13
+ """
14
+
15
+ __version__ = "0.3.0rc2"
16
+
17
+ from .core import Result, SearchResponse, SourceStatus # noqa: E402
18
+ from .engine import DEFAULT_POLICY, KINDS, SearchEngine, TierPolicy # noqa: E402
19
+
20
+ __all__ = ["SearchEngine", "TierPolicy", "DEFAULT_POLICY", "KINDS", "Result", "SearchResponse",
21
+ "SourceStatus", "__version__"]