skdev-websearch-mcp 1.0.0__tar.gz

skdev_websearch_mcp-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SK-DEV-AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skdev_websearch_mcp-1.0.0/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: skdev-websearch-mcp
+Version: 1.0.0
+Summary: Multi-engine web search + fetch MCP server for AI agents (10 tools)
+Author: SK-DEV-AI
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/SK-DEV-AI/websearch
+Project-URL: Repository, https://github.com/SK-DEV-AI/websearch
+Project-URL: Issues, https://github.com/SK-DEV-AI/websearch/issues
+Keywords: mcp,mcp-server,web-search,search-engine,scraping,reranker,llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx==0.28.1
+Requires-Dist: mcp>=2.0.0
+Requires-Dist: trafilatura>=2.3.1
+Requires-Dist: markdownify>=1.0.0
+Requires-Dist: protego>=0.7.0
+Requires-Dist: aiosqlite>=0.20.0
+Requires-Dist: pymupdf>=1.24.0
+Requires-Dist: ebooklib>=0.18
+Requires-Dist: python-docx>=1.0.0
+Requires-Dist: lxml>=6.0.0
+Requires-Dist: extruct>=0.18.0
+Requires-Dist: html-to-markdown>=3.0.0
+Requires-Dist: ddgs>=9.0.0
+Requires-Dist: websockets>=14.0
+Provides-Extra: scraping
+Requires-Dist: scrapling>=0.4.15; extra == "scraping"
+Provides-Extra: extract
+Requires-Dist: crawl4ai>=0.9.4; extra == "extract"
+Provides-Extra: ocr
+Requires-Dist: docling==2.118.1; extra == "ocr"
+Provides-Extra: all
+Requires-Dist: scrapling>=0.4.15; extra == "all"
+Requires-Dist: crawl4ai>=0.9.4; extra == "all"
+Requires-Dist: docling==2.118.1; extra == "all"
+Dynamic: license-file
+
+# websearch MCP Server
+
+[![skills.sh](https://skills.sh/b/SK-DEV-AI/skills)](https://skills.sh/SK-DEV-AI/skills) · `npx skills add SK-DEV-AI/skills`
+
+Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.
+
+## Tools (10)
+
+- **search** — Open research questions. depth=1 snippets, depth≥2 full-page fetch + rerank. `synthesize` returns a Groq answer with [N] citations.
+- **fetch** — URL to markdown/text. Direct fetch with auto-fallback to the Helium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (`offset`). For structured JSON use `extract`.
+- **screenshot** — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
+- **browser_action** — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use `fetch`.
+- **wikipedia** — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
+- **arxiv** — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch `papers`).
+- **extract** — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
+- **pdf_extract** — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
+- **map_site** — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
+- **ping** — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.
+
+## Setup
+
+Requires Python 3.14+.
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+cp run.example run   # fill in your API keys (run is gitignored — never commit it)
+chmod +x run
+```
+
+Point your MCP client at `run` (stdio):
+
+```json
+{ "websearch": { "command": "/path/to/websearch/run" } }
+```
+
+### API keys (`run`)
+
+| Key | Used for | Required? |
+|---|---|---|
+| `GROQ_API_KEYS` | query rewrite, synthesis, grounding checks | Recommended (search works without, no synthesis) |
+| `TAVILY_KEYS` | Tavily engine lane | Optional |
+| `NV_KEY` | NVIDIA embedding endpoint | Optional |
+
+Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.
+
+## Notes
+
+- **Reranker (optional, local):** ranking uses a local `gte-reranker-modernbert-base` cross-encoder over a Unix socket (`RERANKER_MODEL` overrides the model). Without it, results return unranked with `relevance_score 0.0` — everything still works.
+- **Helium CDP (optional):** bot-walled pages escalate to a Chrome DevTools browser at `127.0.0.1:9222` when available; without it those pages return a terminal verdict instead of hanging.
+- **Caching:** fetch results cache for 1h (`cache_ttl=0` forces fresh). No database, no state beyond the cache dir.
+- **Safety:** all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
+- **Groq model split:** synthesis + grounding + extract-llm use `openai/gpt-oss-120b`; rewrite/classify/expand/decompose use `openai/gpt-oss-20b` (the previous `llama-3.1-8b-instant` was retired by Groq and now 404s). Constants `GROQ_SMART_MODEL`/`GROQ_CHEAP_MODEL` in config.py.
+- **Keyless lanes (all advisory-only):** OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
+- **Local reranker:** a `gte-reranker-modernbert-base` cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
+- **Optional keys (graceful keyless fallback):** `SE_KEY`/`STACKEXCHANGE_KEY` (10k/day SE), `S2_KEY`/`S2_API_KEY` (dedicated S2 1 RPS).
+- **Marginalia:** best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired (kontakt@marginalia.nu for one).
+

skdev_websearch_mcp-1.0.0/README.md

@@ -0,0 +1,58 @@
+# websearch MCP Server
+
+[![skills.sh](https://skills.sh/b/SK-DEV-AI/skills)](https://skills.sh/SK-DEV-AI/skills) · `npx skills add SK-DEV-AI/skills`
+
+Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.
+
+## Tools (10)
+
+- **search** — Open research questions. depth=1 snippets, depth≥2 full-page fetch + rerank. `synthesize` returns a Groq answer with [N] citations.
+- **fetch** — URL to markdown/text. Direct fetch with auto-fallback to the Helium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (`offset`). For structured JSON use `extract`.
+- **screenshot** — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
+- **browser_action** — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use `fetch`.
+- **wikipedia** — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
+- **arxiv** — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch `papers`).
+- **extract** — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
+- **pdf_extract** — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
+- **map_site** — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
+- **ping** — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.
+
+## Setup
+
+Requires Python 3.14+.
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+cp run.example run   # fill in your API keys (run is gitignored — never commit it)
+chmod +x run
+```
+
+Point your MCP client at `run` (stdio):
+
+```json
+{ "websearch": { "command": "/path/to/websearch/run" } }
+```
+
+### API keys (`run`)
+
+| Key | Used for | Required? |
+|---|---|---|
+| `GROQ_API_KEYS` | query rewrite, synthesis, grounding checks | Recommended (search works without, no synthesis) |
+| `TAVILY_KEYS` | Tavily engine lane | Optional |
+| `NV_KEY` | NVIDIA embedding endpoint | Optional |
+
+Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.
+
+## Notes
+
+- **Reranker (optional, local):** ranking uses a local `gte-reranker-modernbert-base` cross-encoder over a Unix socket (`RERANKER_MODEL` overrides the model). Without it, results return unranked with `relevance_score 0.0` — everything still works.
+- **Helium CDP (optional):** bot-walled pages escalate to a Chrome DevTools browser at `127.0.0.1:9222` when available; without it those pages return a terminal verdict instead of hanging.
+- **Caching:** fetch results cache for 1h (`cache_ttl=0` forces fresh). No database, no state beyond the cache dir.
+- **Safety:** all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
+- **Groq model split:** synthesis + grounding + extract-llm use `openai/gpt-oss-120b`; rewrite/classify/expand/decompose use `openai/gpt-oss-20b` (the previous `llama-3.1-8b-instant` was retired by Groq and now 404s). Constants `GROQ_SMART_MODEL`/`GROQ_CHEAP_MODEL` in config.py.
+- **Keyless lanes (all advisory-only):** OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
+- **Local reranker:** a `gte-reranker-modernbert-base` cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
+- **Optional keys (graceful keyless fallback):** `SE_KEY`/`STACKEXCHANGE_KEY` (10k/day SE), `S2_KEY`/`S2_API_KEY` (dedicated S2 1 RPS).
+- **Marginalia:** best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired (kontakt@marginalia.nu for one).
+

skdev_websearch_mcp-1.0.0/arxiv.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+import xml.etree.ElementTree as ET
+from typing import Any
+
+from config import get_http_client
+
+
+async def search_arxiv(query: str, count: int = 3, search_field: str = "all",
+                       sort_by: str = "relevance", sort_order: str = "descending",
+                       start: int = 0, id_list: str = "",
+                       category: str = "", raw_query: str = "") -> dict:
+    try:
+        if raw_query:
+            params = {"search_query": raw_query, "start": start,
+                "max_results": min(count, 50), "sortBy": sort_by, "sortOrder": sort_order}
+        elif id_list:
+            params = {"id_list": id_list, "start": start,
+                "max_results": min(count, 50), "sortBy": sort_by, "sortOrder": sort_order}
+        else:
+            search_q = f"{search_field}:{query}" if search_field != "all" else f"all:{query}"
+            if category:
+                search_q += f" AND cat:{category}"
+            params = {"search_query": search_q, "start": start,
+                "max_results": min(count, 50), "sortBy": sort_by, "sortOrder": sort_order}
+        c = get_http_client()
+        r = await c.get("https://export.arxiv.org/api/query", params=params,
+                        headers={"User-Agent": "mcp-codesearch/1.0"}, timeout=15)
+        if r.status_code != 200:
+            return {"results": [], "total_available": 0}
+        ns = {"atom": "http://www.w3.org/2005/Atom",
+              "arxiv": "http://arxiv.org/schemas/atom"}
+        root = ET.fromstring(r.text)
+        total = root.findtext("opensearch:totalResults", "0",
+                              {"opensearch": "http://a9.com/-/spec/opensearch/1.1/"})
+        results = []
+        for entry in root.findall("atom:entry", ns)[:count]:
+            title = entry.findtext("atom:title", "", ns).replace("\n", " ").strip()
+            summary_text = entry.findtext("atom:summary", "", ns).replace("\n", " ").strip()[:500]
+            url = ""
+            for link in entry.findall("atom:link", ns):
+                if link.get("title") == "pdf":
+                    url = link.get("href", "")
+                    break
+                if link.get("rel") == "alternate":
+                    url = link.get("href", "")
+            authors = [a.findtext("atom:name", "", ns) for a in entry.findall("atom:author", ns)]
+            published = entry.findtext("atom:published", "", ns)[:10]
+            updated = entry.findtext("atom:updated", "", ns)[:10]
+            categories = [c.get("term", "") for c in entry.findall("atom:category", ns)]
+            comment = entry.findtext("arxiv:comment", "", ns)
+            journal_ref = entry.findtext("arxiv:journal_ref", "", ns)
+            doi = entry.findtext("arxiv:doi", "", ns)
+            pc = entry.find("arxiv:primary_category", ns)
+            primary_cat = pc.get("term", "") if pc is not None else ""
+            id_text = entry.findtext("atom:id", "", ns)
+            arxiv_id = id_text.split("/abs/")[-1] if "/abs/" in id_text else ""
+            results.append({
+                "title": title, "url": url, "snippet": summary_text,
+                "authors": authors, "published": published, "updated": updated,
+                "source": "arxiv", "arxiv_id": arxiv_id,
+                "categories": categories, "primary_category": primary_cat,
+                "comment": comment, "journal_ref": journal_ref, "doi": doi,
+            })
+        return {"results": results, "total_available": int(total)}
+    except Exception:
+        return {"results": [], "total_available": 0}

skdev_websearch_mcp-1.0.0/bench_prof.py

@@ -0,0 +1,77 @@
+"""Detailed end-to-end benchmark of websearch search_multi with all engine keys loaded.
+
+Diagnostic only. Loads env from the server's `run` file via bash (so ${VAR:-default}
+expands exactly as the server sees), instruments every engine + phase on the `research`
+namespace, runs search_multi once, prints a per-phase bill.
+"""
+import asyncio, os, subprocess, sys, time, tempfile
+
+WSDIR = "/home/sk/session-root/repos/websearch"
+
+res = subprocess.run(["bash", "-c",
+    f"source <(grep -E '^export ' {WSDIR}/run); env | grep -E '^(GROQ|TAVILY|TINYFISH|ANYSEARCH|NV|RERANKER)_'"],
+    capture_output=True, text=True)
+for line in res.stdout.splitlines():
+    if "=" in line:
+        k, _, v = line.partition("=")
+        os.environ[k] = v
+
+sys.path.insert(0, WSDIR)
+import research
+
+from collections import defaultdict
+TIMES = defaultdict(float)
+COUNTS = defaultdict(int)
+RESULTS = defaultdict(int)
+_t0 = {}
+
+def _timeit(name):
+    def deco(fn):
+        async def wrap(*a, **k):
+            key = name
+            if key not in _t0:
+                _t0[key] = time.monotonic()
+            r = await fn(*a, **k)
+            dt = time.monotonic() - _t0.pop(key, time.monotonic())
+            TIMES[key] += dt
+            COUNTS[key] += 1
+            if isinstance(r, dict):
+                RESULTS[key] += len(r.get("results", []))
+            elif isinstance(r, list):
+                RESULTS[key] += len(r)
+            return r
+        return wrap
+    return deco
+
+_existing = {}
+def patch(mod, name):
+    _existing[name] = getattr(mod, name)
+    setattr(mod, name, _timeit(name)(_existing[name]))
+
+# Engine phases in research namespace
+for fn in ["search_google_rss", "search_tavily", "search_wikipedia", "search_reddit",
+           "search_arxiv", "search_anysearch", "tinyfish_search", "search_ddg",
+           "search_brave"]:
+    if hasattr(research, fn):
+        patch(research, fn)
+# Groq + rerank phases
+for fn in ["rewrite_query", "classify_need", "decompose_query", "expand_query", "_rerank"]:
+    if hasattr(research, fn):
+        patch(research, fn)
+
+query = sys.argv[1] if len(sys.argv) > 1 else "best laptops 2026 under 1000 dollars"
+count = int(sys.argv[2]) if len(sys.argv) > 2 else 50
+
+async def main():
+    t0 = time.monotonic()
+    from config import TAVILY_KEYS, TINYFISH_KEYS, GROQ_API_KEYS, NV_KEY
+    print(f"keys: tavily={len(TAVILY_KEYS)} tinyfish={len(TINYFISH_KEYS)} groq={len(GROQ_API_KEYS)} nv={'y' if NV_KEY else 'n'}", flush=True)
+    r = await research.search_multi(query, count=count, tavily_depth="advanced")
+    twall = time.monotonic() - t0
+    print(f"\n=== SEARCH_MULTI  q={query!r} count={count} ===")
+    print(f"{'phase':<22}{'sec':>8}{'calls':>6}{'res':>6}")
+    for k in sorted(TIMES, key=lambda x: -TIMES[x]):
+        print(f"{k:<22}{TIMES[k]:>8.2f}{COUNTS[k]:>6}{RESULTS[k]:>6}")
+    print(f"\nTOTAL WALL: {twall:.1f}s  results={len(r.get('results', []))}")
+
+asyncio.run(main())