webscout-mcp 0.1.0__tar.gz

webscout_mcp-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 webscout-mcp contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

webscout_mcp-0.1.0/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.4
+Name: webscout-mcp
+Version: 0.1.0
+Summary: A smart web search & fetch MCP server with built-in caching, rate-limiting, and content extraction
+Author: webscout-mcp contributors
+License: MIT
+Keywords: mcp,model-context-protocol,web-search,web-scraping,ai-agent,caching
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp>=1.0.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: lxml>=5.0.0
+Requires-Dist: trafilatura>=1.6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
+Dynamic: license-file
+
+# webscout-mcp
+
+Web search and fetch tools for AI agents, as an MCP server. Search, fetch, crawl, and extract structured data from the web — no API keys, no per-request billing, everything stays on your machine.
+
+## Install
+
+```bash
+pip install webscout-mcp
+```
+
+Requires Python 3.10+.
+
+## Quick start
+
+Add to your MCP client config (Claude Code, Cursor, Codex, etc.):
+
+```json
+{
+  "mcpServers": {
+    "webscout": {
+      "command": "webscout-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+That's it. Your agent gets six tools:
+
+- `web_search` — search via Bing, no key needed
+- `web_fetch` — fetch a page and extract the main article (markdown/text/html)
+- `web_crawl` — bounded BFS crawl with depth and page limits
+- `web_extract` — pull structured data with CSS selectors, attributes, regex
+- `cache_stats` — inspect the local cache
+- `cache_clear` — wipe the cache
+
+## Usage examples
+
+### Search
+
+```json
+web_search(query="best python async libraries", max_results=5)
+```
+
+Returns structured results with title, URL, and snippet.
+
+### Fetch a page
+
+```json
+web_fetch(url="https://example.com", extract=true, output_format="markdown")
+```
+
+`extract=true` runs trafilatura to strip nav, ads, and sidebars — you get clean article content, not raw HTML.
+
+### Extract structured data
+
+```json
+web_extract(
+  url="https://example.com/products",
+  rules='[
+    {"name": "titles", "selector": ".product h2", "multiple": true},
+    {"name": "prices", "selector": ".price", "regex": "\\$([\\d.]+)", "multiple": true},
+    {"name": "links", "selector": "a.product", "attribute": "href", "multiple": true}
+  ]'
+)
+```
+
+Each rule supports `selector`, `attribute`, `multiple`, `regex`, and `default`.
+
+### Crawl a site
+
+```json
+web_crawl(seed_url="https://example.com", max_depth=2, max_pages=10)
+```
+
+Respects same-domain by default. All fetched pages go through the same cache and rate limiter.
+
+## Use as a Python library
+
+```python
+import asyncio
+from webscout_mcp import Config, Fetcher, SearchEngine
+
+async def main():
+    config = Config.from_env()
+    config.ensure_dirs()
+
+    fetcher = Fetcher(config)
+    result = await fetcher.fetch("https://example.com", extract=True)
+    print(result.title)
+    print(result.content[:500])
+    await fetcher.close()
+
+    search = SearchEngine(config)
+    results = await search.search("python async", max_results=5)
+    for r in results:
+        print(f"{r.position}. {r.title} — {r.url}")
+
+asyncio.run(main())
+```
+
+## How it works
+
+- **Search** uses Bing via direct HTTP (no API key). Results are cached by query.
+- **Fetching** uses httpx with exponential-backoff retries, per-domain token-bucket rate limiting, and a 5 MB content cap.
+- **Content extraction** uses trafilatura — the same library behind many read-it-later services.
+- **Caching** is SQLite with TTL and a size cap; old entries are evicted automatically. Repeat fetches and searches cost nothing.
+- **Crawling** is BFS with configurable depth, page count, and same-domain restriction.
+
+Everything runs locally. No data leaves your machine.
+
+## Configuration
+
+All settings have sensible defaults. Override via environment variables (`WEBSCOUT_` prefix) or CLI flags:
+
+| Variable | Default | What it does |
+|---|---|---|
+| `WEBSCOUT_CACHE_DIR` | `~/.cache/webscout` | Where the SQLite cache lives |
+| `WEBSCOUT_CACHE_TTL` | `7200` | Cache entry lifetime in seconds |
+| `WEBSCOUT_CACHE_MAX_SIZE_MB` | `512` | Max cache size before eviction |
+| `WEBSCOUT_REQUEST_TIMEOUT` | `15.0` | HTTP timeout in seconds |
+| `WEBSCOUT_MAX_RETRIES` | `3` | Retry attempts per request |
+| `WEBSCOUT_RATE_LIMIT_PER_SECOND` | `2.0` | Max requests per second per domain |
+| `WEBSCOUT_SEARCH_MAX_RESULTS` | `10` | Default search result count |
+| `WEBSCOUT_CRAWLER_MAX_DEPTH` | `2` | Default crawl depth |
+| `WEBSCOUT_CRAWLER_MAX_PAGES` | `20` | Default max pages per crawl |
+
+CLI flags override env vars:
+
+```bash
+webscout-mcp --cache-ttl 3600 --cache-dir /tmp/webscout
+```
+
+## Transports
+
+```bash
+# stdio (default — works with Claude Code, Cursor, etc.)
+webscout-mcp
+
+# SSE (for remote or browser-based clients)
+webscout-mcp --transport sse --host 0.0.0.0 --port 8000
+```
+
+## Development
+
+```bash
+git clone https://github.com/wxs-lang/webscout-mcp.git
+cd webscout-mcp
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT

webscout_mcp-0.1.0/README.md

@@ -0,0 +1,155 @@
+# webscout-mcp
+
+Web search and fetch tools for AI agents, as an MCP server. Search, fetch, crawl, and extract structured data from the web — no API keys, no per-request billing, everything stays on your machine.
+
+## Install
+
+```bash
+pip install webscout-mcp
+```
+
+Requires Python 3.10+.
+
+## Quick start
+
+Add to your MCP client config (Claude Code, Cursor, Codex, etc.):
+
+```json
+{
+  "mcpServers": {
+    "webscout": {
+      "command": "webscout-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+That's it. Your agent gets six tools:
+
+- `web_search` — search via Bing, no key needed
+- `web_fetch` — fetch a page and extract the main article (markdown/text/html)
+- `web_crawl` — bounded BFS crawl with depth and page limits
+- `web_extract` — pull structured data with CSS selectors, attributes, regex
+- `cache_stats` — inspect the local cache
+- `cache_clear` — wipe the cache
+
+## Usage examples
+
+### Search
+
+```json
+web_search(query="best python async libraries", max_results=5)
+```
+
+Returns structured results with title, URL, and snippet.
+
+### Fetch a page
+
+```json
+web_fetch(url="https://example.com", extract=true, output_format="markdown")
+```
+
+`extract=true` runs trafilatura to strip nav, ads, and sidebars — you get clean article content, not raw HTML.
+
+### Extract structured data
+
+```json
+web_extract(
+  url="https://example.com/products",
+  rules='[
+    {"name": "titles", "selector": ".product h2", "multiple": true},
+    {"name": "prices", "selector": ".price", "regex": "\\$([\\d.]+)", "multiple": true},
+    {"name": "links", "selector": "a.product", "attribute": "href", "multiple": true}
+  ]'
+)
+```
+
+Each rule supports `selector`, `attribute`, `multiple`, `regex`, and `default`.
+
+### Crawl a site
+
+```json
+web_crawl(seed_url="https://example.com", max_depth=2, max_pages=10)
+```
+
+Respects same-domain by default. All fetched pages go through the same cache and rate limiter.
+
+## Use as a Python library
+
+```python
+import asyncio
+from webscout_mcp import Config, Fetcher, SearchEngine
+
+async def main():
+    config = Config.from_env()
+    config.ensure_dirs()
+
+    fetcher = Fetcher(config)
+    result = await fetcher.fetch("https://example.com", extract=True)
+    print(result.title)
+    print(result.content[:500])
+    await fetcher.close()
+
+    search = SearchEngine(config)
+    results = await search.search("python async", max_results=5)
+    for r in results:
+        print(f"{r.position}. {r.title} — {r.url}")
+
+asyncio.run(main())
+```
+
+## How it works
+
+- **Search** uses Bing via direct HTTP (no API key). Results are cached by query.
+- **Fetching** uses httpx with exponential-backoff retries, per-domain token-bucket rate limiting, and a 5 MB content cap.
+- **Content extraction** uses trafilatura — the same library behind many read-it-later services.
+- **Caching** is SQLite with TTL and a size cap; old entries are evicted automatically. Repeat fetches and searches cost nothing.
+- **Crawling** is BFS with configurable depth, page count, and same-domain restriction.
+
+Everything runs locally. No data leaves your machine.
+
+## Configuration
+
+All settings have sensible defaults. Override via environment variables (`WEBSCOUT_` prefix) or CLI flags:
+
+| Variable | Default | What it does |
+|---|---|---|
+| `WEBSCOUT_CACHE_DIR` | `~/.cache/webscout` | Where the SQLite cache lives |
+| `WEBSCOUT_CACHE_TTL` | `7200` | Cache entry lifetime in seconds |
+| `WEBSCOUT_CACHE_MAX_SIZE_MB` | `512` | Max cache size before eviction |
+| `WEBSCOUT_REQUEST_TIMEOUT` | `15.0` | HTTP timeout in seconds |
+| `WEBSCOUT_MAX_RETRIES` | `3` | Retry attempts per request |
+| `WEBSCOUT_RATE_LIMIT_PER_SECOND` | `2.0` | Max requests per second per domain |
+| `WEBSCOUT_SEARCH_MAX_RESULTS` | `10` | Default search result count |
+| `WEBSCOUT_CRAWLER_MAX_DEPTH` | `2` | Default crawl depth |
+| `WEBSCOUT_CRAWLER_MAX_PAGES` | `20` | Default max pages per crawl |
+
+CLI flags override env vars:
+
+```bash
+webscout-mcp --cache-ttl 3600 --cache-dir /tmp/webscout
+```
+
+## Transports
+
+```bash
+# stdio (default — works with Claude Code, Cursor, etc.)
+webscout-mcp
+
+# SSE (for remote or browser-based clients)
+webscout-mcp --transport sse --host 0.0.0.0 --port 8000
+```
+
+## Development
+
+```bash
+git clone https://github.com/wxs-lang/webscout-mcp.git
+cd webscout-mcp
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT

webscout_mcp-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "webscout-mcp"
+version = "0.1.0"
+description = "A smart web search & fetch MCP server with built-in caching, rate-limiting, and content extraction"
+readme = "README.md"
+requires-python = ">=3.10"
+license = {text = "MIT"}
+authors = [{name = "webscout-mcp contributors"}]
+keywords = ["mcp", "model-context-protocol", "web-search", "web-scraping", "ai-agent", "caching"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "mcp>=1.0.0",
+    "httpx>=0.25.0",
+    "beautifulsoup4>=4.12.0",
+    "lxml>=5.0.0",
+    "trafilatura>=1.6.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-asyncio>=0.21",
+]
+
+[project.scripts]
+webscout-mcp = "webscout_mcp.__main__:main"
+
+[tool.setuptools.packages.find]
+include = ["webscout_mcp*"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

webscout_mcp-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

webscout_mcp-0.1.0/tests/test_basic.py

@@ -0,0 +1,185 @@
+"""Basic tests for webscout-mcp core modules.
+
+These tests focus on pure-logic components that don't require network access.
+Network-dependent tests are marked and can be run with --run-network.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+from pathlib import Path
+
+import pytest
+
+from webscout_mcp.cache import Cache
+from webscout_mcp.config import Config
+from webscout_mcp.extractor import DataExtractor, ExtractionRule
+from webscout_mcp.utils import TokenBucket, is_valid_url, normalize_url, truncate_text
+
+
+# --- Config ---
+
+class TestConfig:
+    def test_defaults(self):
+        cfg = Config()
+        assert cfg.cache_ttl == 7200
+        assert cfg.request_timeout == 15.0
+        assert cfg.search_max_results == 10
+
+    def test_from_env(self, monkeypatch):
+        monkeypatch.setenv("WEBSCOUT_CACHE_TTL", "3600")
+        monkeypatch.setenv("WEBSCOUT_SEARCH_MAX_RESULTS", "5")
+        cfg = Config.from_env()
+        assert cfg.cache_ttl == 3600
+        assert cfg.search_max_results == 5
+
+    def test_ensure_dirs(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            cfg = Config(cache_dir=Path(tmpdir) / "test_cache")
+            cfg.ensure_dirs()
+            assert (Path(tmpdir) / "test_cache").exists()
+
+
+# --- Cache ---
+
+class TestCache:
+    @pytest.fixture
+    def cache(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            c = Cache(Path(tmpdir) / "test.db", ttl=3600, max_size_mb=10)
+            yield c
+
+    def test_set_and_get(self, cache):
+        cache.set("https://example.com", "hello world", "text/plain")
+        result = cache.get("https://example.com")
+        assert result is not None
+        assert result["value"] == "hello world"
+        assert result["cached"] is True
+
+    def test_get_missing(self, cache):
+        assert cache.get("https://nonexistent.com") is None
+
+    def test_expiry(self, cache):
+        cache.set("https://example.com", "data", ttl=0)
+        # ttl=0 means expires immediately
+        import time
+        time.sleep(0.1)
+        assert cache.get("https://example.com") is None
+
+    def test_delete(self, cache):
+        cache.set("https://example.com", "data")
+        cache.delete("https://example.com")
+        assert cache.get("https://example.com") is None
+
+    def test_clear(self, cache):
+        cache.set("https://a.com", "a")
+        cache.set("https://b.com", "b")
+        deleted = cache.clear()
+        assert deleted == 2
+        assert cache.get("https://a.com") is None
+
+    def test_stats(self, cache):
+        cache.set("https://example.com", "hello")
+        stats = cache.stats()
+        assert stats["entries"] == 1
+        assert stats["total_size_bytes"] > 0
+
+
+# --- Utils ---
+
+class TestUtils:
+    def test_normalize_url(self):
+        assert normalize_url("HTTPS://Example.COM/path//to//page/") == "https://example.com/path/to/page/"
+        assert normalize_url("http://example.com:80/path") == "http://example.com/path"
+        assert normalize_url("https://example.com:443/path") == "https://example.com/path"
+
+    def test_normalize_url_strips_fragment(self):
+        result = normalize_url("https://example.com/page#section")
+        assert "#" not in result
+
+    def test_is_valid_url(self):
+        assert is_valid_url("https://example.com")
+        assert is_valid_url("http://example.com/path?q=1")
+        assert not is_valid_url("not a url")
+        assert not is_valid_url("ftp://example.com")
+        assert not is_valid_url("")
+
+    def test_truncate_text(self):
+        short = "hello"
+        assert truncate_text(short, 100) == short
+
+        long_text = "a" * 1000
+        result = truncate_text(long_text, 100)
+        assert len(result) < 1000
+        assert "truncated" in result
+
+    @pytest.mark.asyncio
+    async def test_token_bucket(self):
+        bucket = TokenBucket(rate=10.0, burst=2)
+        # Should be able to acquire burst tokens quickly
+        await bucket.acquire("https://example.com")
+        await bucket.acquire("https://example.com")
+        # Third should require a short wait
+        await bucket.acquire("https://example.com")
+
+
+# --- Extractor ---
+
+class TestExtractor:
+    SAMPLE_HTML = """
+    <html>
+    <head><title>Test Page</title></head>
+    <body>
+        <h1 class="title">Hello World</h1>
+        <p class="price">$29.99</p>
+        <a href="https://example.com/1" class="item">Item 1</a>
+        <a href="https://example.com/2" class="item">Item 2</a>
+        <a href="https://example.com/3" class="item">Item 3</a>
+        <div class="description">This is a <b>great</b> product.</div>
+    </body>
+    </html>
+    """
+
+    def test_extract_single_text(self):
+        extractor = DataExtractor(Config(), None)  # fetcher not needed for HTML extraction
+        rules = [ExtractionRule(name="title", selector="h1.title")]
+        result = extractor.extract_from_html(self.SAMPLE_HTML, rules)
+        assert result["title"] == "Hello World"
+
+    def test_extract_with_regex(self):
+        extractor = DataExtractor(Config(), None)
+        rules = [ExtractionRule(name="price", selector=".price", regex=r"\$([\d.]+)")]
+        result = extractor.extract_from_html(self.SAMPLE_HTML, rules)
+        assert result["price"] == "29.99"
+
+    def test_extract_attribute_multiple(self):
+        extractor = DataExtractor(Config(), None)
+        rules = [ExtractionRule(name="links", selector="a.item", attribute="href", multiple=True)]
+        result = extractor.extract_from_html(self.SAMPLE_HTML, rules)
+        assert result["links"] == [
+            "https://example.com/1",
+            "https://example.com/2",
+            "https://example.com/3",
+        ]
+
+    def test_extract_default_on_missing(self):
+        extractor = DataExtractor(Config(), None)
+        rules = [ExtractionRule(name="missing", selector=".nonexistent", default="N/A")]
+        result = extractor.extract_from_html(self.SAMPLE_HTML, rules)
+        assert result["missing"] == "N/A"
+
+
+# --- MCP Server (import test) ---
+
+class TestServer:
+    def test_create_server(self):
+        """Verify the server can be created without errors."""
+        with tempfile.TemporaryDirectory() as tmpdir:
+            cfg = Config(cache_dir=Path(tmpdir))
+            from webscout_mcp.server import create_server
+            mcp = create_server(cfg)
+            assert mcp is not None
+            # FastMCP stores tools internally
+            assert hasattr(mcp, "tool")

webscout_mcp-0.1.0/webscout_mcp/__init__.py

@@ -0,0 +1,11 @@
+"""webscout-mcp: A smart web search & fetch MCP server."""
+
+__version__ = "0.1.0"
+__all__ = ["__version__", "create_server", "Config", "Fetcher", "SearchEngine", "Crawler", "DataExtractor"]
+
+from .config import Config
+from .crawler import Crawler
+from .extractor import DataExtractor
+from .fetcher import Fetcher
+from .search import SearchEngine
+from .server import create_server

webscout_mcp-0.1.0/webscout_mcp/__main__.py

@@ -0,0 +1,79 @@
+"""Command-line entry point for webscout-mcp.
+
+Usage:
+    webscout-mcp                  # Start MCP server over stdio (default)
+    webscout-mcp --transport sse  # Start MCP server over SSE
+    webscout-mcp --version        # Print version
+    webscout-mcp --help           # Show help
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from . import __version__
+from .config import Config
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        prog="webscout-mcp",
+        description="A smart web search & fetch MCP server with caching and content extraction.",
+    )
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=f"webscout-mcp {__version__}",
+    )
+    parser.add_argument(
+        "--transport",
+        choices=["stdio", "sse"],
+        default="stdio",
+        help="MCP transport protocol (default: stdio)",
+    )
+    parser.add_argument(
+        "--host",
+        default="127.0.0.1",
+        help="Host for SSE transport (default: 127.0.0.1)",
+    )
+    parser.add_argument(
+        "--port",
+        type=int,
+        default=8000,
+        help="Port for SSE transport (default: 8000)",
+    )
+    parser.add_argument(
+        "--cache-dir",
+        default=None,
+        help="Override cache directory",
+    )
+    parser.add_argument(
+        "--cache-ttl",
+        type=int,
+        default=None,
+        help="Override cache TTL in seconds",
+    )
+
+    args = parser.parse_args()
+
+    # Build config
+    config = Config.from_env()
+    if args.cache_dir:
+        from pathlib import Path
+        config.cache_dir = Path(args.cache_dir)
+    if args.cache_ttl is not None:
+        config.cache_ttl = args.cache_ttl
+
+    # Create and run server
+    from .server import create_server
+    mcp = create_server(config)
+
+    if args.transport == "sse":
+        mcp.run(transport="sse", host=args.host, port=args.port)
+    else:
+        mcp.run(transport="stdio")
+
+
+if __name__ == "__main__":
+    main()