wet-mcp 1.0.0__tar.gz

wet_mcp-1.0.0/.gitignore

@@ -0,0 +1,53 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+
+# IDE
+.idea/
+*.swp
+*.swo
+.cursor/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+.nox/
+
+# Misc
+.DS_Store
+*.log
+.env
+.env.local
+*.bak
+
+# Ruff
+.ruff_cache/
+
+# Build artifacts
+*.whl
+node_modules/

wet_mcp-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 n24q02m
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wet_mcp-1.0.0/PKG-INFO

@@ -0,0 +1,182 @@
+Metadata-Version: 2.4
+Name: wet-mcp
+Version: 1.0.0
+Summary: Open-source MCP Server thay thế Tavily - Web search, extract, crawl với SearXNG
+Project-URL: Homepage, https://github.com/n24q02m/wet-mcp
+Project-URL: Repository, https://github.com/n24q02m/wet-mcp.git
+Project-URL: Issues, https://github.com/n24q02m/wet-mcp/issues
+Author-email: n24q02m <quangminh2422004@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: crawl4ai,mcp,searxng,tavily-alternative,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: ==3.13.*
+Requires-Dist: crawl4ai>=0.8.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: mcp[cli]>=1.0.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-on-whales>=0.73.0
+Description-Content-Type: text/markdown
+
+# WET - Web ExTract MCP Server
+
+[![PyPI version](https://badge.fury.io/py/wet-mcp.svg)](https://badge.fury.io/py/wet-mcp)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+> **Open-source MCP Server thay thế Tavily cho web scraping & multimodal extraction**
+
+Zero-install experience: chỉ cần `uvx wet-mcp` - tự động setup và quản lý SearXNG container.
+
+## Features
+
+| Feature | Description |
+|:--------|:------------|
+| **Web Search** | Tìm kiếm qua SearXNG (metasearch: Google, Bing, DuckDuckGo, Brave) |
+| **Content Extract** | Trích xuất nội dung sạch (Markdown/Text/HTML) |
+| **Deep Crawl** | Đi qua nhiều trang con từ URL gốc với depth control |
+| **Site Map** | Khám phá cấu trúc URL của website |
+| **Media** | List và download images, videos, audio files |
+| **Anti-bot** | Stealth mode bypass Cloudflare, Medium, LinkedIn, Twitter |
+
+## Quick Start
+
+### Prerequisites
+
+- Docker daemon running (for SearXNG)
+- Python 3.13+ (hoặc dùng uvx)
+
+### MCP Client Configuration
+
+**Claude Desktop / Cursor / Windsurf / Antigravity:**
+
+```json
+{
+  "mcpServers": {
+    "wet": {
+      "command": "uvx",
+      "args": ["wet-mcp"]
+    }
+  }
+}
+```
+
+**Đó là tất cả!** Khi MCP client gọi wet-mcp lần đầu:
+1. Tự động install Playwright chromium
+2. Tự động pull SearXNG Docker image
+3. Start `wet-searxng` container
+4. Chạy MCP server
+
+### Without uvx
+
+```bash
+pip install wet-mcp
+wet-mcp
+```
+
+## Tools
+
+| Tool | Actions | Description |
+|:-----|:--------|:------------|
+| `web` | search, extract, crawl, map | Web operations |
+| `media` | list, download | Media discovery & download |
+| `help` | - | Full documentation |
+
+### Examples
+
+```python
+# Search
+{"action": "search", "query": "python web scraping", "max_results": 10}
+
+# Extract content
+{"action": "extract", "urls": ["https://example.com"]}
+
+# Crawl with depth
+{"action": "crawl", "urls": ["https://docs.python.org"], "depth": 2}
+
+# Map site structure
+{"action": "map", "urls": ["https://example.com"]}
+
+# List media
+{"action": "list", "url": "https://github.com/python/cpython"}
+
+# Download media
+{"action": "download", "media_urls": ["https://example.com/image.png"]}
+```
+
+## Tech Stack
+
+| Component | Technology |
+|:----------|:-----------|
+| Language | Python 3.13 |
+| MCP Framework | FastMCP |
+| Web Search | SearXNG (auto-managed Docker) |
+| Web Crawling | Crawl4AI |
+| Docker Management | python-on-whales |
+
+## How It Works
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    MCP Client                           │
+│            (Claude, Cursor, Windsurf)                   │
+└─────────────────────┬───────────────────────────────────┘
+                      │ MCP Protocol
+                      ▼
+┌─────────────────────────────────────────────────────────┐
+│                   WET MCP Server                        │
+│  ┌──────────┐  ┌──────────┐  ┌──────────────────────┐   │
+│  │   web    │  │  media   │  │        help          │   │
+│  │ (search, │  │ (list,   │  │  (full documentation)│   │
+│  │ extract, │  │ download)│  └──────────────────────┘   │
+│  │ crawl,   │  └────┬─────┘                             │
+│  │ map)     │       │                                   │
+│  └────┬─────┘       │                                   │
+│       │             │                                   │
+│       ▼             ▼                                   │
+│  ┌──────────┐  ┌──────────┐                             │
+│  │ SearXNG  │  │ Crawl4AI │                             │
+│  │ (Docker) │  │(Playwright)│                           │
+│  └──────────┘  └──────────┘                             │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Configuration
+
+Environment variables:
+
+| Variable | Default | Description |
+|:---------|:--------|:------------|
+| `WET_AUTO_DOCKER` | `true` | Auto-manage SearXNG container |
+| `WET_SEARXNG_PORT` | `8080` | SearXNG container port |
+| `SEARXNG_URL` | `http://localhost:8080` | External SearXNG URL |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+## Container Management
+
+```bash
+# View SearXNG logs
+docker logs wet-searxng
+
+# Stop SearXNG
+docker stop wet-searxng
+
+# Remove container (will be recreated on next run)
+docker rm wet-searxng
+
+# Reset auto-setup (forces re-install Playwright)
+rm ~/.wet-mcp/.setup-complete
+```
+
+## License
+
+MIT License

wet_mcp-1.0.0/README.md

@@ -0,0 +1,152 @@
+# WET - Web ExTract MCP Server
+
+[![PyPI version](https://badge.fury.io/py/wet-mcp.svg)](https://badge.fury.io/py/wet-mcp)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+> **Open-source MCP Server thay thế Tavily cho web scraping & multimodal extraction**
+
+Zero-install experience: chỉ cần `uvx wet-mcp` - tự động setup và quản lý SearXNG container.
+
+## Features
+
+| Feature | Description |
+|:--------|:------------|
+| **Web Search** | Tìm kiếm qua SearXNG (metasearch: Google, Bing, DuckDuckGo, Brave) |
+| **Content Extract** | Trích xuất nội dung sạch (Markdown/Text/HTML) |
+| **Deep Crawl** | Đi qua nhiều trang con từ URL gốc với depth control |
+| **Site Map** | Khám phá cấu trúc URL của website |
+| **Media** | List và download images, videos, audio files |
+| **Anti-bot** | Stealth mode bypass Cloudflare, Medium, LinkedIn, Twitter |
+
+## Quick Start
+
+### Prerequisites
+
+- Docker daemon running (for SearXNG)
+- Python 3.13+ (hoặc dùng uvx)
+
+### MCP Client Configuration
+
+**Claude Desktop / Cursor / Windsurf / Antigravity:**
+
+```json
+{
+  "mcpServers": {
+    "wet": {
+      "command": "uvx",
+      "args": ["wet-mcp"]
+    }
+  }
+}
+```
+
+**Đó là tất cả!** Khi MCP client gọi wet-mcp lần đầu:
+1. Tự động install Playwright chromium
+2. Tự động pull SearXNG Docker image
+3. Start `wet-searxng` container
+4. Chạy MCP server
+
+### Without uvx
+
+```bash
+pip install wet-mcp
+wet-mcp
+```
+
+## Tools
+
+| Tool | Actions | Description |
+|:-----|:--------|:------------|
+| `web` | search, extract, crawl, map | Web operations |
+| `media` | list, download | Media discovery & download |
+| `help` | - | Full documentation |
+
+### Examples
+
+```python
+# Search
+{"action": "search", "query": "python web scraping", "max_results": 10}
+
+# Extract content
+{"action": "extract", "urls": ["https://example.com"]}
+
+# Crawl with depth
+{"action": "crawl", "urls": ["https://docs.python.org"], "depth": 2}
+
+# Map site structure
+{"action": "map", "urls": ["https://example.com"]}
+
+# List media
+{"action": "list", "url": "https://github.com/python/cpython"}
+
+# Download media
+{"action": "download", "media_urls": ["https://example.com/image.png"]}
+```
+
+## Tech Stack
+
+| Component | Technology |
+|:----------|:-----------|
+| Language | Python 3.13 |
+| MCP Framework | FastMCP |
+| Web Search | SearXNG (auto-managed Docker) |
+| Web Crawling | Crawl4AI |
+| Docker Management | python-on-whales |
+
+## How It Works
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    MCP Client                           │
+│            (Claude, Cursor, Windsurf)                   │
+└─────────────────────┬───────────────────────────────────┘
+                      │ MCP Protocol
+                      ▼
+┌─────────────────────────────────────────────────────────┐
+│                   WET MCP Server                        │
+│  ┌──────────┐  ┌──────────┐  ┌──────────────────────┐   │
+│  │   web    │  │  media   │  │        help          │   │
+│  │ (search, │  │ (list,   │  │  (full documentation)│   │
+│  │ extract, │  │ download)│  └──────────────────────┘   │
+│  │ crawl,   │  └────┬─────┘                             │
+│  │ map)     │       │                                   │
+│  └────┬─────┘       │                                   │
+│       │             │                                   │
+│       ▼             ▼                                   │
+│  ┌──────────┐  ┌──────────┐                             │
+│  │ SearXNG  │  │ Crawl4AI │                             │
+│  │ (Docker) │  │(Playwright)│                           │
+│  └──────────┘  └──────────┘                             │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Configuration
+
+Environment variables:
+
+| Variable | Default | Description |
+|:---------|:--------|:------------|
+| `WET_AUTO_DOCKER` | `true` | Auto-manage SearXNG container |
+| `WET_SEARXNG_PORT` | `8080` | SearXNG container port |
+| `SEARXNG_URL` | `http://localhost:8080` | External SearXNG URL |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+## Container Management
+
+```bash
+# View SearXNG logs
+docker logs wet-searxng
+
+# Stop SearXNG
+docker stop wet-searxng
+
+# Remove container (will be recreated on next run)
+docker rm wet-searxng
+
+# Reset auto-setup (forces re-install Playwright)
+rm ~/.wet-mcp/.setup-complete
+```
+
+## License
+
+MIT License

wet_mcp-1.0.0/pyproject.toml

@@ -0,0 +1,89 @@
+[project]
+name = "wet-mcp"
+version = "1.0.0"
+description = "Open-source MCP Server thay thế Tavily - Web search, extract, crawl với SearXNG"
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "n24q02m", email = "quangminh2422004@gmail.com" }]
+keywords = ["mcp", "web-scraping", "searxng", "crawl4ai", "tavily-alternative"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+requires-python = "==3.13.*"
+dependencies = [
+    # MCP Server
+    "mcp[cli]>=1.0.0",
+    # Docker Management
+    "python-on-whales>=0.73.0",
+    # Web Crawling
+    "crawl4ai>=0.8.0",
+    # HTTP Client
+    "httpx>=0.27.0",
+    # Config
+    "pydantic>=2.0.0",
+    "pydantic-settings>=2.0.0",
+    # Logging
+    "loguru>=0.7.0",
+]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.25.0",
+    "ruff>=0.8.0",
+    "ty>=0.0.1a10",
+    "pre-commit>=4.0.0",
+]
+
+[project.scripts]
+wet-mcp = "wet_mcp:main"
+
+[project.urls]
+Homepage = "https://github.com/n24q02m/wet-mcp"
+Repository = "https://github.com/n24q02m/wet-mcp.git"
+Issues = "https://github.com/n24q02m/wet-mcp/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/wet_mcp"]
+
+[tool.hatch.build.targets.wheel.sources]
+"src" = ""
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/wet_mcp/**/*.py",
+    "src/wet_mcp/**/*.yml",
+    "src/wet_mcp/**/*.md",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+
+[tool.ty]
+rules = { unresolved-import = "ignore", unresolved-attribute = "ignore", possibly-missing-attribute = "ignore" }
+
+[tool.ruff]
+line-length = 88
+target-version = "py313"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B", "C4"]
+ignore = ["E501"]
+
+[tool.ruff.format]
+quote-style = "double"

wet_mcp-1.0.0/src/wet_mcp/__init__.py

@@ -0,0 +1,6 @@
+"""WET MCP Server - Web ExTract for AI Agents."""
+
+from wet_mcp.server import main, mcp
+
+__version__ = "0.1.0"
+__all__ = ["mcp", "main", "__version__"]

wet_mcp-1.0.0/src/wet_mcp/__main__.py

@@ -0,0 +1,6 @@
+"""WET MCP Server entry point."""
+
+from wet_mcp.server import main
+
+if __name__ == "__main__":
+    main()

wet_mcp-1.0.0/src/wet_mcp/config.py

@@ -0,0 +1,32 @@
+"""Configuration settings for WET MCP Server."""
+
+from pydantic_settings import BaseSettings
+
+
+class Settings(BaseSettings):
+    """WET MCP Server configuration."""
+
+    # SearXNG
+    searxng_url: str = "http://localhost:8080"
+    searxng_timeout: int = 30
+
+    # Crawler
+    crawler_headless: bool = True
+    crawler_timeout: int = 60
+
+    # Docker Management
+    wet_auto_docker: bool = True
+    wet_container_name: str = "wet-searxng"
+    wet_searxng_image: str = "searxng/searxng:latest"
+    wet_searxng_port: int = 8080
+
+    # Media
+    download_dir: str = "~/.wet-mcp/downloads"
+
+    # Logging
+    log_level: str = "INFO"
+
+    model_config = {"env_prefix": "", "case_sensitive": False}
+
+
+settings = Settings()

wet_mcp-1.0.0/src/wet_mcp/docker_manager.py

@@ -0,0 +1,146 @@
+"""Docker container management for SearXNG."""
+
+import socket
+from importlib.resources import files
+from pathlib import Path
+
+from loguru import logger
+
+from wet_mcp.config import settings
+
+
+def _find_available_port(start_port: int, max_tries: int = 10) -> int:
+    """Find an available port starting from start_port."""
+    for offset in range(max_tries):
+        port = start_port + offset
+        try:
+            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+                s.bind(("localhost", port))
+                return port
+        except OSError:
+            continue
+    # Fallback to original port
+    return start_port
+
+
+def _get_settings_path() -> Path:
+    """Get path to SearXNG settings file.
+
+    Copies bundled settings.yml to user config directory for Docker mounting.
+    Uses ~/.wet-mcp/ which is typically shared with Docker.
+    """
+    config_dir = Path.home() / ".wet-mcp"
+    config_dir.mkdir(parents=True, exist_ok=True)
+
+    settings_file = config_dir / "searxng_settings.yml"
+
+    # Copy bundled settings if not exists
+    if not settings_file.exists():
+        bundled = files("wet_mcp").joinpath("searxng_settings.yml")
+        settings_file.write_text(bundled.read_text())
+        logger.debug(f"Copied SearXNG settings to: {settings_file}")
+
+    return settings_file
+
+
+def ensure_searxng() -> str:
+    """Start SearXNG container if not running. Returns URL.
+
+    This function handles:
+    - Automatic container creation if it doesn't exist
+    - Port conflict resolution (tries next available port)
+    - SearXNG configuration for JSON API format via settings.yml mount
+    - Graceful fallback to external SearXNG URL if Docker unavailable
+    """
+    if not settings.wet_auto_docker:
+        logger.info("Auto Docker disabled, using external SearXNG")
+        return settings.searxng_url
+
+    try:
+        from python_on_whales import DockerException, docker
+    except ImportError:
+        logger.warning("python-on-whales not installed, using external SearXNG")
+        return settings.searxng_url
+
+    container_name = settings.wet_container_name
+    image = settings.wet_searxng_image
+    preferred_port = settings.wet_searxng_port
+
+    try:
+        if docker.container.exists(container_name):
+            container = docker.container.inspect(container_name)
+            if container.state.running:
+                logger.debug(f"SearXNG container already running: {container_name}")
+                # Extract port from running container
+                ports = container.network_settings.ports
+                if ports and "8080/tcp" in ports and ports["8080/tcp"]:
+                    port = int(ports["8080/tcp"][0].get("HostPort", preferred_port))
+                else:
+                    port = preferred_port
+            else:
+                logger.info(f"Starting stopped container: {container_name}")
+                docker.container.start(container_name)
+                port = preferred_port
+        else:
+            # Find available port to avoid conflicts
+            port = _find_available_port(preferred_port)
+            if port != preferred_port:
+                logger.info(f"Port {preferred_port} in use, using {port}")
+
+            # Get settings file path
+            settings_path = _get_settings_path()
+
+            logger.info(f"Starting SearXNG container: {container_name}")
+            docker.run(
+                image,
+                name=container_name,
+                detach=True,
+                publish=[(port, 8080)],
+                volumes=[(str(settings_path), "/etc/searxng/settings.yml", "ro")],
+                envs={
+                    "SEARXNG_SECRET": "wet-internal",
+                },
+            )
+            logger.info(f"SearXNG container started on port {port}")
+
+        return f"http://localhost:{port}"
+
+    except DockerException as e:
+        logger.warning(f"Docker not available: {e}")
+        logger.warning("Falling back to external SearXNG URL")
+        return settings.searxng_url
+    except Exception as e:
+        logger.error(f"Failed to start SearXNG: {e}")
+        return settings.searxng_url
+
+
+def stop_searxng() -> None:
+    """Stop SearXNG container if running."""
+    if not settings.wet_auto_docker:
+        return
+
+    try:
+        from python_on_whales import docker
+
+        container_name = settings.wet_container_name
+        if docker.container.exists(container_name):
+            logger.info(f"Stopping container: {container_name}")
+            docker.container.stop(container_name)
+    except Exception as e:
+        logger.debug(f"Failed to stop container: {e}")
+
+
+def remove_searxng() -> None:
+    """Stop and remove SearXNG container."""
+    if not settings.wet_auto_docker:
+        return
+
+    try:
+        from python_on_whales import docker
+
+        container_name = settings.wet_container_name
+        if docker.container.exists(container_name):
+            logger.info(f"Removing container: {container_name}")
+            docker.container.remove(container_name, force=True)
+    except Exception as e:
+        logger.debug(f"Failed to remove container: {e}")

wet_mcp-1.0.0/src/wet_mcp/docs/__init__.py

@@ -0,0 +1 @@
+"""Docs package for WET MCP Server."""