fast-web-search-mcp 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.1.0] - 2026-07-13
+
+### Added
+
+- MCP server with stdio transport
+- `web_search` tool with DuckDuckGo + Bing fallback
+- `news_search` tool for recent news articles
+- `fetch_url` tool with SSRF protection and Readability extraction
+- `search_and_fetch` combined research workflow
+- `diagnostics` tool (disabled by default)
+- DuckDuckGo HTML provider adapter
+- Bing HTML provider adapter
+- Wikipedia REST API provider
+- SearXNG optional provider
+- In-memory LRU cache with TTL
+- Request coalescing for concurrent identical searches
+- Per-provider rate limiting with exponential backoff
+- Result deduplication and ranking
+- Domain include/exclude filters
+- URL canonicalization with tracking parameter removal
+- Content extraction pipeline (Readability → fallback → Markdown)
+- SSRF protection blocking private networks
+- Structured error codes
+- Environment-based configuration
+- Comprehensive test suite (177 tests)

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 fast-web-search-mcp contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,581 @@
+<p align="center">
+  <h1 align="center">fast-web-search-mcp</h1>
+  <p align="center">
+    A production-ready Model Context Protocol server for free web search, news search, and webpage content extraction.
+  </p>
+  <p align="center">
+    <a href="#features">Features</a> ·
+    <a href="#quick-start">Quick Start</a> ·
+    <a href="#integration">Integration</a> ·
+    <a href="#configuration">Configuration</a> ·
+    <a href="#security">Security</a> ·
+    <a href="#documentation">Docs</a>
+  </p>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/version-0.1.0-blue" alt="Version">
+  <img src="https://img.shields.io/badge/node-%3E%3D22-brightgreen" alt="Node.js">
+  <img src="https://img.shields.io/badge/license-MIT-green" alt="License">
+  <img src="https://img.shields.io/badge/tests-177%20passing-brightgreen" alt="Tests">
+  <img src="https://img.shields.io/badge/MCP-1.29.0-orange" alt="MCP SDK">
+  <img src="https://img.shields.io/badge/typescript-6.0-blue" alt="TypeScript">
+  <img src="https://img.shields.io/badge/build-passing-brightgreen" alt="Build">
+</p>
+
+---
+
+## Why fast-web-search-mcp?
+
+Most MCP search servers require paid API keys, Docker containers, or headless browsers. **fast-web-search-mcp** provides free, fast, local web search with zero external dependencies beyond Node.js.
+
+| Capability | fast-web-search-mcp | Paid APIs | Headless Browser |
+|------------|:-------------------:|:---------:|:----------------:|
+| Free to use | ✅ | ❌ | ✅ |
+| No API key | ✅ | ❌ | ✅ |
+| No Docker | ✅ | ✅ | ❌ |
+| Sub-second response | ✅ | ✅ | ❌ |
+| SSRF protection | ✅ | N/A | ❌ |
+| Result caching | ✅ | ❌ | ❌ |
+| Multiple providers | ✅ | ❌ | ❌ |
+| Works on Windows | ✅ | ✅ | ⚠️ |
+
+---
+
+## Features
+
+- **Web Search** — DuckDuckGo primary with Bing fallback and Wikipedia reference lookup
+- **News Search** — Date-sensitive news results with publication metadata
+- **Page Fetching** — Readability extraction with Markdown or plain text output
+- **Combined Research** — Search, rank, and fetch top results in one call
+- **Intelligent Caching** — In-memory LRU cache with configurable TTL
+- **Request Coalescing** — Deduplicates concurrent identical searches
+- **Rate Limiting** — Per-provider throttling with exponential backoff
+- **Result Deduplication** — URL canonicalization and title similarity matching
+- **SSRF Protection** — Blocks private IPs, localhost, cloud metadata endpoints
+- **Provider Fallback** — Automatic failover when a provider is unavailable
+- **Zero Runtime Dependencies** — No Python, no Docker, no browser engine
+
+---
+
+## Quick Start
+
+### Install
+
+```bash
+# Global install (recommended)
+npm install -g fast-web-search-mcp
+
+# Or run directly without installing
+npx -y fast-web-search-mcp
+```
+
+### Requirements
+
+| Dependency | Version |
+|------------|---------|
+| Node.js | ≥ 22.0 |
+| npm | ≥ 10.0 |
+
+### Verify Installation
+
+```bash
+# Confirm the binary is available
+fast-web-search-mcp --version
+
+# Or run directly
+npx fast-web-search-mcp --version
+```
+
+---
+
+## Integration
+
+Add `fast-web-search-mcp` to your AI coding assistant. Select your platform below:
+
+<details>
+<summary><strong>Claude Code</strong> (Anthropic CLI)</summary>
+
+**Project scope** — add to `.claude/settings.json` in your project root:
+
+```json
+{
+  "mcpServers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "env": {
+        "FWSMCP_LOG_LEVEL": "warn",
+        "FWSMCP_REDACT_QUERIES": "true"
+      }
+    }
+  }
+}
+```
+
+**User scope** — add to `~/.claude/settings.json` for global availability:
+
+```json
+{
+  "mcpServers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "env": {
+        "FWSMCP_DEFAULT_REGION": "us-en",
+        "FWSMCP_DEFAULT_LANGUAGE": "en",
+        "FWSMCP_REDACT_QUERIES": "true"
+      }
+    }
+  }
+}
+```
+
+**Or use the CLI:**
+
+```bash
+claude mcp add fast-web-search -- fast-web-search-mcp
+```
+
+> **Windows users:** Wrap the command with `cmd /c`:
+> ```bash
+> claude mcp add fast-web-search -- cmd /c fast-web-search-mcp
+> ```
+
+</details>
+
+<details>
+<summary><strong>Claude Desktop</strong> (Anthropic GUI)</summary>
+
+Add to your configuration file:
+
+- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
+- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "env": {
+        "FWSMCP_LOG_LEVEL": "warn",
+        "FWSMCP_DEFAULT_REGION": "us-en",
+        "FWSMCP_DEFAULT_LANGUAGE": "en"
+      }
+    }
+  }
+}
+```
+
+Restart Claude Desktop after saving.
+
+</details>
+
+<details>
+<summary><strong>OpenAI Codex CLI</strong></summary>
+
+Add to `~/.codex/config.toml` (global) or `.codex/config.toml` (project):
+
+```toml
+[mcp_servers.fast-web-search]
+command = "fast-web-search-mcp"
+env = { FWSMCP_LOG_LEVEL = "warn", FWSMCP_REDACT_QUERIES = "true" }
+```
+
+**Or use the CLI:**
+
+```bash
+codex mcp add fast-web-search -- fast-web-search-mcp
+```
+
+</details>
+
+<details>
+<summary><strong>OpenCode</strong></summary>
+
+Add to `~/.config/opencode/opencode.json`:
+
+```json
+{
+  "mcp": {
+    "fast-web-search": {
+      "type": "local",
+      "command": ["fast-web-search-mcp"],
+      "enabled": true,
+      "environment": {
+        "FWSMCP_DEFAULT_REGION": "us-en",
+        "FWSMCP_DEFAULT_LANGUAGE": "en",
+        "FWSMCP_REDACT_QUERIES": "true",
+        "FWSMCP_SAFE_SEARCH": "moderate"
+      }
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><strong>Antigravity</strong> (Google IDE / CLI)</summary>
+
+**Antigravity IDE:**
+
+1. Click **...** in the agent side panel → **MCP Servers** → **Manage MCP Servers** → **View raw config**
+2. Add the server to the opened `mcp_config.json` file
+
+**Antigravity CLI:**
+
+Type `/mcp` in the prompt panel to open the interactive MCP Manager.
+
+**Configuration file:**
+
+- **Global:** `~/.gemini/config/mcp_config.json`
+- **Project:** `.agents/mcp_config.json`
+
+```json
+{
+  "mcpServers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "args": [],
+      "env": {
+        "FWSMCP_LOG_LEVEL": "warn",
+        "FWSMCP_REDACT_QUERIES": "true"
+      }
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><strong>Cursor</strong></summary>
+
+Add to `~/.cursor/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "env": {
+        "FWSMCP_LOG_LEVEL": "warn"
+      }
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><strong>VS Code / GitHub Copilot</strong></summary>
+
+Add to `.vscode/mcp.json` in your workspace:
+
+```json
+{
+  "servers": {
+    "fast-web-search": {
+      "command": "fast-web-search-mcp",
+      "env": {
+        "FWSMCP_LOG_LEVEL": "warn"
+      }
+    }
+  }
+}
+```
+
+</details>
+
+---
+
+## MCP Tools
+
+| Tool | Description | Use Case |
+|------|-------------|----------|
+| `web_search` | Search the general web via DuckDuckGo + Bing fallback | General research, fact-finding |
+| `news_search` | Search for recent news articles with publication dates | Current events, monitoring |
+| `fetch_url` | Fetch a URL and extract clean Markdown or plain text | Article reading, documentation |
+| `search_and_fetch` | Combined search + fetch workflow with character budget | Deep research, analysis |
+| `diagnostics` | Return non-sensitive server diagnostics | Troubleshooting (disabled by default) |
+
+### Tool Parameters
+
+<details>
+<summary><strong>web_search</strong></summary>
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | *required* | Search query (1-500 characters) |
+| `max_results` | integer | 10 | Maximum results to return (1-25) |
+| `strategy` | string | `"fallback"` | Provider strategy: `fallback` or `merge` |
+| `region` | string | `"us-en"` | Region code for localized results |
+| `language` | string | `"en"` | Language code |
+| `time_range` | string | `"any"` | Time filter: `day`, `week`, `month`, `year`, `any` |
+| `safe_search` | string | `"moderate"` | Content filter: `off`, `moderate`, `strict` |
+| `include_domains` | array | — | Restrict results to these domains |
+| `exclude_domains` | array | — | Exclude results from these domains |
+| `deduplicate` | boolean | `true` | Remove duplicate results by URL |
+| `timeout_ms` | integer | — | Operation timeout (1000-60000 ms) |
+
+</details>
+
+<details>
+<summary><strong>news_search</strong></summary>
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | *required* | News search query (1-500 characters) |
+| `max_results` | integer | 10 | Maximum results (1-25) |
+| `time_range` | string | `"week"` | Time filter: `day`, `week`, `month` |
+| `region` | string | `"us-en"` | Region code |
+| `language` | string | `"en"` | Language code |
+| `include_domains` | array | — | Restrict to these domains |
+| `exclude_domains` | array | — | Exclude these domains |
+
+</details>
+
+<details>
+<summary><strong>fetch_url</strong></summary>
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `url` | string | *required* | HTTP/HTTPS URL to fetch |
+| `output` | string | `"markdown"` | Output format: `markdown` or `text` |
+| `max_characters` | integer | 20000 | Maximum characters (100-100000) |
+| `include_metadata` | boolean | `true` | Include page metadata |
+| `include_links` | boolean | `false` | Include extracted links |
+| `timeout_ms` | integer | — | Fetch timeout (1000-60000 ms) |
+
+</details>
+
+<details>
+<summary><strong>search_and_fetch</strong></summary>
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | *required* | Search query (1-500 characters) |
+| `search_results` | integer | 8 | Number of results to consider (1-20) |
+| `fetch_results` | integer | 3 | Number of pages to fetch (1-5) |
+| `max_characters_per_page` | integer | 20000 | Character limit per page |
+| `total_character_budget` | integer | 100000 | Total character budget |
+| `timeout_ms` | integer | — | Total timeout (5000-120000 ms) |
+
+</details>
+
+---
+
+## Configuration
+
+All configuration is environment-based. No config files required.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `FWSMCP_LOG_LEVEL` | `warn` | Log verbosity: `debug`, `info`, `warn`, `error` |
+| `FWSMCP_LOG_FORMAT` | `text` | Log format: `text` or `json` |
+| `FWSMCP_REDACT_QUERIES` | `true` | Redact search queries in logs |
+| `FWSMCP_DEFAULT_REGION` | `us-en` | Default region for searches |
+| `FWSMCP_DEFAULT_LANGUAGE` | `en` | Default language for searches |
+| `FWSMCP_SAFE_SEARCH` | `moderate` | Safe search level: `off`, `moderate`, `strict` |
+| `FWSMCP_PROVIDERS` | `duckduckgo,bing,wikipedia` | Comma-separated provider list |
+| `FWSMCP_PROVIDER_STRATEGY` | `fallback` | Strategy: `fallback` (sequential) or `merge` (parallel) |
+| `FWSMCP_SEARCH_TIMEOUT_MS` | `20000` | Search operation timeout (ms) |
+| `FWSMCP_PROVIDER_TIMEOUT_MS` | `10000` | Per-provider timeout (ms) |
+| `FWSMCP_FETCH_TIMEOUT_MS` | `15000` | Page fetch timeout (ms) |
+| `FWSMCP_RESEARCH_TIMEOUT_MS` | `45000` | search_and_fetch total timeout (ms) |
+| `FWSMCP_MAX_SEARCH_RESULTS` | `25` | Maximum results per search |
+| `FWSMCP_MAX_FETCH_CHARACTERS` | `100000` | Maximum characters per fetch |
+| `FWSMCP_CACHE_ENABLED` | `true` | Enable in-memory result cache |
+| `FWSMCP_CACHE_MAX_ENTRIES` | `250` | Maximum cache entries |
+| `FWSMCP_SEARCH_CACHE_TTL_MS` | `900000` | Search cache TTL (15 min) |
+| `FWSMCP_NEWS_CACHE_TTL_MS` | `300000` | News cache TTL (5 min) |
+| `FWSMCP_PAGE_CACHE_TTL_MS` | `1800000` | Page cache TTL (30 min) |
+| `FWSMCP_SEARXNG_URL` | — | Optional SearXNG instance URL |
+| `FWSMCP_DIAGNOSTICS_ENABLED` | `false` | Enable diagnostics tool |
+| `FWSMCP_GLOBAL_EXCLUDE_DOMAINS` | — | Comma-separated domains to always exclude |
+| `FWSMCP_CONFIG_FILE` | — | Path to optional JSON config file |
+
+### Optional JSON Configuration
+
+For complex setups, point `FWSMCP_CONFIG_FILE` to a JSON file:
+
+```json
+{
+  "defaults": {
+    "region": "us-en",
+    "language": "en",
+    "safeSearch": "moderate"
+  },
+  "providers": {
+    "order": ["duckduckgo", "bing", "wikipedia"],
+    "strategy": "fallback"
+  },
+  "cache": {
+    "enabled": true,
+    "maxEntries": 250,
+    "searchTtlMs": 900000
+  },
+  "limits": {
+    "maxSearchResults": 25,
+    "maxFetchCharacters": 100000
+  },
+  "privacy": {
+    "redactQueriesInLogs": true
+  }
+}
+```
+
+> **Priority:** Environment variables > JSON config file > defaults.
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    MCP Client                            │
+│            (Claude Code / Codex / OpenCode)              │
+└─────────────────────┬───────────────────────────────────┘
+                      │ stdio (JSON-RPC)
+                      ▼
+┌─────────────────────────────────────────────────────────┐
+│               fast-web-search-mcp                        │
+│                                                         │
+│  ┌─────────┐  ┌──────────┐  ┌───────────┐  ┌────────┐ │
+│  │  Tools   │  │  Cache   │  │ Coalescer │  │ Limiter│ │
+│  │ 5 tools  │  │ LRU+TTL  │  │ Dedup     │  │ Per-   │ │
+│  │          │  │          │  │ inflight  │  │ provider│ │
+│  └────┬─────┘  └──────────┘  └───────────┘  └────────┘ │
+│       │                                                  │
+│       ▼                                                  │
+│  ┌─────────────────────────────────────────────────┐    │
+│  │              Orchestrator Pipeline                │    │
+│  │  Validate → Cache → Coalesce → Select → Execute  │    │
+│  │  → Normalize → Filter → Dedup → Rank → Return    │    │
+│  └─────────────────────┬───────────────────────────┘    │
+│                        │                                 │
+│       ┌────────────────┼────────────────┐               │
+│       ▼                ▼                ▼               │
+│  ┌─────────┐    ┌──────────┐    ┌────────────┐         │
+│  │ DuckDuck │    │   Bing   │    │ Wikipedia  │         │
+│  │   Go     │    │          │    │ REST API   │         │
+│  │ (HTML)   │    │ (HTML)   │    │            │         │
+│  └─────────┘    └──────────┘    └────────────┘         │
+│       │                │                                 │
+│       ▼                ▼                                 │
+│  ┌─────────────────────────────────────────────────┐    │
+│  │            HTTP Client (Node fetch)               │    │
+│  │   Timeout · Body Limit · Redirect · SSRF Block   │    │
+│  └─────────────────────────────────────────────────┘    │
+└─────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Security
+
+### Protections
+
+| Threat | Mitigation |
+|--------|------------|
+| SSRF / private network access | Blocks localhost, RFC1918, link-local, cloud metadata IPs |
+| Credential leakage | Rejects URLs with embedded username/password |
+| Query exposure | Logs redact queries by default (`FWSMCP_REDACT_QUERIES=true`) |
+| Prompt injection via fetched content | Includes untrusted content warning in all fetch results |
+| Uncontrolled resource consumption | Body size limits, timeout enforcement, character budgets |
+| Process isolation | stdio transport only — no network listener, no HTTP endpoint |
+
+### Disabled by Default
+
+- **Diagnostics tool** — must be explicitly enabled via `FWSMCP_DIAGNOSTICS_ENABLED=true`
+- **SearXNG integration** — requires explicit `FWSMCP_SEARXNG_URL` configuration
+
+See [docs/SECURITY.md](docs/SECURITY.md) for the full threat model.
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/your-org/fast-web-search-mcp.git
+cd fast-web-search-mcp
+npm ci
+npm run build
+npm test
+```
+
+### Available Scripts
+
+| Script | Description |
+|--------|-------------|
+| `npm run build` | Production build with tsup |
+| `npm run dev` | Development mode with tsx |
+| `npm test` | Run full test suite (177 tests) |
+| `npm run test:unit` | Unit tests only |
+| `npm run test:security` | Security tests only |
+| `npm run typecheck` | TypeScript type checking |
+| `npm run lint` | ESLint |
+| `npm run format` | Prettier formatting |
+
+### Project Structure
+
+```
+fast-web-search-mcp/
+├── src/
+│   ├── cache/              # LRU cache, cache key generation
+│   ├── config/             # Defaults, env parsing, config loader
+│   ├── fetch/              # URL validation, content extraction, Markdown
+│   ├── http/               # HTTP client, errors, headers
+│   ├── observability/      # Structured stderr logger
+│   ├── providers/          # DuckDuckGo, Bing, Wikipedia, SearXNG
+│   ├── schemas/            # Zod validation schemas
+│   ├── search/             # Orchestrator, coalescer, rate limiter, dedup
+│   ├── server/             # MCP server factory, tool registration
+│   ├── tools/              # 5 MCP tool implementations
+│   ├── utils/              # Domain, text, date, async utilities
+│   └── index.ts            # Entry point
+├── test/
+│   ├── fixtures/           # HTML fixtures for parser tests
+│   └── unit/               # Unit tests (177 tests)
+├── docs/                   # Documentation and ADRs
+└── examples/               # Config examples
+```
+
+---
+
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Configuration Reference](docs/CONFIGURATION.md) | All environment variables and options |
+| [Provider Guide](docs/PROVIDERS.md) | Provider details, strategy, and SearXNG setup |
+| [Security Model](docs/SECURITY.md) | Threat model and protections |
+| [Troubleshooting](docs/TROUBLESHOOTING.md) | Common issues and debugging |
+| [OpenCode Integration](docs/OPENCODE.md) | OpenCode-specific setup guide |
+| [Changelog](CHANGELOG.md) | Version history |
+
+---
+
+## Roadmap
+
+- [ ] SQLite persistent cache
+- [ ] Brave Search provider
+- [ ] Image search support
+- [ ] Streamable HTTP transport
+- [ ] Result scoring customization
+
+---
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing`)
+3. Commit changes (`git commit -m 'feat: add amazing feature'`)
+4. Push to branch (`git push origin feature/amazing`)
+5. Open a Pull Request
+
+---
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.