searxNcrawl 0.31.0__tar.gz

searxncrawl-0.31.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DDM – Das Digitale Momentum GmbH & Co KG
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

searxncrawl-0.31.0/PKG-INFO

@@ -0,0 +1,390 @@
+Metadata-Version: 2.4
+Name: searxNcrawl
+Version: 0.31.0
+Summary: searxNcrawl web crawler with markdown extraction - no DB, no enrichment, just crawling.
+Author: DDM – Das Digitale Momentum GmbH & Co KG
+License-Expression: MIT
+Project-URL: Homepage, https://www.das-digitale-momentum.de/en/open-source/#searxncrawl
+Project-URL: Repository, https://github.com/DasDigitaleMomentum/searxNcrawl
+Project-URL: Issues, https://github.com/DasDigitaleMomentum/searxNcrawl/issues
+Project-URL: Changelog, https://github.com/DasDigitaleMomentum/searxNcrawl/blob/main/CHANGELOG.md
+Keywords: mcp,mcp-server,searxng,crawl4ai,web-crawler,web-search,markdown,llm-tools
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: crawl4ai>=0.7.4
+Requires-Dist: tldextract>=5.1.2
+Requires-Dist: playwright>=1.40.0
+Requires-Dist: fastmcp>=3.4.3
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: python-dotenv>=1.0.1
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Dynamic: license-file
+
+# searxNcrawl
+
+MCP server and CLI toolkit for web search and crawling, built on [Crawl4AI](https://github.com/unclecode/crawl4ai) and [SearXNG](https://github.com/searxng/searxng).
+
+Published at [github.com/DasDigitaleMomentum/searxNcrawl](https://github.com/DasDigitaleMomentum/searxNcrawl) — maintained by **DDM – Das Digitale Momentum GmbH & Co KG**. Successor to `searxng-mcp`.
+
+<!-- mcp-name: io.github.DasDigitaleMomentum/searxncrawl -->
+
+## Quick Start
+
+Pick your setup:
+
+### Docker Compose
+
+MCP server with Playwright/Chromium, ready in one command. SearXNG required separately for search.
+
+```bash
+cp .env.example .env          # set SEARXNG_URL to your SearXNG instance
+docker compose up --build
+```
+
+➜ MCP server at `http://localhost:9555/mcp`
+
+### uvx (no clone)
+
+Run the MCP server from [PyPI](https://pypi.org/project/searxncrawl/), no clone or virtualenv. Chromium is downloaded automatically on the first crawl.
+
+```bash
+SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl
+```
+
+The latest development version runs straight from GitHub with `uvx --from git+https://github.com/DasDigitaleMomentum/searxNcrawl searxncrawl`.
+
+### pip (standalone)
+
+CLI tools, Python API, and MCP server. SearXNG required for search.
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e .
+playwright install chromium
+```
+
+### uv (standalone)
+
+Same capabilities as pip.
+
+```bash
+uv sync
+uv run playwright install chromium
+```
+
+### What you get
+
+| Feature                 | Docker Compose | pip / uv  |
+| ----------------------- | -------------- | --------- |
+| MCP Server (STDIO)      | —              | ✅        |
+| MCP Server (HTTP)       | ✅             | ✅        |
+| Web Crawl               | ✅             | ✅        |
+| Web Search              | ✅¹            | ✅¹       |
+| CLI Tools               | via `exec`²    | ✅        |
+| Python API              | —              | ✅        |
+| CORS (HTTP)             | ✅             | ✅        |
+
+¹ Requires a SearXNG instance. ² `docker compose exec searxncrawl crawl ...`
+
+## Features
+
+### Crawling
+- Single page, multi-page, and **site crawling** (DFS with depth/page limits)
+- Production-tested extraction config optimized for documentation sites
+- Configurable timeouts with graceful error handling
+
+### Content Quality
+- **Markdown deduplication** — `exact` (default) removes repeated blocks, `off` disables it
+- **Link removal** — strip all links for cleaner LLM context (`--remove-links`)
+- **Dedup guardrails** — non-destructive metadata signals when removal is unusually aggressive
+
+### Web Search
+- SearXNG metasearch integration (privacy-respecting)
+- Configurable language, time range, categories, engines, safe search
+
+### MCP Server
+- **STDIO transport** — for MCP harnesses (Zed, opencode, VS Code, Claude Code, etc.)
+- **HTTP transport** — for remote access and browser clients
+- **CORS support** — configurable origins for browser-based MCP clients
+- Noise-free startup with UTF-8 encoding (cross-platform, incl. Windows)
+
+### CLI Tools
+- `crawl` — crawl pages from the command line
+- `search` — search the web via SearXNG
+- `crawl-capture` — session capture for authenticated crawling
+
+## Installation
+
+### Docker Compose
+
+The Compose stack includes searxNcrawl + Playwright/Chromium. SearXNG must be provided separately.
+
+```bash
+cp .env.example .env
+# Edit .env: set SEARXNG_URL to your SearXNG instance
+docker compose up --build
+```
+
+| Variable    | Default                   | Description                                                   |
+| ----------- | ------------------------- | ------------------------------------------------------------- |
+| `MCP_PORT`                  | `9555`                    | MCP server HTTP port                                          |
+| `LOG_LEVEL`                 | `INFO`                    | MCP server log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
+| `PLAYWRIGHT_AUTO_INSTALL`   | `true`                    | Download Playwright's Chromium automatically before the first browser launch if it is missing. Set to `false` where browsers are provisioned separately |
+| `FASTMCP_HTTP_ALLOWED_HOSTS` | (FastMCP secure defaults) | JSON list of trusted HTTP Host headers, for example `["mcp.example.com"]` |
+
+The MCP server is available at `http://localhost:9555/mcp`.
+
+### pip
+
+```bash
+cd searxNcrawl
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+playwright install chromium
+```
+
+### uv
+
+```bash
+cd searxNcrawl
+uv sync
+uv run playwright install chromium
+```
+
+### SearXNG (search feature)
+
+The `search` tool and CLI command require a SearXNG instance with **JSON output enabled** (`search.formats` in `settings.yml`). For all setups you need your own instance — self-hosting is recommended over public instances (rate limits).
+
+**Environment variables:**
+
+| Variable               | Example / Recommended                 | Description                                                                                                                         |
+| ---------------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| `SEARXNG_URL`          | `http://localhost:8888`               | SearXNG instance URL                                                                                                                |
+| `SEARXNG_USERNAME`     | (none)                                | Optional basic auth user                                                                                                            |
+| `SEARXNG_PASSWORD`     | (none)                                | Optional basic auth pass                                                                                                            |
+| `SEARCH_RESULT_FIELDS` | `title,url,content,publishedDate`     | Comma-separated result fields. Unset = all SearXNG fields. Available: title, url, content, publishedDate, engine, score, category, img_src, thumbnail |
+
+Example `.env`:
+```bash
+SEARXNG_URL=http://localhost:8888
+SEARCH_RESULT_FIELDS=title,url,content,publishedDate
+LOG_LEVEL=INFO
+```
+
+**Config file search order** (CLI tools only):
+
+1. `./.env` — current directory
+2. `~/.config/searxncrawl/.env` — user config
+
+If no `.env` exists, `.env.example` is auto-copied to the user config path.
+
+## Usage
+
+### MCP Server
+
+#### Start the server
+
+```bash
+# STDIO transport (for MCP harnesses)
+python -m crawler.mcp_server
+
+# HTTP transport
+python -m crawler.mcp_server --transport http --port 8000
+
+# HTTP exposed through a specific public hostname
+python -m crawler.mcp_server --transport http --host 0.0.0.0 --allowed-hosts "mcp.example.com"
+
+# HTTP with CORS
+python -m crawler.mcp_server --transport http --allowed-hosts "mcp.example.com" --cors-origins "https://app.example.com"
+
+# Docker (HTTP only)
+docker compose up --build
+```
+
+#### MCP client configuration
+
+**With uvx (no clone, no venv):**
+
+```json
+{
+  "mcpServers": {
+    "crawler": {
+      "command": "uvx",
+      "args": ["searxncrawl"],
+      "env": { "SEARXNG_URL": "http://your-searxng:8888" }
+    }
+  }
+}
+```
+
+The first crawl downloads Playwright's Chromium once (about 550 MB on disk); later starts reuse it. On Linux hosts without the browser's system libraries, run `uvx --from searxncrawl playwright install --with-deps chromium` once.
+
+**Python with venv:**
+
+```json
+{
+  "mcpServers": {
+    "crawler": {
+      "command": "python",
+      "args": ["-m", "crawler.mcp_server"],
+      "cwd": "/path/to/searxNcrawl",
+      "env": { "SEARXNG_URL": "http://your-searxng:8888" }
+    }
+  }
+}
+```
+
+**With uv (no manual venv):**
+
+```json
+{
+  "mcpServers": {
+    "crawler": {
+      "command": "uv",
+      "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
+      "env": { "SEARXNG_URL": "http://your-searxng:8888" }
+    }
+  }
+}
+```
+
+**Docker (HTTP endpoint):**
+
+```json
+{
+  "mcpServers": {
+    "crawler": {
+      "url": "http://localhost:9555/mcp"
+    }
+  }
+}
+```
+
+#### CORS
+
+FastMCP validates the HTTP `Host` header independently of the address on which
+the server listens. For remote access, allow the exact externally visible Host
+header with a comma-separated CLI value:
+
+```bash
+crawl-mcp --transport http --host 0.0.0.0 --allowed-hosts "mcp.example.com,mcp.internal.example"
+```
+
+Alternatively, use FastMCP's environment setting. It uses JSON-list syntax:
+
+```bash
+FASTMCP_HTTP_ALLOWED_HOSTS='["mcp.example.com"]' crawl-mcp --transport http --host 0.0.0.0
+```
+
+Browser Origin validation and CORS response headers are separate from Host
+validation. `--cors-origins` configures both FastMCP's Origin guard and the CORS
+middleware using the same normalized, comma-separated values:
+
+```bash
+crawl-mcp --transport http --cors-origins "http://localhost:3000,https://myapp.com"
+crawl-mcp --transport http --cors-origins "*"   # all origins — local dev only
+```
+
+Omitting either allowlist preserves FastMCP's secure defaults (and permits the
+upstream environment setting to apply). A value of `*` for Hosts or Origins is
+an explicit opt-in to broad access and should only be used when that security
+trade-off is intentional. Without `--cors-origins`, no CORS headers are sent.
+
+### CLI Tools
+
+After `pip install -e .` (or `uv sync`), the following commands are available:
+
+```bash
+# Crawl a page
+crawl https://docs.example.com
+
+# Site crawl with depth limit
+crawl https://docs.example.com --site --max-depth 2 --max-pages 10 -o docs/
+
+# Clean output (no links)
+crawl https://example.com --remove-links
+
+# Search
+search "python tutorials"
+search "Rezepte" --language de --max-results 5
+
+# Session capture for authenticated crawling
+crawl-capture --start-url https://example.com/login \
+    --completion-url 'https://example.com/dashboard.*' \
+    --output ./state.json
+```
+
+See [Session Capture](docs/usage/session-capture.md) for the full `crawl-capture` guide.
+
+### Python API
+
+```python
+from crawler import crawl_page, crawl_page_async, crawl_site, crawl_site_async
+
+# Single page
+doc = await crawl_page_async("https://docs.example.com/intro", dedup_mode="exact")
+print(doc.markdown)
+
+# Site crawl
+result = crawl_site("https://docs.example.com", max_depth=2, max_pages=10)
+for doc in result.documents:
+    print(f"{doc.status}: {doc.final_url}")
+
+# Authenticated crawl
+doc = await crawl_page_async(
+    "https://example.com/private",
+    auth={"storage_state": "/path/to/state.json"},
+)
+```
+
+## Reference
+
+- **[MCP Tools](docs/usage/mcp-tools.md)** — full parameter reference for `crawl`, `crawl_site`, `search`
+- **[Output Formats](docs/usage/output-formats.md)** — Markdown and JSON output structure, including `CrawledDocument`
+- **[Session Capture](docs/usage/session-capture.md)** — manual login flow and CDP session export
+
+## Configuration
+
+Default config is optimized for documentation sites. Customize via overrides:
+
+```python
+from crawler import build_markdown_run_config, RunConfigOverrides
+
+config = build_markdown_run_config(
+    RunConfigOverrides(
+        delay_before_return_html=1.0,
+        mean_delay=1.0,
+        scan_full_page=True,
+    )
+)
+doc = await crawl_page_async("https://example.com", config=config)
+```
+
+## Dependencies
+
+- `crawl4ai>=0.7.4` — crawler engine
+- `playwright>=1.40.0` — browser automation
+- `fastmcp>=3.4.3` — MCP server framework
+- `httpx>=0.27.0` — HTTP client for SearXNG
+- `tldextract>=5.1.2` — domain parsing for site crawls
+
+## License
+
+MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG
+
+---
+
+Maintained by [Das Digitale Momentum](https://www.das-digitale-momentum.de/en/open-source/#searxncrawl) · Much, Germany · [All our open source projects](https://github.com/DasDigitaleMomentum)