mcp-web-engine 1.0.5 → 1.2.0

package/README.md

@@ -1,141 +1,99 @@
-# 🌐 MCP Web Engine & Search Gateway
+# MCP Web Engine
 
-> **Privacy-First, Self-Hostable & SSRF-Hardened MCP Server & Web Engine for AI Agents**
-> **MCP Specification 2026-07-28 Pure Stateless Core & Stdio / HTTP Dual Mode**
+Self-hostable MCP search, public-page fetch and Markdown extraction. Source version: **1.2.0**; examples below run the checkout, not an assumed published release.
 
-[![PyPI Version](https://img.shields.io/badge/pypi-v1.0.5-blue.svg)](https://pypi.org/project/mcp-web-engine/)
-[![npm Version](https://img.shields.io/badge/npm-v1.0.5-red.svg)](https://www.npmjs.com/package/mcp-web-engine)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+Requires Python >=3.11. The optional npm launcher requires Node >=18 and [uv](https://docs.astral.sh/uv/getting-started/installation/) on PATH. Installation resolves dependencies from `pyproject.toml` and may access package indexes. uv isolates Python dependencies; it is not a security sandbox.
 
----
+## Local stdio
 
-## ⚡ Instant Quickstart with `uvx` (Recommended for AI Agents)
-
-AI agents (Claude Desktop, Cursor, Windsurf, TurboLLM, Hermes Agent) can run `mcp-web-engine` in an isolated ephemeral environment with **zero configuration**:
-
-```bash
-# Run stdio MCP server in an isolated environment (10ms startup, 0% system contamination)
-uvx mcp-web-engine
-```
-
-Or run via Git repository:
 ```bash
-uvx --from git+https://github.com/Arbolencio/mcp-web-engine mcp-web-engine
+git clone https://github.com/Arbolencio/mcp-web-engine.git
+cd mcp-web-engine
+uv run mcp-web-engine
 ```
 
----
-
-## 📌 What it does & Why it exists
-
-`mcp-web-engine` is a high-performance Model Context Protocol (MCP) server implementing the **MCP Specification 2026-07-28 Pure Stateless Core**. It provides AI Agents with secure, private, and SSRF-hardened access to multi-engine web search, raw content fetching, and clean HTML-to-Markdown extraction.
+Default transport is newline-delimited JSON-RPC over stdin/stdout. Diagnostics and installer output belong on stderr. Clients should initialize, send `notifications/initialized`, then use `ping`, `tools/list`, and `tools/call`. No search service is required for initialization or listing tools.
 
-### Key Capabilities & Architecture
-- **Dual Execution Mode:**
-  - **Stdio Mode (Default):** Runs JSON-RPC over `stdin`/`stdout` for direct LLM client integration (`uvx mcp-web-engine`).
-  - **HTTP/SSE Server Mode:** Runs FastAPI/Uvicorn server (`mcp-web-engine --serve`).
-- **Pure Stateless Core (Spec 2026-07-28):** Zero sessions, zero state, zero initialize handshake required. Every request is completely independent.
-- **Protocol Methods:** Implements `server/discover`, `tools/list`, and `tools/call` with standard JSON-RPC 2.0.
-- **Zero Third-Party Tracking:** Powered by an internal SearXNG meta-search engine aggregating 70+ sources with DuckDuckGo fallback.
-- **Automatic URL Normalization:** Handled transparently by `normalize_searxng_url()` — any provided `SEARXNG_URL` variation (`http://127.0.0.1:8082`, `http://127.0.0.1:8082/`, `http://127.0.0.1:8082/search`) automatically appends `/search` if missing.
-- **SSRF Hardened:** Pre-request DNS resolution checks prevent agents from accessing internal networks (`localhost`, `127.0.0.1`, `192.168.x.x`, `169.254.169.254`). Step-by-step HTTP 301/302 redirect re-validation.
-- **Zero `--break-system-packages`:** Executed 100% in sandboxed virtual environments created on-the-fly by `uv` or `venv`.
+Client configuration (replace the absolute checkout path):
 
----
-
-## 🔌 Integration Guides
-
-### 1. TurboLLM / Local MCP Catalog
-```typescript
-{
-  id: 'mcp-web-engine',
-  name: 'MCP Web Engine',
-  cat: 'Search',
-  desc: 'Privacy-first, self-hostable & SSRF-hardened web search engine and scraper for AI agents.',
-  cmd: 'uvx mcp-web-engine',
-  uvx: true,
-  argNote: 'Requires a running SearXNG instance.',
-  envs: [
-    { key: 'SEARXNG_URL', desc: 'SearXNG meta-search endpoint (default: http://127.0.0.1:8082/search)', required: true },
-  ],
-}
-```
-
-### 2. Claude Desktop (`claude_desktop_config.json`)
 ```json
 {
   "mcpServers": {
     "mcp-web-engine": {
-      "command": "uvx",
-      "args": ["mcp-web-engine"],
-      "env": {
-        "SEARXNG_URL": "http://127.0.0.1:8082/search"
-      }
+      "command": "uv",
+      "args": ["tool", "run", "--from", "/absolute/path/mcp-web-engine", "mcp-web-engine"],
+      "env": {"SEARXNG_URL": "http://127.0.0.1:8082/search", "ALLOW_DDG_FALLBACK": "false"}
     }
   }
 }
 ```
 
-### 3. Cursor / Windsurf / Hermes Agent
-```json
-{
-  "mcpServers": {
-    "mcp-web-engine": {
-      "command": "uvx",
-      "args": ["mcp-web-engine"]
-    }
-  }
-}
+The npm tarball includes the canonical Python src package and build metadata. `npx --package /absolute/path/mcp-web-engine-1.2.0.tgz mcp-web-engine` delegates to uv; Python/uv remain prerequisites, and first launch may need network access for dependencies.
+
+## HTTP
+
+```bash
+export API_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
+export SEARXNG_URL=http://127.0.0.1:8082/search
+uv run mcp-web-engine --serve
 ```
 
----
+HTTP defaults to `127.0.0.1:5050`; `API_KEY` must contain at least 32 non-whitespace characters. Send `Authorization: Bearer <your-key>` to protected endpoints. Keep the key out of source control and URLs. `/health` is a liveness endpoint, not a SearXNG connectivity check.
 
-## 🛠️ Provided MCP Tools
+`POST /v1/mcp/legacy` provides the initialize-based compatibility path. `/v1/mcp` is a project-specific stateless extension requiring `Mcp-Method` (and `Mcp-Name` for tool calls). Its `2026-07-28` label is an internal extension identifier, **not a claim of official MCP conformance**. SSE support is limited; this is not a complete implementation of every MCP transport or optional feature.
 
-| Tool Name | Parameters | Description |
-|---|---|---|
-| `web_search` | `query` (str), `limit` (int, default: 10) | Multi-engine meta-search via SearXNG returning structured JSON results |
-| `fetch_url` | `url` (str), `max_bytes` (int, optional) | Fetches raw web content after strict SSRF security validation |
-| `extract_markdown` | `url` (str), `max_bytes` (int, optional) | Scrapes a webpage and converts HTML into clean, LLM-optimized Markdown |
+## Tools
 
----
+| Tool | Arguments | Purpose |
+|---|---|---|
+| `web_search` | `query`, optional `limit` | Search through configured SearXNG |
+| `fetch_url` | `url`, optional `max_bytes` | Fetch a public HTTP(S) page |
+| `extract_markdown` | `url`, optional `max_bytes` | Convert fetched HTML to Markdown |
 
-## ⚙️ Configuration & Environment Variables
+Use `tools/list` for authoritative schemas and bounds.
 
-| Variable | Required | Default | Description |
-|---|---|---|---|
-| `SEARXNG_URL` | **Yes** | `http://127.0.0.1:8082/search` | SearXNG meta-search endpoint (Auto-normalizes trailing slashes and missing `/search`) |
-| `PORT` | No | `5050` | HTTP/SSE server port (Used only when running in `--serve` mode) |
-| `HOST` | No | `0.0.0.0` | HTTP/SSE server binding host |
-| `API_KEY` | No | `""` | Optional authentication key for HTTP endpoints |
+## Configuration
 
----
+Process environment is authoritative. CWD `.env` files are **not** loaded automatically. To explicitly load one, set `MCP_WEB_ENGINE_ENV_FILE` to an existing **absolute** file path; process variables override file values.
 
-## 🛠️ Development & Building
+| Variable | Default | Meaning |
+|---|---|---|
+| `SEARXNG_URL` | `http://127.0.0.1:8082/search` | Trusted operator-controlled search backend; search needs it running |
+| `ALLOW_DDG_FALLBACK` | `false` | Opt in to sending queries to DuckDuckGo if SearXNG fails |
+| `API_KEY` | empty | Operator-generated HTTP bearer key; mandatory for CLI HTTP mode |
+| `HOST` / `PORT` | `127.0.0.1` / `5050` | HTTP bind address and port |
+| `RATE_LIMIT_PER_MINUTE` | `120` | Per-key HTTP admission limit, persisted in local SQLite |
+| `DEFAULT_TIMEOUT_SEC` | `10` | Outbound timeout, up to 120 seconds |
+| `MAX_PAYLOAD_BYTES` | `2097152` | Payload ceiling, up to 16 MiB |
+| `MCP_WEB_ENGINE_STATE_DB` | `$XDG_STATE_HOME/mcp-web-engine/usage.sqlite3` (or `~/.local/state/...`) | Local SQLite admission quotas and per-key tool telemetry; share one local file across workers |
+| `MCP_WEB_ENGINE_BETA_KEYS_FILE` | Same state directory, `beta_keys.json` | Operator-managed credentials, separate from telemetry; use an absolute path |
+| `LOG_LEVEL` / `ENV` | `INFO` / `production` | Logging and environment settings |
+
+## Docker
 
 ```bash
-# Clone Repository
-git clone https://github.com/Arbolencio/mcp-web-engine.git
-cd mcp-web-engine
-
-# Build Wheel for PyPI
-python3 -m build --wheel
+# With API_KEY already exported as above:
+docker compose up --build -d
+```
 
-# Run locally in stdio mode via uv
-uv run python -m mcp_web_engine.main
+The image installs the Python package, runs as a non-root user and explicitly starts `mcp-web-engine --serve` on port 5050. Compose publishes **only host loopback**. Container `HOST=0.0.0.0` is necessary for that port mapping; do not confuse it with the host binding. Set `SEARXNG_URL` to a reachable backend: compose defaults to `http://host.docker.internal:8082/search`, using the host gateway. An instance bound only to host loopback may not be reachable from Docker. No SearXNG instance or credentials are bundled. See [operations and limitations](docs/operations.md).
 
-# Run locally in HTTP server mode via uv
-uv run python -m mcp_web_engine.main --serve
-```
+## Security controls and limitations
 
----
+Public fetches validate schemes, resolved addresses and redirects, rejecting loopback/private/reserved destinations. Bounded payloads, timeouts and tool argument validation reduce resource exposure. These are defense-in-depth controls, not a guarantee against all SSRF, DNS, routing, proxy or upstream failures; enforce outbound network policy independently.
 
-## 🛡️ Security & Environment Isolation
+SearXNG is an intentional trusted-backend exception to public-fetch restrictions and can use a private address. Its operator and upstream engines receive queries; self-hosting does not keep searches entirely on-host. DuckDuckGo fallback is off by default, but ordinary SearXNG searches still involve upstream services. Fetched pages are untrusted input and may contain prompt injection. Dependency isolation does not isolate filesystem/network access. HTTP authentication is not TLS or a multi-host distributed rate limiter; use a restricted network and a TLS reverse proxy when needed. Do not expose the service directly to the Internet.
 
-- **SSRF Hardening:** Blocks `localhost`, loopbacks, private subnets (`10.0.0.0/8`, `192.168.0.0/16`, `172.16.0.0/12`), percent-encoding bypasses (`%31%32%37...`), and re-validates `Location` headers on HTTP redirects step-by-step.
-- **Zero Package Pollution:** Execution via `uvx` or `bin/cli.js` never modifies global site-packages or uses dangerous flags like `--break-system-packages`.
+## Development and artifact verification
 
----
+```bash
+uv sync --extra test
+uv run pytest -q
+uv run --with build python -m build
+npm pack
+python3 tests/artifact_smoke.py -- uv tool run --from /absolute/path/to/package.whl mcp-web-engine
+```
 
-## 📄 License
+Tests are offline by default. The artifact smoke sends no search query: initialize, silent notification, ping, tool listing, and a rejected loopback fetch. CI tests Python 3.11–3.13 and builds/installs wheel and npm artifacts outside the checkout. Package dependency installation still requires package-index access unless cached.
 
-MIT License © 2026 MCP Web Engine Contributors.
+MIT License. See [LICENSE](LICENSE).

package/bin/cli.js

@@ -1,90 +1,22 @@
 #!/usr/bin/env node
-
-/**
- * MCP Web Engine CLI Entry Point (npm / npx fallback)
- * Spawns MCP Web Engine in an ISOLATED Python environment without polluting global system packages.
- * 
- * Recommended execution for AI agents:
- *   uvx mcp-web-engine
- */
-
-const { spawn, execSync } = require('child_process');
-const path = require('path');
-const fs = require('fs');
-const os = require('os');
-
-const reqTxt = path.join(__dirname, '..', 'requirements.txt');
-
-function forwardChild(child) {
-  child.on('error', (err) => {
-    console.error('[mcp-web-engine] Failed to start process:', err.message);
-    process.exit(1);
-  });
-  child.on('exit', (code, signal) => {
-    if (signal) {
-      process.exit(128 + (signal === 'SIGINT' ? 2 : 15));
-    }
-    process.exit(code ?? 0);
-  });
-}
-
-// 1. Try running with 'uv' if available (fastest, cleanest, isolated)
-function tryUv() {
-  try {
-    execSync('uv --version', { stdio: 'ignore' });
-    const repoRoot = path.join(__dirname, '..');
-    const child = spawn('uv', ['run', '--directory', repoRoot, 'python', '-m', 'mcp_web_engine.main', ...process.argv.slice(2)], {
-      stdio: 'inherit',
-      env: process.env
-    });
-    forwardChild(child);
-    return true;
-  } catch (e) {
-    return false;
-  }
-}
-
-// 2. Fallback to dedicated isolated virtualenv in user cache dir (~/.cache/mcp-web-engine/venv)
-function runInIsolatedVenv() {
-  const cacheDir = process.env.XDG_CACHE_HOME || path.join(os.homedir(), '.cache');
-  const venvDir = path.join(cacheDir, 'mcp-web-engine', 'venv');
-  const venvPython = process.platform === 'win32' 
-    ? path.join(venvDir, 'Scripts', 'python.exe')
-    : path.join(venvDir, 'bin', 'python');
-  const venvPip = process.platform === 'win32'
-    ? path.join(venvDir, 'Scripts', 'pip.exe')
-    : path.join(venvDir, 'bin', 'pip');
-
-  function findSystemPython() {
-    for (const cmd of ['python3', 'python']) {
-      try {
-        execSync(`${cmd} --version`, { stdio: 'ignore' });
-        return cmd;
-      } catch (e) {}
-    }
-    throw new Error('Python 3 is required but was not found on PATH.');
-  }
-
-  // Ensure isolated venv exists and has requirements installed
-  if (!fs.existsSync(venvPython)) {
-    console.log('[mcp-web-engine] Creating isolated Python venv at:', venvDir);
-    const sysPython = findSystemPython();
-    fs.mkdirSync(path.dirname(venvDir), { recursive: true });
-    execSync(`${sysPython} -m venv "${venvDir}"`, { stdio: 'inherit' });
-    console.log('[mcp-web-engine] Installing dependencies into isolated venv...');
-    execSync(`"${venvPip}" install -r "${reqTxt}"`, { stdio: 'inherit' });
-  }
-
-  const repoRoot = path.join(__dirname, '..');
-  const env = { ...process.env, PYTHONPATH: path.join(repoRoot, 'src') };
-
-  const child = spawn(venvPython, ['-m', 'mcp_web_engine.main', ...process.argv.slice(2)], {
-    stdio: 'inherit',
-    env: env
-  });
-  forwardChild(child);
-}
-
-if (!tryUv()) {
-  runInIsolatedVenv();
+// Ship the Python src package in the npm tarball and let uv isolate it.
+// No pip/bootstrap chatter can ever enter MCP stdout.
+const { spawn } = require('node:child_process');
+const path = require('node:path');
+
+const root = path.resolve(__dirname, '..');
+const child = spawn('uv', ['tool', 'run', '--from', root, 'mcp-web-engine', ...process.argv.slice(2)], {
+  stdio: 'inherit',
+  env: process.env,
+});
+child.on('error', (error) => {
+  console.error(`[mcp-web-engine] Cannot launch uv: ${error.message}. Install uv: https://docs.astral.sh/uv/getting-started/installation/`);
+  process.exitCode = 1;
+});
+const signals = ['SIGINT', 'SIGTERM'];
+for (const signal of signals) {
+  process.on(signal, () => { if (!child.killed) child.kill(signal); });
 }
+child.on('exit', (code, signal) => {
+  process.exitCode = code ?? (signal === 'SIGINT' ? 130 : 143);
+});

package/docs/operations.md

@@ -0,0 +1,15 @@
+# Operating boundaries
+
+- Configure an operator-owned SearXNG backend. Queries go to that backend and its search engines even when DuckDuckGo fallback is disabled. Never accept arbitrary clients' backend URLs as trusted configuration.
+- Keep HTTP on loopback or a restricted network. Generate a unique API_KEY (at least 32 characters); terminate TLS at a reverse proxy for remote access. Health is liveness only. HTTP admission limits, beta quotas and per-key tool telemetry persist in SQLite, serializing workers that share the same local file. Set MCP_WEB_ENGINE_STATE_DB to an absolute writable path. This is not multi-host coordination: do not put SQLite on a distributed network mount. Process summary metrics remain process-local. Set MCP_WEB_ENGINE_BETA_KEYS_FILE to the separate operator-managed credential file; telemetry never rewrites that file. Preserve state across container replacement to preserve quotas; deleting database rows resets usage.
+- Explicit env files require MCP_WEB_ENGINE_ENV_FILE with an absolute path. Compose itself may interpolate a local .env file; that is Docker Compose behavior, not automatic application env loading. Do not bake secrets into images or npm tarballs.
+- The Docker context uses a deny-by-default allowlist for Python source and build metadata. Keep credentials outside src/; review actual artifacts before distribution. Do not put secrets in Python source or README metadata.
+- Public fetch validation and redirect checks do not replace firewall egress policy. Restrict cloud metadata, internal services and proxy configuration at the network layer. Treat fetched content as untrusted, including prompt injection.
+- uv/npm installation can access package registries and write dependency caches. It is not an execution sandbox. Pin and audit dependencies for reproducible deployments.
+- The stateless protocol label is project-specific, not an official specification certification. SSE is limited; validate your client's transport requirements. Stdio uses JSON lines rather than Content-Length framing.
+
+## Verify a local Docker image without touching deployed services
+
+Compose mounts the named `usage-state` volume for SQLite and operator-managed beta credentials. Container replacement preserves it; `docker compose down -v` explicitly destroys it and resets usage. Back up the volume and never use that command as routine deployment cleanup.
+
+Build a separately tagged image, run it with an ephemeral loopback port and a disposable key, probe /health and protected routes, then remove only that container. Never use compose down against an existing deployment as a test cleanup step.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "mcp-web-engine",
-  "version": "1.0.5",
-  "description": "Privacy-First, Self-Hostable & SSRF-Hardened MCP Server & Web Engine for AI Agents",
+  "version": "1.2.0",
+  "description": "Self-hostable MCP search, public-page fetch and Markdown extraction",
   "main": "bin/cli.js",
   "bin": {
     "mcp-web-engine": "bin/cli.js"
@@ -11,7 +11,9 @@
   },
   "files": [
     "bin",
-    "*.py",
+    "src/mcp_web_engine/*.py",
+    "pyproject.toml",
+    "docs/operations.md",
     "requirements.txt",
     "README.md",
     "LICENSE"

package/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["hatchling>=1.27,<2"]
+build-backend = "hatchling.build"
+
+[project]
+name = "mcp-web-engine"
+version = "1.2.0"
+description = "Self-hostable MCP search, public-page fetch and Markdown extraction"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+authors = [
+    { name = "Arbolencio" }
+]
+keywords = [
+    "mcp",
+    "mcp-server",
+    "searxng",
+    "web-search",
+    "ai-agent",
+    "turbollm",
+    "web-scraper",
+    "ssrf-protection"
+]
+dependencies = [
+    "fastapi>=0.162.0,<1",
+    "uvicorn>=0.54.0,<1",
+    "pydantic>=2.14.3,<3",
+    "pydantic-settings>=2.16.2,<3",
+    "curl-cffi>=0.17.0,<0.18",
+    "beautifulsoup4>=4.15.0,<5",
+    "html2text>=2025.4.15,<2027",
+    "sse-starlette>=3.6.0,<4"
+]
+
+[project.scripts]
+mcp-web-engine = "mcp_web_engine.main:cli_entrypoint"
+
+[project.urls]
+Homepage = "https://github.com/Arbolencio/mcp-web-engine"
+Repository = "https://github.com/Arbolencio/mcp-web-engine.git"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/mcp_web_engine"]
+
+[project.optional-dependencies]
+test = ["pytest>=8,<10", "pytest-asyncio>=0.26,<2", "httpx>=0.28,<1"]
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "I"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"

package/requirements.txt

@@ -1,9 +1 @@
-fastapi>=0.109.0
-uvicorn>=0.27.0
-pydantic>=2.5.0
-pydantic-settings>=2.1.0
-curl-cffi>=0.6.0
-beautifulsoup4>=4.12.0
-html2text>=2024.1.14
-sse-starlette>=2.0.0
-pytest>=8.0.0
+-e .

package/src/mcp_web_engine/__init__.py

@@ -0,0 +1,3 @@
+"""MCP Web Engine package."""
+
+__version__ = "1.2.0"

package/src/mcp_web_engine/config.py

@@ -0,0 +1,43 @@
+"""
+Centralized Configuration Module (Pydantic Settings / Environment Variables)
+"""
+import os
+from pathlib import Path
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class Settings(BaseSettings):
+    model_config = SettingsConfigDict(env_file=None, extra="ignore")
+
+    API_KEY: str = Field(default="", description="Operator-generated HTTP bearer key; required for --serve")
+    SEARXNG_URL: str = Field(default="http://127.0.0.1:8082/search", description="Internal SearXNG URL")
+    PORT: int = Field(default=5050, ge=1, le=65535, description="Service listening port")
+    HOST: str = Field(default="127.0.0.1", min_length=1, description="Service listening host")
+    
+    # Rate Limiting & Limits
+    RATE_LIMIT_PER_MINUTE: int = Field(default=120, ge=1, le=100000, description="Max requests per minute per API key")
+    DEFAULT_TIMEOUT_SEC: float = Field(default=10.0, gt=0, le=120, allow_inf_nan=False, description="Default HTTP request timeout in seconds")
+    MAX_PAYLOAD_BYTES: int = Field(default=2 * 1024 * 1024, ge=1, le=16 * 1024 * 1024, description="Max payload size limit (2 MB)")
+
+    # Privacy: the DuckDuckGo HTML fallback is deliberately opt-in, so that a query
+    # never leaves the host unless the operator explicitly allows it.
+    ALLOW_DDG_FALLBACK: bool = Field(default=False, description="Fall back to html.duckduckgo.com when SearXNG is unavailable (privacy trade-off, off by default)")
+    
+    # Environment & Logging
+    ENV: str = Field(default="production", description="Environment (development/production)")
+    LOG_LEVEL: str = Field(default="INFO", description="Log level")
+
+def load_settings():
+    """Only an explicitly selected absolute env file may supplement process env."""
+    selected = os.environ.get("MCP_WEB_ENGINE_ENV_FILE")
+    if selected:
+        path = Path(selected).expanduser()
+        if not path.is_absolute() or not path.is_file():
+            raise ValueError("MCP_WEB_ENGINE_ENV_FILE must name an existing absolute file")
+        return Settings(_env_file=path, _env_file_encoding="utf-8")
+    return Settings()
+
+
+settings = load_settings()

package/src/mcp_web_engine/logging_obs.py

@@ -0,0 +1,142 @@
+"""Secret-safe JSON logs and separate durable tool telemetry."""
+import hashlib
+import json
+import logging
+import re
+import threading
+from datetime import datetime, timezone
+from typing import Any, Optional
+
+from . import state_store
+from .config import settings
+
+
+class JSONFormatter(logging.Formatter):
+    _sensitive_key = re.compile(r"key|token|secret|auth|password|cookie|credential", re.I)
+    _url = re.compile(r"(?:[a-z][a-z0-9+.-]*://|www\.)[^\s<>\"']+", re.I)
+    _credentials = re.compile(
+        r"\b(?:Bearer\s+\S+|sk_mcp_beta_[A-Za-z0-9_-]+|"
+        r"(?:password|token|secret|api[_-]?key|authorization|cookie)\s*[:=]\s*\S+)", re.I)
+
+    def _text(self, value):
+        value = str(value)
+        master = settings.API_KEY
+        if isinstance(master, str) and master:
+            value = value.replace(master, "[REDACTED]")
+        value = self._url.sub("[REDACTED_URL]", value)
+        value = self._credentials.sub("[REDACTED]", value)
+        # Bound log size and remove terminal/control injection.
+        return "".join(c if ord(c) >= 32 and ord(c) != 127 else " " for c in value[:4096])
+
+    def format(self, record):
+        try:
+            message = "Operation failed" if record.exc_info else self._text(record.getMessage())
+        except Exception:
+            message = "Unformattable log message"
+        log_obj: dict[str, Any] = {
+            "timestamp": datetime.now(timezone.utc).isoformat(),
+            "level": record.levelname,
+            "module": self._text(record.module),
+            "message": message,
+        }
+        if hasattr(record, "extra_data"):
+            log_obj["extra"] = self.sanitize_data(record.extra_data)
+        return json.dumps(log_obj, allow_nan=False)
+
+    def sanitize_data(self, data, _depth=0):
+        if _depth >= 8:
+            return "[TRUNCATED]"
+        if isinstance(data, dict):
+            return {self._text(k): "REDACTED" if self._sensitive_key.search(str(k))
+                    else self.sanitize_data(v, _depth + 1) for k, v in list(data.items())[:100]}
+        if isinstance(data, (list, tuple)):
+            return [self.sanitize_data(v, _depth + 1) for v in data[:100]]
+        if data is None or type(data) in (bool, int):
+            return data
+        if isinstance(data, str):
+            return self._text(data)
+        # No repr/str of arbitrary exceptions or objects: may contain credentials.
+        if isinstance(data, float):
+            import math
+            return data if math.isfinite(data) else None
+        return "[REDACTED_OBJECT]"
+
+
+logger = logging.getLogger("mcp_web_engine")
+logger.setLevel(getattr(logging, settings.LOG_LEVEL.upper(), logging.INFO))
+handler = logging.StreamHandler()
+handler.setFormatter(JSONFormatter())
+if not any(isinstance(h.formatter, JSONFormatter) for h in logger.handlers):
+    logger.addHandler(handler)
+logger.propagate = False
+
+
+class MetricsTracker:
+    """Bounded process summary; per-key tool counters persist independently.
+
+    Call record once per tool completion, and check_rate_limit once per HTTP
+    admission. Telemetry requests are tool calls, never the admission quota.
+    """
+    def __init__(self):
+        self.total_requests = 0
+        self.total_errors = 0
+        self.tool_calls = {"web_search": 0, "fetch_url": 0, "extract_markdown": 0}
+        self.total_units_consumed = 0
+        self.latencies_ms = []
+        self._lock = threading.Lock()
+
+    def record(self, tool_name: str, latency_ms: float, success: bool, units: int = 1, api_key: Optional[str] = None):
+        with self._lock:
+            self.total_requests += 1
+            if tool_name in self.tool_calls:
+                self.tool_calls[tool_name] += 1
+            self.total_errors += int(not success)
+            self.total_units_consumed += units
+            self.latencies_ms.append(latency_ms)
+            if len(self.latencies_ms) > 1000:
+                del self.latencies_ms[:-1000]
+        if api_key:
+            self.update_beta_telemetry(api_key, tool_name, latency_ms, success, units)
+
+    def update_beta_telemetry(self, api_key: str, tool_name: str, latency_ms: float, success: bool, units: int = 1):
+        """Compatibility name; now records any authenticated key without JSON IO."""
+        try:
+            state_store.record_telemetry(hashlib.sha256(api_key.encode("utf-8")).hexdigest(),
+                                         tool_name, latency_ms, success, units)
+        except (state_store.StateUnavailable, state_store.AdmissionDenied, ValueError):
+            logger.error("Telemetry persistence failed")
+
+    def get_beta_telemetry(self, api_key: str):
+        """Durable per-key dashboard, no credential IO.
+
+        requests/admissions are HTTP admissions; tool_requests counts completed
+        tool calls (possibly several per HTTP batch). Raises StateUnavailable
+        rather than falsely returning zero if the state database cannot be read.
+        """
+        usage = state_store.read_usage(hashlib.sha256(api_key.encode("utf-8")).hexdigest())
+        tools = usage.get("requests", 0)
+        return {
+            "requests": usage.get("admissions", 0),
+            "admissions": usage.get("admissions", 0),
+            "tool_requests": tools,
+            "errors": usage.get("errors", 0),
+            "web_search": usage.get("web_search", 0),
+            "fetch_url": usage.get("fetch_url", 0),
+            "extract_markdown": usage.get("extract_markdown", 0),
+            "total_units_consumed": usage.get("units", 0),
+            "avg_latency_ms": round(usage.get("latency_total", 0) / tools, 2) if tools else 0.0,
+            "last_seen": datetime.fromtimestamp(usage["last_seen"], timezone.utc).isoformat() if usage.get("last_seen") else None,
+        }
+
+    def get_summary(self):
+        with self._lock:
+            return {
+                "total_requests": self.total_requests,
+                "total_errors": self.total_errors,
+                "tool_calls": dict(self.tool_calls),
+                "total_units_consumed": self.total_units_consumed,
+                "avg_latency_ms": round(sum(self.latencies_ms) / len(self.latencies_ms), 2) if self.latencies_ms else 0.0,
+            }
+
+
+metrics = MetricsTracker()