fetch-guard 0.9.0__tar.gz

fetch_guard-0.9.0/.github/workflows/ci.yml

@@ -0,0 +1,31 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: ruff check fetch_guard/ tests/
+
+      - name: Test
+        run: pytest -q

fetch_guard-0.9.0/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        run: pip install uv
+
+      - name: Build
+        run: uv build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

fetch_guard-0.9.0/.gitignore

@@ -0,0 +1,44 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+dist/
+build/
+*.whl
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+*.cover
+
+# Ruff
+.ruff_cache/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+Thumbs.db
+Desktop.ini
+.DS_Store
+
+# MCP local config
+.mcp.json
+
+# Plans / scope docs
+plans/
+
+# AI workspace
+CLAUDE.md

fetch_guard-0.9.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Owen Sterling
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fetch_guard-0.9.0/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.4
+Name: fetch-guard
+Version: 0.9.0
+Summary: Fetch URLs and return clean, LLM-ready markdown with metadata and prompt injection defense
+Project-URL: Homepage, https://github.com/Erodenn/fetch-guard
+Project-URL: Repository, https://github.com/Erodenn/fetch-guard
+Project-URL: Issues, https://github.com/Erodenn/fetch-guard/issues
+Author-email: Owen Sterling <owen@erodenn.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Requires-Python: >=3.10
+Requires-Dist: beautifulsoup4
+Requires-Dist: extruct
+Requires-Dist: mcp
+Requires-Dist: requests
+Requires-Dist: trafilatura
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: js
+Requires-Dist: playwright; extra == 'js'
+Description-Content-Type: text/markdown
+
+# Fetch Guard
+
+[![License: MIT](https://badgen.net/github/license/Erodenn/fetch-guard)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+
+An [MCP](https://modelcontextprotocol.io/) server and CLI tool that fetches URLs and returns clean, LLM-ready markdown. Not a simple HTTP client, not a browser automation wrapper. A purpose-built extraction pipeline that sanitizes HTML, pulls structured metadata, detects prompt injection attempts, and handles the edge cases that break naive fetchers: bot blocks, paywalls, login walls, non-HTML content types, and pages that require JavaScript to render.
+
+The core problem is straightforward: LLMs need web content, but raw HTML is noisy and potentially hostile. Fetched pages can contain hidden text, invisible Unicode, off-screen elements, and outright prompt injection attempts embedded in the content itself. This pipeline strips all of that before the content reaches the model.
+
+Three layers handle the injection defense specifically:
+
+1. **Pre-extraction sanitization** removes hidden elements (`display:none`, `visibility:hidden`, `opacity:0`), off-screen positioned content, `aria-hidden` elements, `<noscript>` tags, and 18 categories of non-printing Unicode characters. This happens before content extraction, so trafilatura never sees the attack vectors.
+2. **Pattern scanning** runs 15 compiled regex patterns against the extracted text, covering system prompt overrides, ignore-previous instructions, role injection, fake conversation tags, hidden instruction markers (`[INST]`, `<<SYS>>`), and suspicious base64 blocks.
+3. **Session-salted output wrapping** generates a random 8-character hex salt per invocation and wraps the body in `<fetch-content-{salt}>` tags. Since the salt is unpredictable, injected content cannot spoof the wrapper boundaries.
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.10+
+- pip
+
+### Install
+
+```bash
+pip install fetch-guard
+```
+
+For JavaScript rendering (optional):
+
+```bash
+pip install 'fetch-guard[js]' && playwright install chromium
+```
+
+### Configure Your MCP Client
+
+Add the following to your MCP client config. Works with Claude Code, Claude Desktop, Cursor, or any MCP-compatible client.
+
+**Via uvx (recommended):**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "uvx",
+      "args": ["fetch-guard"]
+    }
+  }
+}
+```
+
+**Via pip install:**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "fetch-guard"
+    }
+  }
+}
+```
+
+**From source:**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "python",
+      "args": ["-m", "fetch_guard.server"]
+    }
+  }
+}
+```
+
+### Verify
+
+Ask your AI assistant to fetch any URL. If it returns structured content with a status header, metadata, and risk assessment, you're connected.
+
+### CLI
+
+```bash
+fetch-guard-cli <url> [options]
+# or: python -m fetch_guard.cli <url> [options]
+```
+
+| Flag | Default | Description |
+|---|---|---|
+| `--timeout N` | 180 | Request timeout in seconds |
+| `--max-words N` | none | Word cap on extracted body content |
+| `--js` | off | Use Playwright for JS-rendered pages |
+| `--strict` | off | Exit code 2 on high-risk injection |
+| `--links MODE` | `domains` | `domains` for unique external domains, `full` for all URLs with anchor text |
+
+### Claude Code Skill
+
+Copy `resources/fetch-guard/` to `.claude/skills/fetch-guard/` in your project, or use the standalone command file `resources/fetch-guard.md` as a Claude Code command.
+
+## What It Does
+
+The pipeline runs a 13-step sequence from URL to structured output:
+
+1. **`/llms.txt` preflight.** Checks the domain root for `/llms.txt` before the full fetch. If the requested URL is a domain root and `/llms.txt` exists, that content replaces the normal HTML pipeline entirely. This respects the emerging convention for LLM-friendly site summaries.
+
+2. **Fetch.** Static HTTP request via `requests`, or Playwright-driven browser rendering if `--js` is set. No automatic fallback between the two: `--js` is explicit opt-in.
+
+3. **Edge detection.** Classifies the response for bot blocks (Cloudflare challenges, 403/429/503 with block signatures, LinkedIn's custom 999), paywalls (subscription prompts, premium overlays), and login walls (sign-in redirects, members-only patterns).
+
+4. **Automatic retry.** Bot blocks trigger one retry with a full Chrome User-Agent string before reporting. Paywalls and login walls are reported immediately with no retry.
+
+5. **Content-type routing.** Non-HTML responses get a fast path: JSON is rendered as a fenced code block, RSS/Atom feeds are parsed into structured summaries, CSV becomes a markdown table (capped at 2,000 rows), and plain text passes through directly. Binary content types are rejected.
+
+6. **HTML sanitization.** Strips hidden elements, off-screen positioned content, `aria-hidden` nodes, `<noscript>` tags, and non-printing Unicode. Returns a tally of everything removed.
+
+7. **Content extraction.** trafilatura converts sanitized HTML to markdown with link preservation.
+
+8. **Metadata extraction.** Pulls title, author, date, description, canonical URL, and image from three sources in priority order: JSON-LD, Open Graph, then meta tags.
+
+9. **Link extraction.** Two modes: `domains` returns a sorted list of unique external domains, `full` returns all external URLs grouped by domain with anchor text.
+
+10. **Injection scanning.** Runs all 15 patterns against the extracted markdown. Each match records the pattern name, severity (high/medium), and a 60-character context snippet.
+
+11. **Truncation.** If `--max-words` is set, the body is truncated after extraction but before output wrapping.
+
+12. **Salt wrapping.** The body gets wrapped in session-salted tags for defense-in-depth.
+
+13. **Output formatting.** CLI produces five plaintext sections (status header, body, metadata, links, injection details). MCP server returns a structured JSON dict with the same data.
+
+## Output
+
+### CLI
+
+Five sections, printed to stdout:
+
+- **Status header:** URL, fetch timestamp, risk flag (`OK` or `INJECTION WARNING`), sanitization tally, edge case info if detected
+- **Body:** clean markdown wrapped in `<fetch-content-{salt}>` tags
+- **Metadata:** JSON block with title, author, date, description, canonical URL, image
+- **External links:** domain list or full URL breakdown by domain
+- **Injection details:** pattern name, severity, and context snippet for each match (only present when patterns detected)
+
+### MCP Server
+
+Returns a structured dict:
+
+```
+url, fetched_at, body, content_type, metadata, links, links_mode,
+risk_level, injection_matches, edge_cases, sanitization,
+llms_txt_available, llms_txt_replaced, js_rendered, js_hint,
+retried, truncated_at
+```
+
+When `--strict` is set and the risk level is `HIGH`, the CLI exits with code 2 and the MCP server raises an error response. The full result is still available in both cases.
+
+## Exit Codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Fetch error (network failure, empty response, binary content) |
+| 2 | High-risk injection detected (`--strict` only) |
+
+## Architecture
+
+```
+fetch_guard/
+├── pipeline.py             # Core orchestration — 13-step sequence, shared by CLI and server
+├── cli.py                  # CLI entry point — arg parsing, pipeline call, output
+├── server.py               # MCP server — FastMCP wrapper over the same pipeline
+│
+├── http/                   # HTTP fetching layer
+│   ├── client.py           # Static HTTP fetch via requests
+│   ├── playwright.py       # JS rendering via Playwright (optional)
+│   └── llms_txt.py         # /llms.txt preflight check
+│
+├── extraction/             # Content extraction and edge detection
+│   ├── content.py          # trafilatura wrapper — HTML to markdown
+│   ├── content_type.py     # Non-HTML routing — JSON, XML/RSS, CSV, plain text
+│   ├── edges.py            # Bot block, paywall, login wall classification
+│   ├── links.py            # External link extraction (domain list or full URLs)
+│   └── metadata.py         # JSON-LD, Open Graph, meta tag extraction
+│
+├── security/               # Injection defense
+│   ├── guard.py            # Salt generation, content wrapping, pattern scanning
+│   ├── patterns.py         # 15 compiled regex patterns — single source of truth
+│   └── sanitizer.py        # Hidden element and non-printing character removal
+│
+└── output/                 # Formatting
+    └── formatter.py        # CLI output assembly
+```
+
+Each module is a single-responsibility unit with a public function as its interface. `pipeline.py` is the shared core: both `cli.py` and `server.py` call `pipeline.run()` and handle the result in their own way.
+
+## Development
+
+```bash
+# Run tests (217 unit tests, all mocked — no network calls)
+pytest
+
+# Run live integration tests (hits real URLs)
+pytest -m live
+
+# Lint
+ruff check fetch_guard/ tests/
+```
+
+CI runs on push and PR to `main` via GitHub Actions, testing against Python 3.10, 3.12, and 3.13.
+
+## Acknowledgements
+
+Developed with [Claude Code](https://claude.ai/code).
+
+## License
+
+[MIT](LICENSE)

fetch_guard-0.9.0/README.md

@@ -0,0 +1,215 @@
+# Fetch Guard
+
+[![License: MIT](https://badgen.net/github/license/Erodenn/fetch-guard)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+
+An [MCP](https://modelcontextprotocol.io/) server and CLI tool that fetches URLs and returns clean, LLM-ready markdown. Not a simple HTTP client, not a browser automation wrapper. A purpose-built extraction pipeline that sanitizes HTML, pulls structured metadata, detects prompt injection attempts, and handles the edge cases that break naive fetchers: bot blocks, paywalls, login walls, non-HTML content types, and pages that require JavaScript to render.
+
+The core problem is straightforward: LLMs need web content, but raw HTML is noisy and potentially hostile. Fetched pages can contain hidden text, invisible Unicode, off-screen elements, and outright prompt injection attempts embedded in the content itself. This pipeline strips all of that before the content reaches the model.
+
+Three layers handle the injection defense specifically:
+
+1. **Pre-extraction sanitization** removes hidden elements (`display:none`, `visibility:hidden`, `opacity:0`), off-screen positioned content, `aria-hidden` elements, `<noscript>` tags, and 18 categories of non-printing Unicode characters. This happens before content extraction, so trafilatura never sees the attack vectors.
+2. **Pattern scanning** runs 15 compiled regex patterns against the extracted text, covering system prompt overrides, ignore-previous instructions, role injection, fake conversation tags, hidden instruction markers (`[INST]`, `<<SYS>>`), and suspicious base64 blocks.
+3. **Session-salted output wrapping** generates a random 8-character hex salt per invocation and wraps the body in `<fetch-content-{salt}>` tags. Since the salt is unpredictable, injected content cannot spoof the wrapper boundaries.
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.10+
+- pip
+
+### Install
+
+```bash
+pip install fetch-guard
+```
+
+For JavaScript rendering (optional):
+
+```bash
+pip install 'fetch-guard[js]' && playwright install chromium
+```
+
+### Configure Your MCP Client
+
+Add the following to your MCP client config. Works with Claude Code, Claude Desktop, Cursor, or any MCP-compatible client.
+
+**Via uvx (recommended):**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "uvx",
+      "args": ["fetch-guard"]
+    }
+  }
+}
+```
+
+**Via pip install:**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "fetch-guard"
+    }
+  }
+}
+```
+
+**From source:**
+
+```json
+{
+  "mcpServers": {
+    "fetch-guard": {
+      "command": "python",
+      "args": ["-m", "fetch_guard.server"]
+    }
+  }
+}
+```
+
+### Verify
+
+Ask your AI assistant to fetch any URL. If it returns structured content with a status header, metadata, and risk assessment, you're connected.
+
+### CLI
+
+```bash
+fetch-guard-cli <url> [options]
+# or: python -m fetch_guard.cli <url> [options]
+```
+
+| Flag | Default | Description |
+|---|---|---|
+| `--timeout N` | 180 | Request timeout in seconds |
+| `--max-words N` | none | Word cap on extracted body content |
+| `--js` | off | Use Playwright for JS-rendered pages |
+| `--strict` | off | Exit code 2 on high-risk injection |
+| `--links MODE` | `domains` | `domains` for unique external domains, `full` for all URLs with anchor text |
+
+### Claude Code Skill
+
+Copy `resources/fetch-guard/` to `.claude/skills/fetch-guard/` in your project, or use the standalone command file `resources/fetch-guard.md` as a Claude Code command.
+
+## What It Does
+
+The pipeline runs a 13-step sequence from URL to structured output:
+
+1. **`/llms.txt` preflight.** Checks the domain root for `/llms.txt` before the full fetch. If the requested URL is a domain root and `/llms.txt` exists, that content replaces the normal HTML pipeline entirely. This respects the emerging convention for LLM-friendly site summaries.
+
+2. **Fetch.** Static HTTP request via `requests`, or Playwright-driven browser rendering if `--js` is set. No automatic fallback between the two: `--js` is explicit opt-in.
+
+3. **Edge detection.** Classifies the response for bot blocks (Cloudflare challenges, 403/429/503 with block signatures, LinkedIn's custom 999), paywalls (subscription prompts, premium overlays), and login walls (sign-in redirects, members-only patterns).
+
+4. **Automatic retry.** Bot blocks trigger one retry with a full Chrome User-Agent string before reporting. Paywalls and login walls are reported immediately with no retry.
+
+5. **Content-type routing.** Non-HTML responses get a fast path: JSON is rendered as a fenced code block, RSS/Atom feeds are parsed into structured summaries, CSV becomes a markdown table (capped at 2,000 rows), and plain text passes through directly. Binary content types are rejected.
+
+6. **HTML sanitization.** Strips hidden elements, off-screen positioned content, `aria-hidden` nodes, `<noscript>` tags, and non-printing Unicode. Returns a tally of everything removed.
+
+7. **Content extraction.** trafilatura converts sanitized HTML to markdown with link preservation.
+
+8. **Metadata extraction.** Pulls title, author, date, description, canonical URL, and image from three sources in priority order: JSON-LD, Open Graph, then meta tags.
+
+9. **Link extraction.** Two modes: `domains` returns a sorted list of unique external domains, `full` returns all external URLs grouped by domain with anchor text.
+
+10. **Injection scanning.** Runs all 15 patterns against the extracted markdown. Each match records the pattern name, severity (high/medium), and a 60-character context snippet.
+
+11. **Truncation.** If `--max-words` is set, the body is truncated after extraction but before output wrapping.
+
+12. **Salt wrapping.** The body gets wrapped in session-salted tags for defense-in-depth.
+
+13. **Output formatting.** CLI produces five plaintext sections (status header, body, metadata, links, injection details). MCP server returns a structured JSON dict with the same data.
+
+## Output
+
+### CLI
+
+Five sections, printed to stdout:
+
+- **Status header:** URL, fetch timestamp, risk flag (`OK` or `INJECTION WARNING`), sanitization tally, edge case info if detected
+- **Body:** clean markdown wrapped in `<fetch-content-{salt}>` tags
+- **Metadata:** JSON block with title, author, date, description, canonical URL, image
+- **External links:** domain list or full URL breakdown by domain
+- **Injection details:** pattern name, severity, and context snippet for each match (only present when patterns detected)
+
+### MCP Server
+
+Returns a structured dict:
+
+```
+url, fetched_at, body, content_type, metadata, links, links_mode,
+risk_level, injection_matches, edge_cases, sanitization,
+llms_txt_available, llms_txt_replaced, js_rendered, js_hint,
+retried, truncated_at
+```
+
+When `--strict` is set and the risk level is `HIGH`, the CLI exits with code 2 and the MCP server raises an error response. The full result is still available in both cases.
+
+## Exit Codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Fetch error (network failure, empty response, binary content) |
+| 2 | High-risk injection detected (`--strict` only) |
+
+## Architecture
+
+```
+fetch_guard/
+├── pipeline.py             # Core orchestration — 13-step sequence, shared by CLI and server
+├── cli.py                  # CLI entry point — arg parsing, pipeline call, output
+├── server.py               # MCP server — FastMCP wrapper over the same pipeline
+│
+├── http/                   # HTTP fetching layer
+│   ├── client.py           # Static HTTP fetch via requests
+│   ├── playwright.py       # JS rendering via Playwright (optional)
+│   └── llms_txt.py         # /llms.txt preflight check
+│
+├── extraction/             # Content extraction and edge detection
+│   ├── content.py          # trafilatura wrapper — HTML to markdown
+│   ├── content_type.py     # Non-HTML routing — JSON, XML/RSS, CSV, plain text
+│   ├── edges.py            # Bot block, paywall, login wall classification
+│   ├── links.py            # External link extraction (domain list or full URLs)
+│   └── metadata.py         # JSON-LD, Open Graph, meta tag extraction
+│
+├── security/               # Injection defense
+│   ├── guard.py            # Salt generation, content wrapping, pattern scanning
+│   ├── patterns.py         # 15 compiled regex patterns — single source of truth
+│   └── sanitizer.py        # Hidden element and non-printing character removal
+│
+└── output/                 # Formatting
+    └── formatter.py        # CLI output assembly
+```
+
+Each module is a single-responsibility unit with a public function as its interface. `pipeline.py` is the shared core: both `cli.py` and `server.py` call `pipeline.run()` and handle the result in their own way.
+
+## Development
+
+```bash
+# Run tests (217 unit tests, all mocked — no network calls)
+pytest
+
+# Run live integration tests (hits real URLs)
+pytest -m live
+
+# Lint
+ruff check fetch_guard/ tests/
+```
+
+CI runs on push and PR to `main` via GitHub Actions, testing against Python 3.10, 3.12, and 3.13.
+
+## Acknowledgements
+
+Developed with [Claude Code](https://claude.ai/code).
+
+## License
+
+[MIT](LICENSE)

fetch_guard-0.9.0/fetch_guard/__init__.py

@@ -0,0 +1 @@
+"""Fetch Guard — LLM-ready web fetching with prompt injection defense."""