ai-crawler-radar 0.1.0__tar.gz

ai_crawler_radar-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Surendran B
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai_crawler_radar-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE
+include README.md
+include SECURITY.md
+recursive-include src/ai_crawler_radar/data *.json
+recursive-include skills *.md

ai_crawler_radar-0.1.0/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: ai-crawler-radar
+Version: 0.1.0
+Summary: Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
+Author-email: Surendran B <reachsuren@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/surendranb/ai-crawler-radar
+Project-URL: Repository, https://github.com/surendranb/ai-crawler-radar
+Project-URL: Issues, https://github.com/surendranb/ai-crawler-radar/issues
+Keywords: mcp,ai-crawler,llms-txt,robots-txt,turnstile,waf,cloudflare-radar,searchgpt,perplexity
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp<3,>=2.0.0
+Requires-Dist: httpx>=0.27.0
+Provides-Extra: test
+Requires-Dist: pytest>=8.0.0; extra == "test"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
+Provides-Extra: publish
+Requires-Dist: build>=1.2.2; extra == "publish"
+Requires-Dist: twine>=5.1.1; extra == "publish"
+Dynamic: license-file
+
+# ⚡ ai-crawler-radar
+
+[![CI](https://github.com/surendranb/ai-crawler-radar/actions/workflows/ci.yml/badge.svg)](https://github.com/surendranb/ai-crawler-radar/actions)
+[![PyPI version](https://img.shields.io/pypi/v/ai-crawler-radar.svg?style=flat-square&color=blue)](https://pypi.org/project/ai-crawler-radar/)
+[![NPM version](https://img.shields.io/npm/v/@surendranb/ai-crawler-radar.svg?style=flat-square&color=green)](https://www.npmjs.com/package/@surendranb/ai-crawler-radar)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](LICENSE)
+
+> **Autonomous AI Search, Training & Agent Crawler Diagnostic Engine.**  
+> Probes live web pages with authentic AI bot signatures, audits `robots.txt` across 3 tiers, identifies WAF hard-blocks & Turnstile challenges, calculates token bloat multipliers, and validates `/llms.txt`.
+
+Available in **two architectural deployment variants** designed for maximum flexibility and privacy:
+1. **Local MCP (stdio)**: Zero-auth, runs on your machine via `uvx` / `npx`, cold boot < 50ms.
+2. **Remote MCP (Streamable HTTP / SSE)**: Hosted edge gateway (`https://crawler.builditwithai.xyz/mcp`) delivering continuous tool, schema, prompt, and playbook updates without needing PyPI/NPM package upgrades. Data never leaves your computer; zero cloud compute burden.
+
+---
+
+## ⚡ 1-Line Universal Installer
+
+Automatically detects and configures all installed AI agent harnesses (Claude Desktop, Cursor, Claude Code, Antigravity, OpenCode, Zed, Windsurf):
+
+```bash
+curl -fsSL https://crawler.builditwithai.xyz/install | bash
+```
+
+---
+
+## 🛠️ MCP Configuration: Choose Your Variant
+
+### Variant A: Remote MCP (Streamable HTTP / SSE)
+> **Recommended for instant edge updates**: Tools, schemas, prompts, and operational playbooks update dynamically at the edge without requiring package reinstallations or PyPI/NPM updates. Zero cloud compute burden; data never leaves your computer.
+
+#### 🤖 Claude Desktop / Claude Code
+Add to `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "url": "https://crawler.builditwithai.xyz/mcp"
+    }
+  }
+}
+```
+
+#### 💻 Cursor / Windsurf / Antigravity
+Add to your IDE MCP settings:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "url": "https://crawler.builditwithai.xyz/mcp"
+    }
+  }
+}
+```
+
+#### 🌐 OpenCode
+Add to `opencode.jsonc`:
+```jsonc
+{
+  "mcp": {
+    "ai-crawler-radar": {
+      "type": "remote",
+      "url": "https://crawler.builditwithai.xyz/mcp",
+      "enabled": true
+    }
+  }
+}
+```
+
+---
+
+### Variant B: Local MCP (stdio)
+> **Recommended for offline / air-gapped environments**: Runs natively on your machine via Python (`uvx`) or Node (`npx`). Zero dependencies, cold boot < 50ms.
+
+#### 🤖 Claude Desktop / Claude Code
+Add to `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "command": "uvx",
+      "args": ["ai-crawler-radar"]
+    }
+  }
+}
+```
+
+#### 💻 Cursor / Windsurf / Antigravity
+Add to your IDE MCP settings:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "command": "uvx",
+      "args": ["ai-crawler-radar"]
+    }
+  }
+}
+```
+
+#### 🌐 OpenCode
+Add to `opencode.jsonc`:
+```jsonc
+{
+  "mcp": {
+    "ai-crawler-radar": {
+      "type": "local",
+      "command": ["uvx", "ai-crawler-radar"],
+      "enabled": true
+    }
+  }
+}
+```
+
+*(Node.js users can substitute `uvx ai-crawler-radar` with `npx -y @surendranb/ai-crawler-radar`)*
+
+---
+
+## 🧭 The 3-Tier AI Bot Model
+
+`ai-crawler-radar` categorizes and evaluates crawler access across three distinct operational tiers:
+
+| Tier | Primary Crawlers | Strategic Purpose | Typical Posture |
+| :--- | :--- | :--- | :--- |
+| **Tier 1: Model Pre-Training** | `GPTBot`, `ClaudeBot`, `Google-Extended`, `Meta-ExternalAgent`, `Bytespider`, `CCBot` | Bulk offline data harvesting to train foundation models | Block if protecting proprietary intellectual property |
+| **Tier 2: AI Search Indexing** | `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `Applebot`, `Bingbot` | Real-time indexing to power SearchGPT, Perplexity & Siri citations | **Allow** to drive citations and conversational discovery |
+| **Tier 3: Live User Fetchers** | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | Real-time URL fetching triggered by an active human chat prompt | **Allow** for link previews and interactive agent assistance |
+
+---
+
+## 📡 Available Tools & Prompts
+
+### Tools
+| Tool Name | Parameters | Description |
+| :--- | :--- | :--- |
+| `probe_url` | `url`, `tier`, `check_markdown_twin` | Probes URL using authentic headers across bot tiers; diagnoses WAF hard-blocks, Turnstile challenges, CSR empty shells, and token bloat multipliers. |
+| `audit_robots` | `domain`, `categories` | RFC 9309 rules parser evaluating permissions across Training, Search, and Live Fetch tiers with strategic recommendations. |
+| `compare_readiness` | `urls` | Multi-domain side-by-side benchmark comparing AI crawler readiness, token efficiency, and `/llms.txt` adoption. |
+| `bot_directory` | `query`, `category` | Search and inspect 758 verified bots from Cloudflare Radar Bot Directory. |
+| `generate_llms_txt` | `domain` | Validates existing `/llms.txt` against CommonMark V2 standard, or generates a compliant starter specification. |
+| `skill_read` | `skill_name` | Read bundled operational playbooks dynamically. |
+| `skills_list` | *(none)* | List available operational skills. |
+
+### Dynamic Prompts (Remote Variant)
+- `audit_domain_ai_posture`: Comprehensive 3-tier crawler readiness and robots.txt audit.
+- `diagnose_crawler_block`: Root-cause investigation into WAF, Turnstile, or CSR blocks.
+- `generate_llms_specification`: Automated generation of CommonMark V2 `/llms.txt` and `/llms-full.txt`.
+
+---
+
+## 🖥️ Command Line (CLI) Usage
+
+```bash
+# Probe URL with authentic AI bot signatures
+ai-crawler-radar probe https://docs.anthropic.com
+
+# Audit domain robots.txt across 3 bot tiers
+ai-crawler-radar audit github.com
+
+# Search Cloudflare Radar Bot Directory (758 verified bots)
+ai-crawler-radar bots --query openai
+ai-crawler-radar bots --category AI_SEARCH
+
+# Validate or generate CommonMark V2 /llms.txt
+ai-crawler-radar llmstxt https://fastmcp.org
+ai-crawler-radar llmstxt example.com --generate
+
+# Output pure JSON for automated CI/CD pipelines
+ai-crawler-radar probe https://example.com --json
+```
+
+---
+
+## 🔬 Diagnostic Engine Architecture
+
+1. **WAF Hard-Blocks**: Identifies Cloudflare (`CF-Ray`, Error 1020), AWS WAF, Akamai Ghost, and Imperva access blocks.
+2. **Turnstile & Challenges**: Detects Cloudflare Turnstile widgets, managed JS puzzles, and interactive CAPTCHAs that halt crawlers.
+3. **CSR Empty Shells**: Flags single-page apps (`<div id="root">`, `<div id="app">`) where dynamic JavaScript rendering leaves initial HTML blank for crawlers.
+4. **Token Bloat Multiplier**: Computes ratio of raw DOM tokens to extracted readable content tokens (e.g. 35x bloat) to measure LLM ingestion efficiency.
+5. **Content Negotiation & Cache Poisoning**: Detects missing `Vary: Accept` headers on markdown twin responses that cause CDN cache collisions.
+
+---
+
+## 🔒 Telemetry & Privacy
+
+Data never leaves your computer. In both local and remote variants, private target data is never stored on external servers.
+
+To opt out of anonymous usage telemetry:
+```bash
+export DO_NOT_TRACK=1
+# or
+export AI_CRAWLER_RADAR_TELEMETRY=off
+```
+
+---
+
+## 📄 License
+
+MIT License. Free and open source forever for all developers and AI agents.

ai_crawler_radar-0.1.0/README.md

@@ -0,0 +1,199 @@
+# ⚡ ai-crawler-radar
+
+[![CI](https://github.com/surendranb/ai-crawler-radar/actions/workflows/ci.yml/badge.svg)](https://github.com/surendranb/ai-crawler-radar/actions)
+[![PyPI version](https://img.shields.io/pypi/v/ai-crawler-radar.svg?style=flat-square&color=blue)](https://pypi.org/project/ai-crawler-radar/)
+[![NPM version](https://img.shields.io/npm/v/@surendranb/ai-crawler-radar.svg?style=flat-square&color=green)](https://www.npmjs.com/package/@surendranb/ai-crawler-radar)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](LICENSE)
+
+> **Autonomous AI Search, Training & Agent Crawler Diagnostic Engine.**  
+> Probes live web pages with authentic AI bot signatures, audits `robots.txt` across 3 tiers, identifies WAF hard-blocks & Turnstile challenges, calculates token bloat multipliers, and validates `/llms.txt`.
+
+Available in **two architectural deployment variants** designed for maximum flexibility and privacy:
+1. **Local MCP (stdio)**: Zero-auth, runs on your machine via `uvx` / `npx`, cold boot < 50ms.
+2. **Remote MCP (Streamable HTTP / SSE)**: Hosted edge gateway (`https://crawler.builditwithai.xyz/mcp`) delivering continuous tool, schema, prompt, and playbook updates without needing PyPI/NPM package upgrades. Data never leaves your computer; zero cloud compute burden.
+
+---
+
+## ⚡ 1-Line Universal Installer
+
+Automatically detects and configures all installed AI agent harnesses (Claude Desktop, Cursor, Claude Code, Antigravity, OpenCode, Zed, Windsurf):
+
+```bash
+curl -fsSL https://crawler.builditwithai.xyz/install | bash
+```
+
+---
+
+## 🛠️ MCP Configuration: Choose Your Variant
+
+### Variant A: Remote MCP (Streamable HTTP / SSE)
+> **Recommended for instant edge updates**: Tools, schemas, prompts, and operational playbooks update dynamically at the edge without requiring package reinstallations or PyPI/NPM updates. Zero cloud compute burden; data never leaves your computer.
+
+#### 🤖 Claude Desktop / Claude Code
+Add to `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "url": "https://crawler.builditwithai.xyz/mcp"
+    }
+  }
+}
+```
+
+#### 💻 Cursor / Windsurf / Antigravity
+Add to your IDE MCP settings:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "url": "https://crawler.builditwithai.xyz/mcp"
+    }
+  }
+}
+```
+
+#### 🌐 OpenCode
+Add to `opencode.jsonc`:
+```jsonc
+{
+  "mcp": {
+    "ai-crawler-radar": {
+      "type": "remote",
+      "url": "https://crawler.builditwithai.xyz/mcp",
+      "enabled": true
+    }
+  }
+}
+```
+
+---
+
+### Variant B: Local MCP (stdio)
+> **Recommended for offline / air-gapped environments**: Runs natively on your machine via Python (`uvx`) or Node (`npx`). Zero dependencies, cold boot < 50ms.
+
+#### 🤖 Claude Desktop / Claude Code
+Add to `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "command": "uvx",
+      "args": ["ai-crawler-radar"]
+    }
+  }
+}
+```
+
+#### 💻 Cursor / Windsurf / Antigravity
+Add to your IDE MCP settings:
+```json
+{
+  "mcpServers": {
+    "ai-crawler-radar": {
+      "command": "uvx",
+      "args": ["ai-crawler-radar"]
+    }
+  }
+}
+```
+
+#### 🌐 OpenCode
+Add to `opencode.jsonc`:
+```jsonc
+{
+  "mcp": {
+    "ai-crawler-radar": {
+      "type": "local",
+      "command": ["uvx", "ai-crawler-radar"],
+      "enabled": true
+    }
+  }
+}
+```
+
+*(Node.js users can substitute `uvx ai-crawler-radar` with `npx -y @surendranb/ai-crawler-radar`)*
+
+---
+
+## 🧭 The 3-Tier AI Bot Model
+
+`ai-crawler-radar` categorizes and evaluates crawler access across three distinct operational tiers:
+
+| Tier | Primary Crawlers | Strategic Purpose | Typical Posture |
+| :--- | :--- | :--- | :--- |
+| **Tier 1: Model Pre-Training** | `GPTBot`, `ClaudeBot`, `Google-Extended`, `Meta-ExternalAgent`, `Bytespider`, `CCBot` | Bulk offline data harvesting to train foundation models | Block if protecting proprietary intellectual property |
+| **Tier 2: AI Search Indexing** | `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `Applebot`, `Bingbot` | Real-time indexing to power SearchGPT, Perplexity & Siri citations | **Allow** to drive citations and conversational discovery |
+| **Tier 3: Live User Fetchers** | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | Real-time URL fetching triggered by an active human chat prompt | **Allow** for link previews and interactive agent assistance |
+
+---
+
+## 📡 Available Tools & Prompts
+
+### Tools
+| Tool Name | Parameters | Description |
+| :--- | :--- | :--- |
+| `probe_url` | `url`, `tier`, `check_markdown_twin` | Probes URL using authentic headers across bot tiers; diagnoses WAF hard-blocks, Turnstile challenges, CSR empty shells, and token bloat multipliers. |
+| `audit_robots` | `domain`, `categories` | RFC 9309 rules parser evaluating permissions across Training, Search, and Live Fetch tiers with strategic recommendations. |
+| `compare_readiness` | `urls` | Multi-domain side-by-side benchmark comparing AI crawler readiness, token efficiency, and `/llms.txt` adoption. |
+| `bot_directory` | `query`, `category` | Search and inspect 758 verified bots from Cloudflare Radar Bot Directory. |
+| `generate_llms_txt` | `domain` | Validates existing `/llms.txt` against CommonMark V2 standard, or generates a compliant starter specification. |
+| `skill_read` | `skill_name` | Read bundled operational playbooks dynamically. |
+| `skills_list` | *(none)* | List available operational skills. |
+
+### Dynamic Prompts (Remote Variant)
+- `audit_domain_ai_posture`: Comprehensive 3-tier crawler readiness and robots.txt audit.
+- `diagnose_crawler_block`: Root-cause investigation into WAF, Turnstile, or CSR blocks.
+- `generate_llms_specification`: Automated generation of CommonMark V2 `/llms.txt` and `/llms-full.txt`.
+
+---
+
+## 🖥️ Command Line (CLI) Usage
+
+```bash
+# Probe URL with authentic AI bot signatures
+ai-crawler-radar probe https://docs.anthropic.com
+
+# Audit domain robots.txt across 3 bot tiers
+ai-crawler-radar audit github.com
+
+# Search Cloudflare Radar Bot Directory (758 verified bots)
+ai-crawler-radar bots --query openai
+ai-crawler-radar bots --category AI_SEARCH
+
+# Validate or generate CommonMark V2 /llms.txt
+ai-crawler-radar llmstxt https://fastmcp.org
+ai-crawler-radar llmstxt example.com --generate
+
+# Output pure JSON for automated CI/CD pipelines
+ai-crawler-radar probe https://example.com --json
+```
+
+---
+
+## 🔬 Diagnostic Engine Architecture
+
+1. **WAF Hard-Blocks**: Identifies Cloudflare (`CF-Ray`, Error 1020), AWS WAF, Akamai Ghost, and Imperva access blocks.
+2. **Turnstile & Challenges**: Detects Cloudflare Turnstile widgets, managed JS puzzles, and interactive CAPTCHAs that halt crawlers.
+3. **CSR Empty Shells**: Flags single-page apps (`<div id="root">`, `<div id="app">`) where dynamic JavaScript rendering leaves initial HTML blank for crawlers.
+4. **Token Bloat Multiplier**: Computes ratio of raw DOM tokens to extracted readable content tokens (e.g. 35x bloat) to measure LLM ingestion efficiency.
+5. **Content Negotiation & Cache Poisoning**: Detects missing `Vary: Accept` headers on markdown twin responses that cause CDN cache collisions.
+
+---
+
+## 🔒 Telemetry & Privacy
+
+Data never leaves your computer. In both local and remote variants, private target data is never stored on external servers.
+
+To opt out of anonymous usage telemetry:
+```bash
+export DO_NOT_TRACK=1
+# or
+export AI_CRAWLER_RADAR_TELEMETRY=off
+```
+
+---
+
+## 📄 License
+
+MIT License. Free and open source forever for all developers and AI agents.

ai_crawler_radar-0.1.0/SECURITY.md

@@ -0,0 +1,17 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.1.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability within `ai-crawler-radar`, please report it privately:
+
+1. **Email**: reachsuren@gmail.com
+2. Include reproduction steps, relevant URL payloads, and environment details.
+3. Please do not open public GitHub issues for security vulnerabilities.
+
+We take security seriously and will acknowledge reports within 48 hours.

ai_crawler_radar-0.1.0/pyproject.toml

@@ -0,0 +1,57 @@
+[project]
+name = "ai-crawler-radar"
+version = "0.1.0"
+description = "Autonomous AI Search, Training & Agent Crawler Diagnostic Engine"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [{ name = "Surendran B", email = "reachsuren@gmail.com" }]
+keywords = ["mcp", "ai-crawler", "llms-txt", "robots-txt", "turnstile", "waf", "cloudflare-radar", "searchgpt", "perplexity"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+  "mcp>=2.0.0,<3",
+  "httpx>=0.27.0",
+]
+
+[project.optional-dependencies]
+test = [
+  "pytest>=8.0.0",
+  "pytest-asyncio>=0.23.0",
+]
+publish = [
+  "build>=1.2.2",
+  "twine>=5.1.1",
+]
+
+[project.urls]
+Homepage = "https://github.com/surendranb/ai-crawler-radar"
+Repository = "https://github.com/surendranb/ai-crawler-radar"
+Issues = "https://github.com/surendranb/ai-crawler-radar/issues"
+
+[project.scripts]
+ai-crawler-radar = "ai_crawler_radar.cli:main"
+ai-crawler-radar-mcp = "ai_crawler_radar.server:main"
+
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+ai_crawler_radar = ["data/*.json"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

ai_crawler_radar-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ai_crawler_radar-0.1.0/skills/ai_crawler_radar_skill.md

@@ -0,0 +1,100 @@
+---
+name: ai-crawler-radar
+description: Operational playbook for auditing and optimizing websites for AI search engines, training scrapers, and autonomous agents.
+---
+
+# AI Crawler Radar Operational Playbook
+
+## 1. Executive Overview
+
+Modern AI engines interact with web properties across three distinct operational tiers:
+
+1. **Tier 1: Model Pre-Training Crawlers** (`GPTBot`, `ClaudeBot`, `Google-Extended`, `Meta-ExternalAgent`, `Bytespider`, `CCBot`)
+   - High volume batch scrapers ingesting entire domains to train next-generation foundation models.
+   - Typical business decision: Many publishers block this tier via `robots.txt` to protect proprietary intellectual property from uncompensated training.
+
+2. **Tier 2: Real-Time AI Search Indexers** (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `Applebot`, `Bingbot`)
+   - Targeted crawlers refreshing indices for immediate search grounding (ChatGPT Search, Perplexity Search, Siri Intelligence).
+   - Typical business decision: Publishers want this tier **allowed** to drive citations, referral traffic, and brand visibility in conversational search.
+
+3. **Tier 3: Live User Fetchers** (`ChatGPT-User`, `Claude-User`, `Perplexity-User`)
+   - Interactive retrieval triggered in real-time when an end-user pastes a link or asks an agent to browse a specific URL.
+   - Typical business decision: Should be permitted unless the page is behind authentication.
+
+---
+
+## 2. The 5 Root-Cause Failure Modes
+
+When an AI engine fails to cite or browse a page, it is rarely a model hallucination. It is almost always one of these 5 infrastructure flaws:
+
+### Failure Mode 1: WAF Hard-Block (Cloudflare, AWS WAF, Akamai)
+- **Symptom**: HTTP 403 Forbidden with `cf-ray`, `Error 1020`, or AWS WAF headers.
+- **Root Cause**: Edge firewall has default bot-management rules active that block non-browser ASNs or synthetic User-Agents.
+- **Remediation**: Create a WAF Custom Exception Rule allowing verified bots matching `OAI-SearchBot` or `PerplexityBot`.
+
+### Failure Mode 2: Turnstile / JavaScript Soft-Challenge
+- **Symptom**: HTTP 200 or 403 returning `<div class="cf-turnstile">` or Cloudflare challenge interstitial (`Just a moment...`).
+- **Root Cause**: Turnstile or JS challenge intercepts requests before server content renders. AI bots do not execute interactive CAPTCHAs.
+- **Remediation**: Exclude AI search user agents from interactive challenge actions.
+
+### Failure Mode 3: CSR Empty Shell (`<div id="root">`)
+- **Symptom**: Clean 200 OK response, but visible content length is < 150 characters, accompanied by heavy JavaScript script bundles.
+- **Root Cause**: Pure Client-Side Rendering without Server-Side Rendering (SSR) or Static Site Generation (SSG). AI search bots cannot parse dynamic single-page applications efficiently.
+- **Remediation**: Implement SSG/SSR or publish pre-rendered Markdown equivalents.
+
+### Failure Mode 4: Severe Token Bloat Multiplier
+- **Symptom**: 25,000 raw HTML tokens yielding only 600 tokens of substantive reading content (> 40x bloat).
+- **Root Cause**: Massive inline SVG icons, inline JSON data blobs, mega-navigation headers, and nested un-semantic `<div>` wrappers.
+- **Remediation**: Support content negotiation (`Accept: text/markdown`) or deploy `/llms.txt`.
+
+### Failure Mode 5: Missing `Vary: Accept`
+- **Symptom**: Content negotiation works locally, but production users intermittently receive cached HTML when requesting Markdown.
+- **Root Cause**: Edge CDN caches do not vary cache keys on the `Accept` header.
+- **Remediation**: Ensure upstream origin sends `Vary: Accept` whenever serving differentiated responses.
+
+---
+
+## 3. Recommended robots.txt Configuration
+
+To allow AI search discovery while protecting proprietary data from foundation model training:
+
+```txt
+# Block AI model pre-training & data harvesting
+User-agent: GPTBot
+Disallow: /
+
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: Meta-ExternalAgent
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: CCBot
+Disallow: /
+
+# Allow AI Search & Conversational Discovery
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Claude-SearchBot
+Allow: /
+
+User-agent: Applebot
+Allow: /
+
+# Standard baseline
+User-agent: *
+Disallow: /admin/
+Disallow: /private/
+
+Sitemap: https://yourdomain.com/sitemap.xml
+```

ai_crawler_radar-0.1.0/src/ai_crawler_radar/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: MIT
+"""ai-crawler-radar: Autonomous AI Search, Training & Agent Crawler Diagnostic Engine."""
+
+__version__ = "0.1.0"