llm-crawler-audit 0.1.0__tar.gz

llm_crawler_audit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Surendran B
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_crawler_audit-0.1.0/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: llm-crawler-audit
+Version: 0.1.0
+Summary: AI & LLM crawler readiness MCP server: audit robots.txt, validate llms.txt, and verify markdown mirrors for GEO/AEO optimization.
+Author-email: Surendran B <reachsuren@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://llm-crawler.builditwithai.xyz
+Project-URL: Repository, https://github.com/surendranb/llm-crawler-audit.git
+Project-URL: Issues, https://github.com/surendranb/llm-crawler-audit/issues
+Keywords: mcp,mcp-server,llm-crawler,robots-txt,llms-txt,geo,aeo,ai-search,gptbot,claudebot,perplexitybot
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp>=1.3.0
+Provides-Extra: test
+Requires-Dist: pytest>=8.0.0; extra == "test"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
+Dynamic: license-file
+
+# LLM Crawler & AI Readiness Audit MCP Server
+
+Production Model Context Protocol (MCP) server for evaluating, diagnosing, and optimizing websites for AI search engines, answer bots, and LLM crawlers (GEO/AEO).
+
+[![PyPI version](https://img.shields.io/pypi/v/llm-crawler-audit.svg)](https://pypi.org/project/llm-crawler-audit/)
+[![npm version](https://img.shields.io/npm/v/llm-crawler-audit.svg)](https://www.npmjs.com/package/llm-crawler-audit)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+Designed for Claude Code, Cursor, and agentic workflows to diagnose crawler accessibility, validate structured `/llms.txt` knowledge bases, and verify markdown mirror content negotiation.
+
+---
+
+## Capabilities & Tools
+
+1. **`audit_robots_for_ai(url: str)`**: Fetches `/robots.txt` and simulates RFC 9309 crawler permissions across 9 AI user-agents: `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `anthropic-ai`, `PerplexityBot`, `Applebot-Extended`, `Bytespider`, `Google-Extended`, and `CCBot`. Returns allow/disallow/unspecified status per agent, crawl delay, sitemaps, and citation impact.
+2. **`validate_llms_txt(url: str)`**: Fetches and verifies `/llms.txt` and `/llms-full.txt` compliance against the [llmstxt.org](https://llmstxt.org) standard (H1 title, summary blockquote, link structure, file size, token estimates, and compliance score).
+3. **`audit_markdown_mirror(url: str)`**: Checks for `<link rel="alternate" type="text/markdown">` in HTML `<head>`, parses HTTP `Link` headers, and verifies `Accept: text/markdown` content negotiation with `Vary: Accept` caching.
+4. **`audit_full_ai_readiness(url: str)`**: Computes a composite 0-100 GEO/AEO readiness score across robots permissions (35%), llms.txt compliance (35%), and markdown mirror parity (30%) with a prioritized action punchlist.
+5. **`doctor()`**: Runs diagnostics on network resolution, environment variables, local skills, and testbeds (`surendranb.com`, `builditwithai.xyz`).
+6. **`skills_list()` / `skill_read(name: str)`**: Access dynamic workflow playbooks (`ai-robots-txt-playbook`, `llms-txt-authoring`, `markdown-mirror-setup`) loaded from GitHub raw with bundled local fallback.
+7. **`check_for_updates()`**: Verify against PyPI for newer releases.
+
+---
+
+## Installation
+
+### Python (`uvx` - Recommended)
+
+```bash
+uvx llm-crawler-audit
+```
+
+### Node / npm (`npx`)
+
+```bash
+npx llm-crawler-audit
+```
+
+### Direct Pip
+
+```bash
+pip install llm-crawler-audit
+llm-crawler-audit
+```
+
+---
+
+## Client Configuration
+
+### Claude Desktop
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "llm-crawler-audit": {
+      "command": "uvx",
+      "args": ["llm-crawler-audit"]
+    }
+  }
+}
+```
+
+### Cursor IDE
+
+Add under **Settings > Features > MCP**:
+
+- **Name**: `llm-crawler-audit`
+- **Type**: `command`
+- **Command**: `uvx llm-crawler-audit`
+
+---
+
+## Error Classification & Absorption
+
+All errors returned by tools are classified into actionable envelopes:
+- `[INPUT_FIXABLE]`: Malformed URLs, missing schemes, or empty parameters.
+- `[ENVIRONMENT_FIXABLE: STOP & ASK HUMAN]`: Network proxy blocks, blocked outbound ports, or local filesystem permission issues.
+- `[TRANSIENT]`: DNS timeouts, SSL handshake failures, or upstream 503/504 errors safe for agent retries.
+
+---
+
+## Telemetry & Privacy
+
+This package includes anonymous usage telemetry (Schema v2) to track tool execution reliability, latency, and error rates. No secret keys, authorization tokens, or private URLs are ever transmitted.
+
+To opt out, set either environment variable:
+```bash
+export DO_NOT_TRACK=1
+# or
+export MCP_TELEMETRY_OPT_OUT=1
+```
+
+---
+
+## License
+
+MIT © [Surendran B](https://surendranb.com)

llm_crawler_audit-0.1.0/README.md

@@ -0,0 +1,99 @@
+# LLM Crawler & AI Readiness Audit MCP Server
+
+Production Model Context Protocol (MCP) server for evaluating, diagnosing, and optimizing websites for AI search engines, answer bots, and LLM crawlers (GEO/AEO).
+
+[![PyPI version](https://img.shields.io/pypi/v/llm-crawler-audit.svg)](https://pypi.org/project/llm-crawler-audit/)
+[![npm version](https://img.shields.io/npm/v/llm-crawler-audit.svg)](https://www.npmjs.com/package/llm-crawler-audit)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+Designed for Claude Code, Cursor, and agentic workflows to diagnose crawler accessibility, validate structured `/llms.txt` knowledge bases, and verify markdown mirror content negotiation.
+
+---
+
+## Capabilities & Tools
+
+1. **`audit_robots_for_ai(url: str)`**: Fetches `/robots.txt` and simulates RFC 9309 crawler permissions across 9 AI user-agents: `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `anthropic-ai`, `PerplexityBot`, `Applebot-Extended`, `Bytespider`, `Google-Extended`, and `CCBot`. Returns allow/disallow/unspecified status per agent, crawl delay, sitemaps, and citation impact.
+2. **`validate_llms_txt(url: str)`**: Fetches and verifies `/llms.txt` and `/llms-full.txt` compliance against the [llmstxt.org](https://llmstxt.org) standard (H1 title, summary blockquote, link structure, file size, token estimates, and compliance score).
+3. **`audit_markdown_mirror(url: str)`**: Checks for `<link rel="alternate" type="text/markdown">` in HTML `<head>`, parses HTTP `Link` headers, and verifies `Accept: text/markdown` content negotiation with `Vary: Accept` caching.
+4. **`audit_full_ai_readiness(url: str)`**: Computes a composite 0-100 GEO/AEO readiness score across robots permissions (35%), llms.txt compliance (35%), and markdown mirror parity (30%) with a prioritized action punchlist.
+5. **`doctor()`**: Runs diagnostics on network resolution, environment variables, local skills, and testbeds (`surendranb.com`, `builditwithai.xyz`).
+6. **`skills_list()` / `skill_read(name: str)`**: Access dynamic workflow playbooks (`ai-robots-txt-playbook`, `llms-txt-authoring`, `markdown-mirror-setup`) loaded from GitHub raw with bundled local fallback.
+7. **`check_for_updates()`**: Verify against PyPI for newer releases.
+
+---
+
+## Installation
+
+### Python (`uvx` - Recommended)
+
+```bash
+uvx llm-crawler-audit
+```
+
+### Node / npm (`npx`)
+
+```bash
+npx llm-crawler-audit
+```
+
+### Direct Pip
+
+```bash
+pip install llm-crawler-audit
+llm-crawler-audit
+```
+
+---
+
+## Client Configuration
+
+### Claude Desktop
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "llm-crawler-audit": {
+      "command": "uvx",
+      "args": ["llm-crawler-audit"]
+    }
+  }
+}
+```
+
+### Cursor IDE
+
+Add under **Settings > Features > MCP**:
+
+- **Name**: `llm-crawler-audit`
+- **Type**: `command`
+- **Command**: `uvx llm-crawler-audit`
+
+---
+
+## Error Classification & Absorption
+
+All errors returned by tools are classified into actionable envelopes:
+- `[INPUT_FIXABLE]`: Malformed URLs, missing schemes, or empty parameters.
+- `[ENVIRONMENT_FIXABLE: STOP & ASK HUMAN]`: Network proxy blocks, blocked outbound ports, or local filesystem permission issues.
+- `[TRANSIENT]`: DNS timeouts, SSL handshake failures, or upstream 503/504 errors safe for agent retries.
+
+---
+
+## Telemetry & Privacy
+
+This package includes anonymous usage telemetry (Schema v2) to track tool execution reliability, latency, and error rates. No secret keys, authorization tokens, or private URLs are ever transmitted.
+
+To opt out, set either environment variable:
+```bash
+export DO_NOT_TRACK=1
+# or
+export MCP_TELEMETRY_OPT_OUT=1
+```
+
+---
+
+## License
+
+MIT © [Surendran B](https://surendranb.com)

llm_crawler_audit-0.1.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "llm-crawler-audit"
+version = "0.1.0"
+description = "AI & LLM crawler readiness MCP server: audit robots.txt, validate llms.txt, and verify markdown mirrors for GEO/AEO optimization."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [{ name = "Surendran B", email = "reachsuren@gmail.com" }]
+keywords = ["mcp", "mcp-server", "llm-crawler", "robots-txt", "llms-txt", "geo", "aeo", "ai-search", "gptbot", "claudebot", "perplexitybot"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "mcp>=1.3.0",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.23.0",
+]
+
+[project.urls]
+Homepage = "https://llm-crawler.builditwithai.xyz"
+Repository = "https://github.com/surendranb/llm-crawler-audit.git"
+Issues = "https://github.com/surendranb/llm-crawler-audit/issues"
+
+[project.scripts]
+llm-crawler-audit = "llm_crawler_audit.server:main"
+llm-crawler-mcp = "llm_crawler_audit.server:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+llm_crawler_audit = ["skills/*.md"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

llm_crawler_audit-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm_crawler_audit-0.1.0/src/llm_crawler_audit/__init__.py

@@ -0,0 +1,3 @@
+"""llm-crawler-audit: AI & LLM crawler readiness MCP server."""
+
+__version__ = "0.1.0"

llm_crawler_audit-0.1.0/src/llm_crawler_audit/crawler_rules.py

@@ -0,0 +1,356 @@
+# SPDX-License-Identifier: MIT
+"""Core engine for RFC 9309 simulation, llms.txt verification, and markdown mirror auditing."""
+from __future__ import annotations
+
+import re
+import urllib.parse
+from html.parser import HTMLParser
+from typing import Any, Dict, List, Optional, Tuple
+
+AI_USER_AGENTS: List[Dict[str, str]] = [
+    {
+        "name": "GPTBot",
+        "operator": "OpenAI",
+        "category": "training_and_search",
+        "description": "OpenAI crawler for model training and ChatGPT Search indexing",
+    },
+    {
+        "name": "ChatGPT-User",
+        "operator": "OpenAI",
+        "category": "live_search",
+        "description": "User-directed live browsing & real-time search citations",
+    },
+    {
+        "name": "ClaudeBot",
+        "operator": "Anthropic",
+        "category": "live_search",
+        "description": "Anthropic crawler for Claude products and search citations",
+    },
+    {
+        "name": "anthropic-ai",
+        "operator": "Anthropic",
+        "category": "training",
+        "description": "Anthropic model training and corpus crawler",
+    },
+    {
+        "name": "PerplexityBot",
+        "operator": "Perplexity AI",
+        "category": "live_search",
+        "description": "Perplexity AI real-time search indexer and source citation bot",
+    },
+    {
+        "name": "Applebot-Extended",
+        "operator": "Apple",
+        "category": "training",
+        "description": "Apple Intelligence training and Siri answer ingestion",
+    },
+    {
+        "name": "Bytespider",
+        "operator": "ByteDance",
+        "category": "training",
+        "description": "ByteDance TikTok/Douyin LLM training and search crawler",
+    },
+    {
+        "name": "Google-Extended",
+        "operator": "Google",
+        "category": "training",
+        "description": "Google Gemini and Vertex AI training crawler",
+    },
+    {
+        "name": "CCBot",
+        "operator": "Common Crawl",
+        "category": "training",
+        "description": "Common Crawl open web dataset used across open-weights models",
+    },
+]
+
+class RobotsGroup:
+    def __init__(self):
+        self.user_agents: List[str] = []
+        self.rules: List[Tuple[str, str]] = []  # ('allow' | 'disallow', path)
+        self.crawl_delay: Optional[float] = None
+
+class RobotsTxt:
+    """RFC 9309 compliant parser and crawler permission simulator."""
+
+    def __init__(self, raw_text: str):
+        self.raw_text = raw_text
+        self.groups: List[RobotsGroup] = []
+        self.sitemaps: List[str] = []
+        self.llm_directives: List[str] = []
+        self._parse()
+
+    def _parse(self):
+        current_group: Optional[RobotsGroup] = None
+        in_directive_section = False
+
+        for raw_line in self.raw_text.splitlines():
+            line = raw_line.strip()
+            # Catch comments that might reference llms.txt or AI policies
+            if line.startswith("#"):
+                comment = line[1:].strip()
+                if any(k in comment.lower() for k in ("llm", "ai", "llms.txt")):
+                    self.llm_directives.append(comment)
+                continue
+
+            # Strip inline comments
+            if "#" in line:
+                line = line.split("#", 1)[0].strip()
+
+            if not line:
+                if current_group and current_group.user_agents and current_group.rules:
+                    self.groups.append(current_group)
+                    current_group = None
+                    in_directive_section = False
+                continue
+
+            if ":" not in line:
+                continue
+
+            field, value = line.split(":", 1)
+            field = field.strip().lower()
+            value = value.strip()
+
+            if field == "user-agent":
+                if in_directive_section or current_group is None:
+                    if current_group and current_group.user_agents and (current_group.rules or current_group.crawl_delay is not None):
+                        self.groups.append(current_group)
+                    current_group = RobotsGroup()
+                    in_directive_section = False
+                current_group.user_agents.append(value)
+            elif field in ("allow", "disallow"):
+                if current_group is None:
+                    continue
+                in_directive_section = True
+                current_group.rules.append((field, value))
+            elif field == "crawl-delay":
+                if current_group is None:
+                    continue
+                in_directive_section = True
+                try:
+                    current_group.crawl_delay = float(value)
+                except ValueError:
+                    pass
+            elif field == "sitemap":
+                if value and value not in self.sitemaps:
+                    self.sitemaps.append(value)
+
+        if current_group and current_group.user_agents:
+            self.groups.append(current_group)
+
+    @staticmethod
+    def _path_matches(pattern: str, path: str) -> bool:
+        if not pattern:
+            return False
+        # RFC 9309 pattern matching: * matches anything, $ matches end
+        regex_pattern = "^"
+        i = 0
+        while i < len(pattern):
+            c = pattern[i]
+            if c == "*":
+                regex_pattern += ".*"
+            elif c == "$":
+                if i == len(pattern) - 1:
+                    regex_pattern += "$"
+                else:
+                    regex_pattern += re.escape(c)
+            else:
+                regex_pattern += re.escape(c)
+            i += 1
+        return re.match(regex_pattern, path) is not None
+
+    def simulate_agent(self, agent_name: str, path: str = "/") -> Dict[str, Any]:
+        """Simulate crawler permissions for a specific agent under RFC 9309 rules."""
+        target_group: Optional[RobotsGroup] = None
+        matched_token: Optional[str] = None
+
+        # 1. Look for specific user-agent group (case-insensitive)
+        for group in self.groups:
+            for ua in group.user_agents:
+                if ua.lower() == agent_name.lower():
+                    target_group = group
+                    matched_token = ua
+                    break
+            if target_group:
+                break
+
+        # 2. Fall back to wildcard '*' group if no specific group matched
+        if not target_group:
+            for group in self.groups:
+                for ua in group.user_agents:
+                    if ua == "*":
+                        target_group = group
+                        matched_token = "*"
+                        break
+                if target_group:
+                    break
+
+        # 3. If no matching group, default is ALLOW unspecified
+        if not target_group:
+            return {
+                "agent": agent_name,
+                "status": "allowed",
+                "matched_group": "none",
+                "rule_type": "default_allow",
+                "rule_pattern": None,
+                "crawl_delay": None,
+            }
+
+        # 4. Longest match rule in RFC 9309 section 2.2.2
+        best_rule_type: Optional[str] = None
+        best_pattern: Optional[str] = None
+        best_len = -1
+
+        for rule_type, pattern in target_group.rules:
+            # Empty disallow means allow all
+            if rule_type == "disallow" and not pattern:
+                rule_type = "allow"
+                pattern = "/"
+
+            if self._path_matches(pattern, path):
+                pat_len = len(pattern)
+                if pat_len > best_len:
+                    best_len = pat_len
+                    best_rule_type = rule_type
+                    best_pattern = pattern
+                elif pat_len == best_len and rule_type == "allow" and best_rule_type == "disallow":
+                    # Tie-breaker: Allow wins
+                    best_rule_type = "allow"
+                    best_pattern = pattern
+
+        if best_rule_type == "disallow":
+            status = "disallowed"
+        else:
+            status = "allowed"
+
+        return {
+            "agent": agent_name,
+            "status": status,
+            "matched_group": matched_token,
+            "rule_type": best_rule_type or "implicit_allow",
+            "rule_pattern": best_pattern,
+            "crawl_delay": target_group.crawl_delay,
+        }
+
+class HeadLinkParser(HTMLParser):
+    """Extracts alternate link tags from HTML head."""
+
+    def __init__(self):
+        super().__init__()
+        self.in_head = False
+        self.alternate_markdown_links: List[str] = []
+        self.describedby_links: List[str] = []
+
+    def handle_starttag(self, tag: str, attrs: List[Tuple[str, Optional[str]]]):
+        ltag = tag.lower()
+        if ltag == "head":
+            self.in_head = True
+        elif ltag == "link":
+            attr_dict = {k.lower(): (v or "") for k, v in attrs}
+            rel = attr_dict.get("rel", "").lower()
+            href = attr_dict.get("href", "")
+            type_attr = attr_dict.get("type", "").lower()
+
+            if "alternate" in rel and type_attr in ("text/markdown", "text/x-markdown"):
+                if href and href not in self.alternate_markdown_links:
+                    self.alternate_markdown_links.append(href)
+            if "describedby" in rel:
+                if href and href not in self.describedby_links:
+                    self.describedby_links.append(href)
+
+    def handle_endtag(self, tag: str):
+        if tag.lower() == "head":
+            self.in_head = False
+
+def parse_link_headers(link_header: str) -> List[Dict[str, str]]:
+    """Parse HTTP Link header values (RFC 8288)."""
+    results: List[Dict[str, str]] = []
+    if not link_header:
+        return results
+
+    parts = link_header.split(",")
+    for part in parts:
+        segments = part.strip().split(";")
+        if not segments:
+            continue
+        url_match = re.match(r"<([^>]+)>", segments[0].strip())
+        if not url_match:
+            continue
+        target_url = url_match.group(1)
+        params: Dict[str, str] = {"url": target_url}
+        for seg in segments[1:]:
+            if "=" in seg:
+                k, v = seg.strip().split("=", 1)
+                params[k.strip().lower()] = v.strip().strip('"\'')
+        results.append(params)
+    return results
+
+def validate_llms_txt_content(content: str) -> Dict[str, Any]:
+    """Validate structure, H1, blockquote, and links according to llmstxt.org."""
+    lines = [line.strip() for line in content.splitlines()]
+    non_empty = [line for line in lines if line]
+
+    h1_title: Optional[str] = None
+    summary_block: Optional[str] = None
+    sections: List[str] = []
+    links: List[Dict[str, str]] = []
+
+    # First non-empty line should be H1
+    if non_empty and non_empty[0].startswith("# "):
+        h1_title = non_empty[0][2:].strip()
+
+    # Look for blockquote summary (> ...)
+    for line in non_empty[1:6]:
+        if line.startswith(">"):
+            summary_block = line.lstrip("> ").strip()
+            break
+
+    # Extract sections and structured markdown links
+    link_regex = re.compile(r"-\s*\[([^\]]+)\]\(([^)]+)\)(?::\s*(.*))?")
+    for line in lines:
+        if line.startswith("## "):
+            sections.append(line[3:].strip())
+        else:
+            m = link_regex.match(line)
+            if m:
+                links.append({
+                    "title": m.group(1).strip(),
+                    "url": m.group(2).strip(),
+                    "description": (m.group(3) or "").strip(),
+                })
+
+    has_h1 = bool(h1_title)
+    has_summary = bool(summary_block)
+    has_links = len(links) > 0
+
+    # Calculate compliance score
+    score = 0
+    if len(content) > 10:
+        score += 40
+    if has_h1:
+        score += 20
+    if has_summary:
+        score += 20
+    if has_links:
+        score += 20
+
+    warnings: List[str] = []
+    if not has_h1:
+        warnings.append("Missing top-level H1 header (# Title) on first non-empty line")
+    if not has_summary:
+        warnings.append("Missing blockquote summary (> Description) directly beneath H1")
+    if not has_links:
+        warnings.append("No structured markdown links (- [Title](url): Description) discovered")
+
+    return {
+        "valid": has_h1 and has_summary,
+        "compliance_score": score,
+        "title": h1_title,
+        "summary": summary_block,
+        "sections": sections,
+        "links_count": len(links),
+        "links": links[:10],
+        "character_count": len(content),
+        "estimated_tokens": max(1, round(len(content) / 4)),
+        "warnings": warnings,
+    }