aivisible-fix 0.2.0__tar.gz

aivisible_fix-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 aivisible-fix contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aivisible_fix-0.2.0/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: aivisible-fix
+Version: 0.2.0
+Summary: Audit and auto-fix a website's visibility to AI crawlers and answer engines (GPTBot, ClaudeBot, PerplexityBot, and more).
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# aivisible-fix
+
+Audits a website for the specific things that make it invisible to AI
+crawlers and AI answer engines (ChatGPT, Claude, Perplexity, Gemini/AI
+Overviews, and others) — and, unlike the free "checker" tools already on
+the market, **generates the fix**, not just a score.
+
+```
+$ aivisible-fix audit https://example.com
+AI-visibility audit for https://example.com
+Score: 40/100
+
+  [FAIL] robots_reachable: robots.txt missing or unreachable ...
+  [PASS] no_ai_crawlers_blocked: No known AI crawler is explicitly blocked.
+  [FAIL] llms_txt_present: No /llms.txt ...
+  ...
+
+$ aivisible-fix fix https://example.com --name "Example Co" --description "..."
+Wrote robots.fixed.txt
+Wrote llms.txt
+Review both files, then deploy them at your site root. Nothing was auto-published.
+```
+
+## What it checks
+
+1. **robots.txt reachable** — the baseline signal every crawler reads first.
+2. **No AI crawler is explicitly blocked** — checked per-agent (GPTBot,
+   ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai,
+   PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider,
+   Amazonbot, Applebot-Extended, cohere-ai, Meta-ExternalAgent, Diffbot,
+   YouBot — see `aivisible_fix/crawlers.py`), not just a blanket `Disallow: /`.
+3. **llms.txt present** — the emerging convention (llmstxt.org) for a short,
+   LLM-readable summary of what a site is and which pages matter.
+4. **sitemap.xml present and valid**, and **referenced from robots.txt**.
+5. **JSON-LD structured data on the homepage** — what answer engines lean on
+   to extract facts (org, product, FAQ, etc.) without guessing from prose.
+
+Score is out of 100, weighted toward the two things that, per current
+published research, cause the overwhelming majority of AI-search invisibility:
+blocked crawlers (40 pts) and missing discovery trail — sitemap + llms.txt
+(40 pts combined).
+
+## Why this instead of the free checkers
+
+Search "AI visibility checker" and you'll find eight-plus free tools
+(isvisible.ai, llmpulse.ai, siftly.ai, and others) that already do the
+scoring for free — that part of the market is a commodity and not worth
+competing on head to head.
+
+None of them **write the fix**. This tool's differentiator is that
+`aivisible-fix fix` outputs a ready-to-deploy patched `robots.txt` (every
+existing line preserved, blocked AI agents explicitly re-allowed) and a
+starter `llms.txt` — output, not just diagnosis. The natural extension
+(not built yet — see ROADMAP below) is a GitHub App that opens this as a
+pull request automatically, the way Dependabot does for dependency bumps,
+plus scheduled re-audits so a clean report doesn't silently rot as new
+crawlers ship.
+
+## Badge
+
+```
+$ aivisible-fix badge https://example.com
+Wrote aivisible-badge.svg (score 40/100). Host it yourself and embed:
+  <a href="https://example.com"><img src="/aivisible-badge.svg" alt="AI-visible: 40/100"></a>
+```
+
+Static SVG, generated locally — no server or account required. Color tracks
+score (red under 50, amber under 80, green at 80+).
+
+## Install
+
+```
+pip install -e .
+```
+
+No dependencies beyond the Python 3.10+ standard library — deliberately, so
+it's trivial to also ship as a single-file GitHub Action.
+
+## Status
+
+v0.1 — functional CLI, tested against live sites (see commit history / this
+was built and smoke-tested against example.com and nytimes.com, correctly
+detecting NYT's well-documented blanket block of AI crawlers). Not yet
+published to PyPI or packaged as a GitHub Action — see BUSINESS_PLAN.md for
+what's next and what needs a human to do it (PyPI account, GitHub App
+registration, a payment processor for the paid tier).

aivisible_fix-0.2.0/README.md

@@ -0,0 +1,86 @@
+# aivisible-fix
+
+Audits a website for the specific things that make it invisible to AI
+crawlers and AI answer engines (ChatGPT, Claude, Perplexity, Gemini/AI
+Overviews, and others) — and, unlike the free "checker" tools already on
+the market, **generates the fix**, not just a score.
+
+```
+$ aivisible-fix audit https://example.com
+AI-visibility audit for https://example.com
+Score: 40/100
+
+  [FAIL] robots_reachable: robots.txt missing or unreachable ...
+  [PASS] no_ai_crawlers_blocked: No known AI crawler is explicitly blocked.
+  [FAIL] llms_txt_present: No /llms.txt ...
+  ...
+
+$ aivisible-fix fix https://example.com --name "Example Co" --description "..."
+Wrote robots.fixed.txt
+Wrote llms.txt
+Review both files, then deploy them at your site root. Nothing was auto-published.
+```
+
+## What it checks
+
+1. **robots.txt reachable** — the baseline signal every crawler reads first.
+2. **No AI crawler is explicitly blocked** — checked per-agent (GPTBot,
+   ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai,
+   PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider,
+   Amazonbot, Applebot-Extended, cohere-ai, Meta-ExternalAgent, Diffbot,
+   YouBot — see `aivisible_fix/crawlers.py`), not just a blanket `Disallow: /`.
+3. **llms.txt present** — the emerging convention (llmstxt.org) for a short,
+   LLM-readable summary of what a site is and which pages matter.
+4. **sitemap.xml present and valid**, and **referenced from robots.txt**.
+5. **JSON-LD structured data on the homepage** — what answer engines lean on
+   to extract facts (org, product, FAQ, etc.) without guessing from prose.
+
+Score is out of 100, weighted toward the two things that, per current
+published research, cause the overwhelming majority of AI-search invisibility:
+blocked crawlers (40 pts) and missing discovery trail — sitemap + llms.txt
+(40 pts combined).
+
+## Why this instead of the free checkers
+
+Search "AI visibility checker" and you'll find eight-plus free tools
+(isvisible.ai, llmpulse.ai, siftly.ai, and others) that already do the
+scoring for free — that part of the market is a commodity and not worth
+competing on head to head.
+
+None of them **write the fix**. This tool's differentiator is that
+`aivisible-fix fix` outputs a ready-to-deploy patched `robots.txt` (every
+existing line preserved, blocked AI agents explicitly re-allowed) and a
+starter `llms.txt` — output, not just diagnosis. The natural extension
+(not built yet — see ROADMAP below) is a GitHub App that opens this as a
+pull request automatically, the way Dependabot does for dependency bumps,
+plus scheduled re-audits so a clean report doesn't silently rot as new
+crawlers ship.
+
+## Badge
+
+```
+$ aivisible-fix badge https://example.com
+Wrote aivisible-badge.svg (score 40/100). Host it yourself and embed:
+  <a href="https://example.com"><img src="/aivisible-badge.svg" alt="AI-visible: 40/100"></a>
+```
+
+Static SVG, generated locally — no server or account required. Color tracks
+score (red under 50, amber under 80, green at 80+).
+
+## Install
+
+```
+pip install -e .
+```
+
+No dependencies beyond the Python 3.10+ standard library — deliberately, so
+it's trivial to also ship as a single-file GitHub Action.
+
+## Status
+
+v0.1 — functional CLI, tested against live sites (see commit history / this
+was built and smoke-tested against example.com and nytimes.com, correctly
+detecting NYT's well-documented blanket block of AI crawlers). Not yet
+published to PyPI or packaged as a GitHub Action — see BUSINESS_PLAN.md for
+what's next and what needs a human to do it (PyPI account, GitHub App
+registration, a payment processor for the paid tier).

aivisible_fix-0.2.0/aivisible_fix/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

aivisible_fix-0.2.0/aivisible_fix/audit.py

@@ -0,0 +1,110 @@
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from urllib.parse import urljoin
+
+from .crawlers import AI_CRAWLERS
+from .fetch import fetch
+from .robots import agent_status, parse_robots_txt
+
+# Each check's point value. Weighted toward the two things that, per current
+# published data, cause the overwhelming majority of AI-search invisibility:
+# blocked crawlers and a missing sitemap/structured data trail to follow.
+POINTS = {
+    "robots_reachable": 5,
+    "no_ai_crawlers_blocked": 40,
+    "llms_txt_present": 20,
+    "sitemap_present": 15,
+    "sitemap_referenced_in_robots": 5,
+    "structured_data_present": 15,
+}
+
+
+@dataclass
+class Finding:
+    check: str
+    passed: bool
+    points: int
+    detail: str
+
+
+@dataclass
+class AuditReport:
+    base_url: str
+    findings: list[Finding] = field(default_factory=list)
+    blocked_agents: list[str] = field(default_factory=list)
+    score: int = 0
+
+    def add(self, check: str, passed: bool, detail: str) -> None:
+        points = POINTS[check] if passed else 0
+        self.findings.append(Finding(check, passed, points, detail))
+        self.score += points
+
+
+def run_audit(base_url: str) -> AuditReport:
+    base_url = base_url.rstrip("/")
+    if not base_url.lower().startswith("https://"):
+        raise ValueError(
+            f"Refusing to audit '{base_url}': only https:// URLs are supported. "
+            "This tool never fetches over plaintext HTTP, for both your safety "
+            "and the target site's."
+        )
+    report = AuditReport(base_url=base_url)
+
+    robots_res = fetch(urljoin(base_url + "/", "robots.txt"))
+    if not robots_res.ok:
+        report.add("robots_reachable", False, "robots.txt missing or unreachable — AI crawlers default to allowed, but you have no way to signal intent or steer them to a sitemap.")
+        groups = []
+    else:
+        report.add("robots_reachable", True, "robots.txt found.")
+        groups = parse_robots_txt(robots_res.text)
+
+    blocked = []
+    for agent in AI_CRAWLERS:
+        status = agent_status(groups, agent)
+        if status == "blocked":
+            blocked.append(agent)
+    report.blocked_agents = blocked
+    if blocked:
+        report.add(
+            "no_ai_crawlers_blocked",
+            False,
+            f"{len(blocked)} AI crawler(s) explicitly blocked: {', '.join(blocked)}.",
+        )
+    else:
+        report.add("no_ai_crawlers_blocked", True, "No known AI crawler is explicitly blocked.")
+
+    llms_res = fetch(urljoin(base_url + "/", "llms.txt"))
+    report.add(
+        "llms_txt_present",
+        llms_res.ok and len(llms_res.text.strip()) > 0,
+        "llms.txt found." if llms_res.ok else "No /llms.txt — the emerging convention for telling an LLM what your site is and which pages matter most.",
+    )
+
+    sitemap_url = None
+    m = re.search(r"(?im)^sitemap:\s*(\S+)", robots_res.text) if robots_res.ok else None
+    if m:
+        sitemap_url = m.group(1).strip()
+    sitemap_res = fetch(sitemap_url or urljoin(base_url + "/", "sitemap.xml"))
+    sitemap_looks_valid = sitemap_res.ok and ("<urlset" in sitemap_res.text or "<sitemapindex" in sitemap_res.text)
+    report.add(
+        "sitemap_present",
+        sitemap_looks_valid,
+        "sitemap.xml found and looks valid." if sitemap_looks_valid else "No reachable/valid sitemap.xml — AI crawlers (and search engines) have no efficient way to discover your full page list.",
+    )
+    report.add(
+        "sitemap_referenced_in_robots",
+        m is not None,
+        "robots.txt points crawlers at your sitemap." if m else "robots.txt doesn't reference a Sitemap: line, even if sitemap.xml exists at the default path.",
+    )
+
+    home_res = fetch(base_url + "/")
+    has_jsonld = home_res.ok and bool(re.search(r'<script[^>]+type=["\']application/ld\+json["\']', home_res.text, re.I))
+    report.add(
+        "structured_data_present",
+        has_jsonld,
+        "JSON-LD structured data found on the homepage." if has_jsonld else "No JSON-LD structured data on the homepage — AI answer engines lean on schema.org markup to extract facts (org name, product, FAQ, etc.) reliably.",
+    )
+
+    return report

aivisible_fix-0.2.0/aivisible_fix/badge.py

@@ -0,0 +1,50 @@
+"""Static SVG badge generator — a shields.io-style badge, generated locally
+and hosted by whoever runs the tool (no server of ours required). This is
+the zero-account distribution mechanic: a site that scores well has a
+concrete, embeddable reason to link back, the same way "Deploys on Vercel"
+or CI-passing badges spread.
+"""
+from __future__ import annotations
+
+LABEL = "AI-visible"
+
+
+def _color_for(score: int) -> str:
+    if score >= 80:
+        return "#2E8B57"  # allow-green
+    if score >= 50:
+        return "#E8A23A"  # amber
+    return "#C6432E"  # block-red
+
+
+def _text_width(text: str) -> int:
+    # Rough monospace-ish estimate — good enough for a badge, no font
+    # metrics library required (this stays stdlib-only by design).
+    return max(6, int(len(text) * 6.7)) + 10
+
+
+def generate_badge_svg(score: int) -> str:
+    value_text = f"{score}/100"
+    label_text = LABEL
+    label_w = _text_width(label_text)
+    value_w = _text_width(value_text)
+    total_w = label_w + value_w
+    color = _color_for(score)
+
+    return f'''<svg xmlns="http://www.w3.org/2000/svg" width="{total_w}" height="20" role="img" aria-label="{label_text}: {value_text}">
+  <linearGradient id="s" x2="0" y2="100%">
+    <stop offset="0" stop-color="#fff" stop-opacity=".08"/>
+    <stop offset="1" stop-opacity=".08"/>
+  </linearGradient>
+  <clipPath id="r"><rect width="{total_w}" height="20" rx="3" fill="#fff"/></clipPath>
+  <g clip-path="url(#r)">
+    <rect width="{label_w}" height="20" fill="#3a3f3c"/>
+    <rect x="{label_w}" width="{value_w}" height="20" fill="{color}"/>
+    <rect width="{total_w}" height="20" fill="url(#s)"/>
+  </g>
+  <g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
+    <text x="{label_w / 2}" y="14">{label_text}</text>
+    <text x="{label_w + value_w / 2}" y="14">{value_text}</text>
+  </g>
+</svg>
+'''

aivisible_fix-0.2.0/aivisible_fix/cli.py

@@ -0,0 +1,101 @@
+from __future__ import annotations
+
+import argparse
+import sys
+from urllib.parse import urljoin
+
+from .audit import run_audit
+from .badge import generate_badge_svg
+from .fetch import fetch
+from .fix import generate_llms_txt, generate_robots_patch
+
+
+def cmd_audit(args: argparse.Namespace) -> int:
+    try:
+        report = run_audit(args.url)
+    except ValueError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 2
+    print(f"\nAI-visibility audit for {report.base_url}")
+    print(f"Score: {report.score}/100\n")
+    for f in report.findings:
+        mark = "PASS" if f.passed else "FAIL"
+        print(f"  [{mark}] {f.check}: {f.detail}")
+    if report.blocked_agents:
+        print(f"\nBlocked crawlers: {', '.join(report.blocked_agents)}")
+    print()
+    return 0 if report.score >= 70 else 1
+
+
+def cmd_fix(args: argparse.Namespace) -> int:
+    try:
+        report = run_audit(args.url)
+    except ValueError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 2
+    robots_res = fetch(urljoin(report.base_url + "/", "robots.txt"))
+    existing = robots_res.text if robots_res.ok else ""
+
+    patched_robots = generate_robots_patch(existing, report)
+    robots_out = args.out_robots or "robots.fixed.txt"
+    with open(robots_out, "w") as fh:
+        fh.write(patched_robots)
+    print(f"Wrote {robots_out}")
+
+    if not any(f.check == "llms_txt_present" and f.passed for f in report.findings):
+        llms_out = args.out_llms or "llms.txt"
+        llms_txt = generate_llms_txt(
+            site_name=args.name or report.base_url,
+            site_description=args.description or f"{args.name or report.base_url} — description needed, edit this file.",
+            key_pages=[],
+        )
+        with open(llms_out, "w") as fh:
+            fh.write(llms_txt)
+        print(f"Wrote {llms_out} (starter — edit key pages in before publishing)")
+
+    print("\nReview both files, then deploy them at your site root. Nothing was auto-published.")
+    return 0
+
+
+def cmd_badge(args: argparse.Namespace) -> int:
+    try:
+        report = run_audit(args.url)
+    except ValueError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 2
+    svg = generate_badge_svg(report.score)
+    out = args.out or "aivisible-badge.svg"
+    with open(out, "w") as fh:
+        fh.write(svg)
+    print(f"Wrote {out} (score {report.score}/100). Host it yourself and embed:")
+    print(f'  <a href="https://example.com"><img src="/aivisible-badge.svg" alt="AI-visible: {report.score}/100"></a>')
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="aivisible-fix", description="Audit and fix a site's visibility to AI crawlers/answer engines.")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_audit = sub.add_parser("audit", help="Score a site's AI-crawler visibility")
+    p_audit.add_argument("url", help="e.g. https://example.com")
+    p_audit.set_defaults(func=cmd_audit)
+
+    p_fix = sub.add_parser("fix", help="Generate a patched robots.txt and starter llms.txt")
+    p_fix.add_argument("url")
+    p_fix.add_argument("--name", help="Site/company name for llms.txt")
+    p_fix.add_argument("--description", help="One-line description for llms.txt")
+    p_fix.add_argument("--out-robots", help="Output path for patched robots.txt")
+    p_fix.add_argument("--out-llms", help="Output path for generated llms.txt")
+    p_fix.set_defaults(func=cmd_fix)
+
+    p_badge = sub.add_parser("badge", help="Generate a static embeddable score badge (SVG)")
+    p_badge.add_argument("url")
+    p_badge.add_argument("--out", help="Output path for the badge SVG")
+    p_badge.set_defaults(func=cmd_badge)
+
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

aivisible_fix-0.2.0/aivisible_fix/crawlers.py

@@ -0,0 +1,24 @@
+# Known AI crawler / agent user-agent tokens, as used in robots.txt.
+# Kept as a single source of truth so audit.py and fix.py never drift apart.
+# Sources: each vendor's own published crawler docs (OpenAI, Anthropic,
+# Perplexity, Google, Common Crawl, ByteDance, Amazon, Apple, Cohere, Meta).
+
+AI_CRAWLERS = {
+    "GPTBot": "OpenAI — trains/improves models on crawled content",
+    "ChatGPT-User": "OpenAI — fetches pages a ChatGPT user links to",
+    "OAI-SearchBot": "OpenAI — powers ChatGPT search result citations",
+    "ClaudeBot": "Anthropic — trains/improves Claude on crawled content",
+    "Claude-Web": "Anthropic — fetches pages referenced in a Claude chat",
+    "anthropic-ai": "Anthropic — legacy training crawler token",
+    "PerplexityBot": "Perplexity — indexes content for AI search answers",
+    "Perplexity-User": "Perplexity — fetches pages a user asked about",
+    "Google-Extended": "Google — controls use in Gemini / AI Overviews (separate from Googlebot)",
+    "CCBot": "Common Crawl — public dataset used to train many LLMs",
+    "Bytespider": "ByteDance — crawls for AI training",
+    "Amazonbot": "Amazon — crawls for Alexa/AI features",
+    "Applebot-Extended": "Apple — controls use in Apple Intelligence (separate from Applebot)",
+    "cohere-ai": "Cohere — trains/improves models on crawled content",
+    "Meta-ExternalAgent": "Meta — crawls for AI training and search features",
+    "Diffbot": "Diffbot — powers third-party AI/knowledge-graph products",
+    "YouBot": "You.com — indexes content for AI search answers",
+}

aivisible_fix-0.2.0/aivisible_fix/fetch.py

@@ -0,0 +1,33 @@
+"""Tiny stdlib-only HTTP helper — no third-party deps required to run this tool."""
+from __future__ import annotations
+
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+
+USER_AGENT = "aivisible-fix/0.1 (+https://github.com/; audit tool, respects robots.txt itself)"
+
+
+@dataclass
+class FetchResult:
+    ok: bool
+    status: int | None
+    text: str
+
+
+def fetch(url: str, timeout: float = 10.0) -> FetchResult:
+    if not url.lower().startswith("https://"):
+        # Refuse plaintext HTTP: this tool fetches third-party sites and must
+        # not be usable to relay/downgrade requests over an unencrypted
+        # channel, and it never has a legitimate reason to fetch anything
+        # other than a public https:// URL.
+        return FetchResult(ok=False, status=None, text="")
+    req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            body = resp.read().decode("utf-8", errors="replace")
+            return FetchResult(ok=200 <= resp.status < 300, status=resp.status, text=body)
+    except urllib.error.HTTPError as e:
+        return FetchResult(ok=False, status=e.code, text="")
+    except (urllib.error.URLError, TimeoutError, OSError):
+        return FetchResult(ok=False, status=None, text="")

aivisible_fix-0.2.0/aivisible_fix/fix.py

@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+from .audit import AuditReport
+from .crawlers import AI_CRAWLERS
+
+
+def generate_robots_patch(existing_robots_text: str, report: AuditReport) -> str:
+    """Return a full replacement robots.txt: keeps every line the site already
+    had, and appends an explicit Allow-all group for each blocked AI crawler.
+    Non-destructive by construction — nothing existing is deleted or edited.
+    """
+    if not report.blocked_agents:
+        return existing_robots_text
+
+    lines = existing_robots_text.rstrip("\n").splitlines() if existing_robots_text.strip() else []
+    lines.append("")
+    lines.append("# --- Added by aivisible-fix: explicitly allow AI crawlers ---")
+    lines.append("# Remove any of these blocks if you deliberately don't want that")
+    lines.append("# crawler training on / citing your content.")
+    for agent in report.blocked_agents:
+        lines.append("")
+        lines.append(f"User-agent: {agent}")
+        lines.append("Allow: /")
+    return "\n".join(lines) + "\n"
+
+
+def generate_llms_txt(site_name: str, site_description: str, key_pages: list[tuple[str, str]]) -> str:
+    """key_pages: list of (title, url) tuples for the handful of pages worth
+    pointing an LLM at directly (docs, pricing, about) — per the llms.txt spec
+    (llmstxt.org): an H1, a one-line summary, then link sections.
+    """
+    lines = [f"# {site_name}", "", f"> {site_description}", ""]
+    if key_pages:
+        lines.append("## Key pages")
+        for title, url in key_pages:
+            lines.append(f"- [{title}]({url})")
+        lines.append("")
+    lines.append(
+        "<!-- Generated by aivisible-fix. Edit freely — this is a starting point, "
+        "not a final file. Keep it short: link out rather than duplicating content. -->"
+    )
+    return "\n".join(lines) + "\n"
+
+
+def suggest_jsonld_snippet(site_name: str, site_url: str, description: str) -> str:
+    return f"""<script type="application/ld+json">
+{{
+  "@context": "https://schema.org",
+  "@type": "Organization",
+  "name": "{site_name}",
+  "url": "{site_url}",
+  "description": "{description}"
+}}
+</script>"""
+
+
+def summarize_unknown_crawler_coverage() -> str:
+    return (
+        f"This tool checks {len(AI_CRAWLERS)} known AI crawler user-agents. "
+        "New ones ship periodically — re-run the audit monthly rather than treating "
+        "a clean report as permanent."
+    )

aivisible_fix-0.2.0/aivisible_fix/robots.py

@@ -0,0 +1,80 @@
+"""Minimal robots.txt parser that groups directives by user-agent.
+
+Deliberately not using urllib.robotparser: that API answers "can THIS
+one agent fetch THIS one URL", but an audit needs to enumerate, for every
+known AI crawler, whether it has its own group and whether that group (or
+the wildcard group) disallows the whole site.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Group:
+    agents: list[str] = field(default_factory=list)
+    disallow: list[str] = field(default_factory=list)
+    allow: list[str] = field(default_factory=list)
+
+
+def parse_robots_txt(text: str) -> list[Group]:
+    groups: list[Group] = []
+    current: Group | None = None
+    seen_directive_since_agent = True  # forces first "User-agent" to start a group
+
+    for raw_line in text.splitlines():
+        line = raw_line.split("#", 1)[0].strip()
+        if not line or ":" not in line:
+            continue
+        field_name, _, value = line.partition(":")
+        field_name = field_name.strip().lower()
+        value = value.strip()
+
+        if field_name == "user-agent":
+            if current is None or not seen_directive_since_agent:
+                current = Group()
+                groups.append(current)
+            current.agents.append(value)
+            seen_directive_since_agent = False
+        elif field_name == "disallow" and current is not None:
+            current.disallow.append(value)
+            seen_directive_since_agent = True
+        elif field_name == "allow" and current is not None:
+            current.allow.append(value)
+            seen_directive_since_agent = True
+        # sitemap / crawl-delay / other fields intentionally ignored here;
+        # sitemap presence is checked separately by audit.py.
+
+    return groups
+
+
+def blocks_everything(group: Group) -> bool:
+    """True if this group disallows the whole site and doesn't carve out
+    an override Allow for it (Allow: / or Allow: with empty/root path)."""
+    has_blanket_disallow = any(d.strip() == "/" for d in group.disallow)
+    if not has_blanket_disallow:
+        return False
+    has_override_allow = any(a.strip() == "/" for a in group.allow)
+    return not has_override_allow
+
+
+def agent_status(groups: list[Group], agent: str) -> str:
+    """Return 'blocked', 'allowed-explicit', or 'allowed-default' for one AI agent.
+
+    'allowed-default' means the site has no group naming this agent and no
+    wildcard (*) group blocking everything — the crawler is free to fetch
+    under robots.txt's default-allow behaviour, but the site owner never
+    made an explicit decision either way.
+    """
+    agent_lower = agent.lower()
+    named_group = next(
+        (g for g in groups if any(a.lower() == agent_lower for a in g.agents)),
+        None,
+    )
+    if named_group is not None:
+        return "blocked" if blocks_everything(named_group) else "allowed-explicit"
+
+    wildcard_group = next((g for g in groups if "*" in g.agents), None)
+    if wildcard_group is not None and blocks_everything(wildcard_group):
+        return "blocked"
+    return "allowed-default"

aivisible_fix-0.2.0/aivisible_fix.egg-info/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: aivisible-fix
+Version: 0.2.0
+Summary: Audit and auto-fix a website's visibility to AI crawlers and answer engines (GPTBot, ClaudeBot, PerplexityBot, and more).
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# aivisible-fix
+
+Audits a website for the specific things that make it invisible to AI
+crawlers and AI answer engines (ChatGPT, Claude, Perplexity, Gemini/AI
+Overviews, and others) — and, unlike the free "checker" tools already on
+the market, **generates the fix**, not just a score.
+
+```
+$ aivisible-fix audit https://example.com
+AI-visibility audit for https://example.com
+Score: 40/100
+
+  [FAIL] robots_reachable: robots.txt missing or unreachable ...
+  [PASS] no_ai_crawlers_blocked: No known AI crawler is explicitly blocked.
+  [FAIL] llms_txt_present: No /llms.txt ...
+  ...
+
+$ aivisible-fix fix https://example.com --name "Example Co" --description "..."
+Wrote robots.fixed.txt
+Wrote llms.txt
+Review both files, then deploy them at your site root. Nothing was auto-published.
+```
+
+## What it checks
+
+1. **robots.txt reachable** — the baseline signal every crawler reads first.
+2. **No AI crawler is explicitly blocked** — checked per-agent (GPTBot,
+   ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai,
+   PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider,
+   Amazonbot, Applebot-Extended, cohere-ai, Meta-ExternalAgent, Diffbot,
+   YouBot — see `aivisible_fix/crawlers.py`), not just a blanket `Disallow: /`.
+3. **llms.txt present** — the emerging convention (llmstxt.org) for a short,
+   LLM-readable summary of what a site is and which pages matter.
+4. **sitemap.xml present and valid**, and **referenced from robots.txt**.
+5. **JSON-LD structured data on the homepage** — what answer engines lean on
+   to extract facts (org, product, FAQ, etc.) without guessing from prose.
+
+Score is out of 100, weighted toward the two things that, per current
+published research, cause the overwhelming majority of AI-search invisibility:
+blocked crawlers (40 pts) and missing discovery trail — sitemap + llms.txt
+(40 pts combined).
+
+## Why this instead of the free checkers
+
+Search "AI visibility checker" and you'll find eight-plus free tools
+(isvisible.ai, llmpulse.ai, siftly.ai, and others) that already do the
+scoring for free — that part of the market is a commodity and not worth
+competing on head to head.
+
+None of them **write the fix**. This tool's differentiator is that
+`aivisible-fix fix` outputs a ready-to-deploy patched `robots.txt` (every
+existing line preserved, blocked AI agents explicitly re-allowed) and a
+starter `llms.txt` — output, not just diagnosis. The natural extension
+(not built yet — see ROADMAP below) is a GitHub App that opens this as a
+pull request automatically, the way Dependabot does for dependency bumps,
+plus scheduled re-audits so a clean report doesn't silently rot as new
+crawlers ship.
+
+## Badge
+
+```
+$ aivisible-fix badge https://example.com
+Wrote aivisible-badge.svg (score 40/100). Host it yourself and embed:
+  <a href="https://example.com"><img src="/aivisible-badge.svg" alt="AI-visible: 40/100"></a>
+```
+
+Static SVG, generated locally — no server or account required. Color tracks
+score (red under 50, amber under 80, green at 80+).
+
+## Install
+
+```
+pip install -e .
+```
+
+No dependencies beyond the Python 3.10+ standard library — deliberately, so
+it's trivial to also ship as a single-file GitHub Action.
+
+## Status
+
+v0.1 — functional CLI, tested against live sites (see commit history / this
+was built and smoke-tested against example.com and nytimes.com, correctly
+detecting NYT's well-documented blanket block of AI crawlers). Not yet
+published to PyPI or packaged as a GitHub Action — see BUSINESS_PLAN.md for
+what's next and what needs a human to do it (PyPI account, GitHub App
+registration, a payment processor for the paid tier).

aivisible_fix-0.2.0/aivisible_fix.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+README.md
+pyproject.toml
+aivisible_fix/__init__.py
+aivisible_fix/audit.py
+aivisible_fix/badge.py
+aivisible_fix/cli.py
+aivisible_fix/crawlers.py
+aivisible_fix/fetch.py
+aivisible_fix/fix.py
+aivisible_fix/robots.py
+aivisible_fix.egg-info/PKG-INFO
+aivisible_fix.egg-info/SOURCES.txt
+aivisible_fix.egg-info/dependency_links.txt
+aivisible_fix.egg-info/entry_points.txt
+aivisible_fix.egg-info/top_level.txt

aivisible_fix-0.2.0/aivisible_fix.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

aivisible_fix-0.2.0/aivisible_fix.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+aivisible-fix = aivisible_fix.cli:main

aivisible_fix-0.2.0/aivisible_fix.egg-info/top_level.txt

@@ -0,0 +1 @@
+aivisible_fix

aivisible_fix-0.2.0/pyproject.toml

@@ -0,0 +1,15 @@
+[project]
+name = "aivisible-fix"
+version = "0.2.0"
+description = "Audit and auto-fix a website's visibility to AI crawlers and answer engines (GPTBot, ClaudeBot, PerplexityBot, and more)."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+dependencies = []
+
+[project.scripts]
+aivisible-fix = "aivisible_fix.cli:main"
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"

aivisible_fix-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+