@hanzlaa/rcode 4.3.3 → 4.4.0

package/rcode/skills/seo/technical-seo-checker/references/llm-crawler-handling.md

@@ -0,0 +1,257 @@
+# LLM Crawler Handling (GPTBot / ClaudeBot / PerplexityBot / etc.)
+
+Referenced from [SKILL.md](https://github.com/aaron-he-zhu/seo-geo-claude-skills/blob/main/optimize/technical-seo-checker/SKILL.md). Use during technical audits to decide whether (and how) to allow AI-engine crawlers. As of 2026, this is a required technical-SEO decision, not an optional one.
+
+---
+
+## Why this matters
+
+AI engines (ChatGPT Search, Claude Search, Perplexity, Google AI Overview, Gemini) crawl the web to populate training data AND real-time answer retrieval. Blocking them can:
+
+- Prevent citation in AI answers (GEO visibility loss)
+- Reduce brand mentions in AI-generated summaries
+- Lock out traffic from the fastest-growing discovery channel
+
+Allowing them can:
+
+- Expose content to training without compensation
+- Increase server load
+- Leak competitive content to rivals who scrape via AI
+
+**The decision is per-organization.** Common stances:
+
+| Stance | Typical choice | Rationale |
+|--------|----------------|-----------|
+| Default-open (marketing-led orgs) | Allow all AI crawlers | GEO visibility > training-data concerns |
+| Default-closed (IP-heavy orgs) | Block all | Proprietary research, legal docs, customer data |
+| Split (most common in 2026) | Allow retrieval bots, block training bots | Best of both — see mapping below |
+
+---
+
+## Known crawler inventory (2026)
+
+### OpenAI
+
+| User-Agent | Purpose | Typical robots.txt choice |
+|-----------|---------|---------------------------|
+| `GPTBot` | Training data for future ChatGPT models | Most orgs block |
+| `ChatGPT-User` | Real-time retrieval when ChatGPT answers with browsing | Most orgs allow (loses citation if blocked) |
+| `OAI-SearchBot` | ChatGPT Search (retrieval-focused) | Allow for GEO visibility |
+
+### Anthropic
+
+| User-Agent | Purpose | Typical choice |
+|-----------|---------|-----------------|
+| `ClaudeBot` / `Claude-Web` | Training data | Most orgs block |
+| `Claude-User` | Real-time fetch when user asks Claude to look up a URL | Allow |
+| `anthropic-ai` (legacy) | Legacy tag; treat as `ClaudeBot` | Block if blocking training |
+
+### Google
+
+| User-Agent | Purpose | Typical choice |
+|-----------|---------|-----------------|
+| `Googlebot` | Search + AI Overview | Always allow |
+| `Google-Extended` | Opt-out for Bard / Gemini training (does NOT affect Search ranking) | Block if opting out of training only |
+| `GoogleOther` | Internal research / product testing | Allow |
+
+### Perplexity
+
+| User-Agent | Purpose | Typical choice |
+|-----------|---------|-----------------|
+| `PerplexityBot` | Crawl for retrieval answers | Allow for GEO visibility |
+| `Perplexity-User` | Real-time fetch for user queries | Allow |
+
+### Common Crawl (downstream training data for many LLMs)
+
+| User-Agent | Purpose | Typical choice |
+|-----------|---------|-----------------|
+| `CCBot` | Feeds Common Crawl dataset used by many LLMs | Blocking here is indirect training opt-out |
+
+### Other 2026 notable
+
+| User-Agent | Purpose |
+|-----------|---------|
+| `Bytespider` | TikTok / Doubao / ByteDance LLMs |
+| `Applebot-Extended` | Apple Intelligence training opt-out |
+| `cohere-ai` | Cohere training |
+| `Meta-ExternalAgent` | Meta Llama training |
+| `Diffbot` | B2B data scraping (may power enterprise LLMs) |
+| `omgili` | Data broker — often blocked |
+| `DataForSeoBot`, `AhrefsBot`, `SemrushBot` | SEO tool crawlers — allow if you use the tool |
+
+---
+
+## Recommended robots.txt patterns
+
+### Pattern 1 — Default-open (maximize GEO visibility, accept training)
+
+```txt
+# Allow all AI engines for both retrieval and training
+User-agent: GPTBot
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Applebot-Extended
+Allow: /
+
+User-agent: cohere-ai
+Allow: /
+
+# Block only data resellers / scrapers
+User-agent: omgili
+Disallow: /
+```
+
+### Pattern 2 — Default-closed (training opt-out, retrieval in)
+
+```txt
+# Allow retrieval bots (needed for GEO visibility)
+User-agent: ChatGPT-User
+Allow: /
+
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: Claude-User
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Perplexity-User
+Allow: /
+
+User-agent: Googlebot
+Allow: /
+
+# Block training bots
+User-agent: GPTBot
+Disallow: /
+
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: Applebot-Extended
+Disallow: /
+
+User-agent: cohere-ai
+Disallow: /
+
+User-agent: Meta-ExternalAgent
+Disallow: /
+
+User-agent: CCBot
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+```
+
+### Pattern 3 — Section-specific (e.g., allow blog, block pricing)
+
+```txt
+User-agent: GPTBot
+Allow: /blog/
+Allow: /guides/
+Disallow: /pricing/
+Disallow: /customers/
+Disallow: /
+# The trailing Disallow: / is the default; Allow: entries above it take precedence
+```
+
+---
+
+## Legal layer beyond robots.txt
+
+Robots.txt alone is **not a legal opt-out** under EU law. For full compliance as content rights holder:
+
+### EU DSM Directive Art 4(3) — TDM reservation
+Machine-readable reservation required (robots.txt is advisory, not legally binding for TDM):
+
+```html
+<!-- In page HEAD -->
+<meta name="tdm-reservation" content="1" />
+```
+
+Or HTTP response header:
+
+```
+X-Robots-Tag: noai, notrain
+```
+
+Reference: [W3C TDM Reservation Protocol](https://www.w3.org/2022/tdmrep/).
+
+### EU AI Act Art 53(1)(c) — GPAI provider obligations
+
+> **Phased application of the AI Act**: the Act entered into force 2024-08; prohibited-practice provisions applied from 2025-02; **GPAI provider obligations (Art 51-55, including Art 53 summary-of-training-data) applied from 2025-08-02**; general applicability to high-risk systems phases through 2026-08. The date below refers specifically to when Art 53 GPAI obligations became applicable, not the whole Act.
+
+Art 53 applicable date: **2025-08-02**. General-Purpose AI Model providers (OpenAI, Anthropic, Google, etc.) must:
+- Publish summary of training data content
+- Respect opt-out signals including `tdm-reservation`
+- Establish copyright policy
+
+Content owners: monitor published training data summaries; file DMCA/EU copyright complaints if your content appears despite opt-out.
+
+### CCPA — 2026 California extension
+California AG 2025 guidance extends §1798.135 "right to opt-out of sale/share" to training data. Use `Global Privacy Control (GPC)` HTTP header on requests + respect inbound GPC signals if you process user data.
+
+### Post-training content removal
+If content was scraped before opt-out was in place:
+- **OpenAI**: submit removal at https://platform.openai.com/privacy-removal-request (documented as of 2024)
+- **Google Bard/Gemini training**: opt-out via Search Console > Settings > Crawling (Google-Extended block)
+- **Perplexity**: email legal@perplexity.ai with URL + copyright assertion
+- **Anthropic**: email privacy@anthropic.com (no public removal form as of 2026)
+
+### Enforcement timeline summary
+| Jurisdiction | Regulation | Effective | Status |
+|---|---|---|---|
+| EU | DSM Art 4 | 2019 / 2021 transposition deadline | Active |
+| EU | AI Act Art 53 GPAI | 2025-08 | Active |
+| US CA | CCPA training data | 2025 AG guidance | Enforceable |
+| UK | TDM exception | 2025 consultation | Pending |
+
+## Diagnostic signals during a technical audit
+
+| Signal | Action |
+|--------|--------|
+| No `User-agent: GPTBot` / `ClaudeBot` / `PerplexityBot` rules in robots.txt | Site is using default-allow; confirm this is intentional with user |
+| `Disallow: /` under `User-agent: *` | All bots blocked including Googlebot — usually accidental |
+| Rules in robots.txt but no `<meta name="robots">` counterpart on high-value pages | Inconsistent — AI bots may honor meta tag differently |
+| Server logs show `GPTBot` / `ClaudeBot` 429s or 403s | Firewall / CDN rate-limiting the bot (Cloudflare's default AI scraper rule, for example) — decide explicitly |
+| Cloudflare "Block AI scrapers" toggle on | Check if user expects this — it overrides robots.txt |
+
+## Cloudflare-specific (2026)
+
+Cloudflare added a one-click "Block AI scrapers" toggle that blocks GPTBot, ClaudeBot, CCBot, and others at the edge — **before** robots.txt is evaluated. Audit step:
+
+1. Log into Cloudflare dashboard
+2. Security → Bots → AI Scrapers and Crawlers
+3. Verify the toggle state matches the org's stance
+
+If Cloudflare is blocking but robots.txt allows, the bot will never reach robots.txt — Cloudflare wins.
+
+## Handoff addition
+
+When the technical audit covers LLM crawler handling, include in the handoff:
+
+- `ai_crawler_stance`: `default-open` | `default-closed` | `mixed` | `unknown`
+- `ai_crawler_blocked`: list of bot user-agents blocked (e.g., `[GPTBot, ClaudeBot, CCBot]`)
+- `ai_crawler_allowed`: list allowed for retrieval (e.g., `[ChatGPT-User, PerplexityBot]`)
+- `ai_crawler_edge_override`: `true` if Cloudflare / Cloudfront is enforcing rules ahead of robots.txt
+- Open loop: ask user to confirm or modify stance if unknown
+
+## See also
+
+- [geo-content-optimizer](https://github.com/aaron-he-zhu/seo-geo-claude-skills/blob/main/build/geo-content-optimizer/SKILL.md) — downstream skill that depends on AI engines actually seeing your content
+- [entity-optimizer](https://github.com/aaron-he-zhu/seo-geo-claude-skills/blob/main/cross-cutting/entity-optimizer/SKILL.md) — blocking retrieval bots breaks AI entity recognition

package/rcode/skills/seo/technical-seo-checker/references/pre-migration-playbook.md

@@ -0,0 +1,137 @@
+# Technical SEO — Pre-Migration Playbook
+
+Referenced from [SKILL.md](https://github.com/aaron-he-zhu/seo-geo-claude-skills/blob/main/optimize/technical-seo-checker/SKILL.md). Use when the user is planning a migration (platform, domain, URL structure, or framework change) and needs an audit + risk assessment before the change ships.
+
+---
+
+## When to use
+
+- WordPress → Headless (Next.js, Astro, Remix, Gatsby, etc.)
+- Shopify → custom / BigCommerce / Magento
+- Subdomain consolidation (blog.example.com → example.com/blog)
+- Domain change (oldbrand.com → newbrand.com)
+- URL structure rewrite (/category/product → /product)
+- HTTP → HTTPS (rare now, still applicable on legacy sites)
+- CMS upgrade that rewrites URLs (e.g., major theme refactor)
+
+## The 6 pre-migration stages
+
+### Stage 1 — Freeze current state (baseline snapshot)
+
+Goal: capture the **current** SEO signals so you can diff after migration.
+
+Capture in `memory/audits/pre-migration-YYYY-MM-DD.md`:
+
+1. **URL inventory**
+   - Full crawl via `~~web crawler` MCP or Screaming Frog / Sitebulb export
+   - Canonical list of every indexable URL, with response code, canonical tag, redirect chain
+   - Total indexable count
+
+2. **Ranking baseline**
+   - Top 100 ranking keywords (from `~~SEO tool` or `~~search console`)
+   - For each: URL, position, volume, CTR, clicks last 90 days
+   - Save as CSV: `memory/monitoring/pre-migration-ranks.csv`
+
+3. **Traffic baseline**
+   - Top 50 URLs by organic sessions (from `~~analytics` or `~~search console`)
+   - Last 30 + 90-day session count per URL
+   - Pages with >1% of total traffic flagged for VIP redirect review
+
+4. **Backlink baseline**
+   - Top 100 backlinks by referring domain authority (from `~~SEO tool`)
+   - Top 50 linked URLs (on your site)
+   - Flag any URL with 10+ backlinks as HIGH-VALUE — needs exact 301 after migration
+
+5. **Schema & entity snapshot**
+   - Current structured data (JSON-LD extracted from head) per page template
+   - Entity representation (Organization, Person, Product, Article, etc.)
+
+### Stage 2 — Risk map
+
+For each proposed migration change, score risk:
+
+| Change | Risk | Impact if mishandled |
+|--------|------|---------------------|
+| URL structure change | HIGH | traffic loss 20-40% for 2-12 weeks |
+| Template rewrite (meta, headers) | HIGH | keyword targeting loss |
+| Domain change | CRITICAL | full PageRank reset without redirects |
+| Schema change | MEDIUM | rich result loss |
+| JS-rendered content | MEDIUM | partial deindex if not SSR |
+| robots.txt / noindex change | CRITICAL | accidental deindex |
+| Internal link restructure | MEDIUM | topic cluster weakness |
+
+Produce a **GO / NOGO** recommendation per change.
+
+### Stage 3 — Redirect map
+
+Before any URL changes, build a one-to-one redirect map:
+
+```csv
+old_url,new_url,reason,priority
+/category/blue-shoes,/shop/blue-shoes,url structure,P0
+/product/abc-123,/product/blue-shoe,slug change,P0
+/blog/2020/seo-tips,/blog/seo-tips,year removed,P1
+```
+
+Rules:
+- Every URL in the Stage 1 inventory MUST have a mapping or a documented decision to 410 / 404
+- No chains (`A → B → C` must become `A → C` and `B → C`)
+- No loops (never `A → B → A`)
+- HIGH-VALUE URLs (Stage 1 flagged) verified manually
+
+Save to `memory/audits/redirect-map-YYYY-MM-DD.csv`.
+
+### Stage 4 — Staging QA
+
+Before production cutover, QA the staging environment:
+
+1. **Robots / indexing**: is `noindex` set on staging? Will it flip to `index` on launch? Any accidental `Disallow: /` in the production robots.txt draft?
+2. **Template parity**: do new templates produce equivalent `<title>`, `<meta description>`, `<h1>`, canonical, schema for each page type?
+3. **Internal linking**: run a crawl of staging; verify topic clusters intact; check for orphan pages introduced by the new template.
+4. **Core Web Vitals**: run PSI / Lighthouse / WebPageTest on staging. LCP, INP, CLS per template. Compare to Stage 1 baseline.
+5. **Schema validation**: validate every page-type template's JSON-LD with Schema.org validator and Google Rich Results Test.
+6. **Pagination / facets**: if the site has collection pages, verify pagination canonical / facet canonical behavior matches or improves on the baseline.
+
+### Stage 5 — Cutover day checklist
+
+At launch:
+
+1. Deploy redirects BEFORE touching DNS / robots (if possible)
+2. Update `robots.txt` to production
+3. Submit new `sitemap.xml` to Search Console and Bing Webmaster
+4. Unblock crawling (remove any staging noindex)
+5. Monitor: tail access logs for 5xx spikes, 404 spikes on old URLs
+6. First-day rollback trigger: if 404 rate on old URLs exceeds 5% of old-URL traffic, immediately audit redirect map
+
+### Stage 6 — Post-migration diff (T+1, T+7, T+30)
+
+Run the same inventory as Stage 1 and diff:
+
+- **T+1**: full crawl. Flag any URL that returns non-2xx when it should redirect.
+- **T+7**: compare traffic per URL vs baseline. Flag URLs with >30% drop.
+- **T+30**: compare rankings vs baseline for top 100 keywords. Flag URLs lost from top 10 / top 20.
+
+Deliverables:
+
+- `memory/audits/post-migration-T+1-YYYY-MM-DD.md`
+- `memory/audits/post-migration-T+7-YYYY-MM-DD.md`
+- `memory/audits/post-migration-T+30-YYYY-MM-DD.md`
+
+## Handoff
+
+- **Status**: DONE | DONE_WITH_CONCERNS | BLOCKED
+- **Objective**: "Pre-migration audit for <change_description>"
+- **Key Findings / Output**: baseline snapshot refs + risk map + redirect map + QA checklist
+- **Evidence**: crawl counts, ranking CSV path, redirect CSV path, Core Web Vitals before/after (if staging ready)
+- **Open Loops**: HIGH-VALUE URLs pending redirect confirmation, schema not validated for templates X/Y, rollback trigger thresholds not yet agreed with engineering
+- **Recommended Next Skill**: `content-refresher` if template rewrite surfaced content-quality gaps; `schema-markup-generator` if new templates need schema authored; otherwise `rank-tracker` for post-launch monitoring
+
+## Red-flag patterns (STOP — do not ship until fixed)
+
+- Any redirect chain >1 hop
+- Missing 301 on a URL with >100 monthly organic sessions
+- `robots.txt` has `Disallow: /` anywhere in production draft
+- Canonical tag on new URL points back to old URL
+- JS-only content with no SSR fallback on key landing pages
+- New template omits `<h1>` or has multiple `<h1>`
+- Schema type changed without rich-result eligibility check