@braedenbuilds/crawl-sim 1.0.5 → 1.2.0

package/.claude-plugin/marketplace.json

@@ -0,0 +1,15 @@
+{
+  "name": "crawl-sim",
+  "owner": {
+    "name": "BraedenBDev",
+    "url": "https://github.com/BraedenBDev"
+  },
+  "plugins": [
+    {
+      "name": "crawl-sim",
+      "source": "./",
+      "description": "Multi-bot web crawler simulator — audit how Googlebot, GPTBot, ClaudeBot, and PerplexityBot see your site",
+      "version": "1.2.0"
+    }
+  ]
+}

package/.claude-plugin/plugin.json

@@ -0,0 +1,13 @@
+{
+  "name": "crawl-sim",
+  "version": "1.2.0",
+  "description": "Multi-bot web crawler simulator — audit how Googlebot, GPTBot, ClaudeBot, and PerplexityBot see your site",
+  "author": {
+    "name": "BraedenBDev",
+    "url": "https://github.com/BraedenBDev"
+  },
+  "homepage": "https://github.com/BraedenBDev/crawl-sim#readme",
+  "repository": "https://github.com/BraedenBDev/crawl-sim",
+  "license": "MIT",
+  "keywords": ["seo", "crawler", "ai-visibility", "claude-code-skill", "googlebot", "gptbot", "claudebot", "perplexitybot"]
+}

package/README.md

@@ -44,15 +44,20 @@ The concept was validated manually: a curl-as-GPTBot + Claude analysis caught a
 
 ## Quick start
 
-### In Claude Code (recommended)
+### As a Claude Code plugin (recommended)
 
-```bash
-npm install -g @braedenbuilds/crawl-sim
-crawl-sim install              # → ~/.claude/skills/crawl-sim/
-crawl-sim install --project    # → .claude/skills/crawl-sim/
+```
+/plugin install BraedenBDev/crawl-sim@github
+```
+
+Or add as a marketplace for easy updates:
+
+```
+/plugin marketplace add BraedenBDev/crawl-sim
+/plugin install crawl-sim@crawl-sim
 ```
 
-Then in Claude Code:
+Then invoke:
 
 ```
 /crawl-sim https://yoursite.com
@@ -60,7 +65,15 @@ Then in Claude Code:
 
 Claude runs the full pipeline, interprets the results, and returns a score card plus prioritized findings.
 
-> **Why `npm install -g` instead of `npx`?** Recent versions of npx have a [known issue](https://github.com/npm/cli/issues) linking bins for scoped single-bin packages in ephemeral installs. A persistent global install avoids the problem entirely. If you want a one-shot install without touching global state, the tarball URL form also works: `npx https://registry.npmjs.org/@braedenbuilds/crawl-sim/-/crawl-sim-1.0.3.tgz install`.
+### Via npm (alternative)
+
+```bash
+npm install -g @braedenbuilds/crawl-sim
+crawl-sim install              # → ~/.claude/skills/crawl-sim/
+crawl-sim install --project    # → .claude/skills/crawl-sim/
+```
+
+> **Why `npm install -g` instead of `npx`?** Recent versions of npx have a known issue linking bins for scoped single-bin packages in ephemeral installs. A persistent global install avoids the problem entirely. The git clone path below is the zero-npm fallback.
 
 ### As a standalone CLI
 
@@ -88,10 +101,13 @@ git clone https://github.com/BraedenBDev/crawl-sim.git ~/.claude/skills/crawl-si
 
 - **Multi-bot simulation.** Nine verified bot profiles covering Google, OpenAI, Anthropic, and Perplexity — including the bot-vs-user-agent distinction (e.g., `ChatGPT-User` officially ignores robots.txt; `claude-user` respects it).
 - **Quantified scoring.** Each bot is graded 0–100 across five categories with letter grades A through F, plus a weighted composite score.
+- **Page-type-aware rubric.** The structured-data category derives the page type from the URL (`root` / `detail` / `archive` / `faq` / `about` / `contact` / `generic`) and applies a per-type schema rubric. A homepage shipping `Organization` + `WebSite` scores 100 without being penalized for not having `BreadcrumbList` or `FAQPage`. Override the detection with `--page-type <type>` when the URL heuristic picks wrong.
+- **Self-explaining scores.** Every `structuredData` block in the JSON report ships `pageType`, `expected`, `optional`, `forbidden`, `present`, `missing`, `extras`, `violations`, `calculation`, and `notes` — so the narrative layer reads the scorer's reasoning directly instead of guessing what was penalized.
 - **Agent-native interpretation.** The Claude Code skill reads raw data, identifies root causes (framework signals, hydration boundaries, soft-404s), and recommends specific fixes.
 - **Three-layer output.** Terminal score card, prose narrative, and structured JSON — so humans and CI both get what they need.
 - **Confidence transparency.** Every claim is tagged `official`, `observed`, or `inferred`. The skill notes when recommendations depend on observed-but-undocumented behavior.
 - **Shell-native core.** All checks use only `curl` + `jq`. No Node, no Python, no Docker. Each script is independently invokable.
+- **Regression-tested.** `npm test` runs a 37-assertion scoring suite against synthetic fixtures, covering URL→page-type detection, per-type rubrics, missing/forbidden schema flagging, and golden non-structured output.
 - **Extensible.** Drop a new profile JSON into `profiles/` and it's auto-discovered.
 
 ---
@@ -107,6 +123,8 @@ git clone https://github.com/BraedenBDev/crawl-sim.git ~/.claude/skills/crawl-si
 /crawl-sim https://yoursite.com --json                     # JSON only (for CI)
 ```
 
+The skill auto-detects page type from the URL. Pass `--page-type root|detail|archive|faq|about|contact|generic` to the underlying `compute-score.sh` when the URL heuristic picks the wrong type (e.g., a homepage at `/en/` that URL-parses as `generic`).
+
 Output is a three-layer report:
 
 1. **Score card** — ASCII overview with per-bot and per-category scores.
@@ -126,6 +144,7 @@ Every script is standalone and outputs JSON to stdout:
 ./scripts/check-llmstxt.sh   https://yoursite.com
 ./scripts/check-sitemap.sh   https://yoursite.com
 ./scripts/compute-score.sh   /tmp/audit-data/
+./scripts/compute-score.sh   --page-type root /tmp/audit-data/   # override URL heuristic
 ```
 
 ### CI/CD
@@ -148,7 +167,7 @@ Each bot is scored 0–100 across five weighted categories:
 |----------|:------:|----------|
 | **Accessibility** | 25 | robots.txt allows, HTTP 200, response time |
 | **Content Visibility** | 30 | server HTML word count, heading structure, internal links, image alt text |
-| **Structured Data** | 20 | JSON-LD presence, validity, page-appropriate `@type` |
+| **Structured Data** | 20 | JSON-LD presence, validity, page-type-aware `@type` rubric (root / detail / archive / faq / about / contact / generic) |
 | **Technical Signals** | 15 | title / description / canonical / OG meta, sitemap inclusion |
 | **AI Readiness** | 10 | `llms.txt` structure, content citability |
 
@@ -218,6 +237,7 @@ crawl-sim/
 ├── bin/install.js         # npm installer
 ├── profiles/              # 9 verified bot profiles (JSON)
 ├── scripts/
+│   ├── _lib.sh            # shared helpers (URL parsing, page-type detection)
 │   ├── fetch-as-bot.sh    # curl with bot UA → JSON (status/headers/body/timing)
 │   ├── extract-meta.sh    # title, description, OG, headings, images
 │   ├── extract-jsonld.sh  # JSON-LD @type detection
@@ -227,8 +247,11 @@ crawl-sim/
 │   ├── check-sitemap.sh   # sitemap.xml URL inclusion
 │   ├── diff-render.sh     # optional Playwright server-vs-rendered comparison
 │   └── compute-score.sh   # aggregates all checks → per-bot + per-category scores
+├── test/
+│   ├── run-scoring-tests.sh  # 37-assertion bash harness (run with `npm test`)
+│   └── fixtures/             # synthetic RUN_DIR fixtures for regression tests
 ├── research/              # Verified bot data sources
-└── docs/specs/            # Design docs
+└── docs/                  # Design docs, issues, accuracy handoffs
 ```
 
 The shell scripts are the plumbing. The Claude Code skill is the intelligence — it reads the raw JSON, understands framework context (Next.js, Nuxt, SPAs), identifies root causes, and writes actionable recommendations.

package/bin/install.js

@@ -14,6 +14,7 @@ const os = require('os');
 const { execFileSync } = require('child_process');
 
 const SOURCE_DIR = path.resolve(__dirname, '..');
+const SKILL_ROOT = path.resolve(SOURCE_DIR, 'skills', 'crawl-sim');
 const SKILL_FILES = ['SKILL.md'];
 const SKILL_DIRS = ['profiles', 'scripts'];
 
@@ -80,7 +81,9 @@ function install(target) {
   fs.mkdirSync(target, { recursive: true });
 
   for (const file of SKILL_FILES) {
-    const src = path.join(SOURCE_DIR, file);
+    // Look in skills/crawl-sim/ first (canonical), fallback to root (symlink)
+    let src = path.join(SKILL_ROOT, file);
+    if (!fs.existsSync(src)) src = path.join(SOURCE_DIR, file);
     const dest = path.join(target, file);
     if (fs.existsSync(src)) {
       fs.copyFileSync(src, dest);
@@ -92,7 +95,8 @@ function install(target) {
   }
 
   for (const dir of SKILL_DIRS) {
-    const src = path.join(SOURCE_DIR, dir);
+    let src = path.join(SKILL_ROOT, dir);
+    if (!fs.existsSync(src)) src = path.join(SOURCE_DIR, dir);
     const dest = path.join(target, dir);
     if (fs.existsSync(src)) {
       if (fs.existsSync(dest)) {

package/package.json

@@ -1,10 +1,13 @@
 {
   "name": "@braedenbuilds/crawl-sim",
-  "version": "1.0.5",
+  "version": "1.2.0",
   "description": "Agent-native multi-bot web crawler simulator. See your site through the eyes of Googlebot, GPTBot, ClaudeBot, and PerplexityBot.",
   "bin": {
     "crawl-sim": "bin/install.js"
   },
+  "scripts": {
+    "test": "./test/run-scoring-tests.sh"
+  },
   "keywords": [
     "seo",
     "crawler",
@@ -37,9 +40,11 @@
   },
   "files": [
     "bin/",
+    "skills/",
+    ".claude-plugin/",
     "SKILL.md",
-    "profiles/",
-    "scripts/",
+    "profiles",
+    "scripts",
     "README.md",
     "LICENSE"
   ]

package/{SKILL.md → skills/crawl-sim/SKILL.md}

@@ -40,7 +40,10 @@ command -v curl >/dev/null 2>&1 || { echo "ERROR: curl is required"; exit 1; }
 command -v jq   >/dev/null 2>&1 || { echo "ERROR: jq is required (brew install jq)"; exit 1; }
 ```
 
-Locate the skill directory: typically `~/.claude/skills/crawl-sim/` or `.claude/skills/crawl-sim/`. Use `$CLAUDE_PLUGIN_ROOT` if set, otherwise find the directory containing this `SKILL.md`.
+Locate the skill directory. Check in this order:
+1. `$CLAUDE_PLUGIN_ROOT/skills/crawl-sim` (plugin install)
+2. `~/.claude/skills/crawl-sim/` (global npm install)
+3. `.claude/skills/crawl-sim/` (project-level install)
 
 ## Orchestration — five narrated stages
 
@@ -51,7 +54,16 @@ Split the work into **five Bash invocations**, each with a clear `description` f
 Tell the user: "Fetching as Googlebot, GPTBot, ClaudeBot, and PerplexityBot..."
 
 ```bash
-SKILL_DIR="$HOME/.claude/skills/crawl-sim"  # or wherever this SKILL.md lives
+# Resolve skill directory
+if [ -n "${CLAUDE_PLUGIN_ROOT:-}" ] && [ -d "$CLAUDE_PLUGIN_ROOT/skills/crawl-sim" ]; then
+  SKILL_DIR="$CLAUDE_PLUGIN_ROOT/skills/crawl-sim"
+elif [ -d "$HOME/.claude/skills/crawl-sim" ]; then
+  SKILL_DIR="$HOME/.claude/skills/crawl-sim"
+elif [ -d ".claude/skills/crawl-sim" ]; then
+  SKILL_DIR=".claude/skills/crawl-sim"
+else
+  echo "ERROR: cannot find crawl-sim skill directory" >&2; exit 1
+fi
 RUN_DIR=$(mktemp -d -t crawl-sim.XXXXXX)
 URL="<user-provided-url>"
 for bot in googlebot gptbot claudebot perplexitybot; do
@@ -115,6 +127,14 @@ Tell the user: "Computing per-bot scores and finalizing the report..."
 cp "$RUN_DIR/score.json" ./crawl-sim-report.json
 ```
 
+**Page-type awareness.** `compute-score.sh` derives a page type from the target URL (`root` / `detail` / `archive` / `faq` / `about` / `contact` / `generic`) and picks a schema rubric accordingly. Root pages are expected to ship `Organization` + `WebSite` — penalizing them for missing `BreadcrumbList` or `FAQPage` would be wrong, so the scorer doesn't. If the URL heuristic picks the wrong type (e.g., a homepage at `/en/` that URL-parses as generic), pass `--page-type <type>`:
+
+```bash
+"$SKILL_DIR/scripts/compute-score.sh" --page-type root "$RUN_DIR" > "$RUN_DIR/score.json"
+```
+
+Valid values: `root`, `detail`, `archive`, `faq`, `about`, `contact`, `generic`. The detected (or overridden) page type is exposed on `score.pageType`, and `score.pageTypeOverridden` flips `true` when `--page-type` was used.
+
 ## Output Layer 1 — Score Card (ASCII)
 
 Print a boxed score card to the terminal:
@@ -168,6 +188,7 @@ Then produce **prioritized findings** ranked by total point impact across bots:
 ### Interpretation rules
 
 - **Cross-bot deltas are the headline.** Compare `visibility.effectiveWords` across bots — if Googlebot has significantly more than the AI bots, that's finding #1. The raw delta is in `visibility.missedWordsVsRendered`.
+- **Trust the structuredData rubric.** Every `bots.<bot>.categories.structuredData` block now carries `pageType`, `expected`, `optional`, `forbidden`, `present`, `missing`, `extras`, `violations`, `calculation`, and `notes`. Read `missing` and `violations` directly — never guess what the scorer was penalizing for. If `notes` says the page scores 100 with no action needed, that IS the finding; don't invent fixes. If the rubric looks wrong for this specific page (e.g., a homepage detected as `generic` because the URL ends in `/en/`), rerun with `--page-type <correct-type>` instead of arguing with the score. Never recommend adding a schema that already appears in `present` or `extras`.
 - **Confidence transparency.** If a claim depends on a bot profile's `rendersJavaScript: false` at `observed` confidence (not `official`), note it: *"Based on observed behavior, not official documentation."*
 - **Framework detection.** Scan the HTML body for signals: `<meta name="next-head-count">` or `_next/static` → Next.js (Pages Router or App Router respectively), `<div id="__nuxt">` → Nuxt, `<div id="app">` with thin content → SPA (Vue/React CSR), `<!--$-->` placeholder tags → React 18 Suspense. Use these to tailor fix recommendations.
 - **No speculation beyond the data.** If server HTML has 0 `<a>` tags inside a component, say "component not present in server HTML" — not "JavaScript hydration failed" unless the diff-render data proves it.

package/{scripts → skills/crawl-sim/scripts}/_lib.sh

@@ -41,6 +41,36 @@ count_words() {
   sed 's/<[^>]*>//g' "$1" | tr -s '[:space:]' '\n' | grep -c '[a-zA-Z0-9]' || true
 }
 
+# Detect the structural page type of a URL based on its path.
+# Returns one of: root, detail, archive, faq, about, contact, generic.
+#
+# Used by compute-score.sh to pick a schema rubric, but also exposed here
+# so other tooling (narrative layer, planned multi-URL mode) can classify
+# URLs consistently without re-implementing the heuristic.
+page_type_for_url() {
+  local url="$1"
+  local path
+  path=$(path_from_url "$url" | sed 's#[?#].*##')
+  if [ "$path" = "/" ]; then
+    echo "root"
+    return
+  fi
+  local trimmed lower
+  trimmed=$(printf '%s' "$path" | sed 's#^/##' | sed 's#/$##')
+  lower=$(printf '%s' "$trimmed" | tr '[:upper:]' '[:lower:]')
+  case "$lower" in
+    "") echo "root" ;;
+    work|journal|blog|articles|news|careers|projects|case-studies|cases)
+      echo "archive" ;;
+    work/*|articles/*|journal/*|blog/*|news/*|case-studies/*|cases/*|case/*|careers/*|projects/*)
+      echo "detail" ;;
+    *faq*) echo "faq" ;;
+    *about*|*team*|*purpose*|*who-we-are*) echo "about" ;;
+    *contact*) echo "contact" ;;
+    *) echo "generic" ;;
+  esac
+}
+
 # Fetch a URL to a local file and return the HTTP status code on stdout.
 # Usage: status=$(fetch_to_file <url> <output-file> [timeout-seconds])
 fetch_to_file() {