pagesight 0.9.0 → 0.11.0

package/README.md

@@ -8,111 +8,45 @@ See your site the way search engines and AI see it.
 npm install pagesight
 ```
 
-Most SEO tools flag "title over 60 characters" and "only one H1 allowed." [Google's own engineers say those rules don't exist.](#why-not-other-seo-tools) Pagesight skips the myths and goes to the sources.
-
-## Tools
-
-### `inspect`
-
-Ask Google: is this page indexed? What canonical did you choose? Any crawl errors? Structured data issues?
-
-Returns index status, canonical (yours vs Google's), crawl status, rich results validation, sitemaps, and referring URLs.
-
-### `pagespeed`
-
-Run Google Lighthouse on any URL:
-
-- **Scores**: performance, accessibility, best-practices, seo
-- **Core Web Vitals (lab)**: FCP, LCP, TBT, CLS, Speed Index, TTI
-- **CrUX field data**: real Chrome user metrics (page + origin)
-- **Opportunities**: ranked by severity with potential savings
-- **Strategy**: `mobile` or `desktop`
-
-### `crux`
-
-Real-world Core Web Vitals from Chrome users (28-day rolling window):
-
-- **Metrics**: LCP, FCP, INP, CLS, TTFB, RTT, navigation types, form factors
-- **Granularity**: by URL or origin, by device (DESKTOP, PHONE, TABLET)
-- **Data**: p75 values + histogram distributions
-
-### `crux_history`
-
-Core Web Vitals trends over time — up to 40 weekly data points (~10 months):
-
-- Trend detection (improved/stable/worse) with percentage change
-- Recent data points table for core metrics
-- Custom period count (1-40)
-
-### `performance`
-
-Google Search Console search analytics:
-
-- **Dimensions**: `query`, `page`, `country`, `device`, `date`, `searchAppearance`, `hour`
-- **Search types**: `web`, `image`, `video`, `news`, `discover`, `googleNews`
-- **Filters**: `equals`, `contains`, `notEquals`, `notContains`, `includingRegex`, `excludingRegex`
-- **Pagination**: up to 25,000 rows
-
-### `robots`
-
-Analyze any site's robots.txt:
-
-- **Syntax validation** per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)
-- **AI crawler audit** — 139+ bots from the [ai-robots-txt](https://github.com/ai-robots-txt/ai.robots.txt) community registry
-- **Bot categories**: training scrapers, AI search crawlers, AI assistants, AI agents
-- **Per-bot status**: blocked or allowed, with the matched rule
+Your AI assistant can write your code. Now it can see your site. Index status, performance, real-user metrics, search traffic, meta tags, structured data, AI crawler access — one package, one call.
 
 ```
-=== robots.txt: https://www.nytimes.com ===
-AI Crawlers: 35 blocked, 104 allowed (of 139 known)
-
-  BLOCKED  GPTBot (OpenAI) — GPT model training
-  BLOCKED  ClaudeBot (Anthropic) — Claude model training
-  ALLOWED  Claude-User (Anthropic) — User-initiated fetching
+=== Site Audit: https://example.com ===
+
+HIGH    Missing canonical URL
+HIGH    7,772 sitemap URLs submitted, 0 indexed
+MEDIUM  Missing og:image — no social preview image
+MEDIUM  Accessibility score: 89/100
+LOW     Missing Twitter Card tags
+LOW     No structured data (JSON-LD) found
 ```
 
-### `sitemaps`
-
-Search Console properties and sitemaps (read-only):
-
-- `list_sites` — all properties with permission level
-- `get_site` — details for a specific property
-- `list_sitemaps` — sitemaps with error/warning counts
-- `get_sitemap` — full details for a specific sitemap
-
-### `setup`
+## Tools
 
-Check auth status or walk through OAuth interactively.
+| Tool | What it does |
+|------|-------------|
+| `audit` | One-call site audit. Runs all checks in parallel. Returns prioritized findings. |
+| `pagespeed` | Lighthouse scores, Core Web Vitals, opportunities, failing audits with fix links. |
+| `metatags` | OG, Twitter Card, canonical, JSON-LD with schema validation, redirect chain, image validation. |
+| `inspect` | Google index status, canonical choice, crawl state, rich results. |
+| `sample_inspect` | Sample URLs from a sitemap and batch-inspect. Diagnoses indexing patterns. |
+| `performance` | Search analytics — clicks, impressions, CTR, position. `compare: true` for period-over-period. |
+| `crux` | Real-user Core Web Vitals (p75, histograms). |
+| `crux_history` | CWV trends over time — up to 40 weekly data points. |
+| `robots` | robots.txt validation (RFC 9309) + AI crawler audit (139+ bots). |
+| `sitemaps` | Search Console properties and sitemaps with submitted/indexed counts. |
+| `setup` | Auth status and OAuth setup. |
 
 ## Setup
 
-### 1. Google Cloud project
-
-1. Go to [Google Cloud Console](https://console.cloud.google.com/)
-2. Create a project (or use existing)
-3. Enable: **Search Console API**, **PageSpeed Insights API**, **Chrome UX Report API**
-4. Create **OAuth client ID** (Desktop app) — for Search Console
-5. Create **API key** — for PageSpeed and CrUX
-
-### 2. Configure
-
-```env
-GSC_CLIENT_ID=your-client-id.apps.googleusercontent.com
-GSC_CLIENT_SECRET=your-client-secret
-GSC_REFRESH_TOKEN=your-refresh-token
-GOOGLE_API_KEY=your-api-key
-```
-
-The `robots` tool works without any credentials.
-
-### 3. Use with your AI assistant
+Add to your MCP config:
 
 ```json
 {
   "mcpServers": {
     "pagesight": {
-      "command": "bun",
-      "args": ["run", "/path/to/pagesight/src/index.ts"],
+      "command": "npx",
+      "args": ["pagesight"],
       "env": {
         "GSC_CLIENT_ID": "your-client-id",
         "GSC_CLIENT_SECRET": "your-secret",
@@ -124,36 +58,30 @@ The `robots` tool works without any credentials.
 }
 ```
 
-Then just ask:
+`robots`, `metatags`, and `pagespeed` work without credentials.
 
-```
-"Is https://mysite.com indexed?"
-"Run pagespeed on my homepage"
-"Which AI crawlers can access my site?"
-"How have my Core Web Vitals changed?"
-"Which queries bring traffic to this page?"
-```
+### Full setup
 
-## Why not other SEO tools?
+1. [Google Cloud Console](https://console.cloud.google.com/) — enable Search Console API, PageSpeed Insights API, Chrome UX Report API
+2. Create OAuth client ID (Desktop app) + API key
+3. Configure:
 
-We checked every common SEO "rule" against official Google documentation:
+```env
+GSC_CLIENT_ID=your-client-id.apps.googleusercontent.com
+GSC_CLIENT_SECRET=your-client-secret
+GSC_REFRESH_TOKEN=your-refresh-token
+GOOGLE_API_KEY=your-api-key
+```
 
-- **"Title must be under 60 characters"** — Google: "there's no limit." Gary Illyes: "an externally made-up metric."
-- **"Meta description must be 155 characters"** — Google: "there's no limit on how long a meta description can be."
-- **"Only one H1 per page"** — John Mueller: "You can use H1 tags as often as you want. There's no limit."
-- **"Minimum 300 words per page"** — Mueller: "the number of words on a page is not a quality factor."
-- **"Text-to-HTML ratio matters"** — Mueller: "it makes absolutely no sense at all for SEO."
+## Why Pagesight
 
-Tools that flag these are reporting their opinions. Pagesight only reports what the sources actually return.
+Every data point comes from a verifiable source. Google's APIs, real Chrome users, RFC 9309, schema.org, a community-maintained bot registry. No invented scores. No rules we can't cite.
 
-## Development
+- **"Title must be under 60 characters"** — Gary Illyes: "an externally made-up metric."
+- **"Only one H1 per page"** — John Mueller: "You can use H1 tags as often as you want."
+- **"Minimum 300 words per page"** — Mueller: "not a quality factor."
 
-```bash
-bun install
-bun run start     # start server
-bun run lint      # biome check
-bun run format    # biome format
-```
+Pagesight reports what the sources report. Nothing more.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pagesight",
-  "version": "0.9.0",
+  "version": "0.11.0",
   "description": "See your site the way search engines and AI see it.",
   "keywords": [
     "seo",

package/src/tools/inspect.ts

@@ -34,6 +34,11 @@ function formatInspection(url: string, siteUrl: string, r: InspectionResult): st
     lines.push(`\nReferring URLs: ${idx.referringUrls.join(", ")}`);
   }
 
+  if (idx.verdict !== "PASS") {
+    const gscUrl = `https://search.google.com/search-console/inspect?resource_id=${encodeURIComponent(siteUrl)}&id=${encodeURIComponent(url)}`;
+    lines.push("", `→ This page is not indexed. Request indexing manually in Google Search Console:`, `  ${gscUrl}`);
+  }
+
   // Rich Results
   if (r.richResultsResult) {
     lines.push("", "--- Rich Results ---", "");

package/src/tools/pagespeed.ts

@@ -16,6 +16,10 @@ function scoreLabel(score: number | null): string {
   return `${pct} (poor)`;
 }
 
+function scorePct(score: number | null): number | null {
+  return score === null ? null : Math.round(score * 100);
+}
+
 function cwvRating(category: string): string {
   if (category === "FAST") return "good";
   if (category === "AVERAGE") return "needs improvement";
@@ -91,7 +95,7 @@ function formatOpportunities(audits: Record<string, PsiAudit>): string[] {
 }
 
 function formatDiagnostics(audits: Record<string, PsiAudit>): string[] {
-  const failing: Array<{ title: string; displayValue: string }> = [];
+  const failing: PsiAudit[] = [];
 
   for (const audit of Object.values(audits)) {
     if (
@@ -100,15 +104,31 @@ function formatDiagnostics(audits: Record<string, PsiAudit>): string[] {
       (audit.scoreDisplayMode === "numeric" || audit.scoreDisplayMode === "metricSavings") &&
       audit.displayValue
     ) {
-      failing.push({ title: audit.title, displayValue: audit.displayValue });
+      failing.push(audit);
     }
   }
 
   if (failing.length === 0) return [];
 
+  failing.sort((a, b) => (a.score ?? 0) - (b.score ?? 0));
+
   const lines: string[] = ["--- Diagnostics ---", ""];
-  for (const item of failing.slice(0, 10)) {
-    lines.push(`${item.title}: ${item.displayValue}`);
+  for (const audit of failing.slice(0, 10)) {
+    lines.push(`${audit.title}: ${audit.displayValue}`);
+
+    const linkMatch = audit.description?.match(/\[.*?\]\((https?:\/\/[^)]+)\)/);
+    if (linkMatch) lines.push(`  Learn more: ${linkMatch[1]}`);
+
+    const items = audit.details?.items;
+    if (items && items.length > 0) {
+      for (const item of items.slice(0, 3)) {
+        lines.push(...formatDetailItem(item));
+      }
+      if (items.length > 3) {
+        lines.push(`  ... and ${items.length - 3} more`);
+      }
+    }
+    lines.push("");
   }
 
   return lines;
@@ -174,6 +194,8 @@ function formatFailingAudits(audits: Record<string, PsiAudit>, categoryRefs: str
   return lines;
 }
 
+// --- Single URL formatting (existing) ---
+
 function formatPagespeed(url: string, result: PsiResult): string {
   const lhr = result.lighthouseResult;
   const lines: string[] = [
@@ -258,12 +280,234 @@ function formatPagespeed(url: string, result: PsiResult): string {
   return lines.join("\n");
 }
 
+// --- Batch formatting ---
+
+function shortUrl(url: string, allUrls: string[]): string {
+  try {
+    const u = new URL(url);
+    const path = u.pathname + u.search;
+    const hasDuplicate = allUrls.some(
+      (other) => other !== url && new URL(other).pathname + new URL(other).search === path,
+    );
+    return hasDuplicate ? u.hostname + path : path;
+  } catch {
+    return url;
+  }
+}
+
+function formatDelta(a: number | null, b: number | null): string {
+  if (a === null || b === null) return "";
+  const diff = b - a;
+  if (diff === 0) return "  (=)";
+  return diff > 0 ? `  (+${diff})` : `  (${diff})`;
+}
+
+function formatBatchCompare(results: Array<{ url: string; result: PsiResult }>, strategy: string): string {
+  const [a, b] = results;
+  const lhrA = a.result.lighthouseResult;
+  const lhrB = b.result.lighthouseResult;
+
+  const lines: string[] = [
+    `=== PageSpeed Compare (${strategy}) ===`,
+    ``,
+    `A: ${a.url}`,
+    `B: ${b.url}`,
+    `Lighthouse: ${lhrA.lighthouseVersion}`,
+    "",
+    "--- Scores ---",
+    "",
+  ];
+
+  // Score comparison
+  const catIds = Object.keys(lhrA.categories);
+  for (const id of catIds) {
+    const catA = lhrA.categories[id];
+    const catB = lhrB.categories[id];
+    if (!catA || !catB) continue;
+    const pA = scorePct(catA.score);
+    const pB = scorePct(catB.score);
+    const delta = formatDelta(pA, pB);
+    lines.push(`${catA.title}: ${pA ?? "N/A"} → ${pB ?? "N/A"}${delta}`);
+  }
+  lines.push("");
+
+  // CWV comparison
+  const cwvIds = [
+    "first-contentful-paint",
+    "largest-contentful-paint",
+    "total-blocking-time",
+    "cumulative-layout-shift",
+    "speed-index",
+    "interactive",
+  ];
+  const cwvLines: string[] = [];
+  for (const id of cwvIds) {
+    const auditA = lhrA.audits[id];
+    const auditB = lhrB.audits[id];
+    if (!auditA?.displayValue || !auditB?.displayValue) continue;
+    cwvLines.push(`${auditA.title}: ${auditA.displayValue} → ${auditB.displayValue}`);
+  }
+  if (cwvLines.length > 0) {
+    lines.push("--- Core Web Vitals (Lab) ---", "", ...cwvLines, "");
+  }
+
+  // Opportunities unique to each / shared
+  const oppsA = collectOpportunityIds(lhrA.audits);
+  const oppsB = collectOpportunityIds(lhrB.audits);
+  const onlyA = [...oppsA].filter((id) => !oppsB.has(id));
+  const onlyB = [...oppsB].filter((id) => !oppsA.has(id));
+  const shared = [...oppsA].filter((id) => oppsB.has(id));
+
+  if (onlyA.length > 0) {
+    lines.push(`--- Opportunities (A only) ---`, "");
+    for (const id of onlyA) lines.push(`  ${lhrA.audits[id].title}: ${lhrA.audits[id].displayValue ?? ""}`);
+    lines.push("");
+  }
+  if (onlyB.length > 0) {
+    lines.push(`--- Opportunities (B only) ---`, "");
+    for (const id of onlyB) lines.push(`  ${lhrB.audits[id].title}: ${lhrB.audits[id].displayValue ?? ""}`);
+    lines.push("");
+  }
+  if (shared.length > 0) {
+    lines.push(`--- Shared Opportunities ---`, "");
+    for (const id of shared) {
+      lines.push(
+        `  ${lhrA.audits[id].title}: ${lhrA.audits[id].displayValue ?? ""} → ${lhrB.audits[id].displayValue ?? ""}`,
+      );
+    }
+    lines.push("");
+  }
+
+  const timeA = (lhrA.timing.total / 1000).toFixed(1);
+  const timeB = (lhrB.timing.total / 1000).toFixed(1);
+  lines.push(`Analysis took ${timeA}s + ${timeB}s`);
+
+  return lines.join("\n");
+}
+
+function formatBatchTable(results: Array<{ url: string; result: PsiResult }>, strategy: string): string {
+  const lines: string[] = [`=== Batch PageSpeed (${results.length} URLs, ${strategy}) ===`, ""];
+
+  // Collect all category IDs from first result
+  const catIds = Object.keys(results[0].result.lighthouseResult.categories);
+  const catNames = catIds.map((id) => results[0].result.lighthouseResult.categories[id].title);
+
+  // Score table
+  lines.push("--- Scores ---", "");
+
+  // Header
+  const urlCol = "URL";
+  const allUrls = results.map((r) => r.url);
+  const urlWidth = Math.max(urlCol.length, ...results.map((r) => shortUrl(r.url, allUrls).length));
+  const colWidth = Math.max(...catNames.map((n) => n.length), 4);
+  lines.push(`${urlCol.padEnd(urlWidth)}  ${catNames.map((n) => n.padEnd(colWidth)).join("  ")}`);
+
+  // Rows
+  let bestPerf: { url: string; score: number } | null = null;
+  let worstPerf: { url: string; score: number } | null = null;
+
+  for (const { url, result } of results) {
+    const lhr = result.lighthouseResult;
+    const scores = catIds.map((id) => {
+      const s = scorePct(lhr.categories[id]?.score);
+      return s !== null ? String(s) : "N/A";
+    });
+    lines.push(`${shortUrl(url, allUrls).padEnd(urlWidth)}  ${scores.map((s) => s.padEnd(colWidth)).join("  ")}`);
+
+    const perf = scorePct(lhr.categories.performance?.score);
+    if (perf !== null) {
+      if (!bestPerf || perf > bestPerf.score) bestPerf = { url: shortUrl(url, allUrls), score: perf };
+      if (!worstPerf || perf < worstPerf.score) worstPerf = { url: shortUrl(url, allUrls), score: perf };
+    }
+  }
+
+  lines.push("");
+  if (bestPerf) lines.push(`Best: ${bestPerf.url} (${bestPerf.score})`);
+  if (worstPerf && worstPerf.url !== bestPerf?.url) lines.push(`Worst: ${worstPerf.url} (${worstPerf.score})`);
+  lines.push("");
+
+  // Shared opportunities across pages
+  const oppCounts = new Map<string, { title: string; count: number }>();
+  for (const { result } of results) {
+    for (const id of collectOpportunityIds(result.lighthouseResult.audits)) {
+      const existing = oppCounts.get(id);
+      if (existing) {
+        existing.count++;
+      } else {
+        oppCounts.set(id, { title: result.lighthouseResult.audits[id].title, count: 1 });
+      }
+    }
+  }
+
+  const sharedOpps = [...oppCounts.entries()].filter(([, v]) => v.count >= 2).sort((a, b) => b[1].count - a[1].count);
+  if (sharedOpps.length > 0) {
+    lines.push("--- Shared Opportunities ---", "");
+    for (const [, { title, count }] of sharedOpps.slice(0, 10)) {
+      lines.push(`  ${title} (${count}/${results.length} pages)`);
+    }
+    lines.push("");
+  }
+
+  const totalTime = results.reduce((sum, r) => sum + r.result.lighthouseResult.timing.total, 0);
+  lines.push(`Total analysis time: ${(totalTime / 1000).toFixed(1)}s`);
+
+  return lines.join("\n");
+}
+
+function collectOpportunityIds(audits: Record<string, PsiAudit>): Set<string> {
+  const ids = new Set<string>();
+  for (const [id, audit] of Object.entries(audits)) {
+    if (audit.score === null || audit.score >= 1) continue;
+    const mode = audit.scoreDisplayMode;
+    const hasNumeric = audit.numericValue && audit.numericValue > 0;
+    if (mode === "metricSavings" || ((mode === "numeric" || mode === "binary") && hasNumeric)) {
+      if (hasNumeric || (audit.details?.items?.length ?? 0) > 0) {
+        ids.add(id);
+      }
+    }
+  }
+  return ids;
+}
+
+async function runBatch(
+  urls: string[],
+  options: { strategy?: "mobile" | "desktop"; categories?: PsiCategoryType[]; locale?: string },
+): Promise<Array<{ url: string; result?: PsiResult; error?: string }>> {
+  const concurrency = 2;
+  const results: Array<{ url: string; result?: PsiResult; error?: string }> = [];
+  let i = 0;
+
+  while (i < urls.length) {
+    const batch = urls.slice(i, i + concurrency);
+    const settled = await Promise.all(
+      batch.map(async (url) => {
+        try {
+          const result = await runPagespeed(url, options);
+          return { url, result };
+        } catch (err) {
+          return { url, error: err instanceof Error ? err.message : String(err) };
+        }
+      }),
+    );
+    results.push(...settled);
+    i += concurrency;
+  }
+
+  return results;
+}
+
 export function registerPagespeedTool(server: McpServer): void {
   server.tool(
     "pagespeed",
-    "Analyze a page's performance using Google PageSpeed Insights API. Returns Lighthouse scores, Core Web Vitals (lab + field), opportunities, and diagnostics.",
+    "Analyze page performance using Google PageSpeed Insights. Accepts a single URL or multiple URLs (batch mode). With 2 URLs, returns a side-by-side comparison with deltas. With 3-10 URLs, returns a summary table with shared opportunities.",
     {
-      url: z.string().url().describe("The URL to analyze."),
+      url: z.string().url().optional().describe("Single URL to analyze. Use this OR urls, not both."),
+      urls: z
+        .array(z.string().url())
+        .min(2)
+        .max(10)
+        .optional()
+        .describe("Multiple URLs (2-10) for batch analysis. 2 URLs = compare mode, 3+ = summary table."),
       strategy: z.enum(["mobile", "desktop"]).optional().describe("Device strategy. Default: 'mobile'."),
       categories: z
         .array(z.enum(["performance", "accessibility", "best-practices", "seo"]))
@@ -271,18 +515,61 @@ export function registerPagespeedTool(server: McpServer): void {
         .describe("Lighthouse categories to run. Default: all four."),
       locale: z.string().optional().describe("Locale for localized results (e.g., 'pt-BR', 'en')."),
     },
-    async ({ url, strategy, categories, locale }) => {
-      try {
-        const result = await runPagespeed(url, {
-          strategy: strategy as "mobile" | "desktop" | undefined,
-          categories: categories as PsiCategoryType[] | undefined,
-          locale,
-        });
-        return { content: [{ type: "text", text: formatPagespeed(url, result) }] };
-      } catch (err) {
-        const msg = err instanceof Error ? err.message : String(err);
-        return { content: [{ type: "text", text: `Error running PageSpeed analysis: ${msg}` }] };
+    async ({ url, urls, strategy, categories, locale }) => {
+      const strat = (strategy as "mobile" | "desktop") ?? "mobile";
+      const cats = categories as PsiCategoryType[] | undefined;
+      const opts = { strategy: strat, categories: cats, locale };
+
+      // Validate: must provide url or urls, not both
+      if (url && urls) {
+        return {
+          content: [{ type: "text", text: "Error: provide either 'url' (single) or 'urls' (batch), not both." }],
+        };
+      }
+      if (!url && !urls) {
+        return {
+          content: [{ type: "text", text: "Error: provide 'url' for single analysis or 'urls' for batch analysis." }],
+        };
+      }
+
+      // Single URL — existing behavior
+      if (url) {
+        try {
+          const result = await runPagespeed(url, opts);
+          return { content: [{ type: "text", text: formatPagespeed(url, result) }] };
+        } catch (err) {
+          const msg = err instanceof Error ? err.message : String(err);
+          return { content: [{ type: "text", text: `Error running PageSpeed analysis: ${msg}` }] };
+        }
       }
+
+      // Batch mode
+      const batchUrls = urls as string[];
+      const results = await runBatch(batchUrls, opts);
+
+      // Separate successes and failures
+      const successes = results.filter((r): r is { url: string; result: PsiResult } => !!r.result);
+      const failures = results.filter((r): r is { url: string; error: string } => !!r.error);
+
+      if (successes.length === 0) {
+        const errorLines = failures.map((f) => `${f.url}: ${f.error}`);
+        return { content: [{ type: "text", text: `All URLs failed:\n${errorLines.join("\n")}` }] };
+      }
+
+      let output: string;
+      if (successes.length === 2) {
+        output = formatBatchCompare(successes, strat);
+      } else {
+        output = formatBatchTable(successes, strat);
+      }
+
+      // Append any failures
+      if (failures.length > 0) {
+        const errorLines = failures.map((f) => `${f.url}: ${f.error}`);
+        output += `\n\n--- Errors ---\n${errorLines.join("\n")}`;
+      }
+
+      return { content: [{ type: "text", text: output }] };
     },
   );
 }