aeo-ready 1.2.0 → 1.3.0

package/README.md

@@ -1,115 +1,91 @@
-# agent-web
+# aeo-ready
 
-Is your site AI-ready? One scan, two scorecards, one score.
+AEO benchmark aggregator. One scan, every score.
 
 ```bash
-npx agent-web scan --url https://your-site.com
+npx aeo-ready scan yoursite.com
 ```
 
-## What it measures
+## What it does
 
-**Agent Readiness** (0-50) — Can AI agents discover, parse, and act on your site?
-- Discovery: llms.txt, robots.txt AI crawlers, sitemap, meta tags
-- Content Structure: markdown availability, heading hierarchy, token budgets, front-loading
-- Capability Signaling: AGENTS.md, agents.json, OpenAPI, content negotiation
-- Actionable: machine-readable contact, pricing, API endpoints, SDK manifest
+Runs every major AEO (Agentic Engine Optimization) benchmark against your site in one command. Shows per-check pass/fail, company comparisons, and tracks scores over time.
 
-**AI Visibility** (0-50) — Does your content get cited in AI-generated responses?
-- Structured Data: schema.org, FAQ markup, rich schemas, Open Graph
-- Citation Readiness: direct answer formatting, question headings, citable structure
-- Authority: E-E-A-T signals, entity optimization, external validation
-- Freshness: modification dates, publication cadence, content recency
+## Sources
 
-**Overall: X/100** with letter grade (A-F).
+| Benchmark | What it checks | Checks |
+|-----------|---------------|--------|
+| **agentic-seo** (Addy Osmani) | Discovery, content structure, token economics, capability signaling, UX bridge | 10 |
+| **Cloudflare** (isitagentready.com) | Discoverability, content accessibility, bot access, API/auth/MCP/A2A discovery, commerce | 19 |
+| **Fern** (afdocs) | llms.txt quality, markdown availability, page size, content structure, URL stability, auth | 23 |
 
 ## Usage
 
 ```bash
-# Audit a live URL
-npx agent-web scan --url https://example.com
+npx aeo-ready scan yoursite.com                   # scan a URL (remote checks)
+npx aeo-ready scan yoursite.com --dir ./public    # full scan (local + remote)
+npx aeo-ready scan yoursite.com --json            # JSON output for CI
+npx aeo-ready scan yoursite.com --threshold 60    # exit 1 if below
+```
 
-# Audit current directory (repo mode)
-npx agent-web scan
+### Why `--dir`?
 
-# JSON output for CI pipelines
-npx agent-web scan --json --threshold 60
+agentic-seo scores ~23/100 in URL-only mode because most checks (content structure, token economics, capability signaling, UX bridge) need filesystem access. Pass `--dir` to your build output or public directory to get the real score.
 
-# Skip benchmark comparison
-npx agent-web scan --no-benchmark
+```
+URL-only:  agentic-seo 23/100 (F)
+With --dir: agentic-seo 92/100 (A)
 ```
 
 ## Output
 
 ```
-  agent-web — AI readiness audit
-
-  Mode: url | Type: saas
+  aeo-ready — AEO benchmark aggregator
 
-  Overall: C 50/100
+  ─── Benchmarks ────────────────────────────────────
 
-  Agent Readiness: 22/50
-    discovery: 3/12
-      + AI-friendly meta tags [3]
-      - llms.txt [0/4]
-        fix: No llms.txt found.
-      - robots.txt AI crawlers [0/3]
-        fix: robots.txt exists but doesn't mention AI crawlers.
+  ███████████████░ agentic-seo      92/100 (A)
+    ✓ Discovery              25/25
+    ◑ Content Structure      19/25
+    ✓ Token Economics        25/25
+    ✓ Capability Signaling   15/15
+    ✓ UX Bridge              8/10
+    compare: Cloudflare 55 · Supabase 52 · Vercel 48
 
-  AI Visibility: 28/50
-    structured Data: 6/12
-      + FAQ markup [3]
-      + Rich schemas [3]
-      - Schema.org markup [0/4]
-        fix: type doesn't match site category.
-
-  Second opinion: agentic-seo scored you 45/100
-```
+  █████████████░░░ Cloudflare       4/5 (B)
+    + robotsTxt, + sitemap, + linkHeaders, + agentSkills...
+    compare: Cloudflare 5 · Vercel 4 · Supabase 3
 
-## CI Mode
+  █████████████░░░ Fern             83/100 (B)
+    + llms-txt-exists, + rendering-strategy, - content-negotiation...
+    compare: Stripe 85 · Supabase 78 · Anthropic 72
 
-Exit with code 1 if below threshold:
+  Average across all sources: 85/100
 
-```bash
-npx agent-web scan --json --threshold 60
+  Fix it:
+    npx agentic-seo init          scaffold llms.txt, AGENTS.md
+    Fern: 6 issues — run npx afdocs https://yoursite.com
 ```
 
-Add to GitHub Actions:
+## CI Mode
 
 ```yaml
-- run: npx agent-web scan --threshold 50
+- run: npx aeo-ready scan yoursite.com --dir ./public --threshold 50
 ```
 
-## Benchmark
+## Dashboard
 
-Runs `npx agentic-seo --json` as an independent second opinion. Shows their score alongside yours. If agentic-seo isn't installed, the benchmark line is skipped gracefully.
-
-## History
+Each scan generates a self-contained HTML dashboard at `.aeo-ready/dashboard.html` with:
+- Score cards for each benchmark
+- Per-check detail (expandable)
+- Company comparisons
+- Score trends over time (inline SVG)
+- Scan history with deltas
 
-Each scan saves to `.agent-web/history.json`. Gitignored by default.
+Auto-opens in browser after each scan.
 
-## Site Type Detection
-
-Automatically infers site type from signals:
-- **SaaS** — pricing + auth pages detected
-- **API/Developer Tool** — /docs, /api, SDK references
-- **Content/Blog** — articles, blog posts, publication cadence
-- **Personal/Portfolio** — portfolio, about me, single person
-
-Scoring adjusts by type (e.g., OpenAPI is required for SaaS, N/A for personal sites).
-
-## Coming Soon
-
-- `--fix` mode: generate missing files (robots.txt, meta tags, structured data)
-- HTML dashboard with score trends over time
-- Citation correlation: query AI models and track whether fixes improve citations
-
-## Also available as a Claude Code skill
-
-```bash
-npx skills add katrinalaszlo/agent-web
-```
+## History
 
-Then use `/agent-web` or `/agent-web https://example.com` inside Claude Code for an interactive audit that also generates missing files.
+Scores persist in `.aeo-ready/history.json`. Re-scan to track improvement over time.
 
 ## Author
 

package/bin/cli.js

@@ -20,7 +20,13 @@ program
 
 program
   .command("scan [url]")
-  .description("Run all AEO benchmarks against a URL")
+  .description(
+    "Run all AEO benchmarks against a URL (add --dir for local scanning)",
+  )
+  .option(
+    "-d, --dir <path>",
+    "Local directory to scan (gives agentic-seo full access)",
+  )
   .option("--json", "Output results as JSON")
   .option(
     "--threshold <number>",
@@ -32,10 +38,15 @@ program
       if (url && !url.startsWith("http")) url = `https://${url}`;
       if (!url) {
         console.error("  Usage: npx aeo-ready scan <url>");
+        console.error("         npx aeo-ready scan <url> --dir ./public");
         process.exit(1);
       }
 
-      const result = await scan({ url, json: opts.json || false });
+      const result = await scan({
+        url,
+        dir: opts.dir || null,
+        json: opts.json || false,
+      });
 
       if (opts.json) {
         process.stdout.write(JSON.stringify(result, null, 2) + "\n");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "aeo-ready",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "AEO benchmark aggregator. One scan, every score. Collects agentic-seo, Cloudflare, and Fern in one report.",
   "type": "module",
   "bin": {

package/src/benchmark/agentic-seo.js

@@ -1,14 +1,109 @@
 import { execSync } from "child_process";
+import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from "fs";
+import { tmpdir } from "os";
+import { join, dirname } from "path";
+
+const KNOWN_FILES = [
+  "robots.txt",
+  "llms.txt",
+  "llms-full.txt",
+  "AGENTS.md",
+  "CLAUDE.md",
+  "skill.md",
+  "agent-permissions.json",
+  "agents.json",
+  "sitemap.xml",
+  ".well-known/ai-plugin.json",
+];
+
+async function fetchText(url) {
+  try {
+    const res = await fetch(url, { redirect: "follow" });
+    if (!res.ok) return null;
+    return await res.text();
+  } catch {
+    return null;
+  }
+}
+
+function parseSitemapUrls(xml, baseUrl) {
+  const urls = [];
+  const matches = xml.matchAll(/<loc>([^<]+)<\/loc>/g);
+  for (const m of matches) {
+    const loc = m[1].trim();
+    if (loc.startsWith(baseUrl)) urls.push(loc);
+  }
+  return urls;
+}
+
+function urlToFilePath(url, baseUrl) {
+  let path = new URL(url).pathname;
+  if (path.endsWith("/")) path += "index.html";
+  else if (!path.includes(".")) path += ".html";
+  return path.replace(/^\//, "");
+}
+
+async function fetchSiteToDir(baseUrl) {
+  const tempDir = mkdtempSync(join(tmpdir(), "aeo-"));
+
+  for (const file of KNOWN_FILES) {
+    const content = await fetchText(`${baseUrl}/${file}`);
+    if (content) {
+      const filePath = join(tempDir, file);
+      mkdirSync(dirname(filePath), { recursive: true });
+      writeFileSync(filePath, content);
+    }
+  }
+
+  const sitemap = await fetchText(`${baseUrl}/sitemap.xml`);
+  if (!sitemap) return tempDir;
+
+  const urls = parseSitemapUrls(sitemap, baseUrl);
+
+  await Promise.all(
+    urls.map(async (url) => {
+      const path = urlToFilePath(url, baseUrl);
+      if (KNOWN_FILES.includes(path)) return;
+
+      const html = await fetchText(url);
+      if (html) {
+        const htmlPath = join(tempDir, path);
+        mkdirSync(dirname(htmlPath), { recursive: true });
+        writeFileSync(htmlPath, html);
+      }
+
+      const mdUrl = url
+        .replace(/\.html$/, ".md")
+        .replace(/\/?$/, (m) => (m === "/" ? "/index.md" : ".md"));
+      if (mdUrl !== url) {
+        const md = await fetchText(mdUrl);
+        if (md) {
+          const mdPath = join(tempDir, path.replace(/\.html$/, ".md"));
+          mkdirSync(dirname(mdPath), { recursive: true });
+          writeFileSync(mdPath, md);
+        }
+      }
+    }),
+  );
+
+  return tempDir;
+}
 
 export async function runBenchmark(target) {
+  let tempDir = null;
+
   try {
-    const args =
-      target && target.startsWith("http")
-        ? `--url ${target} --json`
-        : `${target || "."} --json`;
+    let scanDir;
 
-    const output = execSync(`npx agentic-seo ${args}`, {
-      timeout: 30000,
+    if (target && target.startsWith("http")) {
+      tempDir = await fetchSiteToDir(target);
+      scanDir = tempDir;
+    } else {
+      scanDir = target || ".";
+    }
+
+    const output = execSync(`npx agentic-seo ${scanDir} --json`, {
+      timeout: 60000,
       encoding: "utf8",
       stdio: ["pipe", "pipe", "pipe"],
     });
@@ -36,5 +131,11 @@ export async function runBenchmark(target) {
       available: false,
       reason: err.message?.slice(0, 100),
     };
+  } finally {
+    if (tempDir) {
+      try {
+        rmSync(tempDir, { recursive: true, force: true });
+      } catch {}
+    }
   }
 }

package/src/benchmark/index.js

@@ -27,11 +27,11 @@ const REFERENCE_SCORES = {
   },
 };
 
-export async function runAllBenchmarks(target) {
+export async function runAllBenchmarks(target, dir) {
   const isUrl = target && target.startsWith("http");
 
   const results = await Promise.allSettled([
-    runAgenticSeo(target),
+    runAgenticSeo(dir || target),
     isUrl ? runCloudflare(target) : Promise.resolve(null),
     isUrl ? runFern(target) : Promise.resolve(null),
   ]);

package/src/scan.js

@@ -5,7 +5,7 @@ import { generateDashboard } from "./dashboard/generate.js";
 import { exec } from "child_process";
 
 export async function scan(opts) {
-  const { url, json } = opts;
+  const { url, dir, json } = opts;
 
   if (!json) {
     console.log(
@@ -14,7 +14,7 @@ export async function scan(opts) {
     console.log(chalk.dim(`  Scanning ${url}...\n`));
   }
 
-  const benchmarks = await runAllBenchmarks(url);
+  const benchmarks = await runAllBenchmarks(url, dir);
   const scores = collectScores(benchmarks);
   const averageScore =
     scores.length > 0