@nurkamol/seo-audit 1.33.1 → 1.35.0

package/README.md

@@ -14,6 +14,8 @@
 <p align="center">
   <a href="https://github.com/marketplace/actions/full-site-seo-audit"><img src="https://img.shields.io/badge/GitHub%20Marketplace-Full--site%20SEO%20Audit-f97316?logo=github&logoColor=white" alt="GitHub Marketplace"></a>
   <a href="https://github.com/nurkamol/seo-audit/actions/workflows/test.yml"><img src="https://github.com/nurkamol/seo-audit/actions/workflows/test.yml/badge.svg" alt="tests"></a>
+  <a href="https://github.com/nurkamol/seo-audit/releases/latest"><img src="https://img.shields.io/badge/Download-macOS%20app-0a0a0a?logo=apple&logoColor=white" alt="Download the macOS app"></a>
+  <a href="https://www.npmjs.com/package/@nurkamol/seo-audit"><img src="https://img.shields.io/npm/v/@nurkamol/seo-audit?color=cb3837&logo=npm&logoColor=white&label=npm" alt="npm"></a>
   <a href="https://github.com/nurkamol/seo-audit/releases"><img src="https://img.shields.io/github/v/release/nurkamol/seo-audit?color=f97316" alt="release"></a>
   <img src="https://img.shields.io/badge/node-%E2%89%A518-3c873a" alt="node >= 18">
   <img src="https://img.shields.io/badge/dependencies-0-brightgreen" alt="zero dependencies">
@@ -279,6 +281,23 @@ Interactive by nature, so it is deliberately **not** a GitHub Action input: a
 flag CI can accept and never satisfy is worse than no flag. In CI, set the
 three variables as secrets.
 
+It also brings back **where Google puts each page**, and what it is found for:
+
+- Every finding on a page Google ranks carries that page's average position, so
+  the report can put a template on page two ahead of one nobody has been shown.
+  Positions were in every response Google has ever sent and were being discarded.
+- **`search-console-striking`** names the crawled pages sitting at positions 11
+  to 20 — page two, where the click-through rate is roughly nothing and the
+  ranking is already earned — with the query each one is closest on. It is the
+  only list in this tool that is an opportunity rather than a fault, and every
+  number in it was measured by Google. Moving one of those up two places is
+  usually less work than a new page.
+
+Neither needs a second account, a scrape, or a keyword provider: it is the same
+connection, asked one more question. Google withholds any search too rare to be
+anonymous, so a low-traffic property gets positions and no queries — and the
+report says so rather than leaving silence to read as "found for nothing".
+
 The whole setup, the property-naming trap and what each failure note means:
 [docs/search-console.md](docs/search-console.md). A domain property is named `sc-domain:example.com` rather than by
 its URL. Missing credentials, or a property the account cannot read, are a note
@@ -463,8 +482,10 @@ reason.
 | `--exclude <glob>` | — | Leave URLs out of the crawl. Repeatable; `*` stops at a slash, `**` does not |
 | `--dry-run` | — | Say what would be crawled and stop. A handful of requests instead of hundreds |
 | `--write-sitemap <file>` | — | Write the sitemap this site should have had. Refuses on a crawl that did not see the whole site |
+| `--write-llms <file>` | — | Write the `llms.txt` this site should have had, in the [llmstxt.org](https://llmstxt.org) format, from the site's own titles and descriptions. Nothing is generated or rewritten. Refuses on a partial crawl, for the same reason |
+| `--write-schema <file>` | — | Write the JSON-LD this site could add — `WebSite`, `Organization`, `BreadcrumbList` — built **only** from strings the crawl read off the site. A step it cannot name from the site's own words is skipped, never invented from a slug |
 | `--json <file>` | — | Write a JSON report — findings, the grouped `causes` with their scope lines, and `meta`. Also usable as a baseline, which carries the findings only |
-| `--csv <file>` | — | Write the findings as a spreadsheet, one row each |
+| `--csv <file>` | — | Write the checklist as a spreadsheet: one row per finding with a `points` column for what fixing it is worth, then the checks that passed (`pass`) and the ones that did not apply (`not-checked`) |
 | `--baseline <file>` | — | Compare against a previous `--json` run; show only what changed |
 | `--update-baseline` | — | Rewrite the baseline after comparing |
 | `--limit <n>` | 200 | Maximum pages to check |
@@ -480,7 +501,7 @@ reason.
 | `--psi <urls>` | — | Measure these pages with PageSpeed Insights. A path glob names a section (see below) |
 | `--psi-sample <n>` | 3 | Pages measured per section glob |
 | `--psi-strategy` | `mobile` | `mobile` or `desktop` |
-| `--against <url>` | — | Compare against another deployment now — preview vs production |
+| `--against <url>` | — | Compare against another deployment now — a rebuild against the site it replaces. Findings are matched by path, so the two hosts need not be the same |
 | `--settle <s>` | — | Wait until the site serves consistent HTML before crawling |
 | `--fail-on <level>` | `error` | Exit 1 at `error`, `warn`, `new`, or `never` |
 | `--version` | — | Print the version |
@@ -611,6 +632,31 @@ survivable in CI instead of being switched off in week two.
 
 ---
 
+## The window, on Linux and Windows
+
+The macOS app is a thin client over a local server, and that server runs anywhere Node does:
+
+```bash
+npx @nurkamol/seo-audit --serve
+```
+
+It opens a browser onto the same window the macOS app draws: a sidebar of kept runs down the left, the report beside it, score ring and all. Everything the command line takes is in the form — the same table that decides what the macOS window reaches decides what this draws, so neither can quietly fall behind the other. Finished runs are kept and listed at `/reports`, two of them can be compared, and on macOS it is **the same folder the app uses**: a crawl started in the window is in the browser's list a second later, because there is one folder rather than two.
+
+Nothing leaves the machine. It binds to the loopback address, which is the whole of its security model.
+
+There is also a **native window for Windows and Linux** built on exactly this: a
+Tauri shell that starts the same server and shows the same report, in
+[`desktop/`](desktop/README.md). It ships a Node inside it, so there is nothing
+to install first. Every release attaches a `setup.exe`, a `.deb` and an
+AppImage, each built on its own runner and then **installed and run there**
+before it is attached. It tells you when there is a new version and offers
+whatever is safe for the way you installed it — `winget upgrade` in place, or
+the command to run, or the release page.
+
+`--no-open` if you would rather it did not open a browser. It opens one when a person ran the command and never when something else did, so the macOS window — which spawns this — is unaffected.
+
+---
+
 ## Hosting it, for people who will not open a terminal
 
 Optional, and off the main path. Everything above is free and runs on your own
@@ -741,6 +787,8 @@ Findings come at three levels: **error** (wrong, and costing traffic), **warning
 | No `lastmod` is in the future | warning |
 | A favicon Google can use — the home page declares one that loads, or `/favicon.ico` is there | warning / note |
 | `llms.txt` exists | note |
+| **Which AI crawlers robots.txt lets in** — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and eight more, asked of the same parser Google's rules go through. Split by what blocking costs: an *answering* crawler fetches because somebody asked a question just now, a *training* one does not, and blocking the second changes nothing about being cited today. Always a **note**: refusing an AI crawler is a decision a publisher is entitled to make. It also says whether anybody made it — a block that arrives through `User-agent: *` is usually a CDN or plugin default | note |
+| `llms.txt` and `robots.txt` do not contradict each other — a site that serves a file whose only purpose is to tell an assistant what to read, while disallowing the agent that would read it, has one of the two files wrong | warning |
 | Everything once-per-domain is read on the host that answers — audit `example.com` when the site lives at `www.` and the audit moves there, saying so, rather than reading robots.txt off a 301 | note |
 | `http://`, `www.` and `https://www.` each reach the canonical host in one hop — a variant that answers 429 is reported as **not checked**, never as dead | warning / note |
 | The TLS certificate is not expired, and not expiring within 14 days | error / warning |
@@ -760,12 +808,98 @@ Findings come at three levels: **error** (wrong, and costing traffic), **warning
 
 ---
 
+## What it can write for you
+
+Three of the outputs are not reports. They are the fix, built from what the crawl already read:
+
+| | |
+|---|---|
+| `--write-sitemap` | Every page that answered 200, is HTML, is indexable and is its own canonical |
+| `--write-llms` | The [llms.txt](https://llmstxt.org) — the site's own titles and descriptions, grouped by section |
+| `--write-schema` | `WebSite`, `Organization` and `BreadcrumbList` as JSON-LD |
+
+All three follow one rule and it is absolute: **every value is a string this crawl read off this site.** Nothing is generated, rewritten, summarised or inferred. A page with no description gets a line without one rather than a sentence somebody made up about it; an organisation is named only when the site names itself in `og:site_name`, because a wrong company name is the worst thing in that file to get wrong.
+
+Breadcrumbs make the rule concrete. A `<title>` is not a breadcrumb name — the first live run of this produced `Assets | Jekyll • Simple, blog-aware, static sites` as a step. So a step is named by the page's `<h1>` when it has exactly one, or by the words the site's own navigation uses to link to it, and by nothing else:
+
+```
+  wrote schema.json — 61 block(s) for 61 page(s)
+    60   BreadcrumbList          Jekyll › Docs › Assets
+    1    Organization
+    skipped 142 (a step could not be named from the site's own words)
+    skipped 8   (is at the top of the site, so there is no trail to describe)
+    skipped 1   (already declares a WebSite)
+```
+
+Structured data that describes a site inaccurately is worse than none: it is a machine-readable claim the page does not support, which is a manual-action category at Google. So 142 pages were skipped rather than given a name invented from a URL.
+
+And all three refuse outright on a crawl that did not see the whole site, because a file built from a third of a site is worse than no file — it looks complete. The refusal names the run that would work.
+
+---
+
+## The score
+
+Every report opens with a number out of 100, a grade and a ring:
+
+```
+  https://example.com
+  31 pages · 71 requests · 12.9s
+
+  74/100   C   ████████████████████░░░░░░░░
+  26.1 points across 9 checks · 61 passed · 22 did not apply
+  Clear the errors alone and it is 88.
+```
+
+It is a checklist that has been counted, and the arithmetic is small enough to print:
+
+| | |
+|---|---|
+| **A run starts at 100** and pays for what is wrong with it | Nothing is added for passing; a clean site is at 100 because nothing took points off |
+| **An error-level check costs 12 points, a warning 4** | Not a new judgement — every check already carries a level, argued over check by check when it was written. `scripts/check-levels.mjs` reads those levels back out of the source and a test asserts the table still matches, so a check promoted from warning to error cannot keep its old weight |
+| **A check on some pages costs its share** | A missing `<h1>` on 3 of 40 pages costs a tenth of a warning; the same fault on all 40 costs the whole of one |
+| **Notes cost nothing** | A note is "worth knowing, may be deliberate". An `llms.txt` nobody wanted is not a fault |
+| **A check that could not run is skipped, not passed** | A site with no images has not passed the alt-text check, and a run without `--psi` has not passed the performance ones. Both would be free points for doing less |
+
+So the score is an amount of known, named, locatable work subtracted from a clean sheet — never a prediction of a ranking, an estimate of traffic, or a grade against anybody else's site. What a check costs is what fixing it is worth, and every piece of work under **Start here** carries the points it returns:
+
+```
+    +4.0  ✗ No <h1>                    17 pages under /classes/, 62% of the crawl
+    +2.7  ! og:image is WebP           17 pages under /classes/
+    +1.3  ! Meta description will be cut off   4 pages under /journal/
+```
+
+Two runs of the same site are directly comparable. A run of one site against another is comparable to the degree that the same checks applied to both, which the report says out loud.
+
+### What passed, and what was never checked
+
+A missing finding reads exactly like a passing one, so both are named. **Passing** lists every check the site cleared, in its own words:
+
+```
+  ✓ Every page has a title
+  ✓ Every image has an alt attribute
+  ✓ No og:image is WebP
+```
+
+**Not checked** says why the rest did not apply, grouped by reason:
+
+```
+  · No page declares hreflang. (hreflang-invalid, hreflang-one-way, hreflang-dead, …)
+  · PageSpeed was not asked — run with --psi. (psi-score, psi-lcp, psi-cls, …)
+```
+
+The full checklist — every scored check with its weight, its area and its pass line — is served at `/checks` when the [hosted front end](#hosting-it-for-people-who-will-not-open-a-terminal) is running, so "what does this thing actually check" is a question with a fetchable answer.
+
+---
+
 ## Reading the output
 
 ```
   https://example.com
   31 pages · 71 requests · 12.9s
 
+  74/100   C   ████████████████████░░░░░░░░
+  26.1 points across 9 checks · 61 passed · 22 did not apply
+
   ✗ No <h1> ×17
     The page has no headline.
     · https://example.com/schedule/
@@ -784,7 +918,9 @@ Findings come at three levels: **error** (wrong, and costing traffic), **warning
   20 error  91 warning  12 note
 ```
 
-Findings are grouped by check, not by page, because the fix is usually one change applied everywhere.
+Findings are grouped by check, not by page, because the fix is usually one change applied everywhere. Errors and warnings come first; notes follow under **Worth knowing**, which says out loud that none of them cost the score anything.
+
+Then **Passing**, then **Not checked**.
 
 Some warnings are meant to be lived with. A contact page is *supposed* to be short; a privacy policy has no business carrying editorial links. The tool reports what is true and leaves the judgement to you — it has no way to know which pages are meant to rank.
 
@@ -826,6 +962,13 @@ Two rules that keep the tool trustworthy:
 1. **No false positives.** A check that cries wolf gets the whole report ignored. If a pattern is sometimes legitimate, it is a `note`, not an `error`.
 2. **No dependencies.** It must keep running with a bare `npx` on a machine with nothing installed.
 
+```bash
+npm test          # the engine, the Worker and the extension — no install, any platform
+npm run test:all  # and the macOS app's own Swift suite, where there is a toolchain for it
+```
+
+There are two suites and `npm test` runs one. Keeping it portable is the point — it works on a machine with nothing on it. `test:all` runs both and says plainly when it could not run the second, rather than exiting green having skipped half the work.
+
 See [ROADMAP.md](ROADMAP.md) for what is planned, [CHANGELOG.md](CHANGELOG.md) for what changed.
 
 ## Licence

package/action.yml

@@ -91,6 +91,26 @@ inputs:
       Where to write the findings as a spreadsheet, one row per finding. Off
       unless a path is given.
     required: false
+  write-sitemap:
+    description: >
+      Where to write the sitemap this site should have had — every page that
+      answered 200, is HTML, is indexable and is its own canonical. Refuses on
+      a crawl that did not see the whole site rather than writing a short one.
+      Off unless a path is given.
+    required: false
+  write-llms:
+    description: >
+      Where to write the llms.txt this site should have had, in the
+      llmstxt.org format, from the site's own titles and descriptions. Refuses
+      on a partial crawl for the same reason as write-sitemap. Off unless a
+      path is given.
+    required: false
+  write-schema:
+    description: >
+      Where to write the JSON-LD this site could add — WebSite, Organization
+      and BreadcrumbList, built only from strings the crawl actually read. Off
+      unless a path is given.
+    required: false
   comment:
     description: Post the findings as a sticky comment on the pull request
     required: false
@@ -159,6 +179,9 @@ runs:
         [ -n "$INPUT_BROWSER" ]  && args+=(--browser "$INPUT_BROWSER")
         [ -n "$INPUT_OS" ]       && args+=(--os "$INPUT_OS")
         [ -n "$INPUT_CSV" ]      && args+=(--csv "$INPUT_CSV")
+        [ -n "$INPUT_WRITE_SITEMAP" ] && args+=(--write-sitemap "$INPUT_WRITE_SITEMAP")
+        [ -n "$INPUT_WRITE_LLMS" ] && args+=(--write-llms "$INPUT_WRITE_LLMS")
+        [ -n "$INPUT_WRITE_SCHEMA" ] && args+=(--write-schema "$INPUT_WRITE_SCHEMA")
 
         node "$SEO_AUDIT_BIN" "${args[@]}"
         status=$?

package/bin/seo-audit.mjs

@@ -9,6 +9,8 @@ import { dirname, join } from 'node:path';
 import { serialize, parse, diff } from '../src/baseline.mjs';
 import { parseRedirectMap } from '../src/redirects.mjs';
 import { describe } from '../src/sitemap.mjs';
+import { describeLlms } from '../src/llms.mjs';
+import { describeSchema } from '../src/schema.mjs';
 import { askForSite, isInteractive, invocation } from '../src/prompt.mjs';
 import { userAgentFor, BROWSER_NAMES, OS_NAMES, thisPlatform } from '../src/agents.mjs';
 
@@ -32,6 +34,17 @@ const HELP = `
                        page that answered 200, is HTML, is indexable and is
                        its own canonical. Refuses on a crawl that did not
                        see the whole site, rather than writing a short one
+    --write-llms <file>
+                       write the llms.txt this site should have had — its own
+                       titles and descriptions, grouped by section, in the
+                       llmstxt.org format. Nothing is generated or rewritten,
+                       and it refuses on a partial crawl for the same reason
+    --write-schema <file>
+                       write the JSON-LD this site could add: WebSite,
+                       Organization and BreadcrumbList, built only from
+                       strings the crawl actually read. A page whose trail
+                       has an uncrawled step is skipped rather than given a
+                       name invented from its slug
     --since <date>     crawl only URLs the sitemap says changed on or after
                        this date. Refuses when lastmod cannot answer it —
                        absent, or one build stamp on every URL
@@ -66,6 +79,9 @@ const HELP = `
     --check-external   also check links pointing off the site. Off by default:
                        other people's servers rate-limit and bot-block, so only
                        a 404, a 410 or no answer at all is ever reported
+    --no-open          with --serve, do not open a browser. It opens one when
+                       a person ran the command and never when something else
+                       did, so this is only for the person who wants neither
     --serve [port]     open the same form the hosted version serves, on this
                        machine (default 4321). No account, no bill, and none of
                        the limits a Worker has — the crawl is only bounded by
@@ -140,6 +156,9 @@ function parseArgs(argv) {
     // comma-separated list, and a URL can contain a comma.
     else if (arg === '--exclude') (opts.exclude ??= []).push(value());
     else if (arg === '--write-sitemap') opts.writeSitemap = value();
+    else if (arg === '--write-llms') opts.writeLlms = value();
+    else if (arg === '--write-schema') opts.writeSchema = value();
+    else if (arg === '--no-open') opts.noOpen = true;
     else if (arg === '--md') opts.md = value();
     else if (arg === '--html') opts.html = value();
     else if (arg === '--json') opts.json = value();
@@ -320,6 +339,19 @@ if (opts.serve !== undefined) {
     process.stdin.on('end', () => process.exit(0));
     process.stdin.on('close', () => process.exit(0));
   }
+
+  // Opened for a person, never for a parent. The same distinction the pipe
+  // check above already makes: somebody who typed `--serve` wants the page,
+  // and the macOS window — which spawns this and draws its own report — would
+  // get a browser it never asked for on every launch.
+  //
+  // This is the whole of "the desktop UI for Linux and Windows": a command that
+  // opens a window. Failing to open one is not a reason to refuse to serve, so
+  // the URL is printed either way and nothing here throws.
+  if (!stdinIsPipe && !opts.noOpen) {
+    const { openUrl } = await import('../src/open-url.mjs');
+    if (!openUrl(url)) console.log('  Open that address yourself — this system has no launcher I know.\n');
+  }
 } else {
 
 // --- sign in, and stop ----------------------------------------------------
@@ -402,12 +434,12 @@ if (sites.length > 1) {
     const siteOpts = optionsForSite(opts, site.overrides);
     // Sites run one at a time on purpose: interleaved progress from twenty
     // hosts is unreadable, and each audit is already parallel internally.
-    const { findings, meta } = await audit(site.url, {
+    const { findings, meta, score } = await audit(site.url, {
       ...siteOpts,
       onNote: (m) => !opts.quiet && process.stderr.write(`      ${m}\n`),
       onProgress: live(site.url),
     });
-    runs.push({ findings, meta });
+    runs.push({ findings, meta, score });
   }
 
   if (!opts.quiet) console.log(portfolio(runs));
@@ -417,7 +449,11 @@ if (sites.length > 1) {
     writeFileSync(
       opts.json,
       JSON.stringify(
-        { tool: 'seo-audit', date: runs[0]?.meta.date, sites: runs.map((r) => ({ ...r.meta, findings: r.findings })) },
+        {
+          tool: 'seo-audit',
+          date: runs[0]?.meta.date,
+          sites: runs.map((r) => ({ ...r.meta, score: r.score, findings: r.findings })),
+        },
         null,
         2,
       ),
@@ -455,7 +491,7 @@ if (!opts.quiet && opts.settle) {
   process.stderr.write(`  waiting up to ${opts.settle}s for the site to serve consistent HTML …\n`);
 }
 
-const { findings, meta, sitemap } = await audit(target, {
+const { findings, meta, sitemap, llms, schema, score } = await audit(target, {
   ...opts,
   onNote: (m) => !opts.quiet && process.stderr.write(`  ${m}\n`),
   onProgress: live(target),
@@ -468,6 +504,14 @@ if (opts.writeSitemap && sitemap) {
   if (sitemap.xml) writeFileSync(opts.writeSitemap, sitemap.xml);
   process.stderr.write('\n' + describe(sitemap, opts.writeSitemap));
 }
+if (opts.writeLlms && llms) {
+  if (llms.text) writeFileSync(opts.writeLlms, llms.text);
+  process.stderr.write('\n' + describeLlms(llms, opts.writeLlms));
+}
+if (opts.writeSchema && schema) {
+  if (schema.json) writeFileSync(opts.writeSchema, schema.json);
+  process.stderr.write('\n' + describeSchema(schema, opts.writeSchema));
+}
 
 // --- Compare against another deployment, if asked -----------------------
 let against = null;
@@ -499,12 +543,12 @@ if (opts.baseline && !against) {
 
 // --- Report -------------------------------------------------------------
 if (!opts.quiet) {
-  console.log(comparison ? diffReport(comparison) : terminal(findings, meta));
+  console.log(comparison ? diffReport(comparison) : terminal(findings, meta, { score }));
 }
-if (opts.md) writeFileSync(opts.md, markdown(findings, meta));
-if (opts.html) writeFileSync(opts.html, html(findings, meta));
-if (opts.json) writeFileSync(opts.json, serialize(findings, meta, { full: true }));
-if (opts.csv) writeFileSync(opts.csv, csv(findings, meta));
+if (opts.md) writeFileSync(opts.md, markdown(findings, meta, { score }));
+if (opts.html) writeFileSync(opts.html, html(findings, meta, { score }));
+if (opts.json) writeFileSync(opts.json, serialize(findings, meta, { full: true, score }));
+if (opts.csv) writeFileSync(opts.csv, csv(findings, meta, { score }));
 if (!opts.quiet && (opts.md || opts.html || opts.json)) {
   console.log(`  ${[opts.md, opts.html, opts.json].filter(Boolean).join('  ')}\n`);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nurkamol/seo-audit",
-  "version": "1.33.1",
+  "version": "1.35.0",
   "description": "Crawl a site's sitemap and check every page for SEO, metadata and structured-data problems that single-page graders miss. Zero dependencies.",
   "type": "module",
   "bin": {
@@ -9,7 +9,9 @@
   "exports": {
     ".": "./src/audit.mjs",
     "./causes": "./src/causes.mjs",
+    "./exports": "./src/exports.mjs",
     "./report": "./src/report.mjs",
+    "./score": "./src/score.mjs",
     "./sitemap": "./src/sitemap.mjs",
     "./areas": "./src/areas.mjs",
     "./agents": "./src/agents.mjs",
@@ -28,6 +30,8 @@
   "scripts": {
     "pretest": "node scripts/link-engine.mjs",
     "test": "node --test test/*.test.mjs",
+    "pretest:all": "node scripts/link-engine.mjs",
+    "test:all": "node scripts/test-all.mjs",
     "audit": "node bin/seo-audit.mjs"
   },
   "keywords": [

package/src/agents-ai.mjs

@@ -0,0 +1,110 @@
+// Which AI crawlers a site lets in, and which it does not.
+//
+// The answer engines fetch with their own user agents and obey robots.txt like
+// anything else, so a site's position on being read by them is already written
+// down in a file it already serves — nobody has to be asked and nothing has to
+// be estimated. That is the whole of this module: `robotsVerdict()` has taken
+// an agent since it was written, and this hands it the ten names that matter.
+//
+// **Blocking these is not a fault.** A publisher who does not want their work
+// in a model's training set and says so in robots.txt has done the correct
+// thing, correctly. So the finding is a note, phrased as a fact, and the report
+// is careful never to imply otherwise — the rule about checks that cry wolf
+// applies with particular force to a check somebody could reasonably have
+// meant. What it is for is the case nobody chose: a CDN, a WordPress plugin or
+// a hosting default that added `Disallow` lines the site's owner has never
+// seen, and which quietly costs them every citation in every AI answer.
+
+/** The agents worth asking about, and what each one is for.
+ *
+ *  Two things are deliberately distinguished, because conflating them is the
+ *  usual mistake: **training** crawlers gather text to train on, and blocking
+ *  one changes nothing about whether you can be cited today. **Answering**
+ *  crawlers fetch a page because somebody asked a question just now, and
+ *  blocking one removes you from that answer. A site that meant to opt out of
+ *  training and blocked both has almost certainly not read the difference. */
+export const AI_AGENTS = [
+  { token: 'gptbot', name: 'GPTBot', vendor: 'OpenAI', purpose: 'training' },
+  { token: 'oai-searchbot', name: 'OAI-SearchBot', vendor: 'OpenAI', purpose: 'answering' },
+  { token: 'chatgpt-user', name: 'ChatGPT-User', vendor: 'OpenAI', purpose: 'answering' },
+  { token: 'claudebot', name: 'ClaudeBot', vendor: 'Anthropic', purpose: 'training' },
+  { token: 'claude-searchbot', name: 'Claude-SearchBot', vendor: 'Anthropic', purpose: 'answering' },
+  { token: 'claude-user', name: 'Claude-User', vendor: 'Anthropic', purpose: 'answering' },
+  { token: 'perplexitybot', name: 'PerplexityBot', vendor: 'Perplexity', purpose: 'answering' },
+  { token: 'perplexity-user', name: 'Perplexity-User', vendor: 'Perplexity', purpose: 'answering' },
+  { token: 'google-extended', name: 'Google-Extended', vendor: 'Google', purpose: 'training' },
+  { token: 'applebot-extended', name: 'Applebot-Extended', vendor: 'Apple', purpose: 'training' },
+  { token: 'meta-externalagent', name: 'meta-externalagent', vendor: 'Meta', purpose: 'training' },
+  { token: 'ccbot', name: 'CCBot', vendor: 'Common Crawl', purpose: 'training' },
+  { token: 'bytespider', name: 'Bytespider', vendor: 'ByteDance', purpose: 'training' },
+];
+
+/** Whether an agent's own name appears anywhere in the file.
+ *
+ *  The difference between a decision and a default. `User-agent: *` with
+ *  `Disallow: /private` blocks GPTBot from /private without anybody having
+ *  thought about GPTBot; `User-agent: GPTBot` with `Disallow: /` is somebody
+ *  who did. Only the second is a position, and only the first is worth telling
+ *  a site's owner about. */
+const named = (groups, token) => groups.some((group) => group.agents.includes(token));
+
+/** How each AI agent is treated at the site root.
+ *
+ *  `verdict` is asked of the same function every other robots question in this
+ *  project goes through, so a rule this reads and a rule Google reads are the
+ *  same rule — including the part where a longer `Allow` beats a `Disallow`.
+ *
+ *  @returns {{ agent: object, allowed: boolean, explicit: boolean }[]}
+ */
+export function aiAccess(groups, robotsVerdict, path = '/') {
+  return AI_AGENTS.map((agent) => ({
+    agent,
+    allowed: robotsVerdict(groups, path, agent.token).allowed,
+    explicit: named(groups, agent.token),
+  }));
+}
+
+/** One sentence naming who is shut out, split by what shutting them out costs.
+ *
+ *  Returns null when everybody is let in, because a report saying "nothing is
+ *  blocked" once per site is a line nobody reads and this project has enough
+ *  of those already. */
+export function describeAccess(access) {
+  const blocked = access.filter((row) => !row.allowed);
+  if (!blocked.length) return null;
+
+  const answering = blocked.filter((row) => row.agent.purpose === 'answering');
+  const training = blocked.filter((row) => row.agent.purpose === 'training');
+  const list = (rows) => rows.map((row) => row.agent.name).join(', ');
+
+  const parts = [];
+  if (answering.length) {
+    parts.push(
+      `${list(answering)} fetch a page because somebody asked a question just now, so blocking them ` +
+        'removes the site from those answers today',
+    );
+  }
+  if (training.length) {
+    parts.push(
+      `${list(training)} gather text to train on, and blocking them changes nothing about whether the ` +
+        'site can be cited today',
+    );
+  }
+  // Whether anybody actually decided this. A block that arrives through
+  // `User-agent: *` is usually a default nobody chose.
+  const decided = blocked.some((row) => row.explicit);
+  parts.push(
+    decided
+      ? 'These are named in robots.txt, so this looks deliberate — this is a note, not a fault'
+      : 'None of these are named in robots.txt: they are caught by a `User-agent: *` rule, which is ' +
+        'usually a CDN or plugin default rather than a decision anybody made',
+  );
+
+  return {
+    blocked,
+    answering,
+    training,
+    decided,
+    detail: `${parts.join('. ')}.`,
+  };
+}

package/src/areas.mjs

@@ -22,6 +22,7 @@ export const CATEGORIES = [
   'Structured data',
   'Multilingual',
   'Sitemap & robots',
+  'AI & answer engines',
   'Site & security',
   'Performance',
 ];
@@ -64,6 +65,7 @@ const CATEGORY_OF = {
   'serves-differently': 'Indexability', 'compare-sampled': 'Indexability',
   'search-console': 'Site & security', 'search-console-unconfigured': 'Site & security',
   'search-console-failed': 'Site & security',
+  'search-console-striking': 'Site & security',
   'trailing-slash': 'Redirects', 'meta-refresh': 'Redirects',
   'redirect-dead': 'Redirects', 'redirect-broken': 'Redirects',
   'redirect-not-applied': 'Redirects', 'redirect-hops': 'Redirects',
@@ -105,11 +107,20 @@ const CATEGORY_OF = {
   'robots-missing': 'Sitemap & robots', 'robots-blocks-all': 'Sitemap & robots',
   'robots-no-sitemap': 'Sitemap & robots', 'robots-blocks-sitemap-url': 'Sitemap & robots',
   'sitemap-lastmod-missing': 'Sitemap & robots', 'sitemap-lastmod-identical': 'Sitemap & robots',
-  'sitemap-lastmod-future': 'Sitemap & robots', 'llms-missing': 'Sitemap & robots',
+  'sitemap-lastmod-future': 'Sitemap & robots',
   'sitemap-duplicate-url': 'Sitemap & robots',
   'sitemap-not-indexable': 'Sitemap & robots', 'sitemap-too-many-urls': 'Sitemap & robots',
   'sitemap-too-large': 'Sitemap & robots',
 
+  // AI & answer engines
+  // What the answer engines are allowed to read, which is a question robots.txt
+  // already answers and nothing here estimates. llms.txt moved out of
+  // "Sitemap & robots" when this area arrived: it is addressed to assistants,
+  // not to crawlers, and it belongs beside the agents that read it.
+  'llms-missing': 'AI & answer engines',
+  'ai-crawler-blocked': 'AI & answer engines',
+  'ai-crawler-conflict': 'AI & answer engines',
+
   // Site & security
   'favicon-broken': 'Site & security', 'favicon-missing': 'Site & security',
   'mixed-content': 'Site & security', 'tls-not-checked': 'Site & security',

package/src/audit.mjs

@@ -7,6 +7,9 @@ import { pageChecks, crossPageChecks, sitemapChecks } from './checks.mjs';
 import { certificateExpiry, siteChecks } from './site.mjs';
 import { linkGraph } from './graph.mjs';
 import { compareAgents } from './compare.mjs';
+import { scoreRun } from './score.mjs';
+import { buildLlms } from './llms.mjs';
+import { buildSchema } from './schema.mjs';
 import { searchConsole } from './console.mjs';
 import { applyIgnores, expectationChecks, matchGlob } from './config.mjs';
 import { psiChecks, psiTargets, estimateSeconds } from './psi.mjs';
@@ -632,10 +635,66 @@ export async function audit(target, opts = {}) {
     });
   }
 
+  // What this run was in a position to check at all. A site with no images has
+  // not passed the alt-text check and a run without --psi has not passed the
+  // performance ones, and a score that counted either as a pass would hand out
+  // free points for doing less. Worked out here because this is the only place
+  // that has both the crawl and the options — a front end reading the JSON
+  // gets it in `meta` rather than guessing.
+  const some = (fn) => pages.some((p) => p.doc && fn(p.doc));
+  const applicable = {
+    images: some((d) => d.images.length > 0),
+    hreflang: some((d) => d.hreflang.length > 0),
+    jsonld: some((d) => d.jsonld.length > 0),
+    ogImage: some((d) => Boolean(d.og['og:image'])),
+    twitterImage: some((d) => Boolean(d.twitter['twitter:image'])),
+    fingerprints: some((d) => d.fingerprint !== null),
+    multipage: pages.length > 1,
+    https: origin.startsWith('https:'),
+    // Certificates need a TLS socket. Node has one; the Workers runtime does
+    // not, and says so by handing in a reader that returns nothing.
+    tls: origin.startsWith('https:') && opts.readCertificateExpiry === undefined,
+    sitemap: Boolean(source),
+    // Only a site that serves llms.txt can contradict it. `llms-missing` is
+    // emitted exactly when it is absent, so the run already knows.
+    llmsTxt: !kept.some((finding) => finding.id === 'llms-missing'),
+    expect: Boolean(opts.expect?.length),
+    psi: Boolean(opts.psi?.length),
+    // Field data is Google's, not ours: it exists for a page or it does not,
+    // and PageSpeed says which by returning nothing. One page short of it is
+    // enough to leave the whole family out — under-counting is the honest way
+    // round, since the alternative scores a page on data it never had.
+    psiField: Boolean(opts.psi?.length) && !kept.some((f) => f.id === 'psi-no-field-data'),
+    redirects: Boolean(opts.redirects),
+    external: Boolean(opts.checkExternal),
+    compareAs: Boolean(opts.compareAs),
+  };
+
+  // The llms.txt this site should have had, from the same crawl and by the
+  // same rule as the sitemap: nothing invented, and a refusal rather than a
+  // file built from a fraction of the site.
+  const llms = opts.writeLlms
+    ? buildLlms(pages, { origin, truncated, rateLimited: kept.filter((f) => f.id === 'rate-limited').length })
+    : null;
+
+  // The structured data this site could add, from what it already says and
+  // nothing else. Same refusals, and a fourth rule of its own: every value is
+  // a string this crawl read off this site.
+  const schema = opts.writeSchema
+    ? buildSchema(pages, { origin, truncated, rateLimited: kept.filter((f) => f.id === 'rate-limited').length })
+    : null;
+
   return {
     findings: kept,
     ...(sitemap ? { sitemap } : {}),
+    ...(llms ? { llms } : {}),
+    ...(schema ? { schema } : {}),
+    // How much of the checklist this site passes. Computed here so that the
+    // terminal, the Markdown, the HTML, the window and the extension all show
+    // one number rather than five arithmetics that drift apart.
+    score: scoreRun(kept, { pages: pages.length, applicable }),
     meta: {
+      applicable,
       ignored,
       origin,
       pages: pages.length,