search-auditor 0.1.1 → 0.2.1

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [0.2.1](https://github.com/a-bibikov/search-auditor/compare/v0.2.0...v0.2.1) (2026-11-09)
+
+
+### Documentation
+
+* report screenshot, examples and the current roadmap ([70435c1](https://github.com/a-bibikov/search-auditor/commit/70435c155a16f8be5e6f744b871bfbc576414426))
+
+## [0.2.0](https://github.com/a-bibikov/search-auditor/compare/v0.1.1...v0.2.0) (2026-11-09)
+
+
+### New checks and features
+
+* expand SEO checks and split rules into modules ([#10](https://github.com/a-bibikov/search-auditor/issues/10)) ([f79eddb](https://github.com/a-bibikov/search-auditor/commit/f79eddbd20487f50418ac20c84a1964820877c8c))
+* expand SEO checks and split rules into modules ([#10](https://github.com/a-bibikov/search-auditor/issues/10)) ([41f4cab](https://github.com/a-bibikov/search-auditor/commit/41f4cab66cefe1f06402b40bf58ff8cf9acf7caf))
+
 ## [0.1.1](https://github.com/a-bibikov/search-auditor/compare/v0.1.0...v0.1.1) (2026-11-04)
 
 

package/README.md

@@ -1,72 +1,160 @@
 # search-auditor
 
-**Can Google, Bing and AI search find, read and cite your site?** One crawl, one report: classic SEO checks — status codes, canonical, sitemap, titles, headings, broken links, hreflang, HTTPS — and the AI search checks of [ai-search-auditor](https://github.com/a-bibikov/ai-search-audit), with a score for each and one overall.
+**Can Google, Bing and Yandex reach, index and show your site?** A technical SEO audit in one command: status codes, robots.txt, redirects, canonical, sitemap, titles and descriptions, headings, broken links, images, structured data, hreflang, HTTPS. 80 checks, each with what it costs and how to fix it.
 
 ```bash
 npx search-auditor https://example.com
 ```
 
-No sign-up, no API keys. Node.js 22.18+.
+![The report for albibikov.com: the score, what to do first and the first group of the checklist](docs/report.png)
 
-## Why both in one report
+No sign-up, no API keys. Node.js 22.18+.
 
-People find sites in two ways now: a list of links in Google, Bing or Yandex, and an answer from ChatGPT, Claude, Perplexity or Google AI Overviews that cites a few sources. Both start from the same thing — a crawler that can reach the page, read it and tell what it is about — and then they diverge. A broken canonical drops a page from Google; a heading that glues words together makes AI search misquote it.
+## AI search
 
-Running two tools means two crawls, two reports and two scores that never meet. search-auditor crawls once, runs both rule sets on the same pages and puts the results side by side, so the first three things to fix are the three that cost the most, whichever kind of search they hurt.
+Whether ChatGPT, Claude and Perplexity can read and cite the site is checked by [ai-search-auditor](https://github.com/a-bibikov/ai-search-audit), with a report of its own: `npx ai-search-auditor https://example.com`. Both run on [auditor-engine](https://github.com/a-bibikov/auditor-engine), so the reports look the same.
 
 ## What it checks
 
-24 classic search checks in ten groups, plus the 20 AI search checks of ai-search-auditor (turn them off with `--no-ai`). Nothing is checked twice: robots.txt for search crawlers, `noindex` and link cards are part of the AI search set.
+80 checks in ten groups.
 
 **IDX. Indexing**
 - **IDX-1** Audited pages answer 200
 - **IDX-2** A page that does not exist answers 404, and no «not found» page answers 200
-
-**SEC. HTTPS** (live sites only)
+- **IDX-3** robots.txt answers without errors: no 5xx or timeout (Google then treats the site as closed), plain-text rules, under 500 KB
+- **IDX-4** robots.txt does not block the pages of the site — those in the sitemap and the audited ones — for Googlebot, Bingbot or Yandex
+- **IDX-5** The site has one version: the other host (with or without www.) and the other trailing slash redirect to the main address
+- **IDX-6** Redirects take one step: no chains, no loops
+- **IDX-7** `noindex` does not close the home page and does not contradict the sitemap or the canonical
+- **IDX-8** robots.txt does not block the CSS and JavaScript the page needs to render
+- **IDX-9** The server answers quickly: first byte in under 0.8 s (live sites only)
+
+**SEC. Security and HTTPS** (the connection checks need the live site)
 - **SEC-1** The site is on https, and http:// redirects to it
 - **SEC-2** Pages do not load anything over http://
+- **SEC-3** The redirect from http:// is permanent (301 or 308), not a temporary 302
+- **SEC-4** The certificate is trusted and does not expire within two weeks
+- **SEC-5** HSTS tells browsers to use only https, for half a year at least
+- **SEC-6** The site's own addresses use https://: links, canonical, hreflang, og tags, sitemap
+- **SEC-7** Basic security headers: nosniff, frame protection, Referrer-Policy
 
-**CAN. Canonical**
+**CAN. Canonical: the main address of each page**
 - **CAN-1** Every page has a canonical
 - **CAN-2** The canonical is an absolute address on this site
 - **CAN-3** The canonical leads to a page that answers 200, not to an error or a redirect
-
-**MAP. Sitemap**
+- **CAN-4** One canonical per page: two different ones make Google ignore both
+- **CAN-5** The canonical sits in `<head>`; one in `<body>` is ignored
+- **CAN-6** The canonical in the HTML and in the `Link` header agree
+- **CAN-7** Canonicals lead straight to the main page, not through another canonical
+- **CAN-8** An address with UTM tags names the clean address as canonical
+- **CAN-9** Language versions do not name each other as canonical
+
+**MAP. Sitemap: the map of the site for search engines**
 - **MAP-1** There is a sitemap, and robots.txt points to it
 - **MAP-2** Sitemap addresses do not redirect
 - **MAP-3** Sitemap pages are their own canonical
+- **MAP-4** The sitemap file is valid: XML that answers 200, no bare &, within 50,000 addresses and 50 MB, every file of an index answers
+- **MAP-5** Sitemap addresses are full and on this site
+- **MAP-6** Pages the site links to are in the sitemap
+- **MAP-7** Every sitemap address answers, including up to 50 beyond the audited sample
+- **MAP-8** lastmod is honest: a W3C date, not in the future, not the same build time on every address
 
-**META. Titles and descriptions**
+**META. Meta tags: how a page looks in search results**
 - **META-1** Every page has a title of about 10–60 characters
 - **META-2** Every page has a description of about 50–160 characters
 - **META-3** Titles and descriptions are not repeated across pages
-
-**STR. Page structure**
+- **META-4** One title and one description per page
+- **META-5** The title names the page, not a template: no «Home», «Untitled» or «React App»
+- **META-6** The description adds to the title instead of repeating it
+- **META-7** The page declares its encoding
+- **META-8** The snippet is not forbidden with `nosnippet` or `max-snippet:0`
+- **META-9** The site has a favicon
+
+**STR. Page structure: headings, language, mobile**
 - **STR-1** One H1 per page
 - **STR-2** Heading levels do not skip
 - **STR-3** The page declares its language
 - **STR-4** Pages are set up for mobile
-
-**LNK. Links**
+- **STR-5** Headings are not empty: an H1 that is only a logo reads as no H1
+- **STR-6** The title and the H1 are about the same thing, or Google may rewrite the title
+- **STR-7** The page can be zoomed on a phone
+- **STR-8** `<html lang>` matches the language of the text
+- **STR-9** The main content sits in `<main>`
+- **STR-10** The page starts with `<!doctype html>`
+
+**LNK. Links: how pages link to each other**
 - **LNK-1** Internal links work
 - **LNK-2** Internal links do not go through redirects
+- **LNK-3** Every page is linked from another page: no orphans only the sitemap knows (with `--pages all`)
+- **LNK-4** Links can be followed by a crawler: no `<a>` without href, `#` or `javascript:` for navigation
+- **LNK-5** Internal links do not carry `nofollow`
+- **LNK-6** Link text says where the link goes: no «read more», no image links without alt
+- **LNK-7** Links to other sites still work: up to 30 checked, only 404, 410 and missing domains count as dead
 
-**IMG. Images**
+**IMG. Images: image search and loading speed**
 - **IMG-1** Images have alt text
-
-**DATA. Structured data**
+- **IMG-2** Images have width and height, so the layout does not jump (CLS)
+- **IMG-3** Lazy loading where it helps: not on the first image, yes on long pages
+- **IMG-4** Photos are served in modern formats: WebP or AVIF, not JPEG and PNG only
+- **IMG-5** Images open
+- **IMG-6** Images are not heavier than 500 KB
+- **IMG-7** Alt text describes the image: not a file name, «photo» or a paragraph
+
+**DATA. Structured data: rich results**
 - **DATA-1** JSON-LD has no syntax errors
+- **DATA-2** Every block has a schema.org `@context`, every entity a `@type`
+- **DATA-3** Rich-result types have the properties Google requires: Product, Event, BreadcrumbList, Recipe, VideoObject, LocalBusiness, JobPosting, Course, ratings and reviews
+- **DATA-4** Nested pages have breadcrumbs (BreadcrumbList)
+- **DATA-5** Addresses in the markup are full, dates are ISO 8601
+- **DATA-6** The home page names the site with WebSite
 
-**LANG. Language versions** (sites with hreflang)
+**LANG. Language versions: hreflang for multilingual sites** (sites with hreflang)
 - **LANG-1** hreflang codes are valid
 - **LANG-2** Language versions point at each other
 - **LANG-3** There is an x-default
+- **LANG-4** Every version lists itself in hreflang
+- **LANG-5** hreflang leads to working canonical pages: no redirects, errors or copies
+- **LANG-6** The hreflang language matches the `<html lang>` of the page it points to
+- **LANG-7** Each language is listed once
+- **LANG-8** hreflang addresses are full
 
 Levels: `error` — search engines cannot index the page or index the wrong one; `warn` — lowers rankings or clicks; `info` — a recommendation.
 
-The report opens with the overall score and one per part («Classic search», «AI search»), then what to do first — the three changes worth the most, in plain words and with how many pages they touch — then the findings with fixes, and ends with a checklist of every check by group: ✓ OK, ✖ error, ⚠ warning, ℹ note, – not checked. A check that cannot run on a site — hreflang on a site in one language, HTTPS on a local build — is marked «not checked» with the reason, never as passed.
+The report opens with the score and the number of errors, warnings and notes. Then what to do first: the three changes worth the most, each with what to do, on which pages it was found and why it matters. Then a table per group: every check with ✓ OK, ✖ error, ⚠ warning, ℹ note or – not checked, and what was found. It ends with one table of recommendations in checklist order, group by group: the check, its severity, what to do, and a comment on what was found where. A check that cannot run on a site — hreflang on a site in one language, HTTPS on a local build — is marked «not checked» with the reason, never as passed.
+
+```
+### META. Meta tags: how a page looks in search results
+
+7 of 9 OK
+
+|     | Check                                                        | Result                     |
+|-----|--------------------------------------------------------------|----------------------------|
+| ✓   | META-1 Every page has a title of the right length            | OK                         |
+| ℹ   | META-2 Every page has a description of the right length      | 1 note · 1 of 20 pages     |
+| ⚠   | META-3 Titles and descriptions are not repeated across pages | 4 warnings · 4 of 20 pages |
+| ✓   | META-4 One title and one description per page                | OK                         |
+| ✓   | META-5 The title names the page, not a template              | OK                         |
+| ✓   | META-6 The description adds to the title                     | OK                         |
+| ✓   | META-7 The page declares its encoding                        | OK                         |
+| ✓   | META-8 The snippet is not forbidden                          | OK                         |
+| ✓   | META-9 The site has a favicon                                | OK                         |
+```
 
-**Score:** each part is scored like ai-search-auditor: 100 minus 15 for each check with an error and 5 for each check with a warning; a warning that covers at least half of the audited pages costs 10. The overall score is the mean of the parts. A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60, F below; if fewer than two thirds of the checks could run, the grade is capped at B.
+With `--json`, every finding reads on its own: what the check is, what was found against which norm, why it is a problem and how to fix it.
+
+```json
+{
+  "rule": "META-2",
+  "level": "info",
+  "title": "Every page has a description of the right length",
+  "where": "https://albibikov.com/ru/privacy",
+  "message": "description of 165 characters, should be 50–160",
+  "why": "Without a description search engines pick a random snippet, a long one is cut off mid-sentence, and fewer people click.",
+  "fix": "Add `<meta name=\"description\">` of about 50–160 characters: what the page offers and why to open it."
+}
+```
+
+**Score:** 100 minus 15 for each check with an error and 5 for each check with a warning; a warning that covers at least half of the audited pages costs 10. A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60, F below; if fewer than two thirds of the checks could run, the grade is capped at B.
 
 ## Usage
 
@@ -80,8 +168,8 @@ npx search-auditor http://127.0.0.1:3000 --base https://example.com
 # Every page in the sitemap instead of a sample of 20
 npx search-auditor https://example.com --pages all
 
-# Classic search only, report in Russian
-npx search-auditor https://example.com --no-ai --lang ru
+# Report in Russian
+npx search-auditor https://example.com --lang ru
 
 # JSON for scripts and agents
 npx search-auditor https://example.com --json --out report.json
@@ -92,23 +180,21 @@ npx search-auditor https://example.com --json --out report.json
 | `--base <url>` | Public origin when auditing a local build |
 | `--pages <n\|all>` | Pages to audit: a number sampled across templates (default 20), or `all` for every page in the sitemap |
 | `--max-links <n>` | Internal links to request (default 100) |
-| `--no-ai` | Classic search checks only |
 | `--fail-on <level>` | Exit 1 on `error` (default), `warn` or `never` |
 | `--lang <lang>` | Report language: `en` (default), `ru` |
-| `--json` | JSON instead of Markdown |
+| `--width <n>` | Fit tables into n characters; long cells wrap inside their column. Default: the terminal width in a terminal, plain Markdown in a file or a pipe; `0` turns wrapping off |
+| `--json` | JSON instead of Markdown: each finding has its check `title`, the `message`, `why` it is a problem and how to `fix` it |
 | `--out <file>` | Also write the report to a file |
 | `--config <file>` | Config file (default `search-auditor.config.json`) |
 | `-v`, `--verbose` | Show progress |
 
-Config — the options of ai-search-auditor plus two of its own:
+Config:
 
 ```json
 {
-  "disable": ["STR-2", "C-7"],
+  "disable": ["STR-2", "IMG-4"],
   "maxPages": 20,
   "maxLinks": 100,
-  "ai": true,
-  "trainingBots": "any",
   "lang": "en"
 }
 ```
@@ -122,25 +208,36 @@ In CI the Markdown report also goes to the job summary (`GITHUB_STEP_SUMMARY`).
 ```js
 import { auditSearch, toMarkdown } from "search-auditor";
 
-const report = await auditSearch("https://example.com", { config: { lang: "en", ai: true } });
-console.log(report.score, report.sections);
+const report = await auditSearch("https://example.com", { config: { lang: "en" } });
+console.log(report.score, report.grade);
 console.log(toMarkdown(report));
 ```
 
-`ai-search-auditor` is a peer dependency: the crawl, the scoring and the report come from it, so both parts always run on the same engine version. The SEO checks are also exported as `seoPlugin` — rules, texts, groups and the extra requests they need — for runners that bring their own crawl.
+The crawl, the scoring and the report come from [auditor-engine](https://github.com/a-bibikov/auditor-engine). The checks are exported as `seoRules`, with `collect` for the extra requests they need, for your own runner.
+
+## Example
+
+[albibikov.com](https://albibikov.com), a Next.js site in English and Russian (20 pages, local production build): **95/100 (A)**, 70 of 80 checks run. The connection checks — HTTPS, the certificate, HSTS, the server speed — need the live site and are marked «not checked» on a local build; so are external links when the audit has no internet access.
+
+| | Found |
+|---|---|
+| META-3 ⚠ | The same title on `/` and `/cv`, and on `/ru` and `/ru/cv` |
+| META-2 ℹ | A description of 165 characters on `/ru/privacy`, should be 50–160 |
+| DATA-6 ℹ | No WebSite with the site name on the home page |
 
 ## Roadmap
 
-- **IDX-3** `noindex` and canonical do not contradict each other
-- **IDX-4** robots.txt does not block CSS and JavaScript the page needs to render
-- **CAN-4** Canonical chains: a canonical target whose own canonical points further
-- **MAP-4** Every audited page is in the sitemap; the sitemap has no pages blocked by robots.txt
-- **LNK-3** Orphan pages: in the sitemap, but no internal link leads to them
-- **LNK-4** Redirect chains of more than one hop
-- **IMG-2** Images are sized and lazy-loaded, so they do not slow the page down
-- **PERF-1** Server response time and page weight
-- **SOC-1** Twitter / X cards in addition to Open Graph
+**PERF. Speed** — page weight, render-blocking scripts and styles, Core Web Vitals from the Chrome UX Report
+
+**SOC. Link cards** — Open Graph and Twitter / X cards for every page
+
+- **GitHub Action**, like the one of [ai-search-auditor](https://github.com/a-bibikov/ai-search-audit#github-action), with the report in the job summary
+- **CI that reports only new problems,** with a file of accepted exceptions
+- **SARIF output,** so findings show up in GitHub code scanning
+- **HTML report** to share with people who do not read Markdown
+
+Ideas are welcome in [issues](https://github.com/a-bibikov/search-auditor/issues); for pull requests see [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## License
 
-MIT
+MIT © [Aleksandr Bibikov](https://albibikov.com)

package/dist/cli.js

@@ -1,9 +1,11 @@
 #!/usr/bin/env node
 import { appendFileSync, existsSync, readFileSync, writeFileSync } from "node:fs";
 import { parseArgs } from "node:util";
-import { DEFAULT_LANG, LANGS, isLang, parsePages, toMarkdown } from "ai-search-auditor";
+import { DEFAULT_LANG, LANGS, isLang, parsePages, toJSON, toMarkdown } from "auditor-engine";
 import { auditSearch } from "./index.js";
-const HELP = `search-auditor — one crawl, one report: can Google, Bing and AI search (ChatGPT, Claude, Perplexity) find, read and cite your site?
+const HELP = `search-auditor — technical SEO audit: can Google, Bing and Yandex reach, index and show your site?
+
+AI search (ChatGPT, Claude, Perplexity) is checked by ai-search-auditor: npx ai-search-auditor <url>
 
 Usage:
   search-auditor <url> [options]
@@ -12,9 +14,9 @@ Options:
   --base <url>        public origin when auditing a local build (links in sitemap point there)
   --pages <n|all>     pages to audit: a number sampled across templates (default 20), or all pages in the sitemap
   --max-links <n>     internal links to request (default 100)
-  --no-ai             classic search checks only
   --fail-on <level>   exit 1 on: error (default), warn, never
   --lang <lang>       report language: ${Object.keys(LANGS).join(", ")} (default ${DEFAULT_LANG})
+  --width <n>         fit tables into n characters (default: the terminal width; 0 — no wrapping)
   --json              print the JSON report instead of Markdown
   --out <file>        also write the report to a file (.json or .md)
   --config <file>     config file (default: search-auditor.config.json if present)
@@ -24,23 +26,24 @@ Options:
 Examples:
   npx search-auditor https://example.com
   npx search-auditor http://127.0.0.1:3000 --base https://example.com --fail-on warn
-  npx search-auditor https://example.com --lang ru --no-ai
+  npx search-auditor https://example.com --lang ru
   npx search-auditor https://example.com --pages all
 `;
 // An unknown or malformed flag gets one line with the help hint, not a stack trace.
 let parsed;
 try {
     parsed = parseArgs({
+        // --no-ai turned the AI search checks off; they are no longer here, so old scripts keep working without it.
+        args: process.argv.slice(2).filter((a) => a !== "--no-ai"),
         allowPositionals: true,
-        allowNegative: true,
         options: {
             base: { type: "string" },
             pages: { type: "string" },
             "max-links": { type: "string" },
-            ai: { type: "boolean" },
             "fail-on": { type: "string", default: "error" },
             lang: { type: "string" },
             json: { type: "boolean", default: false },
+            width: { type: "string" },
             out: { type: "string" },
             config: { type: "string" },
             verbose: { type: "boolean", short: "v", default: false },
@@ -90,7 +93,6 @@ const config = {
     lang,
     ...(maxPages ? { maxPages } : {}),
     ...(maxLinks ? { maxLinks } : {}),
-    ...(values.ai === false ? { ai: false } : {}),
 };
 // Progress goes to stderr, so `--json > report.json` stays clean.
 const log = values.verbose ? (line) => process.stderr.write(`· ${line}\n`) : undefined;
@@ -102,9 +104,15 @@ catch (e) {
     fail(e.message);
 }
 const markdown = toMarkdown(report);
-process.stdout.write(values.json ? JSON.stringify(report, null, 2) + "\n" : markdown);
+// In a terminal, tables fit the window and long cells wrap; a file and a CI summary keep plain Markdown.
+if (values.width !== undefined && !/^\d+$/.test(values.width))
+    fail(`--width must be a whole number, got «${values.width}»`);
+const columns = values.width !== undefined ? Number(values.width) : process.stdout.isTTY ? process.stdout.columns : 0;
+const screen = columns > 0 ? toMarkdown(report, { width: columns }) : markdown;
+const json = JSON.stringify(toJSON(report), null, 2) + "\n";
+process.stdout.write(values.json ? json : screen);
 if (values.out)
-    writeFileSync(values.out, values.out.endsWith(".json") ? JSON.stringify(report, null, 2) : markdown);
+    writeFileSync(values.out, values.out.endsWith(".json") ? json : markdown);
 if (process.env.GITHUB_STEP_SUMMARY)
     appendFileSync(process.env.GITHUB_STEP_SUMMARY, markdown);
 const failOn = values["fail-on"];

package/dist/collect.d.ts

@@ -1,5 +1,9 @@
-import { type Site } from "ai-search-auditor";
+import { type Site } from "auditor-engine";
 import type { SeoExtras, SeoSite } from "./types.ts";
+/** Parameters every campaign link adds: the page must still name its clean address as canonical. */
+export declare const UTM = "utm_source=search-auditor&utm_medium=audit";
+/** The <head> of a page: up to </head>, or up to <body> when the closing tag is left out. */
+export declare const headOf: (html: string) => string;
 export interface CollectOptions {
     maxLinks: number;
     concurrency?: number;

package/dist/collect.js

@@ -1,10 +1,74 @@
-import { fetchUrl, isSameSite, pool } from "ai-search-auditor";
+import { fetchUrl, isSameSite, parsePage, parseSitemap, pool } from "auditor-engine";
 import { norm } from "./rules/util.js";
+import { certificateOf, firstHop, timeToFirstByte, trace } from "./trace.js";
+/** Redirecting addresses followed hop by hop per run: enough to show a pattern without a request storm. */
+const MAX_CHAINS = 20;
+/** Requests to time the home page: the median of three smooths out one slow answer. */
+const TTFB_SAMPLES = 3;
+/** The same host with www. added or removed. */
+const otherHostOf = (base) => {
+    const u = new URL(base);
+    u.hostname = u.hostname.startsWith("www.") ? u.hostname.slice(4) : `www.${u.hostname}`;
+    return `${u.origin}/`;
+};
+/** Parameters every campaign link adds: the page must still name its clean address as canonical. */
+export const UTM = "utm_source=search-auditor&utm_medium=audit";
+/** Absolute canonical of a fetched page, or "" when it has none or it does not parse. */
+const canonicalOf = (f) => {
+    if (f.status !== 200)
+        return "";
+    const href = parsePage(f).canonical;
+    try {
+        return href ? new URL(href, f.publicUrl).href : "";
+    }
+    catch {
+        return "";
+    }
+};
+/** The <head> of a page: up to </head>, or up to <body> when the closing tag is left out. */
+export const headOf = (html) => {
+    const end = html.search(/<\/head\s*>|<body[\s>]/i);
+    return end < 0 ? html : html.slice(0, end);
+};
+/** Sitemap addresses outside the sample requested per run: enough to catch a pattern of dead pages. */
+const MAX_SITEMAP_EXTRA = 50;
+/** Images of the site requested per run: enough to catch broken and heavy ones. */
+const MAX_IMAGES = 30;
+/** Links to other sites requested per run: enough to catch dead ones without crawling the web. */
+const MAX_EXTERNAL = 30;
+/** Child sitemaps of an index requested per run. */
+const MAX_SITEMAP_CHILDREN = 10;
+/** The sitemap protocol's limits per file: 50,000 addresses and 50 MB uncompressed. Read a little past them. */
+const SITEMAP_MAX_BYTES = 52 * 1024 * 1024;
+/** What a sitemap file is and how it breaks the format, from its raw body. */
+function sitemapFileOf(url, source, f) {
+    const body = f.body;
+    const kind = /\.gz(\?|$)/i.test(url) && !/<(urlset|sitemapindex)\b/i.test(body)
+        ? "gzip"
+        : /<sitemapindex\b/i.test(body)
+            ? "index"
+            : /<urlset\b/i.test(body)
+                ? "urlset"
+                : /^\s*<(!doctype html|html)/i.test(body)
+                    ? "html"
+                    : "other";
+    const count = (body.match(kind === "index" ? /<sitemap\b/gi : /<url\b/gi) ?? []).length;
+    const badAmpersands = [...body.matchAll(/<loc>([\s\S]*?)<\/loc>/gi)].filter((m) => /&(?!(amp|lt|gt|quot|apos|#\d+|#x[0-9a-f]+);)/i.test(m[1])).length;
+    return { url: f.publicUrl, source, status: f.status, ...(f.error ? { error: f.error } : {}), kind, bytes: Buffer.byteLength(body), count, badAmpersands };
+}
+/** A sampled page whose address can be asked with the trailing slash toggled: not the home page, not a file. */
+const slashCandidate = (site) => site.pages.find((p) => {
+    const path = new URL(p.fetched.publicUrl).pathname;
+    return path !== "/" && !/\.[a-z0-9]{2,5}$/i.test(path) && !p.fetched.redirected;
+});
 const probeOf = (f, from) => ({
     status: f.status,
     redirected: Boolean(f.redirected),
     finalUrl: f.finalUrl ?? f.publicUrl,
+    html: /text\/html|application\/xhtml/i.test(f.headers.get("content-type") ?? ""),
+    noindex: f.status === 200 && /\b(noindex|none)\b/i.test(`${f.headers.get("x-robots-tag") ?? ""} ${f.body.match(/<meta[^>]+name=["']?robots["']?[^>]*>/i)?.[0] ?? ""}`),
     ...(f.error ? { error: f.error } : {}),
+    ...(Number(f.headers.get("content-length")) > 0 ? { bytes: Number(f.headers.get("content-length")) } : {}),
     ...(from ? { from } : {}),
 });
 /** Links worth requesting: pages, not files a crawler would not index anyway. */
@@ -38,36 +102,147 @@ export async function collect(site, opts) {
         })
             .filter((u) => u && isSameSite(u, o) && !known.has(norm(u)))),
     ].slice(0, 20);
-    const firstSeen = new Map();
+    // Every audited page that links to each address: a broken link is fixed on the pages that have it.
+    const linkedFrom = new Map();
     for (const p of site.pages)
         for (const href of p.links)
-            if (isSameSite(href, o) && isPageLike(href) && !firstSeen.has(norm(href)))
-                firstSeen.set(norm(href), p.fetched.publicUrl);
-    const linkTargets = [...firstSeen.keys()].filter((u) => !known.has(u));
+            if (isSameSite(href, o) && isPageLike(href)) {
+                const list = linkedFrom.get(norm(href)) ?? [];
+                if (!list.includes(p.fetched.publicUrl))
+                    linkedFrom.set(norm(href), [...list, p.fetched.publicUrl]);
+            }
+    const linkTargets = [...linkedFrom.keys()].filter((u) => !known.has(u));
     const toCheck = linkTargets.slice(0, opts.maxLinks);
-    log(`${canonicalTargets.length} canonical targets, ${toCheck.length} of ${firstSeen.size} internal links`);
+    log(`${canonicalTargets.length} canonical targets, ${toCheck.length} of ${linkedFrom.size} internal links`);
     const light = { maxBytes: 65_536, timeoutMs: 10_000 };
-    const [probe404, httpHome, canonicals, linked] = await Promise.all([
+    const slashPage = slashCandidate(site);
+    const slashUrl = slashPage ? slashPage.fetched.publicUrl.replace(/\/?$/, (m) => (m ? "" : "/")) : null;
+    const liveHttps = !local && site.base.startsWith("https://");
+    const utmPage = site.pages.find((p) => new URL(p.fetched.publicUrl).pathname === "/" && !p.fetched.publicUrl.includes("?")) ?? site.pages[0];
+    const utmUrl = utmPage ? `${utmPage.fetched.publicUrl}${utmPage.fetched.publicUrl.includes("?") ? "&" : "?"}${UTM}` : null;
+    const host = new URL(site.base).hostname;
+    // The icon search results show next to the address: declared in <head>, or /favicon.ico by convention.
+    const declaresIcon = utmPage ? /<link\b[^>]*\brel\s*=\s*["']?[^"'>]*\bicon\b/i.test(headOf(utmPage.fetched.body)) : true;
+    const [probe404, httpHome, canonicals, linked, robotsRaw, otherHost, slashFetched, ttfb, httpHop, cert, utmFetched, favicon] = await Promise.all([
         fetchUrl(`${site.target}/__search-auditor-${Math.random().toString(36).slice(2, 10)}`, o, light),
         !local && site.base.startsWith("https://") ? fetchUrl(`http://${new URL(site.base).host}/`, o, light) : Promise.resolve(null),
         pool(canonicalTargets, conc, (u) => fetchUrl(u, o, light)),
         pool(toCheck, conc, (u) => fetchUrl(u, o, light)),
+        // Up to 600 KB: enough to tell a robots.txt over Google's 500 KB limit.
+        fetchUrl(`${site.base}/robots.txt`, o, { accept: "text/plain", maxBytes: 614_400, timeoutMs: 10_000 }),
+        local ? Promise.resolve(null) : fetchUrl(otherHostOf(site.base), { target: otherHostOf(site.base).slice(0, -1), base: otherHostOf(site.base).slice(0, -1) }, light),
+        slashUrl ? fetchUrl(slashUrl, o, light) : Promise.resolve(null),
+        local ? Promise.resolve(null) : (async () => {
+            const times = [];
+            for (let i = 0; i < TTFB_SAMPLES; i++) {
+                const ms = await timeToFirstByte(`${site.base}/`, o);
+                if (ms !== null)
+                    times.push(ms);
+            }
+            return times.length ? times : null;
+        })(),
+        liveHttps ? firstHop(`http://${new URL(site.base).host}/`) : Promise.resolve(null),
+        liveHttps ? certificateOf(host) : Promise.resolve(null),
+        utmUrl ? fetchUrl(utmUrl, o, light) : Promise.resolve(null),
+        declaresIcon ? Promise.resolve(null) : fetchUrl(`${site.base}/favicon.ico`, o, { ...light, accept: "image/*" }),
     ]);
     const links = new Map();
-    for (const [url, from] of firstSeen) {
+    for (const [url, from] of linkedFrom) {
         const done = known.get(url);
         if (done)
             links.set(url, { ...done, from });
     }
-    toCheck.forEach((url, i) => links.set(url, probeOf(linked[i], firstSeen.get(url))));
+    toCheck.forEach((url, i) => links.set(url, probeOf(linked[i], linkedFrom.get(url))));
+    // Every address known to redirect, with the pages that point at it: sampled pages, links, canonicals, sitemap.
+    const redirecting = new Map();
+    const note = (url, from) => redirecting.set(url, [...new Set([...(redirecting.get(url) ?? []), ...from])]);
+    const listed = new Set((site.sitemap?.urls ?? []).map((u) => norm(u.loc)));
+    for (const f of [...site.pages.map((p) => p.fetched), ...site.failed])
+        if (f.redirected && f.finalUrl && norm(f.finalUrl) !== norm(f.publicUrl))
+            note(f.publicUrl, listed.has(norm(f.publicUrl)) ? ["sitemap.xml"] : []);
+    // A link that failed without an answer may be a redirect loop: fetch gives up on those without saying so.
+    for (const [url, p] of links)
+        if ((p.redirected && norm(p.finalUrl) !== norm(url)) || p.status === 0)
+            note(url, p.from ?? []);
+    if (httpHome?.redirected)
+        note(httpHome.publicUrl, []);
+    const toTrace = [...redirecting.keys()].slice(0, MAX_CHAINS);
+    const traced = await pool(toTrace, conc, (u) => trace(u, o));
+    const chains = toTrace.map((url, i) => ({ url, ...traced[i], from: redirecting.get(url) ?? [] }));
+    // The sitemap files as robots.txt declares them (or the default address), then the children of an index.
+    const declared = robotsRaw.status === 200 ? [...robotsRaw.body.matchAll(/^\s*sitemap\s*:\s*(\S+)/gim)].map((m) => m[1]) : [];
+    const roots = declared.length
+        ? [...new Set(declared)].slice(0, 5).map((url) => ({ url, source: "robots" }))
+        : [{ url: `${site.base}/sitemap.xml`, source: "default" }];
+    const xml = { accept: "application/xml,text/xml", maxBytes: SITEMAP_MAX_BYTES, timeoutMs: 20_000 };
+    const rootFetched = await pool(roots, conc, (r) => fetchUrl(r.url, o, xml));
+    const sitemapFiles = roots.map((r, i) => sitemapFileOf(r.url, r.source, rootFetched[i]));
+    const childUrls = rootFetched.flatMap((f) => (f.status === 200 ? parseSitemap(f.body).children : [])).slice(0, MAX_SITEMAP_CHILDREN);
+    const childFetched = await pool(childUrls, conc, (u) => fetchUrl(u, o, xml));
+    sitemapFiles.push(...childUrls.map((u, i) => sitemapFileOf(u, "index", childFetched[i])));
+    // Sitemap addresses the crawl did not sample: do they answer at all?
+    const extraUrls = (site.sitemap?.urls ?? [])
+        .map((u) => u.loc)
+        .filter((u) => isSameSite(u, o) && !known.has(norm(u)))
+        .slice(0, MAX_SITEMAP_EXTRA);
+    const extraFetched = await pool(extraUrls, conc, (u) => fetchUrl(u, o, light));
+    const sitemapExtra = new Map(extraUrls.map((u, i) => [norm(u), probeOf(extraFetched[i])]));
+    // Links to other sites, with the pages that have them: do they still exist?
+    const externalFrom = new Map();
+    for (const p of site.pages)
+        for (const href of p.links)
+            if (/^https?:\/\//i.test(href) && !isSameSite(href, o))
+                externalFrom.set(href, [...new Set([...(externalFrom.get(href) ?? []), p.fetched.publicUrl])]);
+    const externalUrls = [...externalFrom.keys()].slice(0, MAX_EXTERNAL);
+    const externalFetched = await pool(externalUrls, conc, (u) => fetchUrl(u, o, { maxBytes: 16_384, timeoutMs: 8_000 }));
+    const external = new Map(externalUrls.map((u, i) => [u, probeOf(externalFetched[i], externalFrom.get(u))]));
+    // Images of this site, with the pages that use them. Only the headers matter: status and Content-Length.
+    const imageFrom = new Map();
+    for (const p of site.pages)
+        for (const img of p.images)
+            if (img.src && isSameSite(img.src, o) && !img.src.startsWith("data:"))
+                imageFrom.set(img.src, [...new Set([...(imageFrom.get(img.src) ?? []), p.fetched.publicUrl])]);
+    const imageUrls = [...imageFrom.keys()].slice(0, MAX_IMAGES);
+    const imageFetched = await pool(imageUrls, conc, (u) => fetchUrl(u, o, { maxBytes: 1024, timeoutMs: 10_000, accept: "image/avif,image/webp,image/*" }));
+    const images = new Map(imageUrls.map((u, i) => [u, probeOf(imageFetched[i], imageFrom.get(u))]));
+    const slashParsed = slashFetched && slashFetched.status === 200 ? parsePage(slashFetched) : null;
+    const abs = (href, base) => {
+        try {
+            return href ? new URL(href, base).href : "";
+        }
+        catch {
+            return "";
+        }
+    };
     const seo = {
         local,
         probe404,
         httpHome,
-        canonicals: new Map(canonicalTargets.map((u, i) => [norm(u), probeOf(canonicals[i])])),
+        canonicals: new Map(canonicalTargets.map((u, i) => [norm(u), { ...probeOf(canonicals[i]), canonical: canonicalOf(canonicals[i]) }])),
         links,
-        linksFound: firstSeen.size,
+        linksFound: linkedFrom.size,
         linksChecked: links.size,
+        robotsRaw,
+        otherHost,
+        slashVariant: slashPage && slashUrl && slashFetched
+            ? {
+                main: slashPage.fetched.publicUrl,
+                mainCanonical: abs(slashPage.canonical, slashPage.fetched.publicUrl),
+                variant: slashUrl,
+                fetched: slashFetched,
+                canonical: slashParsed ? abs(slashParsed.canonical, slashUrl) : "",
+            }
+            : null,
+        chains,
+        ttfb,
+        httpHop,
+        cert,
+        sitemapFiles,
+        sitemapExtra,
+        favicon,
+        external,
+        images,
+        utm: utmPage && utmUrl && utmFetched ? { page: utmPage.fetched.publicUrl, url: utmUrl, status: utmFetched.status, canonical: canonicalOf(utmFetched) } : null,
     };
     return Object.assign(site, { seo });
 }