vagejs 0.1.0 → 0.3.0

package/README.md

@@ -1,12 +1,178 @@
 <p align="center">
-  <img src="assets/brand/vagejs-logo.svg" alt="VAGE JS" width="320">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/agevenkat/vagejs/main/assets/brand/vagejs-logo-dark.svg">
+    <img src="https://raw.githubusercontent.com/agevenkat/vagejs/main/assets/brand/vagejs-logo.svg" alt="vagejs" width="320">
+  </picture>
 </p>
 
-# VAGE JS
+<p align="center">
+  <a href="https://www.npmjs.com/package/vagejs"><img src="https://img.shields.io/npm/v/vagejs?color=4F46E5&label=npm" alt="npm version"></a>
+  <a href="https://github.com/agevenkat/vagejs/actions/workflows/ci.yml"><img src="https://github.com/agevenkat/vagejs/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/npm/l/vagejs?color=2E2A8C" alt="MIT license"></a>
+</p>
+
+# vagejs
+
+An AI-native, static-first web framework. Write pages in Markdown, and vagejs builds a fast static site with SEO, `llms.txt`, accessibility and security handled for you. It's built so AI coding agents can use it without guessing. **Pre-release (0.x).**
+
+- **Static by default**: plain HTML, no client JavaScript unless you add analytics.
+- **SEO built in**: `robots.txt`, `sitemap.xml`, `llms.txt`, canonical URLs, Open Graph and JSON-LD, generated from your pages.
+- **Checks before you ship**: `vage check` (links, SEO, structured data, accessibility) and `vage audit` (page-weight budgets, axe, optional Lighthouse and PageSpeed).
+- **Safe by default**: page content is sanitized; secrets, build output and force-pushes are refused; Cloudflare deploys get a computed Content-Security-Policy.
+- **Agent-friendly**: every site ships with `AGENTS.md`, and `vage info --json` describes every command and config option.
 
-An AI-native, static-first web framework. **Pre-release (0.x).**
+## Quick start
 
-## What works today
+Requires **Node.js 22.12 or newer**.
+
+```bash
+npm create vagejs@latest my-site
+cd my-site
+npm install
+npm run dev
+```
+
+Open the URL it prints. Edit `src/pages/index.md` and the browser reloads.
+
+## Using vagejs
+
+### Project layout
+
+```text
+my-site/
+├── src/
+│   ├── pages/          # every .md file is a page
+│   │   ├── index.md    # → /
+│   │   ├── about.md    # → /about/
+│   │   └── blog/
+│   │       └── first-post.md   # → /blog/first-post/
+│   └── layouts/
+│       └── default.ts  # HTML around each page
+├── public/             # copied to the site as-is (favicons, images)
+├── vage.config.ts      # site settings
+└── AGENTS.md           # instructions for AI coding agents
+```
+
+### Write a page
+
+```markdown
+---
+title: Pricing
+description: Plans and prices.
+---
+
+# Pricing
+
+Our plans start at …
+```
+
+Save it as `src/pages/pricing.md` and it's served at `/pricing/`. `title` is required. `description`, `draft`, `noindex`, `canonical`, `image`, `type: article`, `author`, `datePublished` and `dateModified` are optional; see [`templates/starter/AGENTS.md`](templates/starter/AGENTS.md).
+
+Images in `public/` used as `![Alt text](/images/photo.jpg)` get their `width` and `height` from the file, so the page doesn't jump while loading. With `images: { optimize: true }` (and `npm install --save-dev sharp`), vagejs also makes resized WebP copies of images in page content and offers them with `srcset`.
+
+### Navigation, feeds and AI extras
+
+- `order:` and `navTitle:` in frontmatter, with `tree()`, `breadcrumbs()` and `neighbors()` in layouts, build sidebars, breadcrumbs and previous/next links.
+- `seo: { feeds: [{ under: "/blog/" }] }` writes an RSS feed at `/blog/feed.xml`.
+- `seo: { markdownCopies: true }` publishes a Markdown copy of every listed page (`/docs/cli.md`) for AI tools, without comments or raw HTML; `seo: { lastmod: "git" }` dates the sitemap from git history (CI needs `fetch-depth: 0`).
+
+### Write a layout
+
+Layouts are TypeScript functions. The `html` template tag escapes every value for you; `raw()` inserts trusted HTML such as the rendered page. (Escaping doesn't check URLs, so validate link targets that come from frontmatter.) `pages` lists every published page, and `collection()` picks and sorts them:
+
+```ts
+import { collection, html, raw, z, type Layout } from "vagejs";
+
+// Optional: fields this layout accepts. Pages using it are checked strictly, so typos fail the build.
+export const frontmatter = z.object({ intro: z.string().optional() });
+
+const layout: Layout = ({ content, page, site, pages }) => html`<!doctype html>
+<html lang="${site.lang}">
+  <head><title>${page.title}</title></head>
+  <body>
+    <main>${raw(content)}</main>
+    <h2>Latest posts</h2>
+    <ul>${collection(pages, { under: "/blog/", sort: "date", limit: 5 }).map((p) => html`<li><a href="${p.route}">${p.title}</a></li>`)}</ul>
+  </body>
+</html>`;
+
+export default layout;
+```
+
+### Everyday commands
+
+| Command | What it does |
+| --- | --- |
+| `npm run dev` | Dev server with live reload |
+| `npm run check` | Links, SEO, structured data and basic accessibility; writes nothing |
+| `npm run build` | Static site in `dist/`, plus `robots.txt`, `sitemap.xml`, `llms.txt` |
+| `npm run audit` | Page-weight budgets and automated accessibility (`-- --lighthouse` adds Lighthouse) |
+| `npm run preview` | Serves `dist/` locally, the way a static host would |
+
+Add `--json` to `npx vage build`, `check` or `audit` for machine-readable output. Failures exit with code 1.
+
+### Configure the site
+
+`vage.config.ts`:
+
+```ts
+import { defineConfig } from "vagejs";
+
+export default defineConfig({
+  site: "https://example.com", // your real domain (used for canonical URLs and the sitemap)
+  title: "My Website",
+  description: "What this site is about.",
+  seo: {
+    robots: { aiTraining: "disallow" }, // optional: block AI training crawlers, keep search
+    llmsTxt: { full: false },
+  },
+  redirects: { "/old-page/": "/new-page/" },
+});
+```
+
+Unknown keys are an error, so typos don't silently do nothing. `npx vage info --json` prints the full schema.
+
+### Deploy
+
+```bash
+npx vage deploy:github --remote https://github.com/you/site.git   # check, build, safe commit + push (asks first)
+npx vage deploy                                                  # GitHub Pages via Actions
+npx vage deploy --target cloudflare-workers --project my-site    # Cloudflare Workers: new version, no downtime
+npx vage deploy --target cloudflare-pages --project my-site      # older Cloudflare Pages projects
+```
+
+Every deploy shows its plan and asks for confirmation (`--dry-run` to preview, `--yes` to skip). It reports success only after the host confirms.
+
+Cloudflare now creates new static sites as Workers, so use `cloudflare-workers` for new projects. For a custom domain, add a `wrangler.jsonc` with your `routes`; vagejs uses it and switches versions without re-attaching domains, so the site stays up during deploys.
+
+### Analytics, Search Console and audits of live pages
+
+Optional, and off until configured: GA4 or Google Tag Manager (consent denied by default), Plausible, Microsoft Clarity, IndexNow. `vage connect gsc|ga4` adds read-only Search Console and GA4 reports, and `vage audit --url https://your-site` runs PageSpeed and CrUX with **your own** Google API key. Details are in the generated `AGENTS.md`.
+
+## Updating
+
+vagejs follows semver. While it's 0.x, a minor version (0.1 → 0.2) may include breaking changes, so read the release notes before moving up a minor version.
+
+```bash
+npm outdated vagejs           # see what's available
+npm update vagejs             # latest patch in your range (with ^0.3.0: any 0.3.x)
+npm install vagejs@latest     # move to the newest version, including a new minor
+npx vage --version            # confirm
+npx vage check && npx vage build
+```
+
+If your config pins the rule pack (`guidelines: { rulePack: "…" }`), a build after an update that ships new provider rules stops with a message. Review them with `npx vage guidelines`, then update the pin. New projects always use the latest version through `npm create vagejs@latest`.
+
+## Use with AI coding agents
+
+Every site includes `AGENTS.md` (conventions, recipes, definition of done) and `CLAUDE.md` (imports `AGENTS.md`). Agents can discover the API instead of guessing:
+
+```bash
+npx vage info --json    # commands, options, config and frontmatter JSON Schemas
+npx vage check --json   # findings with file, rule and fix hints
+```
+
+## Command reference
 
 ```bash
 vage create <dir> [--vage-spec <spec>] [--json]   # scaffold a Markdown site
@@ -16,8 +182,9 @@ vage check    [--root .] [--json] [--strict]      # SEO, links, JSON-LD, basic a
 vage audit    [--root .] [--lighthouse] [--url U] [--strategy mobile|desktop|both] [--strict] [--json]
                                                   # budgets + axe a11y; optional Lighthouse; PageSpeed/CrUX for deployed URLs
 vage preview  [--root .] [--port N] [--host]      # serve dist/ locally
-vage deploy:github [--remote URL] [--dry-run] [--yes] [--json]   # check, build, safe commit + push (asks first)
+vage deploy:github [--remote URL] [--branch B] [-m MSG] [--dry-run] [--yes] [--json]   # check, build, safe commit + push (asks first)
 vage deploy        [--remote URL] [--no-wait] [--dry-run] [--yes] # GitHub Pages via Actions; confirms with GitHub
+vage deploy --target cloudflare-workers [--project NAME]          # new Worker version at 100%; custom domains stay up
 vage deploy --target cloudflare-pages --project NAME [--branch B] # Wrangler upload; preview URL per branch
 vage indexnow --changed-since REF | --urls … | --all              # IndexNow URL notification (asks first)
 vage connect gsc|ga4 / vage disconnect google      # read-only Google OAuth (PKCE, loopback); token outside the repo
@@ -27,45 +194,49 @@ vage guidelines [--json]                          # rule pack: provider facts in
 vage info     [--json]                            # commands + config/frontmatter JSON Schemas for agents
 ```
 
-Generated sites include `AGENTS.md` (conventions, recipes, definition of done) and `CLAUDE.md` (imports AGENTS.md).
+## Planned, not implemented
+
+Bing Webmaster Tools API, Netlify/Vercel adapters, CMS content sources, multi-language sites, base paths, sitemap indexes, MCP server, AVIF images.
+
+## Security
+
+Page content is sanitized, dev/preview servers serve only what a site needs, and Cloudflare deploys get security headers with a computed Content-Security-Policy. Report vulnerabilities privately: see [SECURITY.md](SECURITY.md).
 
-Failures exit with code 1. With `--json`, output is `{ "ok": true, ... }` or `{ "ok": false, "error": "..." }`.
+### Protection against spam injection
 
-See [`templates/starter/AGENTS.md`](templates/starter/AGENTS.md) for routing, frontmatter, layouts and config, and [`docs/DECISIONS.md`](docs/DECISIONS.md) for design decisions.
+A static site has no admin panel, database or plugins to break into, so attackers go after your content, your repository or your host instead, usually to plant hidden links and spam pages. vagejs checks for that on every build:
 
-## Getting started
+| Protection | What it does |
+| --- | --- |
+| Outbound link lock | `links.lock.json` lists the domains your site may link to. A link to any other domain fails `vage check` (and so every deploy) until you approve it with `vage links approve <domain>`. It covers links, SVG links, canonical and `og:url`, meta refresh, iframes and forms. With `security.outboundLinks: "require"` (the default for new sites), deleting the file is an error too. |
+| No hidden links | Inline styles, SVG `xlink:href` and legacy tags like `<font>` and `<marquee>` are stripped from page content, and `vage check` fails on external links hidden with CSS, `aria-hidden`, zero-size SVG or no visible content, in content and layouts. |
+| Spam words | `vage check` warns about common injected-spam terms (casino, pharmacy, payday loans…). |
+| Live-site verification | `vage verify` compares the live site with your files: links and other targets, scripts, event handlers, JSON-LD, text (spam words are errors), `robots.txt` and sitemap pages. It also looks at every page as Googlebot, to catch spam shown only to search engines. New sites include a daily GitHub Actions check (opt in with the `VAGE_VERIFY_URL` variable). |
+| Supply chain | New sites run `npm audit` in CI and get Dependabot updates. |
 
 ```bash
-npm create vagejs@latest my-site
-cd my-site
-npm install
-npm run dev
+npx vage links                    # list external domains: approved or NEW
+npx vage links approve github.com # approve one (asks first)
+npx vage verify                   # compare the live site with your files
 ```
 
-## Releasing
-
-See [docs/RELEASING.md](docs/RELEASING.md).
+Also protect the accounts around the site: two-factor authentication on GitHub, npm, your host and your domain registrar; branch protection on `main`; and Google Search Console alerts for security issues.
 
-## Development
+## Contributing
 
 ```bash
+npm install
 npm run typecheck
 npm test            # unit + integration (dev server, preview, build)
 npm run test:e2e    # pack -> create -> npm install/ci -> build (needs npm registry or cache)
 ```
 
-## Planned, not implemented
-
-Bing Webmaster Tools API, Netlify/Vercel adapters, CMS content sources, multi-language sites, base paths, sitemap indexes, MCP server. See `VAGE_JS_BUILD_SPEC.md`.
-
-## Security
-
-Page content is sanitized, dev/preview servers serve only what a site needs, and Cloudflare deploys get security headers with a computed Content-Security-Policy. Report vulnerabilities privately: see [SECURITY.md](SECURITY.md).
+Design decisions: [docs/DECISIONS.md](docs/DECISIONS.md). Provider rules and how they're monitored: [docs/GUIDELINES.md](docs/GUIDELINES.md). Releasing: [docs/RELEASING.md](docs/RELEASING.md).
 
 ## Brand assets
 
 - `assets/brand/vagejs-icon.svg`: the icon, as received (with C2PA content credentials).
-- `assets/brand/vagejs-logo.svg`: the wordmark with **Space Grotesk Bold embedded** (a 1.6 KB WOFF2 subset of the glyphs used), so it renders the same without the font installed. Space Grotesk is © 2020 The Space Grotesk Project Authors, under the SIL Open Font License 1.1 (`assets/brand/fonts/OFL.txt`). Editing the file invalidated its C2PA signature, so that block was removed.
+- `assets/brand/vagejs-logo.svg` and `vagejs-logo-dark.svg`: the wordmark for light and dark backgrounds, with **Space Grotesk Bold embedded** (a 1.6 KB WOFF2 subset of the glyphs used), so it renders the same without the font installed. Space Grotesk is © 2020 The Space Grotesk Project Authors, under the SIL Open Font License 1.1 (`assets/brand/fonts/OFL.txt`). Editing the file invalidated its C2PA signature, so that block was removed. The dark variant only changes the text colors (`#F4F5FA` and `#A5B4FC`).
 - New sites get `favicon.svg`, `favicon.ico` (16/32/48) and `apple-touch-icon.png` (180) in `public/`.
 
 Regenerate (maintainer tools; need Chrome, and Python with `fonttools` and `brotli`):

package/dist/checks/check.d.ts

@@ -18,6 +18,8 @@ export interface CheckResult {
     pagesChecked: number;
     findings: Finding[];
 }
+/** Paths of frontmatter values that came out empty (null), e.g. "faq.items[3].Claude and other AI search?". */
+export declare function emptyValues(value: unknown, at?: string): string[];
 /**
  * Static checks on the rendered site (nothing is written to disk).
  * Passing checks mean the output is technically sound; they do not guarantee

package/dist/checks/check.js

@@ -3,6 +3,7 @@ import path from "node:path";
 import { parse } from "node-html-parser";
 import { createBuildLoader } from "../core/loader.js";
 import { listedPages, renderSite } from "../core/site.js";
+import { externalHost, hiddenReason, isEmptyLink, LINK_LOCK, linkTarget, outboundHosts, readLinkLock, spamHits } from "./integrity.js";
 const DOCS = {
     title: "https://developers.google.com/search/docs/appearance/title-link",
     description: "https://developers.google.com/search/docs/appearance/snippet",
@@ -13,7 +14,19 @@ const DOCS = {
     lang: "https://www.w3.org/WAI/WCAG22/Understanding/language-of-page.html",
     linkPurpose: "https://www.w3.org/WAI/WCAG22/Understanding/link-purpose-in-context.html",
     robots: "https://developers.google.com/search/docs/crawling-indexing/robots/intro",
+    yamlFlow: "https://yaml.org/spec/1.2.2/#74-flow-collection-styles",
 };
+/** Paths of frontmatter values that came out empty (null), e.g. "faq.items[3].Claude and other AI search?". */
+export function emptyValues(value, at = "") {
+    if (value === null)
+        return [at || "(frontmatter)"];
+    if (Array.isArray(value))
+        return value.flatMap((v, i) => emptyValues(v, `${at}[${i}]`));
+    if (typeof value === "object" && value !== undefined) {
+        return Object.entries(value).flatMap(([k, v]) => emptyValues(v, at ? `${at}.${k}` : k));
+    }
+    return [];
+}
 /**
  * Static checks on the rendered site (nothing is written to disk).
  * Passing checks mean the output is technically sound; they do not guarantee
@@ -23,7 +36,7 @@ export async function check(root, options = {}) {
     root = path.resolve(root);
     const loader = await createBuildLoader(root);
     try {
-        const model = await renderSite(root, loader.load);
+        const model = await renderSite(root, loader.load, { collectErrors: true });
         const findings = checkSite(root, model);
         const errors = findings.filter((f) => f.severity === "error").length;
         const warnings = findings.length - errors;
@@ -31,7 +44,7 @@ export async function check(root, options = {}) {
             ok: errors === 0 && (!options.strict || warnings === 0),
             errors,
             warnings,
-            pagesChecked: model.pages.length + (model.notFound ? 1 : 0),
+            pagesChecked: model.pages.length + (model.notFound ? 1 : 0) + model.pageErrors.length,
             findings,
         };
     }
@@ -41,7 +54,13 @@ export async function check(root, options = {}) {
 }
 export function checkSite(root, model) {
     const findings = model.warnings.map((message) => ({ rule: "config", severity: "warning", message }));
+    for (const e of model.pageErrors)
+        findings.push({ rule: "page-error", severity: "error", message: e.message, source: e.source });
     const outputs = outputPaths(root, model);
+    // Pages that failed to render are already reported; links to them aren't broken links on top of that.
+    for (const e of model.pageErrors)
+        if (e.route)
+            outputs.add(`${e.route}index.html`);
     const drafts = new Set(model.draftRoutes);
     const pages = model.notFound ? [...model.pages, model.notFound] : model.pages;
     for (const page of pages) {
@@ -50,10 +69,31 @@ export function checkSite(root, model) {
     }
     findings.push(...checkDuplicates(listedPages(model)));
     findings.push(...checkRobots(model));
+    findings.push(...checkRedirectTargets(model, outputs));
+    findings.push(...checkOutboundLinks(root, model));
     return findings;
 }
 function checkPage(page, doc, model, outputs, drafts, add) {
     const indexable = !page.meta.noindex;
+    if (model.site.security.spamPatterns) {
+        const text = `${doc.querySelector("body")?.text ?? ""} ${doc.querySelectorAll("a[href]").map((a) => a.getAttribute("href")).join(" ")}`;
+        const hits = spamHits(text);
+        if (hits.length) {
+            add({
+                rule: "spam-pattern",
+                severity: "warning",
+                message: `Page contains words common in injected spam (${hits.join(", ")}). If you didn't write them, check this page's history. Turn off with security.spamPatterns: false.`,
+            });
+        }
+    }
+    for (const key of emptyValues(page.frontmatter)) {
+        add({
+            rule: "frontmatter-empty",
+            severity: "warning",
+            message: `Frontmatter "${key}" has no value. If you meant to write text there, check for a missing value; inside { } a comma or colon splits the text, so wrap it in quotes.`,
+            docs: DOCS.yamlFlow,
+        });
+    }
     if (!doc.querySelector("html")?.getAttribute("lang")) {
         add({ rule: "html-lang", severity: "error", message: "<html> has no lang attribute.", docs: DOCS.lang });
     }
@@ -104,6 +144,7 @@ function checkPage(page, doc, model, outputs, drafts, add) {
     }
     for (const a of doc.querySelectorAll("a[href]")) {
         const href = a.getAttribute("href");
+        checkHiddenLink(a, href, model, add);
         if (!a.text.trim() && !a.getAttribute("aria-label") && !a.querySelector("img[alt]:not([alt=''])")) {
             add({ rule: "link-text", severity: "warning", message: `Link to "${href}" has no accessible text.`, docs: DOCS.linkPurpose });
         }
@@ -127,6 +168,11 @@ function checkPage(page, doc, model, outputs, drafts, add) {
             add({ rule: "link-trailing-slash", severity: "warning", message: `Link to "${href}" redirects; use "${target}/".` });
         }
     }
+    // SVG links (a xlink:href) aren't in the loop above; they get the hidden-link check too.
+    for (const a of doc.querySelectorAll("a")) {
+        if (!a.hasAttribute("href") && a.getAttribute("xlink:href"))
+            checkHiddenLink(a, linkTarget(a), model, add);
+    }
     for (const [selector, attr] of [
         ["img[src]", "src"],
         ["script[src]", "src"],
@@ -253,6 +299,64 @@ function checkDuplicates(pages) {
     }
     return findings;
 }
+/** Links visitors can't see but search engines can: the signature of injected spam. */
+function checkHiddenLink(a, href, model, add) {
+    const external = externalHost(href, model.site.site) !== undefined;
+    const hidden = hiddenReason(a) ?? (external && isEmptyLink(a) ? { reason: "empty (no text or image)", strength: "strong" } : undefined);
+    if (!hidden)
+        return;
+    add({
+        rule: "link-hidden",
+        severity: external && hidden.strength === "strong" ? "error" : "warning",
+        message: `Link to "${href}" is ${hidden.reason}: visitors can't see it, but search engines can. Hidden links are a common sign of injected spam. Remove it, or show it.`,
+    });
+}
+/**
+ * Outbound link guard: with security.outboundLinks "lock", every external domain must be
+ * listed in links.lock.json. A domain nobody approved is how injected spam usually shows up.
+ */
+function checkOutboundLinks(root, model) {
+    if (model.site.security.outboundLinks === "off")
+        return [];
+    const hosts = outboundHosts(model);
+    const lock = readLinkLock(root);
+    if (!lock && model.site.security.outboundLinks === "require") {
+        return [{ rule: "links-lock-missing", severity: "error", message: `${LINK_LOCK} is missing, but security.outboundLinks is "require". Restore it from git, or have the site owner recreate it with "npx vage links approve --all" after reviewing "npx vage links".` }];
+    }
+    if (!lock) {
+        return hosts.size
+            ? [{ rule: "links-lock-missing", severity: "warning", message: `The site links to ${hosts.size} external domain(s) and has no ${LINK_LOCK}. The site owner should review them with "npx vage links" and approve them with "npx vage links approve --all"; after that, links to any new domain fail the check.` }]
+            : [];
+    }
+    const approved = new Set(lock.domains);
+    const findings = [];
+    for (const [host, pages] of hosts) {
+        if (approved.has(host))
+            continue;
+        const where = pages.map((p) => p.route);
+        findings.push({
+            rule: "link-unapproved",
+            severity: "error",
+            message: `Links to ${host} (on ${where.slice(0, 3).join(", ")}${where.length > 3 ? ` and ${where.length - 3} more` : ""}) aren't approved in ${LINK_LOCK}. If the site owner added them on purpose, they approve with "npx vage links approve ${host}". Otherwise remove them: unapproved links are how injected spam usually shows up.`,
+            page: pages[0].route,
+            source: pages[0].source,
+        });
+    }
+    return findings;
+}
+/** Redirects to a page on this site must land on something that exists. */
+function checkRedirectTargets(model, outputs) {
+    const findings = [];
+    for (const r of model.redirects) {
+        const target = internalPath(r.to, "/", model.site.site);
+        if (target === undefined)
+            continue;
+        if (target === MALFORMED || resolveOutput(target, outputs) === "missing") {
+            findings.push({ rule: "redirect-target", severity: "error", message: `Redirect from "${r.from}" points to "${r.to}", which isn't a page or file on this site.`, source: "vage.config.ts" });
+        }
+    }
+    return findings;
+}
 function checkRobots(model) {
     const robots = model.files.get("robots.txt");
     if (!robots)

package/dist/checks/crawlability.d.ts

@@ -0,0 +1,34 @@
+import type { Difference } from "./verify.js";
+/**
+ * Can search engines and AI crawlers actually reach the live site? Checks what a site's
+ * own files can't show: CDN/firewall blocks and challenges, robots.txt rules, missing
+ * http→https and www redirects, and noindex headers. Probes send each crawler's
+ * documented user agent (or token), so blocks based on the real crawlers' IP
+ * addresses can't be detected this way.
+ */
+export interface Crawler {
+    token: string;
+    provider: string;
+    purpose: "search" | "ai-search" | "ai-training" | "ai-user";
+    userAgent?: string;
+    source: string;
+}
+export declare const CRAWLERS: Crawler[];
+/** The documented user agent (W.X.Y.Z filled in), or a minimal one carrying the documented token. */
+export declare function probeUserAgent(c: Crawler): string;
+/**
+ * Is the site root ("/") disallowed for this crawler? RFC 9309: the crawler follows the
+ * groups naming its token (merged), else the "*" groups; among rules matching "/", the
+ * longest pattern wins and Allow wins ties. Lines like Sitemap: don't end a group.
+ */
+export declare function robotsBlocks(robots: string, token: string): boolean;
+/** Does an X-Robots-Tag value say noindex for everyone, or for one of these crawlers? */
+export declare function xRobotsNoindex(value: string, tokens: string[]): boolean;
+type Fetcher = typeof fetch;
+export declare function checkCrawlability(origin: string, options?: {
+    fetch?: Fetcher;
+}): Promise<{
+    differences: Difference[];
+    crawlersChecked: number;
+}>;
+export {};

package/dist/checks/crawlability.js

@@ -0,0 +1,142 @@
+import { ruleData } from "../guidelines/pack.js";
+export const CRAWLERS = ruleData("crawlers.probes").crawlers;
+/** The documented user agent (W.X.Y.Z filled in), or a minimal one carrying the documented token. */
+export function probeUserAgent(c) {
+    return c.userAgent ? c.userAgent.replace("W.X.Y.Z", "131.0.0.0") : `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ${c.token}/1.0)`;
+}
+/** A challenge or block page, recognized by its title (not by scripts like reCAPTCHA that normal pages load). */
+const CHALLENGE_TITLE = /<title>\s*(Just a moment|Attention Required|Access denied|Checking your browser)/i;
+/** robots.txt path pattern -> RegExp (RFC 9309: "*" wildcard, "$" end anchor). */
+function patternRegex(pattern) {
+    const anchored = pattern.endsWith("$");
+    const body = (anchored ? pattern.slice(0, -1) : pattern).split("*").map((part) => part.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")).join(".*");
+    return new RegExp(`^${body}${anchored ? "$" : ""}`);
+}
+/**
+ * Is the site root ("/") disallowed for this crawler? RFC 9309: the crawler follows the
+ * groups naming its token (merged), else the "*" groups; among rules matching "/", the
+ * longest pattern wins and Allow wins ties. Lines like Sitemap: don't end a group.
+ */
+export function robotsBlocks(robots, token) {
+    const groups = [];
+    let current;
+    let collectingAgents = false;
+    for (const raw of robots.split(/\r?\n/)) {
+        const m = /^\s*([a-z-]+)\s*:\s*(.*?)\s*$/i.exec(raw.replace(/#.*/, ""));
+        if (!m)
+            continue;
+        const key = m[1].toLowerCase();
+        const value = m[2];
+        if (key === "user-agent") {
+            if (!current || !collectingAgents)
+                groups.push((current = { agents: [], rules: [] }));
+            // "Googlebot/2.1" names the Googlebot product token.
+            current.agents.push(value.toLowerCase().split("/")[0].trim());
+            collectingAgents = true;
+        }
+        else if (key === "allow" || key === "disallow") {
+            collectingAgents = false;
+            if (current && value)
+                current.rules.push({ allow: key === "allow", pattern: value });
+        }
+    }
+    const named = groups.filter((g) => g.agents.includes(token.toLowerCase()));
+    const rules = (named.length ? named : groups.filter((g) => g.agents.includes("*"))).flatMap((g) => g.rules);
+    let best;
+    for (const r of rules) {
+        if (!patternRegex(r.pattern).test("/"))
+            continue;
+        const length = r.pattern.length;
+        if (!best || length > best.length || (length === best.length && r.allow))
+            best = { allow: r.allow, length };
+    }
+    return best !== undefined && !best.allow;
+}
+/** Does an X-Robots-Tag value say noindex for everyone, or for one of these crawlers? */
+export function xRobotsNoindex(value, tokens) {
+    const lower = tokens.map((t) => t.toLowerCase());
+    // "googlebot: noindex, nofollow" targets one bot; "noindex" targets all.
+    return value.split(/,(?=\s*[a-z-]+\s*:)/i).some((part) => {
+        const m = /^\s*([a-z-]+)\s*:\s*(.*)$/i.exec(part);
+        const target = m && !/^(unavailable_after|max-[a-z-]+)$/i.test(m[1]) ? m[1].toLowerCase() : undefined;
+        const directives = target ? m[2] : part;
+        return (!target || lower.includes(target)) && /\b(noindex|none)\b/i.test(directives);
+    });
+}
+export async function checkCrawlability(origin, options = {}) {
+    const doFetch = options.fetch ?? fetch;
+    const out = [];
+    const get = async (url, init = {}, maxChars = 8000) => {
+        try {
+            const res = await doFetch(url, { signal: AbortSignal.timeout(20_000), ...init });
+            // Read the body here too: a reset mid-download is a failed request, not a crash.
+            const body = res.ok ? (await res.text()).slice(0, maxChars) : "";
+            return { status: res.status, ok: res.ok, headers: res.headers, body };
+        }
+        catch {
+            return undefined;
+        }
+    };
+    const blockedReason = (res) => !res ? "no response" : res.headers.get("cf-mitigated") ? `blocked by the CDN (cf-mitigated: ${res.headers.get("cf-mitigated")})` : !res.ok ? `HTTP ${res.status}` : CHALLENGE_TITLE.test(res.body) ? "a challenge page instead of the site" : undefined;
+    // 1. Crawlers reach the home page. Only blocks that a normal browser doesn't get count.
+    const browser = await get(`${origin}/`, { headers: { "user-agent": "Mozilla/5.0 (compatible; vagejs-verify) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", accept: "text/html" } });
+    const browserBlocked = blockedReason(browser);
+    if (browserBlocked) {
+        out.push({ route: "/", severity: "error", kind: "crawler-blocked", detail: `Even a normal browser gets ${browserBlocked}, so crawlers can't reach the site either.` });
+    }
+    else {
+        for (const c of CRAWLERS) {
+            const why = blockedReason(await get(`${origin}/`, { headers: { "user-agent": probeUserAgent(c), accept: "text/html" } }));
+            if (!why)
+                continue;
+            out.push({
+                route: "/",
+                severity: "warning",
+                kind: "crawler-blocked",
+                detail: `${c.token} (${c.provider}, ${c.purpose}) gets ${why}. ${c.purpose === "search"
+                    ? "If your host blocks fake search bots, this probe (which can't come from the real crawler's addresses) may be what's blocked; confirm in Google Search Console or Bing Webmaster Tools."
+                    : "If you didn't block it on purpose, check your host's bot or AI-crawler settings."}`,
+            });
+        }
+    }
+    // 2. robots.txt
+    // Read robots.txt in full (Google processes up to 500 KiB), not just the first few KB.
+    const robotsRes = await get(`${origin}/robots.txt`, {}, 512 * 1024);
+    if (robotsRes?.ok) {
+        for (const c of CRAWLERS.filter((x) => x.purpose !== "ai-user")) {
+            if (robotsBlocks(robotsRes.body, c.token)) {
+                out.push({
+                    route: "/robots.txt",
+                    severity: c.purpose === "search" ? "error" : "warning",
+                    kind: "robots-blocked",
+                    detail: `robots.txt disallows ${c.token} (${c.provider}, ${c.purpose}) from the whole site.${c.purpose === "ai-training" ? " Fine if that's your choice (seo.robots.aiTraining)." : ""}`,
+                });
+            }
+        }
+    }
+    // 3. noindex sent as a header hides the site even when the HTML says nothing.
+    const home = await get(`${origin}/`, { redirect: "manual" });
+    const xRobots = home?.headers.get("x-robots-tag");
+    if (xRobots && xRobotsNoindex(xRobots, ["googlebot", "bingbot"])) {
+        out.push({ route: "/", severity: "error", kind: "noindex-header", detail: `The server sends "X-Robots-Tag: ${xRobots}", which keeps the site out of search results.` });
+    }
+    // 4. One address per page: http should redirect to https, and www/non-www to the site's host.
+    //    Skipped for local addresses (ports, IPs, localhost), where there's no public twin.
+    const url = new URL(origin);
+    const publicHost = !url.port && !/^(localhost|\d+\.\d+\.\d+\.\d+|\[.*\])$/i.test(url.hostname) && url.hostname.includes(".");
+    if (publicHost && url.protocol === "https:") {
+        const plain = await get(`http://${url.host}/`, { redirect: "manual" });
+        const to = plain?.headers.get("location") ?? "";
+        if (plain && !(plain.status >= 300 && plain.status < 400 && to.startsWith("https://"))) {
+            out.push({ route: "/", severity: "warning", kind: "redirect-missing", detail: `http://${url.host}/ answers ${plain.status} instead of redirecting to https. Turn on "Always use HTTPS" (or equivalent) at your host.` });
+        }
+    }
+    if (publicHost) {
+        const other = url.hostname.startsWith("www.") ? url.hostname.slice(4) : `www.${url.hostname}`;
+        const twin = await get(`${url.protocol}//${other}/`, { redirect: "manual" });
+        if (twin && twin.status === 200) {
+            out.push({ route: "/", severity: "warning", kind: "duplicate-host", detail: `${other} serves the site too instead of redirecting to ${url.hostname}. Add a redirect at your host so search engines see one address.` });
+        }
+    }
+    return { differences: out, crawlersChecked: browserBlocked ? 0 : CRAWLERS.length };
+}