pagegraph 0.8.4 → 0.10.0

package/README.md

@@ -43,6 +43,17 @@ OpenCode SDK and its private Effect runtime only when invoked. `vite` stays a pe
 load your app through Vite at runtime — and `lighthouse` is only needed by `pagegraph audit`.
 Importing `pagegraph/audit` programmatically still needs `effect`.
 
+### Agent skill
+
+The Pagegraph CLI serves its bundled TanStack Start integration skill:
+
+```bash
+pagegraph skills list
+pagegraph skills get core
+```
+
+The skill is maintained at [skill-data/core/SKILL.md](skill-data/core/SKILL.md) and compiled into the CLI, so the instructions match the installed Pagegraph version. `skills get core` prints Markdown to stdout; `--json` returns a structured payload.
+
 ## Quick start
 
 **1. Bind your site identity once.** Route files never see an origin or a brand name.
@@ -88,6 +99,7 @@ export const Route = createFileRoute("/pricing")({
       sitemap: { priority: 0.9, changeFrequency: "weekly" },
       related: ["/features", "/docs"],
       link: { title: "Pricing", description: "Simple volume pricing." },
+      modifiedAt: "2026-09-29", // sitemap <lastmod> and the freshness report
     },
   },
   head: (ctx) =>
@@ -349,9 +361,15 @@ severities:
 - **structural** — internally broken declarations; these fail `pagegraph check` (exit 1):
   dead or duplicate edges, path collisions, a redirect in the sitemap, a
   sitemap/noindex contradiction, a `related` target with no `link` card, an
-  instance without a title, or an unmet contextual-link coverage rule.
+  instance without a title, a date that is not ISO 8601, or an unmet
+  contextual-link coverage rule.
 - **editorial** — quality smells, reported but non-failing: duplicate or mis-sized
-  titles and descriptions.
+  titles and descriptions, and pages older than the freshness policy.
+
+Each sitemap entry's `<lastmod>` is the page's last declared change: an
+instance's `modifiedAt`, else its `publishedAt`; a route's `modifiedAt`. A param
+route's date is never inherited by its instances. Undated pages emit no
+`<lastmod>`.
 
 ## Coverage gate (Vite)
 
@@ -401,6 +419,8 @@ export default defineSeoConfig({
 
 ```bash
 pagegraph check                 # CI gate — exit 1 on structural violations
+pagegraph check --site <url>    # also check every declared page's rendered HTML
+pagegraph stale                 # refresh queue: sitemap pages past the freshness policy
 pagegraph graph                 # the graph as a tree · --format mermaid | json
 pagegraph inspect /pricing      # one node: policy, sitemap status, in/out edges
 pagegraph inspect <url> --live  # fetch a deployed page, validate its rendered <head>
@@ -423,7 +443,12 @@ private-IP-blocked HTTP path as `pagegraph audit` — and reports what a crawler
 actually receives: the real homepage depth, the pages with no incoming internal
 edge (rendered orphans), and, when the app has a `seo.config.ts`, the
 declared-vs-rendered link diff. It is bounded with `--limit` and needs no
-framework or config:
+framework or config. Discovery reads same-origin `robots.txt` and bounded
+sitemaps before following page anchors, so a sitemap-only page can appear as an
+orphan. The JSON `crawl` block records skipped URLs, discovery failures, and
+sitemap truncation. Link-followed non-HTML alternates are recorded in `nonHtml`
+without counting as HTML page failures; a sitemap-listed or declared page served
+as non-HTML still blocks a coverage assertion:
 
 ```bash
 pagegraph links verify https://example.com
@@ -456,6 +481,11 @@ and the provenance a gate needs to trust it:
     "bodyTruncated": false,
     "truncatedPages": [],
     "failures": [],
+    "nonHtml": [{ "url": "https://example.com/auth.md", "finalUrl": "https://example.com/auth.md", "contentType": "text/markdown" }],
+    "discoveryFailures": [],
+    "skipped": [],
+    "sitemapTruncated": false,
+    "sitemapUsed": true,
   },
   "nodes": ["/", "/pricing"],
   "edges": [{ "from": "/", "to": "/pricing", "region": "body" }],
@@ -481,6 +511,7 @@ pagegraph links verify --rendered .seo/rendered.json --assert-coverage --json |
 The command exits non-zero when any rule is unmet, and **refuses to assert** (also
 non-zero) rather than reporting a false pass when the crawl outran `--limit`, a
 page failed to fetch (its anchors are missing), a body hit `--max-body-bytes`, the
+robots or sitemap discovery is incomplete, the
 config declares no `coverage`, or the artifact's origin differs from
 `seo.config.ts`. Under `--assert-coverage` the report gains a `coverage` block
 (`{ rules, ok, violations }`); the default `--json` summary is unchanged.
@@ -491,9 +522,11 @@ config declares no `coverage`, or the artifact's origin differs from
 `(source, destination)` pairs that are plausible contextual links and not already
 connected. Pages are grouped into clusters — a shared top-level section, or the
 same `kind` for root-level pages — and only sitemap-eligible pages are proposed.
-Pairs already declared as a `related` edge are excluded, and `--rendered` accepts
-a JSON dump of already-served anchors (`[{ from, to }]` or `{ edges: [...] }`) to
-exclude those too.
+Each direction is evaluated separately. A declared `related` edge or served
+body-region anchor excludes only its own direction. Nav, header, and footer
+anchors do not exclude contextual suggestions. `--rendered` accepts a JSON dump
+of already-served anchors (`[{ from, to, region }]` or `{ edges: [...] }`).
+Legacy edges without a `region` are treated as body links.
 
 The output is a deterministic, reviewable plan: human text by default and
 versioned JSON with `--json`. Nothing is applied; `--limit` and `--cluster`
@@ -507,6 +540,43 @@ pagegraph links candidates --rendered rendered.json --json | jq
 pagegraph improve links --page "/docs/**"
 ```
 
+For content-backed placements, opt in to a bounded same-origin probe. The configured
+`origin` must match `--site`. Each schema-version-2 result carries a sentence
+from served main content, an exact anchor phrase inside it, a target sentence,
+and score components. Unreadable and over-limit pages are reported as skipped.
+Code samples and tabular comparisons are excluded. An anchor is proposed only
+when its distinct content words occur together in a target sentence, so review
+the placement and its context before editing. Repeated passages and clause
+fragments are excluded; named destinations require a destination-specific anchor.
+The home and about pages are omitted as destinations because site navigation already leads there.
+The pair-only command above remains offline and emits schema version 1.
+The page budget selects low-inbound candidate pairs first. Use repeatable
+`--target /exact/path` to put a known weak destination within the budget. Add
+`--source /exact/path` when you have a likely source and want to guarantee the
+pair is probed even on a large site; raise `--page-limit` for more pairs.
+For a site with many legal, comparison, or other low-inbound pages, use
+`--cluster blog` (or another section) to spend the same page budget on an
+editorial cluster. A zero-result broad probe means no placement passed the
+evidence rules within its selected pages; it does not mean the whole site has
+no possible links.
+
+```bash
+pagegraph links verify https://example.com --json
+pagegraph links candidates --site https://example.com --page-limit 25 --json > /tmp/pagegraph-suggestions.json
+pagegraph links candidates --site https://example.com --cluster blog --page-limit 25 --json
+pagegraph links candidates --site https://example.com --source /blog/guide --target /blog/weak --page-limit 25 --json > /tmp/pagegraph-suggestions.json
+pagegraph improve links --suggestions /tmp/pagegraph-suggestions.json --dry-run
+pagegraph improve links --suggestions /tmp/pagegraph-suggestions.json
+```
+
+`improve links` validates the artifact against the configured origin and graph,
+then asks Jev to judge the suggested placements. Only accepted `add` or `update`
+decisions can open an edit turn; skipped and review-band links remain in the run
+artifact without authorizing an edit. Before an accepted edit, Pagegraph checks
+that the source sentence and target passage still occur in served copy. Regenerate
+the artifact if either page changed. For a local site, pass `--allow-private` to
+both `links candidates --site` and `improve links --suggestions`.
+
 ### Contextual-link coverage
 
 Declare a contextual-link coverage policy in `seo.config.ts` (a `coverage` array
@@ -524,6 +594,83 @@ pagegraph check --require-inbound "/pricing=2" --require-inbound "/features/*=1"
 actually served, assert the same policy on the rendered graph — see
 [Assert coverage on the rendered graph](#assert-coverage-on-the-rendered-graph).
 
+### Check rendered content
+
+The declared graph cannot see what a page renders: headings come from
+components, and the `<title>`, JSON-LD, and visible text are composed at render
+time. `pagegraph check --site <url>` fetches every sitemap-eligible page (and
+every concrete page that declares `noindex`) from a running server and checks
+the server-rendered HTML — what a crawler receives before any script runs. A
+server error or failed request is retried once, because a dev server's first
+render of a route can fail while it compiles.
+
+```bash
+pagegraph check --site http://localhost:3000 --allow-private
+pagegraph check --site https://staging.example.com --json | jq '.violations'
+```
+
+| Rule | Severity | Fails when |
+| --- | --- | --- |
+| `rendered-page-unavailable` | structural | The page errors, is not HTML, is truncated, or redirects off its declared path |
+| `rendered-robots-mismatch` | structural | An indexable page renders a `noindex` robots meta, or a `noindex` declaration renders none |
+| `rendered-canonical-mismatch` | structural | The canonical is missing or its path is not the declared path |
+| `missing-h1`, `multiple-h1`, `empty-h1` | structural | The page does not have exactly one H1 with text |
+| `faq-not-visible` | structural | A FAQPage question or answer is not in the rendered text |
+| `offer-price-not-visible` | structural | An Offer or AggregateOffer price does not appear on the page |
+| `heading-level-skip` | editorial | The main-content outline skips a level (h2 → h4) |
+| `h1-title-mismatch` | editorial | The H1 and `<title>` share no significant term |
+| `webpage-head-mismatch` | editorial | A WebPage `name` or `description` disagrees with the head |
+| `thin-content` | editorial | Main content is under the page's `content.minWords` floor |
+
+Google requires structured data to describe content the page visibly shows, so
+a FAQ answer that exists only in JSON-LD or in hydration data (an accordion that
+mounts answers on open, say) is structural. Comparisons ignore punctuation,
+case, and markup; prices match as numbers (`19` matches `$19/mo` and `19.00`,
+not `$199`). Canonicals compare by path, and robots are read from the meta tag
+only, so a local or preview host that sends `X-Robots-Tag: noindex` everywhere
+can still run the check.
+
+Word-count floors are project policy, declared per path glob; a page matching
+several rules must meet the highest:
+
+```ts
+export default defineSeoConfig({
+  // ...
+  content: {
+    minWords: [
+      { path: "/blog/*", minWords: 500 },
+      { path: "/compare/*", minWords: 800 },
+    ],
+  },
+});
+```
+
+`pagegraph audit` runs the heading and structured-data rules on every audited
+HTML page too. The robots, canonical, and word-count rules need the declared
+graph, so they stay in `check`.
+
+### Freshness
+
+Declare a freshness policy to turn old pages into editorial `stale-page`
+findings in `pagegraph check`, and list the refresh queue with `pagegraph stale`:
+
+```ts
+export default defineSeoConfig({
+  // ...
+  freshness: { maxAgeDays: 90 },
+});
+```
+
+```bash
+pagegraph stale                     # stale pages, oldest first; undated pages counted
+pagegraph stale --max-age-days 180 --json
+```
+
+Only sitemap-eligible pages are judged. A page without a date is reported as
+undated rather than stale; give content its `modifiedAt` (or a route its
+`staticData.seo.modifiedAt`) to bring it into the report and the sitemap's
+`<lastmod>`.
+
 ### Audit any website
 
 `pagegraph audit` is framework-independent and does not need `seo.config.ts`. It
@@ -584,6 +731,8 @@ Everything the CLI checks is a pure function you can call from a test:
 ```ts
 import {
   checkGraph,
+  checkPageContent,
+  extractPageContent,
   hasStructuralViolations,
   inspectHtml,
 } from "pagegraph";
@@ -593,6 +742,9 @@ expect(hasStructuralViolations(checkGraph(loadSeoGraph()))).toBe(false);
 // render a page however you like, then assert the head it actually ships
 const report = inspectHtml(url, 200, html);
 expect(report.issues).toEqual([]);
+
+// …and the content it ships: one H1, visible FAQ answers, shown prices
+expect(checkPageContent(extractPageContent(html))).toEqual([]);
 ```
 
 ## TanStack Start example

package/dist/audit.d.ts

@@ -1,3 +1,4 @@
+import "./content-DVL2iyGU.js";
 import * as Context from "effect/Context";
 import * as Effect from "effect/Effect";
 import * as Layer from "effect/Layer";
@@ -103,6 +104,12 @@ interface ProbeOptions {
    * probe field that grows with document size; the audit report leaves it unset.
    */
   readonly captureAnchors?: boolean | undefined;
+  /** Capture bounded text for robots and sitemap discovery. Never enabled for audit reports. */
+  readonly captureBody?: boolean | undefined;
+  /** Refuse cross-origin redirects during a same-origin crawl. */
+  readonly sameOrigin?: string | undefined;
+  /** Optional per-redirect crawl policy, checked before each network request. */
+  readonly allowUrl?: ((url: URL) => boolean) | undefined;
 }
 interface AuditOptions {
   readonly concurrency?: number;

package/dist/audit.js

@@ -1,4 +1,4 @@
-import { a as extractAnchors } from "./links-BFJc-pJ9.js";
+import { l as extractAnchors, r as extractPageContent, t as checkPageContent } from "./content-DbyJoRpj.js";
 import { mkdir, mkdtemp, readFile, rename, rm, writeFile } from "node:fs/promises";
 import { join } from "node:path";
 import * as Context from "effect/Context";
@@ -110,6 +110,22 @@ const pageRules = [
 				return [finding(input, "invalid-canonical", "Page canonical is not a valid URL.", "Use an absolute or resolvable canonical URL.", canonical)];
 			}
 		}
+	},
+	{
+		id: "content",
+		evaluate: (observation, input) => {
+			const content = record(record(pageEvidence(observation.evidence))?.["document"])?.["content"];
+			const list = record(content)?.["findings"];
+			if (!Array.isArray(list)) return [];
+			return list.flatMap((entry) => {
+				const rule = stringAt(entry, "rule");
+				const message = stringAt(entry, "message");
+				const fix = stringAt(entry, "fix");
+				const severity = stringAt(entry, "severity");
+				if (rule === void 0 || message === void 0 || fix === void 0) return [];
+				return [finding(input, rule, message, fix, void 0, severity === "editorial" ? "editorial" : "structural")];
+			});
+		}
 	}
 ];
 const evaluateRules = (rules, observation, input) => rules.flatMap((rule) => rule.evaluate(observation, input));
@@ -864,10 +880,11 @@ const canonicalUrl = (html) => {
 	return null;
 };
 const countMatches = (value, pattern) => [...value.matchAll(pattern)].length;
-const extractDocumentSignals = (body, contentType, requestedUrl) => {
-	const isHtml = contentType.includes("text/html");
+const extractDocumentSignals = (body, contentType, requestedUrl, bodyTruncated = false) => {
+	const isHtml = contentType.includes("text/html") || contentType.includes("application/xhtml+xml");
 	const isMarkdown = contentType.includes("text/markdown");
 	const looksJson = contentType.includes("json") || requestedUrl.pathname.endsWith(".json");
+	const content = isHtml && !bodyTruncated ? extractPageContent(body) : null;
 	const titleMatch = isHtml ? /<title\b[^>]*>([\s\S]*?)<\/title>/i.exec(body) : null;
 	let jsonValid = null;
 	if (looksJson && body.trim().length > 0) try {
@@ -885,8 +902,13 @@ const extractDocumentSignals = (body, contentType, requestedUrl) => {
 		modulePreloadCount: isHtml ? countMatches(body, /<link\b[^>]*rel\s*=\s*["'][^"']*modulepreload[^"']*["'][^>]*>/gi) : 0,
 		scriptCount: isHtml ? countMatches(body, /<script\b/gi) : 0,
 		stylesheetCount: isHtml ? countMatches(body, /<link\b[^>]*rel\s*=\s*["'][^"']*stylesheet[^"']*["'][^>]*>/gi) : 0,
-		wordCount: isMarkdown ? body.trim().split(/\s+/).filter(Boolean).length : null,
-		jsonValid
+		wordCount: isMarkdown ? body.trim().split(/\s+/).filter(Boolean).length : content?.wordCount ?? null,
+		jsonValid,
+		content: content === null ? null : {
+			h1: content.h1,
+			outline: content.outline,
+			findings: checkPageContent(content)
+		}
 	};
 };
 const readBoundedBody = async (response, maxBodyBytes) => {
@@ -960,6 +982,8 @@ const probeHttp = async (request, options) => {
 		let currentUrl = request.url;
 		let response = null;
 		for (let redirectCount = 0; redirectCount <= 10; redirectCount += 1) {
+			if (options.sameOrigin !== void 0 && currentUrl.origin !== options.sameOrigin) throw new Error(`Redirected off-origin to ${currentUrl.href}`);
+			if (options.allowUrl !== void 0 && !options.allowUrl(currentUrl)) throw new Error(`Redirected to robots-disallowed URL ${currentUrl.href}`);
 			response = await requestPinned(currentUrl, request, options);
 			headersAt = performance.now();
 			const location = response.headers.location;
@@ -977,7 +1001,7 @@ const probeHttp = async (request, options) => {
 		} : await readBoundedBody(response, options.maxBodyBytes);
 		const contentType = typeof response.headers["content-type"] === "string" ? response.headers["content-type"] : "";
 		const body = new TextDecoder().decode(bounded.bytes);
-		const anchors = options.captureAnchors === true && contentType.includes("text/html") ? extractAnchors(body, currentUrl.href) : void 0;
+		const anchors = options.captureAnchors === true && (contentType === "" || /text\/html|application\/xhtml\+xml/i.test(contentType)) ? extractAnchors(body, currentUrl.href) : void 0;
 		return {
 			kind: request.kind,
 			method: request.method,
@@ -994,7 +1018,8 @@ const probeHttp = async (request, options) => {
 			capturedBodySha256: bounded.bytes.byteLength === 0 ? null : createHash("sha256").update(bounded.bytes).digest("hex"),
 			bodyTruncated: bounded.truncated,
 			bodyExcerpt: bodyExcerpt(body, contentType),
-			document: body.length === 0 ? null : extractDocumentSignals(body, contentType, currentUrl),
+			...options.captureBody === true ? { body } : {},
+			document: body.length === 0 ? null : extractDocumentSignals(body, contentType, currentUrl, bounded.truncated),
 			anchors,
 			error: null
 		};
@@ -1015,6 +1040,7 @@ const probeHttp = async (request, options) => {
 			capturedBodySha256: null,
 			bodyTruncated: false,
 			bodyExcerpt: null,
+			...options.captureBody === true ? { body: void 0 } : {},
 			document: null,
 			error: errorMessage$2(error)
 		};