pagegraph 0.8.4 → 0.9.0

package/README.md

@@ -43,6 +43,17 @@ OpenCode SDK and its private Effect runtime only when invoked. `vite` stays a pe
 load your app through Vite at runtime — and `lighthouse` is only needed by `pagegraph audit`.
 Importing `pagegraph/audit` programmatically still needs `effect`.
 
+### Agent skill
+
+The Pagegraph CLI serves its bundled TanStack Start integration skill:
+
+```bash
+pagegraph skills list
+pagegraph skills get core
+```
+
+The skill is maintained at [skill-data/core/SKILL.md](skill-data/core/SKILL.md) and compiled into the CLI, so the instructions match the installed Pagegraph version. `skills get core` prints Markdown to stdout; `--json` returns a structured payload.
+
 ## Quick start
 
 **1. Bind your site identity once.** Route files never see an origin or a brand name.
@@ -423,7 +434,12 @@ private-IP-blocked HTTP path as `pagegraph audit` — and reports what a crawler
 actually receives: the real homepage depth, the pages with no incoming internal
 edge (rendered orphans), and, when the app has a `seo.config.ts`, the
 declared-vs-rendered link diff. It is bounded with `--limit` and needs no
-framework or config:
+framework or config. Discovery reads same-origin `robots.txt` and bounded
+sitemaps before following page anchors, so a sitemap-only page can appear as an
+orphan. The JSON `crawl` block records skipped URLs, discovery failures, and
+sitemap truncation. Link-followed non-HTML alternates are recorded in `nonHtml`
+without counting as HTML page failures; a sitemap-listed or declared page served
+as non-HTML still blocks a coverage assertion:
 
 ```bash
 pagegraph links verify https://example.com
@@ -456,6 +472,11 @@ and the provenance a gate needs to trust it:
     "bodyTruncated": false,
     "truncatedPages": [],
     "failures": [],
+    "nonHtml": [{ "url": "https://example.com/auth.md", "finalUrl": "https://example.com/auth.md", "contentType": "text/markdown" }],
+    "discoveryFailures": [],
+    "skipped": [],
+    "sitemapTruncated": false,
+    "sitemapUsed": true,
   },
   "nodes": ["/", "/pricing"],
   "edges": [{ "from": "/", "to": "/pricing", "region": "body" }],
@@ -481,6 +502,7 @@ pagegraph links verify --rendered .seo/rendered.json --assert-coverage --json |
 The command exits non-zero when any rule is unmet, and **refuses to assert** (also
 non-zero) rather than reporting a false pass when the crawl outran `--limit`, a
 page failed to fetch (its anchors are missing), a body hit `--max-body-bytes`, the
+robots or sitemap discovery is incomplete, the
 config declares no `coverage`, or the artifact's origin differs from
 `seo.config.ts`. Under `--assert-coverage` the report gains a `coverage` block
 (`{ rules, ok, violations }`); the default `--json` summary is unchanged.
@@ -491,9 +513,11 @@ config declares no `coverage`, or the artifact's origin differs from
 `(source, destination)` pairs that are plausible contextual links and not already
 connected. Pages are grouped into clusters — a shared top-level section, or the
 same `kind` for root-level pages — and only sitemap-eligible pages are proposed.
-Pairs already declared as a `related` edge are excluded, and `--rendered` accepts
-a JSON dump of already-served anchors (`[{ from, to }]` or `{ edges: [...] }`) to
-exclude those too.
+Each direction is evaluated separately. A declared `related` edge or served
+body-region anchor excludes only its own direction. Nav, header, and footer
+anchors do not exclude contextual suggestions. `--rendered` accepts a JSON dump
+of already-served anchors (`[{ from, to, region }]` or `{ edges: [...] }`).
+Legacy edges without a `region` are treated as body links.
 
 The output is a deterministic, reviewable plan: human text by default and
 versioned JSON with `--json`. Nothing is applied; `--limit` and `--cluster`
@@ -507,6 +531,43 @@ pagegraph links candidates --rendered rendered.json --json | jq
 pagegraph improve links --page "/docs/**"
 ```
 
+For content-backed placements, opt in to a bounded same-origin probe. The configured
+`origin` must match `--site`. Each schema-version-2 result carries a sentence
+from served main content, an exact anchor phrase inside it, a target sentence,
+and score components. Unreadable and over-limit pages are reported as skipped.
+Code samples and tabular comparisons are excluded. An anchor is proposed only
+when its distinct content words occur together in a target sentence, so review
+the placement and its context before editing. Repeated passages and clause
+fragments are excluded; named destinations require a destination-specific anchor.
+The home and about pages are omitted as destinations because site navigation already leads there.
+The pair-only command above remains offline and emits schema version 1.
+The page budget selects low-inbound candidate pairs first. Use repeatable
+`--target /exact/path` to put a known weak destination within the budget. Add
+`--source /exact/path` when you have a likely source and want to guarantee the
+pair is probed even on a large site; raise `--page-limit` for more pairs.
+For a site with many legal, comparison, or other low-inbound pages, use
+`--cluster blog` (or another section) to spend the same page budget on an
+editorial cluster. A zero-result broad probe means no placement passed the
+evidence rules within its selected pages; it does not mean the whole site has
+no possible links.
+
+```bash
+pagegraph links verify https://example.com --json
+pagegraph links candidates --site https://example.com --page-limit 25 --json > /tmp/pagegraph-suggestions.json
+pagegraph links candidates --site https://example.com --cluster blog --page-limit 25 --json
+pagegraph links candidates --site https://example.com --source /blog/guide --target /blog/weak --page-limit 25 --json > /tmp/pagegraph-suggestions.json
+pagegraph improve links --suggestions /tmp/pagegraph-suggestions.json --dry-run
+pagegraph improve links --suggestions /tmp/pagegraph-suggestions.json
+```
+
+`improve links` validates the artifact against the configured origin and graph,
+then asks Jev to judge the suggested placements. Only accepted `add` or `update`
+decisions can open an edit turn; skipped and review-band links remain in the run
+artifact without authorizing an edit. Before an accepted edit, Pagegraph checks
+that the source sentence and target passage still occur in served copy. Regenerate
+the artifact if either page changed. For a local site, pass `--allow-private` to
+both `links candidates --site` and `improve links --suggestions`.
+
 ### Contextual-link coverage
 
 Declare a contextual-link coverage policy in `seo.config.ts` (a `coverage` array

package/dist/audit.d.ts

@@ -103,6 +103,12 @@ interface ProbeOptions {
    * probe field that grows with document size; the audit report leaves it unset.
    */
   readonly captureAnchors?: boolean | undefined;
+  /** Capture bounded text for robots and sitemap discovery. Never enabled for audit reports. */
+  readonly captureBody?: boolean | undefined;
+  /** Refuse cross-origin redirects during a same-origin crawl. */
+  readonly sameOrigin?: string | undefined;
+  /** Optional per-redirect crawl policy, checked before each network request. */
+  readonly allowUrl?: ((url: URL) => boolean) | undefined;
 }
 interface AuditOptions {
   readonly concurrency?: number;

package/dist/audit.js

@@ -1,4 +1,4 @@
-import { a as extractAnchors } from "./links-BFJc-pJ9.js";
+import { a as extractAnchors } from "./links-DAa8KVzg.js";
 import { mkdir, mkdtemp, readFile, rename, rm, writeFile } from "node:fs/promises";
 import { join } from "node:path";
 import * as Context from "effect/Context";
@@ -865,7 +865,7 @@ const canonicalUrl = (html) => {
 };
 const countMatches = (value, pattern) => [...value.matchAll(pattern)].length;
 const extractDocumentSignals = (body, contentType, requestedUrl) => {
-	const isHtml = contentType.includes("text/html");
+	const isHtml = contentType.includes("text/html") || contentType.includes("application/xhtml+xml");
 	const isMarkdown = contentType.includes("text/markdown");
 	const looksJson = contentType.includes("json") || requestedUrl.pathname.endsWith(".json");
 	const titleMatch = isHtml ? /<title\b[^>]*>([\s\S]*?)<\/title>/i.exec(body) : null;
@@ -960,6 +960,8 @@ const probeHttp = async (request, options) => {
 		let currentUrl = request.url;
 		let response = null;
 		for (let redirectCount = 0; redirectCount <= 10; redirectCount += 1) {
+			if (options.sameOrigin !== void 0 && currentUrl.origin !== options.sameOrigin) throw new Error(`Redirected off-origin to ${currentUrl.href}`);
+			if (options.allowUrl !== void 0 && !options.allowUrl(currentUrl)) throw new Error(`Redirected to robots-disallowed URL ${currentUrl.href}`);
 			response = await requestPinned(currentUrl, request, options);
 			headersAt = performance.now();
 			const location = response.headers.location;
@@ -977,7 +979,7 @@ const probeHttp = async (request, options) => {
 		} : await readBoundedBody(response, options.maxBodyBytes);
 		const contentType = typeof response.headers["content-type"] === "string" ? response.headers["content-type"] : "";
 		const body = new TextDecoder().decode(bounded.bytes);
-		const anchors = options.captureAnchors === true && contentType.includes("text/html") ? extractAnchors(body, currentUrl.href) : void 0;
+		const anchors = options.captureAnchors === true && (contentType === "" || /text\/html|application\/xhtml\+xml/i.test(contentType)) ? extractAnchors(body, currentUrl.href) : void 0;
 		return {
 			kind: request.kind,
 			method: request.method,
@@ -994,6 +996,7 @@ const probeHttp = async (request, options) => {
 			capturedBodySha256: bounded.bytes.byteLength === 0 ? null : createHash("sha256").update(bounded.bytes).digest("hex"),
 			bodyTruncated: bounded.truncated,
 			bodyExcerpt: bodyExcerpt(body, contentType),
+			...options.captureBody === true ? { body } : {},
 			document: body.length === 0 ? null : extractDocumentSignals(body, contentType, currentUrl),
 			anchors,
 			error: null
@@ -1015,6 +1018,7 @@ const probeHttp = async (request, options) => {
 			capturedBodySha256: null,
 			bodyTruncated: false,
 			bodyExcerpt: null,
+			...options.captureBody === true ? { body: void 0 } : {},
 			document: null,
 			error: errorMessage$2(error)
 		};