@pseolint/core 0.2.1 → 0.3.0

package/README.md

@@ -1,8 +1,8 @@
 # @pseolint/core
 
-> Programmatic SEO audit engine — 34 rules across 6 categories for SpamBrain risk detection.
+> Programmatic SEO audit engine for SpamBrain-risk detection across large template-generated sites.
 
-The core engine behind [pseolint](https://www.npmjs.com/package/pseolint). Use this package to integrate pSEO auditing into your own tools.
+The core engine behind [pseolint](https://www.npmjs.com/package/pseolint). Use this package to embed pSEO auditing into your own tools, CI pipelines, or SaaS products.
 
 ## Install
 
@@ -12,7 +12,7 @@ npm install @pseolint/core
 
 ## Usage
 
-```typescript
+```ts
 import { auditSource } from "@pseolint/core";
 
 const summary = await auditSource("./out");
@@ -20,34 +20,103 @@ console.log(`Score: ${summary.score}/100`);
 console.log(`Findings: ${summary.findings.length}`);
 ```
 
+`auditSource` accepts a local directory, a single HTML file, a page URL, or a sitemap URL.
+
 ## What It Checks
 
-34 rules across 6 categories:
+42 rules across 8 categories. Seven categories feed the composite score; `data/*` is a separate data-binding family.
 
-- **SpamBrain Risk** — near-duplicate detection (SimHash), entity-swap doorway pages, thin content, boilerplate ratio, template diversity
-- **Content Quality** — unique value per page, heading/meta uniqueness, author attribution, E-E-A-T signals
-- **Internal Linking** — orphan pages, dead ends, cluster connectivity, hub pages, link depth
-- **Technical SEO** — canonical consistency, sitemap completeness, soft 404s, redirect chains, hreflang, Open Graph
-- **Structured Data** — JSON-LD validation, required fields, schema consistency
-- **Cannibalization** — title overlap, keyword collision, URL pattern conflicts
+- **Spam / SpamBrain risk** (8) — near-duplicate (SimHash), entity-swap doorways, thin content, boilerplate ratio, template diversity, template coverage, publication velocity, doorway pattern
+- **Technical SEO** (8) — canonical consistency, canonical/noindex and robots/noindex conflicts, sitemap completeness, robots compliance, redirect chains, soft 404s, Open Graph, hreflang
+- **AEO / AI Overview citability** (8, v0.3.0) — `llms.txt` presence, AI-crawler access in robots.txt, freshness signals, FAQ coverage, answer-first opener, citable-fact density, non-replicable value, content modularity
+- **Content** (5) — unique value, heading / meta uniqueness, author attribution, E-E-A-T signals
+- **Internal linking** (5) — orphan pages, dead ends, cluster connectivity, hub pages, link depth
+- **Structured data** (3) — JSON-LD validity, required fields, cross-page schema consistency
+- **Cannibalization** (3) — title overlap, keyword collision, URL pattern conflicts
+- **Data binding** (2) — verify rendered pages expose values from a source dataset (missing or identical-across-pages bindings)
 
 ## API
 
 ### `auditSource(source, options?)`
 
-Audits a directory path or URL. Returns an `AuditSummary` with score, category scores, and enriched findings.
+Returns an `AuditSummary` with composite score, category scores, enriched findings, and optional cache / state / AI-triage metadata.
+
+Selected options (see `AuditOptions` in `types.ts` for the full surface):
+
+```ts
+await auditSource("https://example.com/sitemap.xml", {
+  concurrency: 5,
+  timeout: 30_000,
+  sampleSize: 200,
+  samplingStrategy: "stratified",        // or "random"
+  ignore: ["**/api/**"],
+  maxFetchBytes: 52_428_800,             // 50 MB hard cap per run
+  cache: { dir: ".pseolint/cache", ttlMs: 7 * 24 * 60 * 60 * 1000 },
+  state: { path: ".pseolint/state.json", since: true, exitOnRegression: true },
+  pageGroups: {
+    blog:     { match: "**/blog/**", rules: ["content/*", "spam/*"] },
+    products: { match: "**/p/**",    overrides: { "spam/thin-content": { thinContentMinWords: 200 } } },
+  },
+  dataSource: { records: [{ url: "/p/*", data: { price: "$19", stock: 12 } }] },
+  entityPatterns: [{ placeholder: "[CITY]", pattern: "\\b(NYC|LA|SF)\\b", flags: "gi" }],
+  ai: { enabled: true, provider: "anthropic", model: "claude-haiku-4-5-20251001", maxCostUsd: 0.1 },
+  telemetry: { enabled: true, path: ".pseolint/telemetry.jsonl" },
+  rules: {
+    nearDuplicateThreshold: 0.85,
+    thinContentMinWords: 300,
+    titleOverlapThreshold: 0.8,
+    // ...
+  },
+});
+```
 
 ### Formatters
 
-```typescript
+```ts
 import { formatConsole, formatJson, formatMarkdown, formatHtml } from "@pseolint/core";
 
-const output = formatConsole(summary);
+const out = formatConsole(summary);
 const json = formatJson(summary);
-const md = formatMarkdown(summary);
+const md   = formatMarkdown(summary);
 const html = formatHtml(summary);
 ```
 
+### AI triage
+
+When `ai.enabled` is set, findings are clustered into root-causes by an LLM. Providers are loaded lazily from optional peer deps — install only the one you need:
+
+```bash
+npm install @ai-sdk/anthropic   # or @ai-sdk/openai, @ai-sdk/google, @ai-sdk/mistral,
+                                #    @ai-sdk/groq, @ai-sdk/xai, @ai-sdk/cohere,
+                                #    ollama-ai-provider-v2
+```
+
+```ts
+import { triageFindings, createLanguageModel, estimateCostUsd } from "@pseolint/core";
+```
+
+Cost and daily-budget caps are enforced pre-flight; results are cached on disk by default.
+
+### Delta runs & regression gating
+
+Pass `state.since: true` to audit only URLs whose content hash changed since the last run, and `state.exitOnRegression: true` to flag a run where a new rule ID fires on any previously clean URL (`summary.hasRegression`).
+
+### Caching
+
+Setting `cache` enables an ETag/Last-Modified-aware disk cache for HTTP fetches. `summary.cacheStats` reports `{ hits, total, bytesSavedEstimate }`.
+
+### Page groups
+
+Classify pages by glob and apply different rule subsets or threshold overrides per group. Results are surfaced in `summary.groupScores` / `summary.groupPageCounts`.
+
+### Rendering
+
+For client-rendered pages, install `playwright-core` and pass `render: { browserWsEndpoint }` to connect to an existing browser endpoint.
+
+## Peer dependencies
+
+All AI providers and `playwright-core` are optional peers — you only install the ones you actually use.
+
 ## License
 
-MIT
+MIT

package/dist/ai/prompt.d.ts

@@ -1,5 +1,5 @@
 import type { RuleResult } from "../types.js";
-export declare const PROMPT_VERSION = "1.0.0";
+export declare const PROMPT_VERSION = "1.1.0";
 export declare const MAX_FINDINGS_IN_PROMPT = 200;
 export interface PromptRequest {
     system: string;

package/dist/ai/prompt.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"prompt.d.ts","sourceRoot":"","sources":["../../src/ai/prompt.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAY,MAAM,aAAa,CAAC;AAExD,eAAO,MAAM,cAAc,UAAU,CAAC;AACtC,eAAO,MAAM,sBAAsB,MAAM,CAAC;AAe1C,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;CACd;AAED,wBAAgB,eAAe,CAAC,CAAC,EAAE,UAAU,GAAG,MAAM,CAMrD;AAmBD,wBAAgB,kBAAkB,CAAC,QAAQ,EAAE,UAAU,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,aAAa,CA8B3F"}
+{"version":3,"file":"prompt.d.ts","sourceRoot":"","sources":["../../src/ai/prompt.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAY,MAAM,aAAa,CAAC;AAExD,eAAO,MAAM,cAAc,UAAU,CAAC;AACtC,eAAO,MAAM,sBAAsB,MAAM,CAAC;AAqB1C,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;CACd;AAED,wBAAgB,eAAe,CAAC,CAAC,EAAE,UAAU,GAAG,MAAM,CAMrD;AAqBD,wBAAgB,kBAAkB,CAAC,QAAQ,EAAE,UAAU,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,aAAa,CAqC3F"}

package/dist/ai/prompt.js

@@ -1,9 +1,15 @@
 import { createHash } from "node:crypto";
-export const PROMPT_VERSION = "1.0.0";
+export const PROMPT_VERSION = "1.1.0";
 export const MAX_FINDINGS_IN_PROMPT = 200;
 const SEVERITY_ORDER = { info: 0, warning: 1, error: 2, critical: 3 };
 const SYSTEM_PROMPT = `You are an SEO audit triage assistant. Given a list of pSEO linter findings, identify 1-5 underlying ROOT CAUSES driving the findings. Group findings by shared underlying problem, not by rule ID. Rank causes by likely SEO impact (highest first).
 
+Findings fall into two distinct threat families — treat them as separate root causes, not one combined cause:
+- SpamBrain penalty risk: spam/*, cannibal/*, content/*, data/*, tech/*, schema/*, links/* — these make Google penalize or demote the site.
+- AI Overview invisibility: aeo/* — these make pages uncitable in AI answer engines (ChatGPT, Perplexity, Gemini, AI Overviews). Sites not cited lose ~68% of traffic vs ~12% for cited sites.
+
+When both families are present, produce at least one root cause from each. Label AEO root causes clearly (e.g. "AI Overviews: ...") so the user can tell them apart from penalty risks.
+
 Rules:
 - Emit rootCauses FIRST, then narrative — do not reverse this order.
 - Keep each rootCause label <= 80 chars and phrase it as a problem statement.
@@ -31,11 +37,17 @@ export function buildPromptRequest(findings, pageCount) {
         pageUrl: f.pageUrl,
         group: f.group,
     }));
+    const countByCategory = {};
+    for (const f of findings) {
+        const cat = f.ruleId.split("/")[0];
+        countByCategory[cat] = (countByCategory[cat] ?? 0) + 1;
+    }
     const payload = {
         totalFindings: total,
         pageCount,
         truncated,
         findings: projected,
+        findingCountByCategory: countByCategory,
     };
     if (truncated) {
         const counts = {};

package/dist/ai/prompt.js.map

@@ -1 +1 @@
-{"version":3,"file":"prompt.js","sourceRoot":"","sources":["../../src/ai/prompt.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAGzC,MAAM,CAAC,MAAM,cAAc,GAAG,OAAO,CAAC;AACtC,MAAM,CAAC,MAAM,sBAAsB,GAAG,GAAG,CAAC;AAE1C,MAAM,cAAc,GAA6B,EAAE,IAAI,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,EAAE,CAAC;AAEhG,MAAM,aAAa,GAAG;;;;;;;;;sDASgC,CAAC;AAOvD,MAAM,UAAU,eAAe,CAAC,CAAa;IAC3C,MAAM,IAAI,GAAG,UAAU,CAAC,QAAQ,CAAC;SAC9B,MAAM,CAAC,CAAC,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,GAAG,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC;SAC3C,MAAM,CAAC,KAAK,CAAC;SACb,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACf,OAAO,GAAG,CAAC,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;AAC/B,CAAC;AAmBD,MAAM,UAAU,kBAAkB,CAAC,QAAsB,EAAE,SAAiB;IAC1E,MAAM,KAAK,GAAG,QAAQ,CAAC,MAAM,CAAC;IAC9B,MAAM,MAAM,GAAG,CAAC,GAAG,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,CAAC,CAAC,QAAQ,CAAC,GAAG,cAAc,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC;IACrG,MAAM,SAAS,GAAG,KAAK,GAAG,sBAAsB,CAAC;IACjD,MAAM,SAAS,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,sBAAsB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAqB,EAAE,CAAC,CAAC;QACvF,EAAE,EAAE,eAAe,CAAC,CAAC,CAAC;QACtB,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,QAAQ,EAAE,CAAC,CAAC,QAAQ;QACpB,OAAO,EAAE,CAAC,CAAC,OAAO;QAClB,OAAO,EAAE,CAAC,CAAC,OAAO;QAClB,KAAK,EAAE,CAAC,CAAC,KAAK;KACf,CAAC,CAAC,CAAC;IAEJ,MAAM,OAAO,GAAkB;QAC7B,aAAa,EAAE,KAAK;QACpB,SAAS;QACT,SAAS;QACT,QAAQ,EAAE,SAAS;KACpB,CAAC;IAEF,IAAI,SAAS,EAAE,CAAC;QACd,MAAM,MAAM,GAA2B,EAAE,CAAC;QAC1C,KAAK,MAAM,CAAC,IAAI,QAAQ;YAAE,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzE,OAAO,CAAC,kBAAkB,GAAG,MAAM,CAAC;IACtC,CAAC;IAED,OAAO;QACL,MAAM,EAAE,aAAa;QACrB,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC;KAC9B,CAAC;AACJ,CAAC"}
+{"version":3,"file":"prompt.js","sourceRoot":"","sources":["../../src/ai/prompt.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAGzC,MAAM,CAAC,MAAM,cAAc,GAAG,OAAO,CAAC;AACtC,MAAM,CAAC,MAAM,sBAAsB,GAAG,GAAG,CAAC;AAE1C,MAAM,cAAc,GAA6B,EAAE,IAAI,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,EAAE,CAAC;AAEhG,MAAM,aAAa,GAAG;;;;;;;;;;;;;;;sDAegC,CAAC;AAOvD,MAAM,UAAU,eAAe,CAAC,CAAa;IAC3C,MAAM,IAAI,GAAG,UAAU,CAAC,QAAQ,CAAC;SAC9B,MAAM,CAAC,CAAC,CAAC,CAAC,OAAO,IAAI,EAAE,CAAC,GAAG,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC;SAC3C,MAAM,CAAC,KAAK,CAAC;SACb,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACf,OAAO,GAAG,CAAC,CAAC,MAAM,IAAI,IAAI,EAAE,CAAC;AAC/B,CAAC;AAqBD,MAAM,UAAU,kBAAkB,CAAC,QAAsB,EAAE,SAAiB;IAC1E,MAAM,KAAK,GAAG,QAAQ,CAAC,MAAM,CAAC;IAC9B,MAAM,MAAM,GAAG,CAAC,GAAG,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,CAAC,CAAC,QAAQ,CAAC,GAAG,cAAc,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC;IACrG,MAAM,SAAS,GAAG,KAAK,GAAG,sBAAsB,CAAC;IACjD,MAAM,SAAS,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,sBAAsB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAqB,EAAE,CAAC,CAAC;QACvF,EAAE,EAAE,eAAe,CAAC,CAAC,CAAC;QACtB,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,QAAQ,EAAE,CAAC,CAAC,QAAQ;QACpB,OAAO,EAAE,CAAC,CAAC,OAAO;QAClB,OAAO,EAAE,CAAC,CAAC,OAAO;QAClB,KAAK,EAAE,CAAC,CAAC,KAAK;KACf,CAAC,CAAC,CAAC;IAEJ,MAAM,eAAe,GAA2B,EAAE,CAAC;IACnD,KAAK,MAAM,CAAC,IAAI,QAAQ,EAAE,CAAC;QACzB,MAAM,GAAG,GAAG,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QACnC,eAAe,CAAC,GAAG,CAAC,GAAG,CAAC,eAAe,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;IACzD,CAAC;IAED,MAAM,OAAO,GAAkB;QAC7B,aAAa,EAAE,KAAK;QACpB,SAAS;QACT,SAAS;QACT,QAAQ,EAAE,SAAS;QACnB,sBAAsB,EAAE,eAAe;KACxC,CAAC;IAEF,IAAI,SAAS,EAAE,CAAC;QACd,MAAM,MAAM,GAA2B,EAAE,CAAC;QAC1C,KAAK,MAAM,CAAC,IAAI,QAAQ;YAAE,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzE,OAAO,CAAC,kBAAkB,GAAG,MAAM,CAAC;IACtC,CAAC;IAED,OAAO;QACL,MAAM,EAAE,aAAa;QACrB,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC;KAC9B,CAAC;AACJ,CAAC"}

package/dist/auditor.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"auditor.d.ts","sourceRoot":"","sources":["../src/auditor.ts"],"names":[],"mappings":"AAoDA,OAAO,KAAK,EAAE,YAAY,EAAE,YAAY,EAAwG,MAAM,YAAY,CAAC;AAktBnK,wBAAsB,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC,CAid/F"}
+{"version":3,"file":"auditor.d.ts","sourceRoot":"","sources":["../src/auditor.ts"],"names":[],"mappings":"AA6DA,OAAO,KAAK,EAAE,YAAY,EAAE,YAAY,EAAwG,MAAM,YAAY,CAAC;AAu0BnK,wBAAsB,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC,CAgf/F"}