@pseolint/core 0.2.0 → 0.2.2

package/README.md

@@ -1,8 +1,8 @@
 # @pseolint/core
 
-> Programmatic SEO audit engine — 34 rules across 6 categories for SpamBrain risk detection.
+> Programmatic SEO audit engine for SpamBrain-risk detection across large template-generated sites.
 
-The core engine behind [pseolint](https://www.npmjs.com/package/pseolint). Use this package to integrate pSEO auditing into your own tools.
+The core engine behind [pseolint](https://www.npmjs.com/package/pseolint). Use this package to embed pSEO auditing into your own tools, CI pipelines, or SaaS products.
 
 ## Install
 
@@ -12,7 +12,7 @@ npm install @pseolint/core
 
 ## Usage
 
-```typescript
+```ts
 import { auditSource } from "@pseolint/core";
 
 const summary = await auditSource("./out");
@@ -20,34 +20,102 @@ console.log(`Score: ${summary.score}/100`);
 console.log(`Findings: ${summary.findings.length}`);
 ```
 
+`auditSource` accepts a local directory, a single HTML file, a page URL, or a sitemap URL.
+
 ## What It Checks
 
-34 rules across 6 categories:
+37+ rules across 7 categories:
 
-- **SpamBrain Risk** — near-duplicate detection (SimHash), entity-swap doorway pages, thin content, boilerplate ratio, template diversity
-- **Content Quality** — unique value per page, heading/meta uniqueness, author attribution, E-E-A-T signals
-- **Internal Linking** — orphan pages, dead ends, cluster connectivity, hub pages, link depth
-- **Technical SEO** — canonical consistency, sitemap completeness, soft 404s, redirect chains, hreflang, Open Graph
-- **Structured Data** — JSON-LD validation, required fields, schema consistency
+- **Spam / SpamBrain risk** — near-duplicate (SimHash), entity-swap doorways, thin content, boilerplate ratio, template diversity, template coverage, publication velocity, doorway pattern
+- **Content** — unique value, heading / meta uniqueness, author attribution, E-E-A-T signals
+- **Internal linking** — orphan pages, dead ends, cluster connectivity, hub pages, link depth, unreachable-from-root
+- **Technical SEO** — canonical consistency, canonical/noindex and robots/noindex conflicts, sitemap completeness, robots compliance, redirect chains, soft 404s, Open Graph, hreflang
+- **Structured data** — JSON-LD validity, required fields, cross-page schema consistency
 - **Cannibalization** — title overlap, keyword collision, URL pattern conflicts
+- **Data binding** — verify rendered pages expose values from a source dataset (catches missing or identical-across-pages bindings)
 
 ## API
 
 ### `auditSource(source, options?)`
 
-Audits a directory path or URL. Returns an `AuditSummary` with score, category scores, and enriched findings.
+Returns an `AuditSummary` with composite score, category scores, enriched findings, and optional cache / state / AI-triage metadata.
+
+Selected options (see `AuditOptions` in `types.ts` for the full surface):
+
+```ts
+await auditSource("https://example.com/sitemap.xml", {
+  concurrency: 5,
+  timeout: 30_000,
+  sampleSize: 200,
+  samplingStrategy: "stratified",        // or "random"
+  ignore: ["**/api/**"],
+  maxFetchBytes: 52_428_800,             // 50 MB hard cap per run
+  cache: { dir: ".pseolint/cache", ttlMs: 7 * 24 * 60 * 60 * 1000 },
+  state: { path: ".pseolint/state.json", since: true, exitOnRegression: true },
+  pageGroups: {
+    blog:     { match: "**/blog/**", rules: ["content/*", "spam/*"] },
+    products: { match: "**/p/**",    overrides: { "spam/thin-content": { thinContentMinWords: 200 } } },
+  },
+  dataSource: { records: [{ url: "/p/*", data: { price: "$19", stock: 12 } }] },
+  entityPatterns: [{ placeholder: "[CITY]", pattern: "\\b(NYC|LA|SF)\\b", flags: "gi" }],
+  ai: { enabled: true, provider: "anthropic", model: "claude-haiku-4-5-20251001", maxCostUsd: 0.1 },
+  telemetry: { enabled: true, path: ".pseolint/telemetry.jsonl" },
+  rules: {
+    nearDuplicateThreshold: 0.85,
+    thinContentMinWords: 300,
+    titleOverlapThreshold: 0.8,
+    // ...
+  },
+});
+```
 
 ### Formatters
 
-```typescript
+```ts
 import { formatConsole, formatJson, formatMarkdown, formatHtml } from "@pseolint/core";
 
-const output = formatConsole(summary);
+const out = formatConsole(summary);
 const json = formatJson(summary);
-const md = formatMarkdown(summary);
+const md   = formatMarkdown(summary);
 const html = formatHtml(summary);
 ```
 
+### AI triage
+
+When `ai.enabled` is set, findings are clustered into root-causes by an LLM. Providers are loaded lazily from optional peer deps — install only the one you need:
+
+```bash
+npm install @ai-sdk/anthropic   # or @ai-sdk/openai, @ai-sdk/google, @ai-sdk/mistral,
+                                #    @ai-sdk/groq, @ai-sdk/xai, @ai-sdk/cohere,
+                                #    ollama-ai-provider-v2
+```
+
+```ts
+import { triageFindings, createLanguageModel, estimateCostUsd } from "@pseolint/core";
+```
+
+Cost and daily-budget caps are enforced pre-flight; results are cached on disk by default.
+
+### Delta runs & regression gating
+
+Pass `state.since: true` to audit only URLs whose content hash changed since the last run, and `state.exitOnRegression: true` to flag a run where a new rule ID fires on any previously clean URL (`summary.hasRegression`).
+
+### Caching
+
+Setting `cache` enables an ETag/Last-Modified-aware disk cache for HTTP fetches. `summary.cacheStats` reports `{ hits, total, bytesSavedEstimate }`.
+
+### Page groups
+
+Classify pages by glob and apply different rule subsets or threshold overrides per group. Results are surfaced in `summary.groupScores` / `summary.groupPageCounts`.
+
+### Rendering
+
+For client-rendered pages, install `playwright-core` and pass `render: { browserWsEndpoint }` to connect to an existing browser endpoint.
+
+## Peer dependencies
+
+All AI providers and `playwright-core` are optional peers — you only install the ones you actually use.
+
 ## License
 
 MIT

package/dist/ai/triage.d.ts

@@ -8,6 +8,12 @@ export interface TriageOptions {
     modelId: string;
     maxInputTokens?: number;
     maxOutputTokens?: number;
+    /** Hard cap on estimated USD for this call. Refuse pre-flight if exceeded. */
+    maxCostUsd?: number;
+    /** USD already spent on successful triages today (used against dailyBudgetUsd). */
+    spentTodayUsd?: number;
+    /** Daily budget ceiling. Refuse pre-flight if spentTodayUsd + this call's estimate > budget. */
+    dailyBudgetUsd?: number;
     cache?: {
         dir: string;
         ttlMs: number;

package/dist/ai/triage.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"triage.d.ts","sourceRoot":"","sources":["../../src/ai/triage.ts"],"names":[],"mappings":"AACA,OAAO,EAAkB,KAAK,aAAa,EAAE,MAAM,IAAI,CAAC;AAExD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AA2B/C,MAAM,WAAW,aAAa;IAC5B,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,aAAa,CAAC;IACrB,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,KAAK,CAAC,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,GAAG,KAAK,CAAC;IAC/C,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,WAAW,aAAa;IAC5B,MAAM,CAAC,EAAE,YAAY,CAAC;IACtB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAeD,wBAAsB,cAAc,CAClC,QAAQ,EAAE,UAAU,EAAE,EACtB,SAAS,EAAE,MAAM,EACjB,OAAO,EAAE,aAAa,GACrB,OAAO,CAAC,aAAa,CAAC,CA+FxB"}
+{"version":3,"file":"triage.d.ts","sourceRoot":"","sources":["../../src/ai/triage.ts"],"names":[],"mappings":"AACA,OAAO,EAAkB,KAAK,aAAa,EAAE,MAAM,IAAI,CAAC;AAExD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AA2B/C,MAAM,WAAW,aAAa;IAC5B,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,aAAa,CAAC;IACrB,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,8EAA8E;IAC9E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,mFAAmF;IACnF,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,gGAAgG;IAChG,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,KAAK,CAAC,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,GAAG,KAAK,CAAC;IAC/C,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,WAAW,aAAa;IAC5B,MAAM,CAAC,EAAE,YAAY,CAAC;IACtB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAeD,wBAAsB,cAAc,CAClC,QAAQ,EAAE,UAAU,EAAE,EACtB,SAAS,EAAE,MAAM,EACjB,OAAO,EAAE,aAAa,GACrB,OAAO,CAAC,aAAa,CAAC,CAuHxB"}

package/dist/ai/triage.js

@@ -16,7 +16,7 @@ const rootCauseSchema = z.object({
 });
 const triagePayloadSchema = z.object({
     rootCauses: z.array(rootCauseSchema).min(1).max(5).describe("1 to 5 root causes, ranked by SEO impact (highest first)."),
-    narrative: z.string().min(1).describe("2-3 sentence overall summary of the audit state."),
+    narrative: z.string().optional().describe("1-2 sentence overall summary. Optional — if output is tight, prioritize rootCauses and omit this."),
 });
 const DEFAULT_MAX_INPUT_TOKENS = 60_000;
 const DEFAULT_MAX_OUTPUT_TOKENS = 4_000;
@@ -39,6 +39,26 @@ export async function triageFindings(findings, pageCount, options) {
     if (estimate > maxInputTokens) {
         return { skipReason: `pre-flight token estimate ${estimate} exceeds cap ${maxInputTokens}` };
     }
+    // Pre-flight cost gate: pessimistic upper bound using (estimated input + max output).
+    const maxOutputTokens = options.maxOutputTokens ?? DEFAULT_MAX_OUTPUT_TOKENS;
+    const preflightCostUsd = estimateCostUsd(options.providerId, options.modelId, {
+        input: estimate,
+        output: maxOutputTokens,
+    });
+    if (options.maxCostUsd !== undefined &&
+        preflightCostUsd !== undefined &&
+        preflightCostUsd > options.maxCostUsd) {
+        return {
+            skipReason: `pre-flight cost $${preflightCostUsd.toFixed(3)} exceeds cap $${options.maxCostUsd.toFixed(3)}`,
+        };
+    }
+    if (options.dailyBudgetUsd !== undefined &&
+        preflightCostUsd !== undefined &&
+        (options.spentTodayUsd ?? 0) + preflightCostUsd > options.dailyBudgetUsd) {
+        return {
+            skipReason: `daily budget exhausted: $${(options.spentTodayUsd ?? 0).toFixed(3)} + $${preflightCostUsd.toFixed(3)} > $${options.dailyBudgetUsd.toFixed(3)}`,
+        };
+    }
     const validIds = new Set();
     for (const f of findings)
         validIds.add(assignFindingId(f));
@@ -58,6 +78,9 @@ export async function triageFindings(findings, pageCount, options) {
             // cache read errors are non-fatal — fall through to fresh call
         }
     }
+    // One-line pre-call estimate so users see what's about to be spent.
+    const costLabel = preflightCostUsd !== undefined ? `~$${preflightCostUsd.toFixed(3)}` : "cost unknown";
+    console.error(`[ai-triage] calling ${options.providerId}:${options.modelId} — ~${estimate.toLocaleString()} input / ≤${maxOutputTokens.toLocaleString()} output tokens, ${costLabel}`);
     let generated;
     try {
         generated = await generateObject({
@@ -65,7 +88,7 @@ export async function triageFindings(findings, pageCount, options) {
             system: req.system,
             prompt: req.user,
             schema: triagePayloadSchema,
-            maxOutputTokens: options.maxOutputTokens ?? DEFAULT_MAX_OUTPUT_TOKENS,
+            maxOutputTokens,
             abortSignal: options.signal,
         });
     }
@@ -74,17 +97,7 @@ export async function triageFindings(findings, pageCount, options) {
         if (err?.name === "AbortError" || (err?.message && /abort/i.test(err.message))) {
             return { skipReason: "aborted during LLM call" };
         }
-        let detail = err?.message ?? String(e);
-        if (err && typeof err === "object" && "text" in err && typeof err.text === "string") {
-            detail += ` | raw=${err.text.slice(0, 400)}`;
-        }
-        if (err?.cause) {
-            try {
-                detail += ` | cause=${JSON.stringify(err.cause).slice(0, 400)}`;
-            }
-            catch { /* ignore */ }
-        }
-        return { skipReason: `LLM call failed: ${detail}` };
+        return { skipReason: `LLM call failed: ${err?.message ?? String(e)}` };
     }
     // Validate relatedFindingIds reference known findings (semantic check
     // beyond the structural schema enforcement performed by generateObject).

package/dist/ai/triage.js.map

@@ -1 +1 @@
-{"version":3,"file":"triage.js","sourceRoot":"","sources":["../../src/ai/triage.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,cAAc,EAAsB,MAAM,IAAI,CAAC;AACxD,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAGxB,OAAO,EACL,cAAc,EACd,sBAAsB,EACtB,eAAe,EACf,kBAAkB,GACnB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,eAAe,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAC/E,OAAO,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAE5C,MAAM,UAAU,GAAG,CAAC,MAAM,EAAE,SAAS,EAAE,OAAO,EAAE,UAAU,CAAU,CAAC;AAErE,MAAM,eAAe,GAAG,CAAC,CAAC,MAAM,CAAC;IAC/B,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,CAAC,uCAAuC,CAAC;IAClF,aAAa,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,wDAAwD,CAAC;IAChH,eAAe,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,2EAA2E,CAAC;IAC1H,QAAQ,EAAE,CAAC,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,QAAQ,CAAC,yEAAyE,CAAC;IAChH,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,uDAAuD,CAAC;IACnG,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,oDAAoD,CAAC;IAC3F,iBAAiB,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,wHAAwH,CAAC;CAC1K,CAAC,CAAC;AAEH,MAAM,mBAAmB,GAAG,CAAC,CAAC,MAAM,CAAC;IACnC,UAAU,EAAE,CAAC,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,2DAA2D,CAAC;IACxH,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,kDAAkD,CAAC;CAC1F,CAAC,CAAC;AAkBH,MAAM,wBAAwB,GAAG,MAAM,CAAC;AACxC,MAAM,yBAAyB,GAAG,KAAK,CAAC;AAExC,SAAS,YAAY,CAAC,QAAsB;IAC1C,MAAM,GAAG,GAAG,QAAQ,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,IAAI,EAAE,CAAC;IACjD,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AAClE,CAAC;AAED,kFAAkF;AAClF,SAAS,cAAc,CAAC,IAAY;IAClC,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;AACpC,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,QAAsB,EACtB,SAAiB,EACjB,OAAsB;IAEtB,IAAI,OAAO,CAAC,MAAM,EAAE,OAAO,EAAE,CAAC;QAC5B,OAAO,EAAE,UAAU,EAAE,+BAA+B,EAAE,CAAC;IACzD,CAAC;IAED,MAAM,GAAG,GAAG,kBAAkB,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;IACpD,MAAM,cAAc,GAAG,QAAQ,CAAC,MAAM,GAAG,sBAAsB,CAAC;IAEhE,MAAM,cAAc,GAAG,OAAO,CAAC,cAAc,IAAI,wBAAwB,CAAC;IAC1E,MAAM,QAAQ,GAAG,cAAc,CAAC,GAAG,CAAC,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC,CAAC;IACvD,IAAI,QAAQ,GAAG,cAAc,EAAE,CAAC;QAC9B,OAAO,EAAE,UAAU,EAAE,6BAA6B,QAAQ,gBAAgB,cAAc,EAAE,EAAE,CAAC;IAC/F,CAAC;IAED,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAU,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,QAAQ;QAAE,QAAQ,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;IAE3D,MAAM,QAAQ,GAAG,cAAc,CAAC;QAC9B,YAAY,EAAE,YAAY,CAAC,QAAQ,CAAC;QACpC,KAAK,EAAE,OAAO,CAAC,OAAO;QACtB,aAAa,EAAE,cAAc;KAC9B,CAAC,CAAC;IAEH,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,EAAE,QAAQ,EAAE,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;YACvF,IAAI,MAAM,EAAE,CAAC;gBACX,OAAO,EAAE,MAAM,EAAE,EAAE,GAAG,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,CAAC;YACnD,CAAC;QACH,CAAC;QAAC,MAAM,CAAC;YACP,+DAA+D;QACjE,CAAC;IACH,CAAC;IAED,IAAI,SAAS,CAAC;IACd,IAAI,CAAC;QACH,SAAS,GAAG,MAAM,cAAc,CAAC;YAC/B,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,MAAM,EAAE,GAAG,CAAC,MAAM;YAClB,MAAM,EAAE,GAAG,CAAC,IAAI;YAChB,MAAM,EAAE,mBAAmB;YAC3B,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,yBAAyB;YACrE,WAAW,EAAE,OAAO,CAAC,MAAM;SAC5B,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,MAAM,GAAG,GAAG,CAAwE,CAAC;QACrF,IAAI,GAAG,EAAE,IAAI,KAAK,YAAY,IAAI,CAAC,GAAG,EAAE,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;YAC/E,OAAO,EAAE,UAAU,EAAE,yBAAyB,EAAE,CAAC;QACnD,CAAC;QACD,IAAI,MAAM,GAAG,GAAG,EAAE,OAAO,IAAI,MAAM,CAAC,CAAC,CAAC,CAAC;QACvC,IAAI,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,MAAM,IAAI,GAAG,IAAI,OAAO,GAAG,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;YACpF,MAAM,IAAI,UAAU,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC;QAC/C,CAAC;QACD,IAAI,GAAG,EAAE,KAAK,EAAE,CAAC;YACf,IAAI,CAAC;gBAAC,MAAM,IAAI,YAAY,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC;YAAC,CAAC;YAAC,MAAM,CAAC,CAAC,YAAY,CAAC,CAAC;QACjG,CAAC;QACD,OAAO,EAAE,UAAU,EAAE,oBAAoB,MAAM,EAAE,EAAE,CAAC;IACtD,CAAC;IAED,sEAAsE;IACtE,yEAAyE;IACzE,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,SAAS,CAAC,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE,EAAE,CAAC;QAC3D,KAAK,MAAM,EAAE,IAAI,CAAC,CAAC,iBAAiB,EAAE,CAAC;YACrC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;gBACtB,OAAO,EAAE,UAAU,EAAE,iDAAiD,CAAC,MAAM,EAAE,EAAE,EAAE,CAAC;YACtF,CAAC;QACH,CAAC;IACH,CAAC;IAED,MAAM,KAAK,GAAG;QACZ,KAAK,EAAE,SAAS,CAAC,KAAK,CAAC,WAAW,IAAI,CAAC;QACvC,MAAM,EAAE,SAAS,CAAC,KAAK,CAAC,YAAY,IAAI,CAAC;KAC1C,CAAC;IAEF,MAAM,MAAM,GAAiB;QAC3B,UAAU,EAAE,SAAS,CAAC,MAAM,CAAC,UAAU;QACvC,SAAS,EAAE,SAAS,CAAC,MAAM,CAAC,SAAS;QACrC,SAAS,EAAE,OAAO,CAAC,OAAO;QAC1B,UAAU,EAAE,OAAO,CAAC,UAAU;QAC9B,UAAU,EAAE,KAAK;QACjB,gBAAgB,EAAE,eAAe,CAAC,OAAO,CAAC,UAAU,EAAE,OAAO,CAAC,OAAO,EAAE,KAAK,CAAC;QAC7E,QAAQ,EAAE,KAAK;QACf,aAAa,EAAE,cAAc;QAC7B,cAAc;KACf,CAAC;IAEF,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC;YACH,MAAM,gBAAgB,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC;QAC9D,CAAC;QAAC,MAAM,CAAC;YACP,mCAAmC;QACrC,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,CAAC;AACpB,CAAC"}
+{"version":3,"file":"triage.js","sourceRoot":"","sources":["../../src/ai/triage.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,cAAc,EAAsB,MAAM,IAAI,CAAC;AACxD,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAGxB,OAAO,EACL,cAAc,EACd,sBAAsB,EACtB,eAAe,EACf,kBAAkB,GACnB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,eAAe,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAC/E,OAAO,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAE5C,MAAM,UAAU,GAAG,CAAC,MAAM,EAAE,SAAS,EAAE,OAAO,EAAE,UAAU,CAAU,CAAC;AAErE,MAAM,eAAe,GAAG,CAAC,CAAC,MAAM,CAAC;IAC/B,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,CAAC,uCAAuC,CAAC;IAClF,aAAa,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,wDAAwD,CAAC;IAChH,eAAe,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,2EAA2E,CAAC;IAC1H,QAAQ,EAAE,CAAC,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,QAAQ,CAAC,yEAAyE,CAAC;IAChH,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,uDAAuD,CAAC;IACnG,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,oDAAoD,CAAC;IAC3F,iBAAiB,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,QAAQ,CAAC,wHAAwH,CAAC;CAC1K,CAAC,CAAC;AAEH,MAAM,mBAAmB,GAAG,CAAC,CAAC,MAAM,CAAC;IACnC,UAAU,EAAE,CAAC,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,2DAA2D,CAAC;IACxH,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mGAAmG,CAAC;CAC/I,CAAC,CAAC;AAwBH,MAAM,wBAAwB,GAAG,MAAM,CAAC;AACxC,MAAM,yBAAyB,GAAG,KAAK,CAAC;AAExC,SAAS,YAAY,CAAC,QAAsB;IAC1C,MAAM,GAAG,GAAG,QAAQ,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,IAAI,EAAE,CAAC;IACjD,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AAClE,CAAC;AAED,kFAAkF;AAClF,SAAS,cAAc,CAAC,IAAY;IAClC,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;AACpC,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,QAAsB,EACtB,SAAiB,EACjB,OAAsB;IAEtB,IAAI,OAAO,CAAC,MAAM,EAAE,OAAO,EAAE,CAAC;QAC5B,OAAO,EAAE,UAAU,EAAE,+BAA+B,EAAE,CAAC;IACzD,CAAC;IAED,MAAM,GAAG,GAAG,kBAAkB,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;IACpD,MAAM,cAAc,GAAG,QAAQ,CAAC,MAAM,GAAG,sBAAsB,CAAC;IAEhE,MAAM,cAAc,GAAG,OAAO,CAAC,cAAc,IAAI,wBAAwB,CAAC;IAC1E,MAAM,QAAQ,GAAG,cAAc,CAAC,GAAG,CAAC,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC,CAAC;IACvD,IAAI,QAAQ,GAAG,cAAc,EAAE,CAAC;QAC9B,OAAO,EAAE,UAAU,EAAE,6BAA6B,QAAQ,gBAAgB,cAAc,EAAE,EAAE,CAAC;IAC/F,CAAC;IAED,sFAAsF;IACtF,MAAM,eAAe,GAAG,OAAO,CAAC,eAAe,IAAI,yBAAyB,CAAC;IAC7E,MAAM,gBAAgB,GAAG,eAAe,CAAC,OAAO,CAAC,UAAU,EAAE,OAAO,CAAC,OAAO,EAAE;QAC5E,KAAK,EAAE,QAAQ;QACf,MAAM,EAAE,eAAe;KACxB,CAAC,CAAC;IACH,IACE,OAAO,CAAC,UAAU,KAAK,SAAS;QAChC,gBAAgB,KAAK,SAAS;QAC9B,gBAAgB,GAAG,OAAO,CAAC,UAAU,EACrC,CAAC;QACD,OAAO;YACL,UAAU,EAAE,oBAAoB,gBAAgB,CAAC,OAAO,CAAC,CAAC,CAAC,iBAAiB,OAAO,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;SAC5G,CAAC;IACJ,CAAC;IACD,IACE,OAAO,CAAC,cAAc,KAAK,SAAS;QACpC,gBAAgB,KAAK,SAAS;QAC9B,CAAC,OAAO,CAAC,aAAa,IAAI,CAAC,CAAC,GAAG,gBAAgB,GAAG,OAAO,CAAC,cAAc,EACxE,CAAC;QACD,OAAO;YACL,UAAU,EAAE,4BAA4B,CAAC,OAAO,CAAC,aAAa,IAAI,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,gBAAgB,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,OAAO,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;SAC5J,CAAC;IACJ,CAAC;IAED,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAU,CAAC;IACnC,KAAK,MAAM,CAAC,IAAI,QAAQ;QAAE,QAAQ,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;IAE3D,MAAM,QAAQ,GAAG,cAAc,CAAC;QAC9B,YAAY,EAAE,YAAY,CAAC,QAAQ,CAAC;QACpC,KAAK,EAAE,OAAO,CAAC,OAAO;QACtB,aAAa,EAAE,cAAc;KAC9B,CAAC,CAAC;IAEH,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,EAAE,QAAQ,EAAE,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;YACvF,IAAI,MAAM,EAAE,CAAC;gBACX,OAAO,EAAE,MAAM,EAAE,EAAE,GAAG,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,CAAC;YACnD,CAAC;QACH,CAAC;QAAC,MAAM,CAAC;YACP,+DAA+D;QACjE,CAAC;IACH,CAAC;IAED,oEAAoE;IACpE,MAAM,SAAS,GAAG,gBAAgB,KAAK,SAAS,CAAC,CAAC,CAAC,KAAK,gBAAgB,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,cAAc,CAAC;IACvG,OAAO,CAAC,KAAK,CACX,uBAAuB,OAAO,CAAC,UAAU,IAAI,OAAO,CAAC,OAAO,OAAO,QAAQ,CAAC,cAAc,EAAE,aAAa,eAAe,CAAC,cAAc,EAAE,mBAAmB,SAAS,EAAE,CACxK,CAAC;IAEF,IAAI,SAAS,CAAC;IACd,IAAI,CAAC;QACH,SAAS,GAAG,MAAM,cAAc,CAAC;YAC/B,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,MAAM,EAAE,GAAG,CAAC,MAAM;YAClB,MAAM,EAAE,GAAG,CAAC,IAAI;YAChB,MAAM,EAAE,mBAAmB;YAC3B,eAAe;YACf,WAAW,EAAE,OAAO,CAAC,MAAM;SAC5B,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,MAAM,GAAG,GAAG,CAAwC,CAAC;QACrD,IAAI,GAAG,EAAE,IAAI,KAAK,YAAY,IAAI,CAAC,GAAG,EAAE,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;YAC/E,OAAO,EAAE,UAAU,EAAE,yBAAyB,EAAE,CAAC;QACnD,CAAC;QACD,OAAO,EAAE,UAAU,EAAE,oBAAoB,GAAG,EAAE,OAAO,IAAI,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC;IACzE,CAAC;IAED,sEAAsE;IACtE,yEAAyE;IACzE,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,SAAS,CAAC,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE,EAAE,CAAC;QAC3D,KAAK,MAAM,EAAE,IAAI,CAAC,CAAC,iBAAiB,EAAE,CAAC;YACrC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;gBACtB,OAAO,EAAE,UAAU,EAAE,iDAAiD,CAAC,MAAM,EAAE,EAAE,EAAE,CAAC;YACtF,CAAC;QACH,CAAC;IACH,CAAC;IAED,MAAM,KAAK,GAAG;QACZ,KAAK,EAAE,SAAS,CAAC,KAAK,CAAC,WAAW,IAAI,CAAC;QACvC,MAAM,EAAE,SAAS,CAAC,KAAK,CAAC,YAAY,IAAI,CAAC;KAC1C,CAAC;IAEF,MAAM,MAAM,GAAiB;QAC3B,UAAU,EAAE,SAAS,CAAC,MAAM,CAAC,UAAU;QACvC,SAAS,EAAE,SAAS,CAAC,MAAM,CAAC,SAAS;QACrC,SAAS,EAAE,OAAO,CAAC,OAAO;QAC1B,UAAU,EAAE,OAAO,CAAC,UAAU;QAC9B,UAAU,EAAE,KAAK;QACjB,gBAAgB,EAAE,eAAe,CAAC,OAAO,CAAC,UAAU,EAAE,OAAO,CAAC,OAAO,EAAE,KAAK,CAAC;QAC7E,QAAQ,EAAE,KAAK;QACf,aAAa,EAAE,cAAc;QAC7B,cAAc;KACf,CAAC;IAEF,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC;YACH,MAAM,gBAAgB,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC;QAC9D,CAAC;QAAC,MAAM,CAAC;YACP,mCAAmC;QACrC,CAAC;IACH,CAAC;IAED,OAAO,EAAE,MAAM,EAAE,CAAC;AACpB,CAAC"}

package/dist/ai/types.d.ts

@@ -14,7 +14,8 @@ export interface RootCause {
 }
 export interface TriageResult {
     rootCauses: RootCause[];
-    narrative: string;
+    /** Optional — may be absent when the model ran tight on output tokens. */
+    narrative?: string;
     modelUsed: string;
     providerId: string;
     tokenUsage: TokenUsage;

package/dist/ai/types.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/ai/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,WAAW,UAAU;IACzB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,SAAS;IACxB,KAAK,EAAE,MAAM,CAAC;IACd,aAAa,EAAE,MAAM,CAAC;IACtB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,QAAQ,EAAE,QAAQ,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE,MAAM,CAAC;IAClB,iBAAiB,EAAE,MAAM,EAAE,CAAC;CAC7B;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,SAAS,EAAE,CAAC;IACxB,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,UAAU,CAAC;IACvB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,QAAQ,EAAE,OAAO,CAAC;IAClB,aAAa,EAAE,MAAM,CAAC;IACtB,cAAc,EAAE,OAAO,CAAC;CACzB"}
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/ai/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,WAAW,UAAU;IACzB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,SAAS;IACxB,KAAK,EAAE,MAAM,CAAC;IACd,aAAa,EAAE,MAAM,CAAC;IACtB,eAAe,EAAE,MAAM,EAAE,CAAC;IAC1B,QAAQ,EAAE,QAAQ,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE,MAAM,CAAC;IAClB,iBAAiB,EAAE,MAAM,EAAE,CAAC;CAC7B;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,SAAS,EAAE,CAAC;IACxB,0EAA0E;IAC1E,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,UAAU,CAAC;IACvB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,QAAQ,EAAE,OAAO,CAAC;IAClB,aAAa,EAAE,MAAM,CAAC;IACtB,cAAc,EAAE,OAAO,CAAC;CACzB"}

package/dist/auditor.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"auditor.d.ts","sourceRoot":"","sources":["../src/auditor.ts"],"names":[],"mappings":"AAmDA,OAAO,KAAK,EAAE,YAAY,EAAE,YAAY,EAAwG,MAAM,YAAY,CAAC;AAktBnK,wBAAsB,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC,CA4a/F"}
+{"version":3,"file":"auditor.d.ts","sourceRoot":"","sources":["../src/auditor.ts"],"names":[],"mappings":"AAoDA,OAAO,KAAK,EAAE,YAAY,EAAE,YAAY,EAAwG,MAAM,YAAY,CAAC;AA8wBnK,wBAAsB,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,YAAY,CAAC,CAod/F"}

package/dist/auditor.js

@@ -26,7 +26,7 @@ import { hreflangConsistencyRule } from "./rules/tech/hreflang-consistency.js";
 import { ogCompletenessRule } from "./rules/tech/og-completeness.js";
 import { robotsNoindexConflictRule } from "./rules/tech/robots-noindex-conflict.js";
 import { sitemapCompletenessRule } from "./rules/tech/sitemap-completeness.js";
-import { robotsComplianceRule } from "./rules/tech/robots-sitemap-presence.js";
+import { robotsComplianceRule, parseDisallowPatterns, isBlockedByPattern, parseCrawlDelaySeconds } from "./rules/tech/robots-sitemap-presence.js";
 import { redirectChainRule } from "./rules/tech/redirect-chain.js";
 import { soft404Rule } from "./rules/tech/soft-404.js";
 import { jsonLdValidRule } from "./rules/schema/json-ld-valid.js";
@@ -43,7 +43,7 @@ import { enrichFindings } from "./enrich-findings.js";
 import { triageFindings } from "./ai/triage.js";
 import { createLanguageModel } from "./ai/adapters/index.js";
 import { promptTriageFeedback } from "./ai/feedback-prompt.js";
-import { generateRunId, appendTelemetryRecord, } from "./telemetry/index.js";
+import { generateRunId, appendTelemetryRecord, todayTriageSpendUsd, } from "./telemetry/index.js";
 import { cachedFetch } from "./cache.js";
 import { stratifiedSample } from "./stratified-sample.js";
 import { readState, writeState, computeContentHash, STATE_SCHEMA_VERSION, } from "./state.js";
@@ -430,7 +430,35 @@ async function collectUrlsFromSitemap(sitemapText, sitemapUrl, visited, timeoutM
     }
     return allUrls;
 }
-async function loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscovery, discoveryBudget, cache, stats) {
+async function fetchRobotsMeta(origin, timeoutMs, cache, stats) {
+    if (!origin)
+        return { disallow: [], crawlDelaySec: 0 };
+    try {
+        const robotsUrl = `${origin}/robots.txt`;
+        const fetched = await fetchTextStrict(robotsUrl, timeoutMs, cache, stats);
+        return {
+            disallow: parseDisallowPatterns(fetched.text),
+            crawlDelaySec: parseCrawlDelaySeconds(fetched.text),
+        };
+    }
+    catch {
+        return { disallow: [], crawlDelaySec: 0 };
+    }
+}
+function sleep(ms) {
+    return new Promise((resolve) => setTimeout(resolve, ms));
+}
+function isDisallowedByRobots(urlPath, patterns) {
+    for (const pat of patterns) {
+        if (isBlockedByPattern(urlPath, pat))
+            return true;
+    }
+    return false;
+}
+function budgetExceeded(b) {
+    return b.cap > 0 && b.used >= b.cap;
+}
+async function loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscovery, discoveryBudget, cache, stats, fillBudgetViaLinkDiscovery = false, byteBudget = { used: 0, cap: 0 }) {
     if (/^https?:\/\//i.test(source)) {
         let text;
         let contentType;
@@ -467,23 +495,38 @@ async function loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscover
                 ? fisherYatesSample(allSitemapUrls, discoveryBudget)
                 : allSitemapUrls;
             const pages = [];
-            await runWithConcurrency(urlsToFetch, concurrency, async (url) => {
+            // Fetch robots.txt once for the origin — reused for Crawl-Delay pacing and Disallow checks.
+            const sourceOrigin = (() => { try {
+                return new URL(source).origin;
+            }
+            catch {
+                return "";
+            } })();
+            const robots = await fetchRobotsMeta(sourceOrigin, timeoutMs, cache, stats);
+            const effectiveConcurrency = robots.crawlDelaySec > 0 ? 1 : concurrency;
+            const delayMs = robots.crawlDelaySec * 1000;
+            await runWithConcurrency(urlsToFetch, effectiveConcurrency, async (url) => {
+                if (budgetExceeded(byteBudget))
+                    return;
                 const result = await fetchPageWithMeta(url, timeoutMs, cache, stats);
                 if (result) {
+                    byteBudget.used += result.html.length;
                     pages.push(result);
                 }
+                if (delayMs > 0)
+                    await sleep(delayMs);
             });
-            // Skip additional crawl discovery when budget is active — sitemap is authoritative
-            if (crawlDiscovery && discoveryBudget === 0) {
+            // Link discovery fills the sample.
+            // Legacy behavior: no budget set + crawlDiscovery true → fill from links (unchanged).
+            // New behavior: budget set + crawlDiscovery true + opt-in flag → top up to budget.
+            const budgetUnderfilled = discoveryBudget > 0 && pages.length < discoveryBudget;
+            const legacyBudgetless = discoveryBudget === 0;
+            const shouldFill = crawlDiscovery && (legacyBudgetless || (budgetUnderfilled && fillBudgetViaLinkDiscovery));
+            if (shouldFill) {
                 const sitemapUrlSet = new Set(allSitemapUrls);
                 const discoveredUrls = new Set();
-                let sourceOrigin;
-                try {
-                    sourceOrigin = new URL(source).origin;
-                }
-                catch {
-                    sourceOrigin = "";
-                }
+                // robots already fetched above; reuse its Disallow patterns here.
+                const disallowPatterns = robots.disallow;
                 for (const page of pages) {
                     const linkMatches = Array.from(page.html.matchAll(/href=["']([^"']+)["']/gi));
                     for (const match of linkMatches) {
@@ -500,9 +543,11 @@ async function loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscover
                             resolvedUrl.search = "";
                             resolvedUrl.hash = "";
                             const normalized = resolvedUrl.href;
-                            if (!sitemapUrlSet.has(normalized) && !discoveredUrls.has(normalized)) {
-                                discoveredUrls.add(normalized);
-                            }
+                            if (sitemapUrlSet.has(normalized) || discoveredUrls.has(normalized))
+                                continue;
+                            if (isDisallowedByRobots(resolvedUrl.pathname, disallowPatterns))
+                                continue;
+                            discoveredUrls.add(normalized);
                         }
                         catch {
                             continue;
@@ -510,11 +555,21 @@ async function loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscover
                     }
                 }
                 if (discoveredUrls.size > 0) {
-                    await runWithConcurrency(Array.from(discoveredUrls), concurrency, async (url) => {
+                    const candidates = Array.from(discoveredUrls);
+                    // Fisher-Yates shuffle so we don't bias toward the first-discovered links (nav/footer).
+                    const shuffled = fisherYatesSample(candidates, candidates.length);
+                    const remaining = discoveryBudget === 0 ? Infinity : discoveryBudget - pages.length;
+                    const toFetch = remaining === Infinity ? shuffled : shuffled.slice(0, remaining);
+                    await runWithConcurrency(toFetch, effectiveConcurrency, async (url) => {
+                        if (budgetExceeded(byteBudget))
+                            return;
                         const result = await fetchPageWithMeta(url, timeoutMs, cache, stats);
                         if (result && result.httpMeta && result.httpMeta.statusCode >= 200 && result.httpMeta.statusCode < 300) {
+                            byteBudget.used += result.html.length;
                             pages.push(result);
                         }
+                        if (delayMs > 0)
+                            await sleep(delayMs);
                     });
                 }
             }
@@ -668,7 +723,10 @@ export async function auditSource(source, options) {
             ttlMs: options.cache.ttlMs ?? 7 * 24 * 60 * 60 * 1000,
         }
         : null;
-    const { pages: loadedPagesRaw, sitemapUrls: sitemapUrlSet, discoveredUrlCount } = await loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscovery, discoveryBudget, cacheConfig, cacheStats);
+    const fillBudgetViaLinkDiscovery = options?.fillBudgetViaLinkDiscovery ?? false;
+    const maxFetchBytes = options?.maxFetchBytes ?? 52_428_800;
+    const fetchByteBudget = { used: 0, cap: maxFetchBytes };
+    const { pages: loadedPagesRaw, sitemapUrls: sitemapUrlSet, discoveredUrlCount } = await loadPagesFromSource(source, concurrency, timeoutMs, crawlDiscovery, discoveryBudget, cacheConfig, cacheStats, fillBudgetViaLinkDiscovery, fetchByteBudget);
     const loadedPages = [...loadedPagesRaw];
     if (discoveredUrlCount && discoveredUrlCount > loadedPages.length) {
         console.error(`Discovered ${discoveredUrlCount} pages, fetched ${loadedPages.length} for audit. Use --sample-size 0 for full crawl.`);
@@ -916,7 +974,13 @@ export async function auditSource(source, options) {
         };
         await writeState(statePath, newState);
     }
+    // Captured for telemetry even when triage is skipped, so users can diagnose
+    // model/provider reliability from their local stats.jsonl.
+    let triageAttempt;
     if (options?.ai?.enabled) {
+        if (options.ai.apiKey) {
+            console.error("[ai-triage] warning: ai.apiKey is set in options. Prefer env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.) — never commit an apiKey to a config file.");
+        }
         try {
             const resolved = await createLanguageModel({
                 provider: options.ai.provider,
@@ -930,6 +994,17 @@ export async function auditSource(source, options) {
                     dir: options.ai.cache?.dir ?? ".pseolint/ai-cache",
                     ttlMs: options.ai.cache?.ttlMs ?? 30 * 24 * 60 * 60 * 1000,
                 };
+            // Daily-budget pre-flight read (best-effort — missing file is fine).
+            let spentTodayUsd;
+            if (options.ai.dailyBudgetUsd !== undefined) {
+                const telemetryPath = options.telemetry?.path ?? ".pseolint/telemetry.jsonl";
+                try {
+                    spentTodayUsd = await todayTriageSpendUsd(telemetryPath);
+                }
+                catch {
+                    spentTodayUsd = 0;
+                }
+            }
             const outcome = await triageFindings(summary.findings, summary.pageCount, {
                 enabled: true,
                 model: resolved.model,
@@ -937,22 +1012,24 @@ export async function auditSource(source, options) {
                 modelId: resolved.modelId,
                 maxInputTokens: options.ai.maxInputTokens,
                 maxOutputTokens: options.ai.maxOutputTokens,
+                maxCostUsd: options.ai.maxCostUsd,
+                dailyBudgetUsd: options.ai.dailyBudgetUsd,
+                spentTodayUsd,
                 cache: cacheConfig,
             });
             if (outcome.skipReason) {
                 console.error(`[ai-triage] skipped: ${outcome.skipReason}`);
+                triageAttempt = { providerId: resolved.providerId, modelId: resolved.modelId, skipReason: outcome.skipReason };
             }
             else {
                 summary.triage = outcome.result;
             }
         }
         catch (e) {
-            if (e instanceof Error) {
-                console.error(`[ai-triage] skipped: ${e.message}`);
-            }
-            else {
-                console.error(`[ai-triage] skipped: unknown error`);
-            }
+            const reason = e instanceof Error ? e.message : "unknown error";
+            console.error(`[ai-triage] skipped: ${reason}`);
+            // No resolved model — providerId/modelId blank.
+            triageAttempt = { providerId: options.ai.provider ?? "", modelId: options.ai.model ?? "", skipReason: reason };
         }
     }
     if (options?.telemetry?.enabled) {
@@ -971,6 +1048,7 @@ export async function auditSource(source, options) {
             ...(summary.cacheStats && { cacheStats: summary.cacheStats }),
             ...(summary.triage && {
                 triage: {
+                    success: true,
                     rootCauseCount: summary.triage.rootCauses.length,
                     providerId: summary.triage.providerId,
                     modelId: summary.triage.modelUsed,
@@ -982,6 +1060,18 @@ export async function auditSource(source, options) {
                     truncatedInput: summary.triage.truncatedInput,
                 },
             }),
+            ...(!summary.triage && triageAttempt && {
+                triage: {
+                    success: false,
+                    skipReason: triageAttempt.skipReason,
+                    rootCauseCount: 0,
+                    providerId: triageAttempt.providerId,
+                    modelId: triageAttempt.modelId,
+                    cacheHit: false,
+                    tokenUsage: { input: 0, output: 0 },
+                    truncatedInput: false,
+                },
+            }),
         };
         await appendTelemetryRecord(telemetryPath, auditRecord);
         // Feedback: only if triage ran