mcp-scraper 0.38.2 → 0.40.1

package/docs/mcp-tool-quality-spec.md

@@ -1,240 +0,0 @@
-# MCP Tool Quality Spec
-
-This spec defines the shipping bar for MCP Scraper tools. It exists because MCP behavior is model-facing: a tool can be technically callable and still fail if the AI cannot infer when to use it, how to fill inputs, or how to chain the result.
-
-## What Actually Steers The AI
-
-The model is primarily affected by what the MCP client receives from `tools/list` and `tools/call`.
-
-1. Tool name
-2. Tool title and annotations
-3. Tool description
-4. Input schema field names, descriptions, defaults, enums, and limits
-5. Output schema and `structuredContent`
-6. Tool result text and next-step guidance
-7. Error shape and retry guidance
-
-README files, website copy, and public skill markdown help humans and skill loaders, but they do not reliably affect runtime AI behavior unless the client explicitly injects those files into context.
-
-## Tool Boundary Rules
-
-Each tool must have one primary job.
-
-- Split tools when the user intent, cost model, result shape, or follow-up workflow differs.
-- Do not overload one tool with unrelated modes if a model could choose the wrong path.
-- Descriptions for adjacent tools must explicitly say when not to use each tool.
-- If a workflow has a natural sequence, encode it in the description and result guidance.
-
-Example:
-
-- `maps_search`: find multiple Google Maps candidates for a category, market, lead list, or "more than the 3-pack".
-- `maps_place_intel`: hydrate one known business or one selected candidate with full profile details and optional reviews.
-
-## Tool Naming
-
-Names must be short, stable, and action-oriented.
-
-- Use domain + action: `maps_search`, `maps_place_intel`, `facebook_ad_search`.
-- Avoid generic names such as `search`, `lookup`, `get_data`, or `run`.
-- Avoid names that imply broader capability than the tool has.
-- Do not rename a public tool without a compatibility plan.
-
-## Tool Descriptions
-
-Descriptions are model instructions. They must be concise but operational.
-
-Every description must include:
-
-- What the tool does.
-- When to use it.
-- When not to use it if there is an adjacent tool.
-- Important defaults and hard caps.
-- Cost-sensitive behavior when relevant.
-- The expected next tool when chaining is common.
-- Whether reports are saved locally.
-
-Bad:
-
-```text
-Search Google Maps.
-```
-
-Good:
-
-```text
-Search Google Maps for multiple businesses/profiles by category, niche, keyword, or local market. Use this when the user asks for several Google Business Profiles, GMBs, GBPs, leads, prospects, competitors, or "more than the 3-pack." Returns up to 50 candidates. Default maxResults is 10; maximum is 50. Use maps_place_intel afterward only when a selected business needs full details and reviews.
-```
-
-## Input Schema
-
-Each input field must have a clear description.
-
-Required:
-
-- Required fields must be actually required in the schema.
-- Defaults must be encoded in the schema, not only in prose.
-- Hard caps must be encoded in the schema.
-- Fields that are often confused must say what not to put there.
-- Location and query fields must tell the model to split location from the topic when possible.
-- Enum fields must explain when to choose each option.
-
-For numeric limits:
-
-- Normal/default value belongs in `.default(...)`.
-- Maximum value belongs in `.max(...)`.
-- Description must say when to use higher values.
-
-## Output Schema And Structured Content
-
-Any tool whose output may be consumed by another tool should have `outputSchema` and return `structuredContent`.
-
-Required for chaining tools:
-
-- Return arrays/objects in `structuredContent`; do not force the model to parse Markdown.
-- Keep `content` as a human-readable report.
-- Ensure `structuredContent` validates against `outputSchema`.
-- Include IDs, URLs, names, positions, counts, and any fields needed for the next tool.
-
-Example:
-
-`maps_search` must return `structuredContent.results[]` with at least:
-
-- `position`
-- `name`
-- `placeUrl`
-- `cid`
-- `cidDecimal`
-- `rating`
-- `reviewCount`
-- `category`
-- `address`
-- `websiteUrl`
-- `directionsUrl`
-- `metadata`
-
-`maps_search` and `directory_workflow` must also return sanitized `attempts[]` records for Maps rotation visibility. Include attempt number, max attempts, outcome, retry flag, proxy mode/source/suffix, proxy target level/location/ZIP, browser session suffix, observed IP city/region, error, result count, and duration. Do not expose full browser session IDs, full proxy IDs, or API keys.
-
-## Tool Annotations
-
-Every public tool should define annotations.
-
-Use:
-
-- `readOnlyHint: true` for research, scrape, search, transcript, and inspect tools.
-- `destructiveHint: false` unless the tool mutates or deletes user data.
-- `idempotentHint: false` for live web searches because results, billing, and anti-bot state can change.
-- `openWorldHint: true` for tools that access public web or external live systems.
-- A human-readable `title`.
-
-Annotations are hints, not a replacement for descriptions.
-
-## Result Text
-
-Human-readable text still matters. It is what users see and what some clients preserve in context.
-
-Every successful result should include:
-
-- Clear title.
-- Returned count versus requested count when relevant.
-- Key result table or summary.
-- Saved report path when report saving is enabled.
-- Next-step guidance for common follow-up actions.
-
-For chained tools, the result text should name the next tool explicitly.
-
-## Error Format
-
-Errors must help the model choose the next action.
-
-Every API error should include:
-
-- `error` or `error_code`
-- Human-readable message
-- Whether retry is reasonable when known
-- Enough context to avoid repeating the same bad call
-
-Common cases:
-
-- Auth failure: tell user API key is invalid or missing. Do not retry.
-- Insufficient balance: return balance, required credits, and top-up URL.
-- CAPTCHA/block: for browser-service Google SERP tools, detect it immediately and treat the session as failed; say it is temporary and retryable with a fresh browser session. Do not wait for a solver.
-- Validation error: identify the bad or missing field.
-- Timeout/cancel: say whether the server attempted cleanup.
-
-## Cost And Concurrency
-
-Tools that cost credits or hold jobs must expose that in metadata or result text.
-
-Required:
-
-- Add cost entry to `CREDIT_COST_CATALOG`.
-- Add ledger operation for billable work.
-- Include refund path on failure where applicable.
-- Respect the account concurrency model.
-- Tool descriptions should warn when a tool is expensive or long-running.
-
-## Docs Surfaces
-
-For a new or changed public tool, update all applicable surfaces:
-
-- `src/mcp/paa-mcp-server.ts`
-- `src/mcp/mcp-tool-schemas.ts`
-- `src/mcp/mcp-response-formatter.ts`
-- API route and API schema
-- `README.md`
-- `public/skill.md`
-- `public/codex-skill.md`
-- `public/skills/mcp-scraper/skill.md`
-- Dashboard UI when users can trigger the workflow there
-- Live protocol tool list tests
-
-## Packaging And Deployment
-
-MCP changes often have two release surfaces.
-
-NPX package:
-
-- Bump `package.json` and `package-lock.json` when publishing npm.
-- Rebuild `dist` with `npx tsup`.
-- Verify `npm pack --dry-run` includes the rebuilt stdio binary.
-- Verify built output contains the new tool name, description, schema, and formatter behavior.
-
-Hosted API:
-
-- Deploy API routes before or with the MCP package.
-- A new MCP tool that calls a new hosted endpoint is broken until production has that endpoint.
-- Answer "was this prod?" by separating API deployment from npm package publication.
-
-## Required Tests
-
-Every new public MCP tool needs tests at the right level for risk.
-
-Minimum:
-
-- Schema default and hard cap tests.
-- Formatter test when result text or `structuredContent` matters.
-- Tool list/protocol test updated with the new tool.
-- Billing/ledger count tests updated for new billable operations.
-- Typecheck.
-- Unit/contract suite.
-
-For live-web tools:
-
-- One live smoke test or saved live evidence for the core workflow.
-- If anti-bot behavior is likely, capture failure mode and retry guidance.
-
-## Definition Of Done
-
-A public MCP tool change is not done until all are true:
-
-- Tool name and boundary are clear.
-- Tool description tells the model when to use it and when not to.
-- Input schema encodes defaults, limits, and field-level instructions.
-- Output is structured when the result will be chained.
-- Result text is useful to humans and names the next tool when appropriate.
-- Errors are actionable.
-- Billing and refunds are correct.
-- Dashboard, docs, skill text, and README are updated where relevant.
-- `dist` is rebuilt for NPX package changes.
-- Production API deployment is accounted for separately from npm publication.
-- Tests and smoke evidence prove the workflow.

package/docs/oauth-legal-review.md

@@ -1,38 +0,0 @@
-# OAuth legal-page review notes
-
-> Counsel review is required before treating these drafts as final legal advice or launching connected-provider access broadly.
-
-Prepared July 11, 2026 for `public/privacy.html` and `public/terms.html`.
-
-## Official policy sources
-
-- Google API Services User Data Policy / Limited Use: <https://developers.google.com/terms/api-services-user-data-policy>
-- Google APIs Terms: <https://developers.google.com/terms/>
-- YouTube API Services Terms: <https://developers.google.com/youtube/terms/api-services-terms-of-service>
-- YouTube API Services Developer Policies: <https://developers.google.com/youtube/terms/developer-policies>
-- YouTube Terms of Service: <https://www.youtube.com/t/terms>
-- Google Privacy Policy: <https://policies.google.com/privacy>
-- Meta Platform Terms: <https://developers.facebook.com/terms/>
-- Meta Privacy Policy: <https://www.facebook.com/privacy/policy/>
-- LinkedIn API Terms of Use: <https://www.linkedin.com/legal/l/api-terms-of-use>
-- X Developer Agreement: <https://docs.x.com/developer-terms/agreement>
-- X Developer Policy: <https://docs.x.com/developer-terms/policy>
-- Nango security and credential storage: <https://nango.dev/docs/guides/platform/security>
-- Nango connection deletion API: <https://nango.dev/docs/reference/backend/http-api/connections/delete>
-
-## Repo-grounded implementation facts
-
-- `src/api/nango-control.ts` sends the signed-in identity to the separate Nango control service and retrieves only connections associated with that identity.
-- The Integrations UI states that connections are private per login and that OAuth tokens remain server-side.
-- `src/api/server.ts` has account deletion/deactivation, Stripe billing, and Resend transactional-email paths.
-- `src/api/db.ts` uses Turso/libSQL in production; `vercel.json` defines the hosted application surface.
-- Scheduled actions bind an exact connection ID to an allowlist of tools.
-
-## Items counsel and the operator must confirm
-
-1. Legal entity name, mailing address, governing law, and dispute forum are not present in the repo and should be added if required.
-2. Confirm actual deletion orchestration removes Nango connections and connected-provider data when `/account/delete` runs; until then, the public policy correctly instructs users to email support for connection deletion confirmation.
-3. Confirm exact operational retention periods, backup lifetime, subprocessor list, international-transfer mechanism, and DPA availability.
-4. Confirm the Service age threshold and jurisdiction-specific consumer/privacy disclosures.
-5. Confirm every provider action has the promised consent gate. In particular, YouTube and X require express action consent, and LinkedIn's general API terms restrict automated posting.
-6. Re-run policy review before adding scopes, providers, advertising uses, model training, or new data recipients.

package/docs/seo-crawl-report-spec.md

@@ -1,287 +0,0 @@
-# SEO Crawl Report — Screaming Frog parity spec (scrape-report scope)
-
-Goal: from our `extract_site` crawl HTML, reproduce the most valuable ~50% of Screaming Frog's
-per-URL crawl reports — **titles, metas, headings, indexability, canonicals, content, structured
-data, and internal-link analysis** — and emit them as structured files in the bulk-scrape folder
-plus an audit report.
-
-Scope boundary (user directive): **scrape reports only** — per-URL crawl data and the link graph we
-can derive from it. Out of scope here: JS-rendering diffs, log-file analysis, GA/GSC API joins,
-spell-check, PageSpeed/Lighthouse, AMP validation, crawl scheduling.
-
----
-
-## 0. Current state (verified)
-
-- `PageData` (`src/api/site-extractor.ts:11`): `url, status, via, title, metaDescription, h1,
-  headings[{level,text}], wordCount, schemaTypes[], canonicalUrl, internalLinks (count),
-  externalLinks (count), bodyMarkdown, schema[]`.
-- `parsePageData(url, html, status, via)` (`src/api/site-extractor.ts:45`) — HTML only, **no response headers**.
-- Fetch layers that DO have headers/timing:
-  - plain: `fetchAndParse` → `res: Response` (`res.headers`).
-  - rotating: `rotating-proxy-crawl.ts` `fetchBatch` → playwright `resp` (`resp.status()`, `resp.headers()`).
-  - `RotatingFetchResult` (`rotating-proxy-crawl.ts:5`) = `{ url, html, status, via }` — must be extended.
-- Bulk output: `saveBulkSite(siteUrl, BulkPage[])` (`mcp-response-formatter.ts:68`); `BulkPage` only
-  carries `url,title,bodyMarkdown,metaDescription,schemaTypes`. Writes `index.md` + `pages/*.md`.
-- `formatExtractSite` (`mcp-response-formatter.ts`) maps `pages` → `BulkPage` for the folder.
-
----
-
-## 1. Screaming Frog feature inventory → coverage decision
-
-Legend: ✅ already captured · ◑ partial · ➕ ADD (in this spec) · 🔗 link-graph (Section 3) · ⛔ out of scope
-
-### 1.1 Response / crawl
-| SF data | Decision | Source |
-|---|---|---|
-| Address (URL) | ✅ | `PageData.url` |
-| Status code + status text | ✅ | `PageData.status` |
-| Content-Type | ➕ | response header |
-| Response time (ms) | ➕ | fetch timing |
-| Size (bytes, HTML transfer) | ➕ | `Content-Length` / `html.length` |
-| Redirect URL + redirect type | ➕ | `Location` header / 3xx |
-| Last-Modified | ➕ | response header |
-| Crawl depth (clicks from start) | 🔗 | computed from link graph |
-| Indexability + reason | ➕ | meta-robots + X-Robots + canonical + status |
-
-### 1.2 On-page elements
-| SF data | Decision | Source |
-|---|---|---|
-| Title 1 + length | ✅/➕ | have title; ADD length + pixel-width estimate |
-| Title — pixel width | ➕ | char-width table estimate (flag approximate) |
-| Meta description + length + pixel width | ✅/➕ | have text; ADD lengths |
-| Meta keywords | ➕ | regex (low value, cheap) |
-| H1 (1st + 2nd) + length | ✅/➕ | have headings[]; derive H1-1/H1-2 + length |
-| H2 (1st/2nd) + count | ✅ | from headings[] |
-| Word count | ✅ | `PageData.wordCount` |
-| Text ratio (text/HTML) | ➕ | `bodyText.length / html.length` |
-| Meta robots | ➕ | regex |
-| X-Robots-Tag | ➕ | response header |
-| Canonical link element | ✅ | `PageData.canonicalUrl` |
-| rel=next / rel=prev | ➕ | regex |
-| hreflang entries | ➕ | regex (list of {lang,href}) |
-| Open Graph tags | ➕ | regex (og:title/description/image/type) |
-| Twitter card tags | ➕ | regex |
-| Mobile/AMP alternate | ➕ | regex `<link rel="amphtml">`, `alternate` |
-
-### 1.3 Images
-| SF data | Decision | Source |
-|---|---|---|
-| Image count | ➕ | count `<img>` |
-| Images missing alt | ➕ | `<img>` without non-empty `alt` |
-| Alt text over N chars | ➕ | alt length check |
-| Image >100KB | ⛔ (needs per-asset fetch) | skip in v1 |
-
-### 1.4 Structured data
-| SF data | Decision | Source |
-|---|---|---|
-| Schema types present | ✅ | `PageData.schemaTypes` |
-| Raw JSON-LD | ✅ | `PageData.schema` |
-| Microdata / RDFa | ◑ | JSON-LD only in v1 |
-| Validation errors | ◑ | shape checks only (not full Google validator) |
-
-### 1.5 Content
-| SF data | Decision | Source |
-|---|---|---|
-| Low content / thin pages | ➕ | wordCount threshold |
-| Exact duplicates | ➕ | hash of normalized body |
-| Near-duplicates | ◑ | simhash (v2) — exact-hash in v1 |
-
-### 1.6 Links — **the high-value gap (user-flagged)**
-| SF data | Decision | Source |
-|---|---|---|
-| Outlinks (count) | ✅ | `internalLinks`/`externalLinks` counts |
-| Outlinks edge list (target, anchor, rel, position) | 🔗➕ | replace counts with edge capture |
-| Inlinks + unique inlinks per URL | 🔗 | computed (invert outlinks) |
-| Anchor text per inlink | 🔗 | from edges |
-| Internal nofollow | 🔗➕ | `rel="nofollow"` on edge |
-| Crawl depth | 🔗 | BFS from start over internal edges |
-| Orphan URLs | 🔗 | in sitemap/crawl set but zero internal inlinks |
-| Broken internal/external links | 🔗 | edge target status ∈ 4xx/5xx |
-| Redirecting links | 🔗 | edge target status ∈ 3xx |
-
-### 1.7 Issue filters (the actionable "reports" — computed cross-page)
-Titles: missing, duplicate, >60 char/>561px, <30 char, multiple, same-as-H1.
-Meta desc: missing, duplicate, >155/>985px, <70, multiple.
-H1: missing, duplicate, multiple, >70 char.
-H2: missing, multiple.
-Canonical: missing, canonicalised (non-self), multiple, non-indexable canonical target.
-Directives: noindex, nofollow.
-Response: 3xx (incl. chains/loops via edges), 4xx broken, 5xx, blocked-by-robots.
-URL: >115 char, uppercase, underscores, params, non-ASCII, duplicate.
-Content: thin (<X words), exact duplicate (hash collision).
-Images: missing alt.
-Structured data: present-but-invalid (shape), missing on key templates.
-Links: broken internal/external, redirected internal, orphan pages.
-
----
-
-## 2. Suggested data model (THE deliverable)
-
-### 2.1 Extend `PageData` (`src/api/site-extractor.ts:11`)
-Add these fields (keep existing):
-
-```ts
-// response/header-derived (require fetch-layer plumbing, Section 3.2)
-contentType:     string | null
-responseTimeMs:  number | null
-sizeBytes:       number | null
-redirectUrl:     string | null      // Location on 3xx
-lastModified:    string | null
-xRobotsTag:      string | null
-
-// on-page additions
-titleLength:     number | null
-titlePixels:     number | null      // estimate, approximate
-metaDescLength:  number | null
-metaKeywords:    string | null
-h1_2:            string | null      // second H1 if present
-h2Count:         number
-metaRobots:      string | null
-relNext:         string | null
-relPrev:         string | null
-hreflang:        Array<{ lang: string; href: string }>
-og:              { title?: string; description?: string; image?: string; type?: string } | null
-twitter:         { card?: string; title?: string; description?: string } | null
-ampHref:         string | null
-
-// images
-imageCount:      number
-imagesMissingAlt: number
-
-// content
-textRatio:       number             // bodyText.length / html.length
-contentHash:     string             // sha1 of normalized visible text
-
-// indexability (derived, see 3.3)
-indexable:       boolean
-indexabilityReason: string | null   // 'noindex' | 'canonicalised' | 'non-200' | 'x-robots-noindex' | null
-
-// LINKS — replace the two counts with the edge list
-outlinks:        Array<{ href: string; anchor: string; rel: string | null; internal: boolean }>
-// keep internalLinks/externalLinks as derived counts for back-compat
-```
-
-### 2.2 Link edge + graph model (post-crawl, computed)
-```ts
-interface LinkEdge { from: string; to: string; anchor: string; rel: string | null; internal: boolean }
-
-interface PageLinkMetrics {
-  url: string
-  inlinks: number
-  uniqueInlinks: number
-  outlinksInternal: number
-  outlinksExternal: number
-  crawlDepth: number | null   // BFS hops from startUrl over internal 200 edges; null = orphan/unreachable
-  orphan: boolean             // crawled/in-sitemap but 0 internal inlinks
-  topAnchors: string[]        // most common inbound anchor texts
-}
-```
-
-### 2.3 Structured outputs written into the bulk folder
-Alongside `index.md` + `pages/*.md`, add:
-- `pages.jsonl` — one JSON line per page = full extended `PageData` minus `bodyMarkdown`/`schema`
-  (those stay in `pages/*.md`). This is the "Internal" tab.
-- `links.jsonl` — one `LinkEdge` per line (internal + external). The "All Outlinks" export.
-- `link-metrics.jsonl` — one `PageLinkMetrics` per URL. The inlinks/depth/orphan view.
-- `issues.json` — the computed issue filters (Section 1.7) as `{ issueKey: { count, urls[] } }`.
-- `report.md` — human summary (counts per issue, top offenders) = the audit deliverable.
-
----
-
-## 3. Implementation blueprint (atomic)
-
-### 3.1 Capture: extend `parsePageData` — `src/api/site-extractor.ts:45`
-Signature change:
-```ts
-function parsePageData(
-  url: string, html: string, status: number, via: 'fetch'|'browser',
-  resp?: { headers?: Record<string,string>; responseTimeMs?: number; redirectUrl?: string|null }
-): PageData
-```
-Add, inside the function, regex/derivations:
-- `titleLength = title?.length`; `titlePixels = estimatePixels(title)` (new helper, char-width table).
-- `metaDescLength`, `metaKeywords` via `<meta name="keywords">`.
-- `h1_2`/`h2Count` from existing `headings` array.
-- `metaRobots` via `<meta name="robots" content="...">`.
-- `relNext`/`relPrev` via `<link rel="next|prev">`.
-- `hreflang[]` via `<link rel="alternate" hreflang="..">`.
-- `og`/`twitter` via `<meta property="og:.."|name="twitter:..">`.
-- `ampHref` via `<link rel="amphtml">`.
-- `imageCount` = matches of `<img`; `imagesMissingAlt` = `<img>` lacking non-empty `alt`.
-- `textRatio = bodyText.length / Math.max(1, html.length)`.
-- `contentHash = sha1(bodyText.replace(/\s+/g,' ').trim())` (`node:crypto`).
-- header fields from `resp`: `contentType, xRobotsTag, lastModified, sizeBytes, responseTimeMs, redirectUrl`.
-- `outlinks[]`: replace the count-only loop (`site-extractor.ts:102-110`) — for each `<a href>` also
-  capture anchor text and `rel`; classify `internal` by origin; keep `internalLinks`/`externalLinks`
-  as `.filter().length` for back-compat.
-- `indexable`/`indexabilityReason`: `false` if status!=200, or metaRobots/xRobots contains `noindex`,
-  or canonical present and != self → reason set accordingly.
-
-New helper `estimatePixels(s)` — sum per-char widths from a static map (approx Arial 'M'≈14, 'i'≈4…);
-mark all pixel fields "approximate" in docs. ~25 lines.
-
-### 3.2 Plumb headers/timing into both fetch paths
-- `RotatingFetchResult` (`rotating-proxy-crawl.ts:5`): add
-  `headers?: Record<string,string>; responseTimeMs?: number; redirectUrl?: string|null`.
-  In `fetchBatch` (`:60-98`): time the `goto`, `resp.headers()`, capture `Location` when 3xx.
-- `fetchAndParse` (`site-extractor.ts:115`): build the same `resp` object from `res.headers` + timing,
-  pass as 5th arg to `parsePageData`.
-- In `extractSite` rotating branch (`site-extractor.ts:~185`): pass `r.headers/responseTimeMs/redirectUrl`
-  into `parsePageData(r.url, r.html, r.status, 'browser', {...})`.
-
-### 3.3 Post-crawl link graph — new file `src/api/seo-link-graph.ts`
-```ts
-export function buildLinkGraph(pages: PageData[], startUrl: string):
-  { edges: LinkEdge[]; metrics: Map<string, PageLinkMetrics> }
-```
-Logic:
-- Flatten `pages[].outlinks` → `edges` (from = page.url).
-- Build `inlinks` map by inverting internal edges; `uniqueInlinks` = distinct `from`.
-- `crawlDepth`: BFS from `startUrl` over internal edges whose target status==200; unreached → null.
-- `orphan`: page in set with 0 internal inlinks and url != startUrl.
-- `topAnchors`: top-3 inbound anchors by frequency.
-- Status join: map target→status from `pages` to flag broken (4xx/5xx) / redirect (3xx) edges.
-
-### 3.4 Issue computation — new file `src/api/seo-issues.ts`
-```ts
-export function computeIssues(pages: PageData[], metrics: Map<string,PageLinkMetrics>):
-  Record<string, { count: number; urls: string[] }>
-```
-Thresholds (SF defaults): title >60/<30 char & >561px; meta >155/<70 & >985px; H1 >70; URL >115;
-thin <200 words (configurable). Duplicates: group by exact `title`/`metaDescription`/`contentHash`,
-flag groups size>1. Broken/redirect/orphan from `metrics`/edges. Indexability from `PageData`.
-
-### 3.5 Write structured outputs — extend `saveBulkSite` (`mcp-response-formatter.ts:68`)
-- Widen `BulkPage` → accept full extended `PageData` (or pass `PageData[]` directly).
-- After writing `pages/*.md`, also `writeFileSync`:
-  - `pages.jsonl` (PageData minus bodyMarkdown/schema),
-  - `links.jsonl`, `link-metrics.jsonl`, `issues.json`, `report.md`.
-- `formatExtractSite` already has full `pages`; pass them through (today it maps to the slim BulkPage —
-  change to pass the full objects, call `buildLinkGraph` + `computeIssues` before `saveBulkSite`).
-- Return extra paths in the bulk summary + `structuredContent` (e.g. `pagesJsonl`, `issuesFile`).
-
-### 3.6 The skill — `seo-crawl-audit`
-Orchestrator (no new server code):
-1. Call `extract_site(url, rotateProxies:true)` → folder path from response.
-2. Read `pages.jsonl` + `issues.json` + `link-metrics.jsonl` from the folder.
-3. Emit a prioritized SEO report: response-code breakdown, title/meta/H1 issues with offender lists,
-   indexability summary, thin/duplicate content, structured-data coverage, and the internal-link
-   section (orphans, deepest pages, most-linked, broken internal links).
-4. Hand the full link graph to the existing `site-architecture-auditor` skill for equity/architecture
-   scoring (it already builds this graph from SF exports — we now feed it ours).
-
----
-
-## 4. Build order (phased, each independently shippable)
-1. **P1 — page fields (no headers):** 3.1 minus header fields + 3.5 `pages.jsonl`. Unlocks titles/metas/
-   H1/canonical/indexability(meta)/thin/dup/schema reports immediately.
-2. **P2 — link graph:** 3.1 outlinks edge capture + 3.3 + `links.jsonl`/`link-metrics.jsonl`. Unlocks
-   inlinks/depth/orphans/broken-links — the user's headline ask.
-3. **P3 — headers/timing:** 3.2 (both fetch paths). Adds content-type/size/response-time/X-Robots/redirects.
-4. **P4 — issues + report:** 3.4 + `issues.json`/`report.md`.
-5. **P5 — skill:** 3.6 + handoff to `site-architecture-auditor`.
-
-## 5. Explicit non-goals (v1)
-Per-asset image weight, microdata/RDFa, full structured-data validation, near-duplicate simhash,
-JS-render diffing, pixel-width exactness (estimate only), redirect-chain hop-by-hop beyond one hop.