mcp-scraper 0.3.14 → 0.3.16

package/docs/seo-crawl-report-spec.md

@@ -0,0 +1,287 @@
+# SEO Crawl Report — Screaming Frog parity spec (scrape-report scope)
+
+Goal: from our `extract_site` crawl HTML, reproduce the most valuable ~50% of Screaming Frog's
+per-URL crawl reports — **titles, metas, headings, indexability, canonicals, content, structured
+data, and internal-link analysis** — and emit them as structured files in the bulk-scrape folder
+plus an audit report.
+
+Scope boundary (user directive): **scrape reports only** — per-URL crawl data and the link graph we
+can derive from it. Out of scope here: JS-rendering diffs, log-file analysis, GA/GSC API joins,
+spell-check, PageSpeed/Lighthouse, AMP validation, crawl scheduling.
+
+---
+
+## 0. Current state (verified)
+
+- `PageData` (`src/api/site-extractor.ts:11`): `url, status, via, title, metaDescription, h1,
+  headings[{level,text}], wordCount, schemaTypes[], canonicalUrl, internalLinks (count),
+  externalLinks (count), bodyMarkdown, schema[]`.
+- `parsePageData(url, html, status, via)` (`src/api/site-extractor.ts:45`) — HTML only, **no response headers**.
+- Fetch layers that DO have headers/timing:
+  - plain: `fetchAndParse` → `res: Response` (`res.headers`).
+  - rotating: `rotating-proxy-crawl.ts` `fetchBatch` → playwright `resp` (`resp.status()`, `resp.headers()`).
+  - `RotatingFetchResult` (`rotating-proxy-crawl.ts:5`) = `{ url, html, status, via }` — must be extended.
+- Bulk output: `saveBulkSite(siteUrl, BulkPage[])` (`mcp-response-formatter.ts:68`); `BulkPage` only
+  carries `url,title,bodyMarkdown,metaDescription,schemaTypes`. Writes `index.md` + `pages/*.md`.
+- `formatExtractSite` (`mcp-response-formatter.ts`) maps `pages` → `BulkPage` for the folder.
+
+---
+
+## 1. Screaming Frog feature inventory → coverage decision
+
+Legend: ✅ already captured · ◑ partial · ➕ ADD (in this spec) · 🔗 link-graph (Section 3) · ⛔ out of scope
+
+### 1.1 Response / crawl
+| SF data | Decision | Source |
+|---|---|---|
+| Address (URL) | ✅ | `PageData.url` |
+| Status code + status text | ✅ | `PageData.status` |
+| Content-Type | ➕ | response header |
+| Response time (ms) | ➕ | fetch timing |
+| Size (bytes, HTML transfer) | ➕ | `Content-Length` / `html.length` |
+| Redirect URL + redirect type | ➕ | `Location` header / 3xx |
+| Last-Modified | ➕ | response header |
+| Crawl depth (clicks from start) | 🔗 | computed from link graph |
+| Indexability + reason | ➕ | meta-robots + X-Robots + canonical + status |
+
+### 1.2 On-page elements
+| SF data | Decision | Source |
+|---|---|---|
+| Title 1 + length | ✅/➕ | have title; ADD length + pixel-width estimate |
+| Title — pixel width | ➕ | char-width table estimate (flag approximate) |
+| Meta description + length + pixel width | ✅/➕ | have text; ADD lengths |
+| Meta keywords | ➕ | regex (low value, cheap) |
+| H1 (1st + 2nd) + length | ✅/➕ | have headings[]; derive H1-1/H1-2 + length |
+| H2 (1st/2nd) + count | ✅ | from headings[] |
+| Word count | ✅ | `PageData.wordCount` |
+| Text ratio (text/HTML) | ➕ | `bodyText.length / html.length` |
+| Meta robots | ➕ | regex |
+| X-Robots-Tag | ➕ | response header |
+| Canonical link element | ✅ | `PageData.canonicalUrl` |
+| rel=next / rel=prev | ➕ | regex |
+| hreflang entries | ➕ | regex (list of {lang,href}) |
+| Open Graph tags | ➕ | regex (og:title/description/image/type) |
+| Twitter card tags | ➕ | regex |
+| Mobile/AMP alternate | ➕ | regex `<link rel="amphtml">`, `alternate` |
+
+### 1.3 Images
+| SF data | Decision | Source |
+|---|---|---|
+| Image count | ➕ | count `<img>` |
+| Images missing alt | ➕ | `<img>` without non-empty `alt` |
+| Alt text over N chars | ➕ | alt length check |
+| Image >100KB | ⛔ (needs per-asset fetch) | skip in v1 |
+
+### 1.4 Structured data
+| SF data | Decision | Source |
+|---|---|---|
+| Schema types present | ✅ | `PageData.schemaTypes` |
+| Raw JSON-LD | ✅ | `PageData.schema` |
+| Microdata / RDFa | ◑ | JSON-LD only in v1 |
+| Validation errors | ◑ | shape checks only (not full Google validator) |
+
+### 1.5 Content
+| SF data | Decision | Source |
+|---|---|---|
+| Low content / thin pages | ➕ | wordCount threshold |
+| Exact duplicates | ➕ | hash of normalized body |
+| Near-duplicates | ◑ | simhash (v2) — exact-hash in v1 |
+
+### 1.6 Links — **the high-value gap (user-flagged)**
+| SF data | Decision | Source |
+|---|---|---|
+| Outlinks (count) | ✅ | `internalLinks`/`externalLinks` counts |
+| Outlinks edge list (target, anchor, rel, position) | 🔗➕ | replace counts with edge capture |
+| Inlinks + unique inlinks per URL | 🔗 | computed (invert outlinks) |
+| Anchor text per inlink | 🔗 | from edges |
+| Internal nofollow | 🔗➕ | `rel="nofollow"` on edge |
+| Crawl depth | 🔗 | BFS from start over internal edges |
+| Orphan URLs | 🔗 | in sitemap/crawl set but zero internal inlinks |
+| Broken internal/external links | 🔗 | edge target status ∈ 4xx/5xx |
+| Redirecting links | 🔗 | edge target status ∈ 3xx |
+
+### 1.7 Issue filters (the actionable "reports" — computed cross-page)
+Titles: missing, duplicate, >60 char/>561px, <30 char, multiple, same-as-H1.
+Meta desc: missing, duplicate, >155/>985px, <70, multiple.
+H1: missing, duplicate, multiple, >70 char.
+H2: missing, multiple.
+Canonical: missing, canonicalised (non-self), multiple, non-indexable canonical target.
+Directives: noindex, nofollow.
+Response: 3xx (incl. chains/loops via edges), 4xx broken, 5xx, blocked-by-robots.
+URL: >115 char, uppercase, underscores, params, non-ASCII, duplicate.
+Content: thin (<X words), exact duplicate (hash collision).
+Images: missing alt.
+Structured data: present-but-invalid (shape), missing on key templates.
+Links: broken internal/external, redirected internal, orphan pages.
+
+---
+
+## 2. Suggested data model (THE deliverable)
+
+### 2.1 Extend `PageData` (`src/api/site-extractor.ts:11`)
+Add these fields (keep existing):
+
+```ts
+// response/header-derived (require fetch-layer plumbing, Section 3.2)
+contentType:     string | null
+responseTimeMs:  number | null
+sizeBytes:       number | null
+redirectUrl:     string | null      // Location on 3xx
+lastModified:    string | null
+xRobotsTag:      string | null
+
+// on-page additions
+titleLength:     number | null
+titlePixels:     number | null      // estimate, approximate
+metaDescLength:  number | null
+metaKeywords:    string | null
+h1_2:            string | null      // second H1 if present
+h2Count:         number
+metaRobots:      string | null
+relNext:         string | null
+relPrev:         string | null
+hreflang:        Array<{ lang: string; href: string }>
+og:              { title?: string; description?: string; image?: string; type?: string } | null
+twitter:         { card?: string; title?: string; description?: string } | null
+ampHref:         string | null
+
+// images
+imageCount:      number
+imagesMissingAlt: number
+
+// content
+textRatio:       number             // bodyText.length / html.length
+contentHash:     string             // sha1 of normalized visible text
+
+// indexability (derived, see 3.3)
+indexable:       boolean
+indexabilityReason: string | null   // 'noindex' | 'canonicalised' | 'non-200' | 'x-robots-noindex' | null
+
+// LINKS — replace the two counts with the edge list
+outlinks:        Array<{ href: string; anchor: string; rel: string | null; internal: boolean }>
+// keep internalLinks/externalLinks as derived counts for back-compat
+```
+
+### 2.2 Link edge + graph model (post-crawl, computed)
+```ts
+interface LinkEdge { from: string; to: string; anchor: string; rel: string | null; internal: boolean }
+
+interface PageLinkMetrics {
+  url: string
+  inlinks: number
+  uniqueInlinks: number
+  outlinksInternal: number
+  outlinksExternal: number
+  crawlDepth: number | null   // BFS hops from startUrl over internal 200 edges; null = orphan/unreachable
+  orphan: boolean             // crawled/in-sitemap but 0 internal inlinks
+  topAnchors: string[]        // most common inbound anchor texts
+}
+```
+
+### 2.3 Structured outputs written into the bulk folder
+Alongside `index.md` + `pages/*.md`, add:
+- `pages.jsonl` — one JSON line per page = full extended `PageData` minus `bodyMarkdown`/`schema`
+  (those stay in `pages/*.md`). This is the "Internal" tab.
+- `links.jsonl` — one `LinkEdge` per line (internal + external). The "All Outlinks" export.
+- `link-metrics.jsonl` — one `PageLinkMetrics` per URL. The inlinks/depth/orphan view.
+- `issues.json` — the computed issue filters (Section 1.7) as `{ issueKey: { count, urls[] } }`.
+- `report.md` — human summary (counts per issue, top offenders) = the audit deliverable.
+
+---
+
+## 3. Implementation blueprint (atomic)
+
+### 3.1 Capture: extend `parsePageData` — `src/api/site-extractor.ts:45`
+Signature change:
+```ts
+function parsePageData(
+  url: string, html: string, status: number, via: 'fetch'|'browser',
+  resp?: { headers?: Record<string,string>; responseTimeMs?: number; redirectUrl?: string|null }
+): PageData
+```
+Add, inside the function, regex/derivations:
+- `titleLength = title?.length`; `titlePixels = estimatePixels(title)` (new helper, char-width table).
+- `metaDescLength`, `metaKeywords` via `<meta name="keywords">`.
+- `h1_2`/`h2Count` from existing `headings` array.
+- `metaRobots` via `<meta name="robots" content="...">`.
+- `relNext`/`relPrev` via `<link rel="next|prev">`.
+- `hreflang[]` via `<link rel="alternate" hreflang="..">`.
+- `og`/`twitter` via `<meta property="og:.."|name="twitter:..">`.
+- `ampHref` via `<link rel="amphtml">`.
+- `imageCount` = matches of `<img`; `imagesMissingAlt` = `<img>` lacking non-empty `alt`.
+- `textRatio = bodyText.length / Math.max(1, html.length)`.
+- `contentHash = sha1(bodyText.replace(/\s+/g,' ').trim())` (`node:crypto`).
+- header fields from `resp`: `contentType, xRobotsTag, lastModified, sizeBytes, responseTimeMs, redirectUrl`.
+- `outlinks[]`: replace the count-only loop (`site-extractor.ts:102-110`) — for each `<a href>` also
+  capture anchor text and `rel`; classify `internal` by origin; keep `internalLinks`/`externalLinks`
+  as `.filter().length` for back-compat.
+- `indexable`/`indexabilityReason`: `false` if status!=200, or metaRobots/xRobots contains `noindex`,
+  or canonical present and != self → reason set accordingly.
+
+New helper `estimatePixels(s)` — sum per-char widths from a static map (approx Arial 'M'≈14, 'i'≈4…);
+mark all pixel fields "approximate" in docs. ~25 lines.
+
+### 3.2 Plumb headers/timing into both fetch paths
+- `RotatingFetchResult` (`rotating-proxy-crawl.ts:5`): add
+  `headers?: Record<string,string>; responseTimeMs?: number; redirectUrl?: string|null`.
+  In `fetchBatch` (`:60-98`): time the `goto`, `resp.headers()`, capture `Location` when 3xx.
+- `fetchAndParse` (`site-extractor.ts:115`): build the same `resp` object from `res.headers` + timing,
+  pass as 5th arg to `parsePageData`.
+- In `extractSite` rotating branch (`site-extractor.ts:~185`): pass `r.headers/responseTimeMs/redirectUrl`
+  into `parsePageData(r.url, r.html, r.status, 'browser', {...})`.
+
+### 3.3 Post-crawl link graph — new file `src/api/seo-link-graph.ts`
+```ts
+export function buildLinkGraph(pages: PageData[], startUrl: string):
+  { edges: LinkEdge[]; metrics: Map<string, PageLinkMetrics> }
+```
+Logic:
+- Flatten `pages[].outlinks` → `edges` (from = page.url).
+- Build `inlinks` map by inverting internal edges; `uniqueInlinks` = distinct `from`.
+- `crawlDepth`: BFS from `startUrl` over internal edges whose target status==200; unreached → null.
+- `orphan`: page in set with 0 internal inlinks and url != startUrl.
+- `topAnchors`: top-3 inbound anchors by frequency.
+- Status join: map target→status from `pages` to flag broken (4xx/5xx) / redirect (3xx) edges.
+
+### 3.4 Issue computation — new file `src/api/seo-issues.ts`
+```ts
+export function computeIssues(pages: PageData[], metrics: Map<string,PageLinkMetrics>):
+  Record<string, { count: number; urls: string[] }>
+```
+Thresholds (SF defaults): title >60/<30 char & >561px; meta >155/<70 & >985px; H1 >70; URL >115;
+thin <200 words (configurable). Duplicates: group by exact `title`/`metaDescription`/`contentHash`,
+flag groups size>1. Broken/redirect/orphan from `metrics`/edges. Indexability from `PageData`.
+
+### 3.5 Write structured outputs — extend `saveBulkSite` (`mcp-response-formatter.ts:68`)
+- Widen `BulkPage` → accept full extended `PageData` (or pass `PageData[]` directly).
+- After writing `pages/*.md`, also `writeFileSync`:
+  - `pages.jsonl` (PageData minus bodyMarkdown/schema),
+  - `links.jsonl`, `link-metrics.jsonl`, `issues.json`, `report.md`.
+- `formatExtractSite` already has full `pages`; pass them through (today it maps to the slim BulkPage —
+  change to pass the full objects, call `buildLinkGraph` + `computeIssues` before `saveBulkSite`).
+- Return extra paths in the bulk summary + `structuredContent` (e.g. `pagesJsonl`, `issuesFile`).
+
+### 3.6 The skill — `seo-crawl-audit`
+Orchestrator (no new server code):
+1. Call `extract_site(url, rotateProxies:true)` → folder path from response.
+2. Read `pages.jsonl` + `issues.json` + `link-metrics.jsonl` from the folder.
+3. Emit a prioritized SEO report: response-code breakdown, title/meta/H1 issues with offender lists,
+   indexability summary, thin/duplicate content, structured-data coverage, and the internal-link
+   section (orphans, deepest pages, most-linked, broken internal links).
+4. Hand the full link graph to the existing `site-architecture-auditor` skill for equity/architecture
+   scoring (it already builds this graph from SF exports — we now feed it ours).
+
+---
+
+## 4. Build order (phased, each independently shippable)
+1. **P1 — page fields (no headers):** 3.1 minus header fields + 3.5 `pages.jsonl`. Unlocks titles/metas/
+   H1/canonical/indexability(meta)/thin/dup/schema reports immediately.
+2. **P2 — link graph:** 3.1 outlinks edge capture + 3.3 + `links.jsonl`/`link-metrics.jsonl`. Unlocks
+   inlinks/depth/orphans/broken-links — the user's headline ask.
+3. **P3 — headers/timing:** 3.2 (both fetch paths). Adds content-type/size/response-time/X-Robots/redirects.
+4. **P4 — issues + report:** 3.4 + `issues.json`/`report.md`.
+5. **P5 — skill:** 3.6 + handoff to `site-architecture-auditor`.
+
+## 5. Explicit non-goals (v1)
+Per-asset image weight, microdata/RDFa, full structured-data validation, near-duplicate simhash,
+JS-render diffing, pixel-width exactness (estimate only), redirect-chain hop-by-hop beyond one hop.