mcp-scraper 0.6.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/dist/bin/api-server.cjs +296 -198
  2. package/dist/bin/api-server.cjs.map +1 -1
  3. package/dist/bin/api-server.js +2 -2
  4. package/dist/bin/mcp-scraper-cli.cjs +1 -1
  5. package/dist/bin/mcp-scraper-cli.cjs.map +1 -1
  6. package/dist/bin/mcp-scraper-cli.js +1 -1
  7. package/dist/bin/mcp-scraper-install.cjs +1 -1
  8. package/dist/bin/mcp-scraper-install.cjs.map +1 -1
  9. package/dist/bin/mcp-scraper-install.js +1 -1
  10. package/dist/bin/mcp-stdio-server.cjs +3 -3
  11. package/dist/bin/mcp-stdio-server.cjs.map +1 -1
  12. package/dist/bin/mcp-stdio-server.js +3 -3
  13. package/dist/bin/paa-harvest.cjs +166 -74
  14. package/dist/bin/paa-harvest.cjs.map +1 -1
  15. package/dist/bin/paa-harvest.js +1 -1
  16. package/dist/chunk-EGWJ74EX.js +7 -0
  17. package/dist/chunk-EGWJ74EX.js.map +1 -0
  18. package/dist/{chunk-APJO2XV5.js → chunk-HL33CGJF.js} +3 -3
  19. package/dist/{chunk-APJO2XV5.js.map → chunk-HL33CGJF.js.map} +1 -1
  20. package/dist/{chunk-AUCXKRRH.js → chunk-RMPPYKUV.js} +167 -75
  21. package/dist/chunk-RMPPYKUV.js.map +1 -0
  22. package/dist/{chunk-BL7BBSYF.js → chunk-RUGJE5EB.js} +2 -2
  23. package/dist/{chunk-3K5GT6VG.js → chunk-YGRZU7IR.js} +3 -3
  24. package/dist/index.cjs +166 -74
  25. package/dist/index.cjs.map +1 -1
  26. package/dist/index.js +1 -1
  27. package/dist/{server-WHM446IZ.js → server-JNY4XPZE.js} +81 -76
  28. package/dist/server-JNY4XPZE.js.map +1 -0
  29. package/dist/{site-extract-repository-U476J44K.js → site-extract-repository-NVSZH35Y.js} +3 -3
  30. package/dist/{worker-KJZ3ZN2N.js → worker-JQTS437L.js} +4 -4
  31. package/dist/worker-JQTS437L.js.map +1 -0
  32. package/package.json +1 -1
  33. package/dist/chunk-AUCXKRRH.js.map +0 -1
  34. package/dist/chunk-F2BXETDX.js +0 -7
  35. package/dist/chunk-F2BXETDX.js.map +0 -1
  36. package/dist/server-WHM446IZ.js.map +0 -1
  37. package/dist/worker-KJZ3ZN2N.js.map +0 -1
  38. package/docs/spec-browser-routing.md +0 -108
  39. package/docs/spec-kernel-computer-controls.md +0 -166
  40. package/docs/spec-mcp-tools.md +0 -146
  41. package/docs/spec-server-instructions.md +0 -94
  42. package/docs/specs/agent-ready-seo-packet-spec.md +0 -237
  43. package/docs/specs/audit-visual-demo.md +0 -309
  44. package/docs/specs/cli-agent-wiring-spec.md +0 -206
  45. package/docs/specs/local-competitive-audit-spec.md +0 -311
  46. package/docs/specs/scheduled-workflows-api-spec.md +0 -304
  47. package/docs/specs/seo-cli-growth-roadmap-spec.md +0 -181
  48. package/docs/specs/seo-workflow-runner-and-reports-spec.md +0 -241
  49. /package/dist/{chunk-BL7BBSYF.js.map → chunk-RUGJE5EB.js.map} +0 -0
  50. /package/dist/{chunk-3K5GT6VG.js.map → chunk-YGRZU7IR.js.map} +0 -0
  51. /package/dist/{site-extract-repository-U476J44K.js.map → site-extract-repository-NVSZH35Y.js.map} +0 -0
@@ -1,237 +0,0 @@
1
- # Agent-Ready SEO Packet — Implementation Spec
2
-
3
- `agent-packet` turns live SEO evidence into a folder an AI agent can use without re-discovering context. It is both a CLI workflow and a product wedge: MCP Scraper becomes the evidence layer for Codex, Claude, Cursor, and other agents.
4
-
5
- ## User promise
6
-
7
- ```bash
8
- mcp-scraper-cli workflow run agent-packet \
9
- --keyword "roof repair Denver" \
10
- --domain example.com \
11
- --location "Denver, CO"
12
- ```
13
-
14
- Output:
15
-
16
- ```text
17
- agent-packet/
18
- report.html
19
- brief.md
20
- evidence.json
21
- sources.csv
22
- competitors.csv
23
- tasks.md
24
- agent-instructions.md
25
- manifest.json
26
- ```
27
-
28
- The packet should answer:
29
-
30
- - What does the live search landscape show?
31
- - Which sources and competitors matter?
32
- - What questions do people ask?
33
- - Is the target domain present?
34
- - What should an AI agent do next?
35
-
36
- ## Inputs
37
-
38
- CLI flags:
39
-
40
- ```bash
41
- --keyword <query> # required
42
- --domain <domain> # optional but recommended
43
- --location <city,state> # optional
44
- --max-questions <n> # default 40, max 150
45
- --include-serp # default true
46
- --include-paa # default true
47
- --include-ai-overview # default true through search/PAA tools
48
- --include-page-snapshots # default false, cap 5
49
- --output-dir <path>
50
- --return-partial # default true
51
- ```
52
-
53
- Structured input:
54
-
55
- ```ts
56
- interface AgentPacketInput {
57
- keyword: string
58
- domain?: string
59
- location?: string
60
- maxQuestions: number
61
- includeSerp: boolean
62
- includePaa: boolean
63
- includeAiOverview: boolean
64
- includePageSnapshots: boolean
65
- pageSnapshotLimit: number
66
- returnPartial: boolean
67
- }
68
- ```
69
-
70
- ## Evidence collection plan
71
-
72
- Use existing tools/API paths:
73
-
74
- 1. `search_serp`
75
- - Capture organic results, local pack when present, AI Overview detection/citations when available.
76
- - Use `location` if provided.
77
- 2. `harvest_paa`
78
- - Capture PAA tree, answers, source titles/sites/cites, forums/videos when available.
79
- - Use `maxQuestions`.
80
- 3. Optional `extract_url`
81
- - Extract target domain page if the target appears in SERP or user passes a URL later.
82
- - Extract top source pages only when `includePageSnapshots` is true.
83
-
84
- No server-side LLM is required. The workflow produces structured evidence and instructions; the calling agent can synthesize if desired.
85
-
86
- ## Artifact formats
87
-
88
- ### `evidence.json`
89
-
90
- ```ts
91
- interface AgentPacketEvidence {
92
- input: AgentPacketInput
93
- serp: {
94
- status: 'ok' | 'failed' | 'skipped'
95
- organic: Array<{ position: number; title: string; url: string; domain: string; snippet?: string | null }>
96
- localPack: Array<{ position: number; name: string; rating?: string | null; reviewCount?: string | null; websiteUrl?: string | null }>
97
- aiOverview: { detected: boolean; text?: string | null; citations: Array<{ title?: string; url?: string; domain?: string }> }
98
- }
99
- paa: {
100
- status: 'ok' | 'failed' | 'skipped'
101
- questions: Array<{ question: string; answer?: string | null; sourceTitle?: string | null; sourceSite?: string | null; sourceUrl?: string | null }>
102
- }
103
- target: {
104
- domain: string | null
105
- organicPositions: number[]
106
- citedInAiOverview: boolean
107
- citedInPaa: boolean
108
- localPackPositions: number[]
109
- }
110
- competitors: Array<{
111
- domain: string
112
- organicPositions: number[]
113
- paaMentions: number
114
- aiOverviewCitations: number
115
- sourceUrls: string[]
116
- }>
117
- warnings: string[]
118
- }
119
- ```
120
-
121
- ### `sources.csv`
122
-
123
- Columns:
124
-
125
- ```text
126
- surface,position,title,url,domain,question,answer_excerpt,is_target,is_competitor
127
- ```
128
-
129
- ### `competitors.csv`
130
-
131
- Columns:
132
-
133
- ```text
134
- domain,organic_best_position,organic_count,paa_mentions,ai_overview_citations,source_url_count
135
- ```
136
-
137
- ### `brief.md`
138
-
139
- Sections:
140
-
141
- ```md
142
- # SEO Evidence Brief
143
-
144
- ## Search Intent
145
- ## SERP Evidence
146
- ## PAA Evidence
147
- ## AI Overview Evidence
148
- ## Target Presence
149
- ## Competitor Sources
150
- ## Content Gaps
151
- ## Recommended Page Additions
152
- ```
153
-
154
- Content should be evidence-grounded. If the workflow cannot determine a gap deterministically, phrase it as an evidence prompt:
155
-
156
- ```md
157
- - Review these competitor source URLs before writing: ...
158
- - The target was not found in the captured AI/PAA sources.
159
- ```
160
-
161
- ### `tasks.md`
162
-
163
- Action checklist for an AI/code agent:
164
-
165
- ```md
166
- # Agent Tasks
167
-
168
- - [ ] Read `evidence.json`.
169
- - [ ] Compare the target domain against `competitors.csv`.
170
- - [ ] Draft or update page sections using only evidence from `sources.csv`.
171
- - [ ] Add citations/links where appropriate.
172
- - [ ] Preserve claims that can be traced to an evidence row.
173
- ```
174
-
175
- ### `agent-instructions.md`
176
-
177
- Stable instructions:
178
-
179
- ```md
180
- You are working from an MCP Scraper SEO evidence packet. Use `evidence.json` and CSV files as source of truth. Do not invent citations. If a recommendation is not supported by evidence, mark it as an assumption.
181
- ```
182
-
183
- ## HTML report
184
-
185
- Report sections:
186
-
187
- 1. Executive summary.
188
- 2. Target visibility.
189
- 3. SERP table.
190
- 4. AI Overview citations.
191
- 5. PAA source table.
192
- 6. Competitor domains.
193
- 7. Agent task list.
194
- 8. Diagnostics and warnings.
195
-
196
- ## Partial-result behavior
197
-
198
- The workflow should return partial artifacts when one surface fails.
199
-
200
- Examples:
201
-
202
- - SERP succeeds, PAA CAPTCHA-exhausts: write report with SERP evidence and PAA warning.
203
- - PAA succeeds, SERP fails: write PAA evidence and source table.
204
- - Target domain omitted: skip target-presence calculations and focus on market evidence.
205
-
206
- ## Files
207
-
208
- New:
209
-
210
- - `src/workflows/workflows/agent-packet.ts`
211
- - `src/workflows/renderers/agent-packet-report.ts`
212
- - `tests/unit/workflow-agent-packet.test.ts`
213
- - `tests/fixtures/workflows/agent-packet/`
214
-
215
- Modified:
216
-
217
- - `src/workflows/registry.ts`
218
- - `docs/specs/seo-workflow-runner-and-reports-spec.md` when implemented.
219
-
220
- ## Tests
221
-
222
- - Fixture SERP + PAA evidence produces stable `evidence.json`.
223
- - Target domain matching handles `www.` and protocol differences.
224
- - CSV escaping handles commas, quotes, and newlines.
225
- - Partial SERP/PAA failures still write manifest and report.
226
- - `agent-instructions.md` warns against unsupported citations.
227
-
228
- ## Future MCP tool
229
-
230
- After the CLI workflow is proven, consider an MCP tool:
231
-
232
- ```text
233
- seo_agent_packet
234
- ```
235
-
236
- Description should say it creates an evidence packet for downstream AI agents and returns local artifact paths. It should use `outputSchema` and structured content containing artifact paths, counts, and warnings.
237
-
@@ -1,309 +0,0 @@
1
- # Spec — Technical SEO Audit visual demo (marketing site)
2
-
3
- ## 0. Problem & decision
4
-
5
- `audit_site` is a Screaming-Frog-class technical audit, but the marketing site only describes it in
6
- bullet text. There is **no visual** of its real output (internal/external link analysis, image audit,
7
- indexability, 30+ issue types). Competitors show their crawl UI; we show prose.
8
-
9
- **Decision:** Build a self-contained, tabbed **`AuditDemo`** React component with five tabs, each
10
- rendering a realistic visual (table + bars) of one audit dimension, using sample data whose shape
11
- mirrors real `audit_site` output. Render it (a) on the **Crawl & Audit feature page** (primary) and
12
- (b) a condensed 1-tab teaser on the **homepage** (secondary). Plain-language labels, complete, not
13
- deceptive — per `[[audience-non-technical-owner]]`.
14
-
15
- All work is in **`public/app.jsx`** (single SPA source) → rebuilt to `public/app.js` via
16
- `npm run build:ui`. No backend changes. Website-only deploy (no version bump, no npm publish).
17
-
18
- ## 1. Ground truth — what audit_site really emits (cite when writing sample data)
19
-
20
- | Dimension | Source file | Fields the demo must mirror |
21
- |---|---|---|
22
- | Issues | `src/api/seo-issues.ts` (`computeIssues`, L34–66) | issue key (e.g. `title.duplicate`), `count`, severity (🔴 broken/error, 🟠 missing-title/h1/indexability/canonical, 🟡 else — see L85–87), example URLs |
23
- | Internal links | `src/api/seo-link-report.ts` (`LinkReportSummary.internal`, L6–15) | `totalLinks`, `pages`, `orphans`, `brokenInternal`, `avgInlinks`, `distribution{zero,oneToTwo,threeToTen,elevenPlus}`, `topByInlinks[]{url,inlinks,outlinksInternal,outlinksExternal}` (top 20) |
24
- | External links | `seo-link-report.ts` (`.external` + `ExtDomainRow` L3) | `totalLinks`, `uniqueDomains`, `domains[]{domain,links,nofollow,pages}` |
25
- | Images | `src/api/image-audit.ts` (`ImageRow` L8–12, `ImageAuditSummary` L17–24) | per-row `{url,size,format,over100kb,legacyFormat}`; summary `{unique,totalSize,avgSize,over100kb,legacyFormat}` |
26
- | Per-page indexability | `pages.jsonl` fields (`PageData`) | `url,status,indexable,canonicalUrl,wordCount,schemaTypes[],inlinks,crawlDepth` |
27
-
28
- Real headline line for reference (`mcp-response-formatter.ts:768`):
29
- `"{internal} internal / {external} external links · {orphans} orphans · {broken} broken · avg {n} inlinks/page · top external: {domain}"`.
30
-
31
- ## 2. New data constant — `AUDIT_DEMO`
32
-
33
- Insert **immediately above** `function FeatureSection(` (currently near `public/app.jsx:2039`, just
34
- before `var f = FEATURES[slug]...`). One module-level const, no interpolation.
35
-
36
- ```jsx
37
- var AUDIT_DEMO = {
38
- site: 'example.com',
39
- pagesCrawled: 1284,
40
- stats: [
41
- { k: 'Pages crawled', v: '1,284' },
42
- { k: 'Issues found', v: '37 types' },
43
- { k: 'Internal links', v: '18,402' },
44
- { k: 'Images', v: '512' },
45
- ],
46
- issues: [
47
- { sev: '🔴', label: 'Broken internal links (404)', key: 'links.brokenInternal', count: 23 },
48
- { sev: '🔴', label: 'Pages returning 5xx errors', key: 'response.error5xx', count: 4 },
49
- { sev: '🟠', label: 'Missing page title', key: 'title.missing', count: 11 },
50
- { sev: '🟠', label: 'Missing H1', key: 'h1.missing', count: 19 },
51
- { sev: '🟠', label: 'Set to noindex', key: 'indexability.noindex', count: 8 },
52
- { sev: '🟠', label: 'Missing canonical tag', key: 'canonical.missing', count: 27 },
53
- { sev: '🟡', label: 'Duplicate title', key: 'title.duplicate', count: 34 },
54
- { sev: '🟡', label: 'Duplicate meta description', key: 'meta.duplicate', count: 41 },
55
- { sev: '🟡', label: 'Thin content (under 200 words)', key: 'content.thin', count: 52 },
56
- { sev: '🟡', label: 'Images missing alt text', key: 'images.missingAlt', count: 88 },
57
- { sev: '🟡', label: 'Missing schema markup', key: 'schema.missing', count: 63 },
58
- { sev: '🟡', label: 'Title longer than 60 chars', key: 'title.tooLong', count: 29 },
59
- ],
60
- internal: {
61
- totalLinks: 18402, orphans: 14, brokenInternal: 23, avgInlinks: 9.7,
62
- distribution: { zero: 14, oneToTwo: 121, threeToTen: 818, elevenPlus: 331 },
63
- top: [
64
- { url: '/', inlinks: 1284, out: 42 },
65
- { url: '/pricing', inlinks: 1190, out: 28 },
66
- { url: '/blog', inlinks: false ? 0 : 642, out: 35 },
67
- { url: '/features/seo-audit', inlinks: 318, out: 22 },
68
- { url: '/docs', inlinks: 291, out: 51 },
69
- { url: '/blog/best-ramen-nyc', inlinks: 47, out: 18 },
70
- ],
71
- },
72
- external: {
73
- totalLinks: 1903, uniqueDomains: 142,
74
- domains: [
75
- { domain: 'twitter.com', links: 311, nofollow: 311, pages: 290 },
76
- { domain: 'linkedin.com', links: 287, nofollow: 287, pages: 281 },
77
- { domain: 'github.com', links: 142, nofollow: 0, pages: 96 },
78
- { domain: 'youtube.com', links: 88, nofollow: 12, pages: 61 },
79
- { domain: 'stripe.com', links: 54, nofollow: 0, pages: 40 },
80
- ],
81
- },
82
- images: {
83
- unique: 512, totalSize: '47.3 MB', avgSize: '94 KB', over100kb: 138, legacyFormat: 74,
84
- heaviest: [
85
- { url: '/img/hero-bg.png', size: '2.4 MB', format: 'png', over: true, legacy: true },
86
- { url: '/img/team-photo.jpg', size: '1.1 MB', format: 'jpeg', over: true, legacy: true },
87
- { url: '/img/product-shot.png', size: '870 KB', format: 'png', over: true, legacy: true },
88
- { url: '/img/og-card.jpg', size: '512 KB', format: 'jpeg', over: true, legacy: true },
89
- { url: '/img/icon-sprite.webp', size: '210 KB', format: 'webp', over: true, legacy: false },
90
- ],
91
- },
92
- pages: [
93
- { url: '/', status: 200, indexable: true, canonical: true, words: 842, schema: 3 },
94
- { url: '/pricing', status: 200, indexable: true, canonical: true, words: 1190, schema: 2 },
95
- { url: '/blog/best-ramen-nyc', status: 200, indexable: true, canonical: true, words: 2847, schema: 4 },
96
- { url: '/old-landing', status: 301, indexable: false, canonical: false, words: 0, schema: 0 },
97
- { url: '/checkout', status: 200, indexable: false, canonical: true, words: 120, schema: 1 },
98
- { url: '/team', status: 404, indexable: false, canonical: false, words: 0, schema: 0 },
99
- ],
100
- };
101
- ```
102
- (Note: the `false ? 0 : 642` form avoids a literal `Math` call — keep plain literals; replace with `642`.)
103
-
104
- ## 3. New component — `AuditDemo`
105
-
106
- Insert as a **standalone function** immediately above `function FeatureSection(`. Uses one
107
- `useState` for the active tab. Reuses existing tokens (`var(--surface)`, `var(--border)`,
108
- `var(--accent)`, `var(--text-muted)`, `var(--font-mono)`) and the existing `data-tab` /
109
- `data-tab--active` classes already in the stylesheet (used at `app.jsx:2384`). One small reusable
110
- `Bar` helper for the horizontal bars.
111
-
112
- ```jsx
113
- function AuditBar({ value, max, color }) {
114
- var pct = max > 0 ? Math.round((value / max) * 100) : 0;
115
- return (
116
- <div style={{height:7,background:'var(--bg-2)',borderRadius:99,overflow:'hidden',flex:1}}>
117
- <div style={{width:pct+'%',height:'100%',background:color||'var(--accent)',borderRadius:99}}/>
118
- </div>
119
- );
120
- }
121
-
122
- function AuditDemo({ compact }) {
123
- var TABS = compact
124
- ? [['issues','Issues'],['internal','Internal links'],['images','Images']]
125
- : [['issues','Issues'],['internal','Internal links'],['external','External domains'],['images','Images'],['pages','Indexability']];
126
- var [tab, setTab] = useState('issues');
127
- var d = AUDIT_DEMO;
128
- var th = { textAlign:'left', fontFamily:'var(--font-mono)', fontSize:10.5, letterSpacing:'.08em',
129
- textTransform:'uppercase', color:'var(--text-subtle)', padding:'7px 10px', fontWeight:600 };
130
- var td = { padding:'8px 10px', fontSize:12.5, borderTop:'1px solid var(--border)', color:'var(--text-muted)' };
131
- var mono = { fontFamily:'var(--font-mono)', fontSize:12, color:'var(--text)' };
132
- var distMax = Math.max(d.internal.distribution.zero, d.internal.distribution.oneToTwo,
133
- d.internal.distribution.threeToTen, d.internal.distribution.elevenPlus);
134
- var issueMax = Math.max.apply(null, d.issues.map(function(i){ return i.count; }));
135
-
136
- return (
137
- <div style={{border:'1px solid var(--border)',borderRadius:'var(--radius-lg)',background:'var(--surface)',overflow:'hidden'}}>
138
- {/* header */}
139
- <div style={{display:'flex',alignItems:'center',justifyContent:'space-between',gap:12,padding:'12px 14px',borderBottom:'1px solid var(--border)',flexWrap:'wrap'}}>
140
- <div style={{fontFamily:'var(--font-mono)',fontSize:12.5,color:'var(--text-muted)'}}>
141
- <span style={{color:'var(--accent)'}}>audit_site</span> {d.site} · {d.pagesCrawled.toLocaleString()} pages
142
- </div>
143
- <div style={{display:'flex',gap:14,flexWrap:'wrap'}}>
144
- {d.stats.map(function(s){ return (
145
- <div key={s.k} style={{fontSize:11.5,color:'var(--text-subtle)'}}>
146
- <b style={{color:'var(--text)',fontFamily:'var(--font-mono)'}}>{s.v}</b> {s.k}
147
- </div>
148
- ); })}
149
- </div>
150
- </div>
151
- {/* tab bar */}
152
- <div style={{display:'flex',gap:6,padding:'10px 12px',borderBottom:'1px solid var(--border)',flexWrap:'wrap'}}>
153
- {TABS.map(function(t){
154
- var on = tab===t[0];
155
- return <button key={t[0]} className={'data-tab'+(on?' data-tab--active':'')} onClick={function(){ setTab(t[0]); }}>{t[1]}</button>;
156
- })}
157
- </div>
158
- {/* panels */}
159
- <div style={{padding:'4px 6px 10px'}}>
160
- {tab==='issues' && (
161
- <table style={{width:'100%',borderCollapse:'collapse'}}>
162
- <thead><tr><th style={th}></th><th style={th}>Issue</th><th style={Object.assign({},th,{textAlign:'right'})}>Pages</th><th style={Object.assign({},th,{width:'34%'})}></th></tr></thead>
163
- <tbody>{d.issues.map(function(i){ return (
164
- <tr key={i.key}>
165
- <td style={Object.assign({},td,{width:24})}>{i.sev}</td>
166
- <td style={td}>{i.label} <span style={{color:'var(--text-subtle)',fontFamily:'var(--font-mono)',fontSize:10.5}}>{i.key}</span></td>
167
- <td style={Object.assign({},td,mono,{textAlign:'right'})}>{i.count}</td>
168
- <td style={td}><AuditBar value={i.count} max={issueMax} color={i.sev==='🔴'?'#dc2626':i.sev==='🟠'?'#ea580c':'var(--accent)'}/></td>
169
- </tr>
170
- ); })}</tbody>
171
- </table>
172
- )}
173
- {tab==='internal' && (
174
- <div>
175
- <div style={{display:'flex',gap:14,flexWrap:'wrap',padding:'8px 10px'}}>
176
- {[['Total internal links',d.internal.totalLinks.toLocaleString()],['Orphan pages',d.internal.orphans],['Broken',d.internal.brokenInternal],['Avg inlinks / page',d.internal.avgInlinks]].map(function(s){
177
- return <div key={s[0]} style={{fontSize:11.5,color:'var(--text-subtle)'}}><b style={{color:'var(--text)',fontFamily:'var(--font-mono)',fontSize:14,display:'block'}}>{s[1]}</b>{s[0]}</div>;
178
- })}
179
- </div>
180
- <div style={{padding:'4px 10px 10px'}}>
181
- <div style={{fontFamily:'var(--font-mono)',fontSize:10.5,letterSpacing:'.1em',textTransform:'uppercase',color:'var(--text-subtle)',margin:'4px 0 8px'}}>Inlink distribution</div>
182
- {[['0 (orphan)',d.internal.distribution.zero,'#dc2626'],['1–2',d.internal.distribution.oneToTwo,'#ea580c'],['3–10',d.internal.distribution.threeToTen,'var(--accent)'],['11+',d.internal.distribution.elevenPlus,'var(--accent)']].map(function(r){
183
- return <div key={r[0]} style={{display:'flex',alignItems:'center',gap:10,marginBottom:6}}>
184
- <span style={{width:64,fontSize:11.5,color:'var(--text-muted)',fontFamily:'var(--font-mono)'}}>{r[0]}</span>
185
- <AuditBar value={r[1]} max={distMax} color={r[2]}/>
186
- <span style={{width:34,textAlign:'right',fontSize:11.5,fontFamily:'var(--font-mono)',color:'var(--text)'}}>{r[1]}</span>
187
- </div>;
188
- })}
189
- </div>
190
- <table style={{width:'100%',borderCollapse:'collapse',marginTop:6}}>
191
- <thead><tr><th style={th}>Top pages by inlinks</th><th style={Object.assign({},th,{textAlign:'right'})}>Inlinks</th><th style={Object.assign({},th,{textAlign:'right'})}>Outlinks</th></tr></thead>
192
- <tbody>{d.internal.top.map(function(p){ return (
193
- <tr key={p.url}><td style={Object.assign({},td,mono)}>{p.url}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{p.inlinks.toLocaleString()}</td><td style={Object.assign({},td,{textAlign:'right'})}>{p.out}</td></tr>
194
- ); })}</tbody>
195
- </table>
196
- </div>
197
- )}
198
- {tab==='external' && (
199
- <table style={{width:'100%',borderCollapse:'collapse'}}>
200
- <thead><tr><th style={th}>External domain</th><th style={Object.assign({},th,{textAlign:'right'})}>Links</th><th style={Object.assign({},th,{textAlign:'right'})}>Nofollow</th><th style={Object.assign({},th,{textAlign:'right'})}>From pages</th></tr></thead>
201
- <tbody>{d.external.domains.map(function(x){ return (
202
- <tr key={x.domain}><td style={Object.assign({},td,mono)}>{x.domain}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{x.links}</td><td style={Object.assign({},td,{textAlign:'right'})}>{x.nofollow}</td><td style={Object.assign({},td,{textAlign:'right'})}>{x.pages}</td></tr>
203
- ); })}</tbody>
204
- </table>
205
- )}
206
- {tab==='images' && (
207
- <div>
208
- <div style={{display:'flex',gap:14,flexWrap:'wrap',padding:'8px 10px'}}>
209
- {[['Unique images',d.images.unique],['Total weight',d.images.totalSize],['Average',d.images.avgSize],['Over 100 KB',d.images.over100kb],['Legacy format',d.images.legacyFormat]].map(function(s){
210
- return <div key={s[0]} style={{fontSize:11.5,color:'var(--text-subtle)'}}><b style={{color:'var(--text)',fontFamily:'var(--font-mono)',fontSize:14,display:'block'}}>{s[1]}</b>{s[0]}</div>;
211
- })}
212
- </div>
213
- <table style={{width:'100%',borderCollapse:'collapse',marginTop:4}}>
214
- <thead><tr><th style={th}>Heaviest images</th><th style={Object.assign({},th,{textAlign:'right'})}>Size</th><th style={th}>Format</th><th style={th}>Flags</th></tr></thead>
215
- <tbody>{d.images.heaviest.map(function(im){ return (
216
- <tr key={im.url}><td style={Object.assign({},td,mono)}>{im.url}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{im.size}</td><td style={td}>{im.format}</td>
217
- <td style={td}>{im.over && <span style={{color:'#ea580c'}}>over 100KB</span>}{im.over && im.legacy ? ' · ' : ''}{im.legacy && <span style={{color:'#dc2626'}}>legacy</span>}</td></tr>
218
- ); })}</tbody>
219
- </table>
220
- </div>
221
- )}
222
- {tab==='pages' && (
223
- <table style={{width:'100%',borderCollapse:'collapse'}}>
224
- <thead><tr><th style={th}>URL</th><th style={th}>Status</th><th style={th}>Indexable</th><th style={th}>Canonical</th><th style={Object.assign({},th,{textAlign:'right'})}>Words</th><th style={Object.assign({},th,{textAlign:'right'})}>Schema</th></tr></thead>
225
- <tbody>{d.pages.map(function(p){ return (
226
- <tr key={p.url}>
227
- <td style={Object.assign({},td,mono)}>{p.url}</td>
228
- <td style={Object.assign({},td,{color:p.status>=400?'#dc2626':p.status>=300?'#ea580c':'var(--text-muted)'})}>{p.status}</td>
229
- <td style={td}>{p.indexable?'✓':'—'}</td>
230
- <td style={td}>{p.canonical?'✓':'—'}</td>
231
- <td style={Object.assign({},td,mono,{textAlign:'right'})}>{p.words.toLocaleString()}</td>
232
- <td style={Object.assign({},td,{textAlign:'right'})}>{p.schema}</td>
233
- </tr>
234
- ); })}</tbody>
235
- </table>
236
- )}
237
- </div>
238
- </div>
239
- );
240
- }
241
- ```
242
-
243
- ## 4. Wiring — render points
244
-
245
- ### 4a. Feature page (primary)
246
- In `FeatureSection` (`app.jsx:2041` render), insert the demo **between** the `skill-section` block
247
- (closes ~`2079`) and the `More capabilities` block (opens ~`2081`). Gate to the audit feature only:
248
-
249
- ```jsx
250
- {slug === 'web' && (
251
- <div style={{marginTop:48}}>
252
- <div className="section-kicker">See it in action</div>
253
- <h2 className="how__title" style={{margin:'6px 0 16px'}}>A full technical SEO audit, not just a crawl</h2>
254
- <AuditDemo />
255
- </div>
256
- )}
257
- ```
258
- `slug` is already in scope (`FeatureSection({ slug, ... })`, `var f = FEATURES[slug]`).
259
-
260
- ### 4b. Homepage teaser (secondary, optional but recommended)
261
- On the landing page, after the existing data-tabs/source-tiles section, drop a compact instance:
262
- ```jsx
263
- <div style={{maxWidth:760,margin:'28px auto 0'}}><AuditDemo compact /></div>
264
- ```
265
- Place near the `sourceTiles` render (search `sourceTiles.map` in `app.jsx`) so it sits with the
266
- other "what you get" proof. Compact mode shows Issues / Internal links / Images only.
267
-
268
- ## 5. Copy updates (same edit pass — make the claims match the visual)
269
-
270
- 1. **Feature bullets** (`FEATURES.web.bullets`, `app.jsx:1955–1958`) — split the single audit bullet
271
- into the specifics the demo now proves:
272
- - `'Technical SEO audit: 30+ issue checks across titles, meta, headings, indexability, schema'`
273
- - `'Internal link analysis: inlinks per page, orphan detection, link distribution, broken links'`
274
- - `'External link profile: top linked domains, nofollow ratio, pages linking out'`
275
- - `'Image audit: weight & format per image, over-100KB and legacy-format flags'`
276
- 2. **Source tile** (`app.jsx:2386`) — keep, it's accurate.
277
- 3. **`FEATURES.web.sub`** (`app.jsx:1952`) — already good; no change.
278
-
279
- ## 6. CSS check
280
- `.data-tab` / `.data-tab--active` already exist (used at `app.jsx:2384`). Confirm they style a pill
281
- button; if `.data-tab` lacks base button reset, the inline usage still works because we only rely on
282
- its existing visual. **No new CSS required.** If tables need zebra/scroll on mobile, wrap each
283
- `<table>` in `<div style={{overflowX:'auto'}}>` (already safe at 414px since columns are few).
284
-
285
- ## 7. Build, verify, deploy (website-only)
286
-
287
- ```bash
288
- cd /Users/vilovieta/Desktop/mcp-scraper
289
- npm run build:ui # rebuild public/app.js from app.jsx
290
- grep -c "AuditDemo" public/app.js # expect >=2 (def + render)
291
- git add public/app.jsx public/app.js
292
- git commit -m "feat(site): tabbed technical-SEO-audit visual demo (issues, links, images, indexability)"
293
- git push origin main
294
- /opt/homebrew/bin/vercel --prod --yes # deploy; no version bump, no npm publish
295
- ```
296
- Then load `https://mcpscraper.dev/features/web` (or whatever the audit feature route is — confirm via
297
- `FEATURE_ORDER` / the router) and click each tab; load the homepage for the compact teaser.
298
-
299
- ## 8. Acceptance criteria
300
- - [ ] Feature page renders a 5-tab audit visual; every tab shows a populated table/bars.
301
- - [ ] Numbers in `AUDIT_DEMO` are internally consistent (e.g. distribution buckets sum ≈ pages).
302
- - [ ] Issue rows show severity color matching `seo-issues.ts` severity rules.
303
- - [ ] Homepage shows the 3-tab compact teaser.
304
- - [ ] Renders cleanly at 414px (mobile) — no horizontal overflow of the page.
305
- - [ ] No backend/test changes; `npm run build:ui` clean; deployed to prod.
306
-
307
- ## 9. Out of scope / follow-ups
308
- - Live "run an audit on your URL" interactive demo (would hit `/extract-site` + cost credits).
309
- - Wiring the demo to real cached audit JSON instead of `AUDIT_DEMO` sample.