mcp-scraper 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/bin/api-server.cjs +296 -198
- package/dist/bin/api-server.cjs.map +1 -1
- package/dist/bin/api-server.js +2 -2
- package/dist/bin/mcp-scraper-cli.cjs +1 -1
- package/dist/bin/mcp-scraper-cli.cjs.map +1 -1
- package/dist/bin/mcp-scraper-cli.js +1 -1
- package/dist/bin/mcp-scraper-install.cjs +1 -1
- package/dist/bin/mcp-scraper-install.cjs.map +1 -1
- package/dist/bin/mcp-scraper-install.js +1 -1
- package/dist/bin/mcp-stdio-server.cjs +3 -3
- package/dist/bin/mcp-stdio-server.cjs.map +1 -1
- package/dist/bin/mcp-stdio-server.js +3 -3
- package/dist/bin/paa-harvest.cjs +166 -74
- package/dist/bin/paa-harvest.cjs.map +1 -1
- package/dist/bin/paa-harvest.js +1 -1
- package/dist/chunk-EGWJ74EX.js +7 -0
- package/dist/chunk-EGWJ74EX.js.map +1 -0
- package/dist/{chunk-APJO2XV5.js → chunk-HL33CGJF.js} +3 -3
- package/dist/{chunk-APJO2XV5.js.map → chunk-HL33CGJF.js.map} +1 -1
- package/dist/{chunk-AUCXKRRH.js → chunk-RMPPYKUV.js} +167 -75
- package/dist/chunk-RMPPYKUV.js.map +1 -0
- package/dist/{chunk-BL7BBSYF.js → chunk-RUGJE5EB.js} +2 -2
- package/dist/{chunk-3K5GT6VG.js → chunk-YGRZU7IR.js} +3 -3
- package/dist/index.cjs +166 -74
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +1 -1
- package/dist/{server-WHM446IZ.js → server-JNY4XPZE.js} +81 -76
- package/dist/server-JNY4XPZE.js.map +1 -0
- package/dist/{site-extract-repository-U476J44K.js → site-extract-repository-NVSZH35Y.js} +3 -3
- package/dist/{worker-KJZ3ZN2N.js → worker-JQTS437L.js} +4 -4
- package/dist/worker-JQTS437L.js.map +1 -0
- package/package.json +1 -1
- package/dist/chunk-AUCXKRRH.js.map +0 -1
- package/dist/chunk-F2BXETDX.js +0 -7
- package/dist/chunk-F2BXETDX.js.map +0 -1
- package/dist/server-WHM446IZ.js.map +0 -1
- package/dist/worker-KJZ3ZN2N.js.map +0 -1
- package/docs/spec-browser-routing.md +0 -108
- package/docs/spec-kernel-computer-controls.md +0 -166
- package/docs/spec-mcp-tools.md +0 -146
- package/docs/spec-server-instructions.md +0 -94
- package/docs/specs/agent-ready-seo-packet-spec.md +0 -237
- package/docs/specs/audit-visual-demo.md +0 -309
- package/docs/specs/cli-agent-wiring-spec.md +0 -206
- package/docs/specs/local-competitive-audit-spec.md +0 -311
- package/docs/specs/scheduled-workflows-api-spec.md +0 -304
- package/docs/specs/seo-cli-growth-roadmap-spec.md +0 -181
- package/docs/specs/seo-workflow-runner-and-reports-spec.md +0 -241
- /package/dist/{chunk-BL7BBSYF.js.map → chunk-RUGJE5EB.js.map} +0 -0
- /package/dist/{chunk-3K5GT6VG.js.map → chunk-YGRZU7IR.js.map} +0 -0
- /package/dist/{site-extract-repository-U476J44K.js.map → site-extract-repository-NVSZH35Y.js.map} +0 -0
|
@@ -1,237 +0,0 @@
|
|
|
1
|
-
# Agent-Ready SEO Packet — Implementation Spec
|
|
2
|
-
|
|
3
|
-
`agent-packet` turns live SEO evidence into a folder an AI agent can use without re-discovering context. It is both a CLI workflow and a product wedge: MCP Scraper becomes the evidence layer for Codex, Claude, Cursor, and other agents.
|
|
4
|
-
|
|
5
|
-
## User promise
|
|
6
|
-
|
|
7
|
-
```bash
|
|
8
|
-
mcp-scraper-cli workflow run agent-packet \
|
|
9
|
-
--keyword "roof repair Denver" \
|
|
10
|
-
--domain example.com \
|
|
11
|
-
--location "Denver, CO"
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
Output:
|
|
15
|
-
|
|
16
|
-
```text
|
|
17
|
-
agent-packet/
|
|
18
|
-
report.html
|
|
19
|
-
brief.md
|
|
20
|
-
evidence.json
|
|
21
|
-
sources.csv
|
|
22
|
-
competitors.csv
|
|
23
|
-
tasks.md
|
|
24
|
-
agent-instructions.md
|
|
25
|
-
manifest.json
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
The packet should answer:
|
|
29
|
-
|
|
30
|
-
- What does the live search landscape show?
|
|
31
|
-
- Which sources and competitors matter?
|
|
32
|
-
- What questions do people ask?
|
|
33
|
-
- Is the target domain present?
|
|
34
|
-
- What should an AI agent do next?
|
|
35
|
-
|
|
36
|
-
## Inputs
|
|
37
|
-
|
|
38
|
-
CLI flags:
|
|
39
|
-
|
|
40
|
-
```bash
|
|
41
|
-
--keyword <query> # required
|
|
42
|
-
--domain <domain> # optional but recommended
|
|
43
|
-
--location <city,state> # optional
|
|
44
|
-
--max-questions <n> # default 40, max 150
|
|
45
|
-
--include-serp # default true
|
|
46
|
-
--include-paa # default true
|
|
47
|
-
--include-ai-overview # default true through search/PAA tools
|
|
48
|
-
--include-page-snapshots # default false, cap 5
|
|
49
|
-
--output-dir <path>
|
|
50
|
-
--return-partial # default true
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
Structured input:
|
|
54
|
-
|
|
55
|
-
```ts
|
|
56
|
-
interface AgentPacketInput {
|
|
57
|
-
keyword: string
|
|
58
|
-
domain?: string
|
|
59
|
-
location?: string
|
|
60
|
-
maxQuestions: number
|
|
61
|
-
includeSerp: boolean
|
|
62
|
-
includePaa: boolean
|
|
63
|
-
includeAiOverview: boolean
|
|
64
|
-
includePageSnapshots: boolean
|
|
65
|
-
pageSnapshotLimit: number
|
|
66
|
-
returnPartial: boolean
|
|
67
|
-
}
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
## Evidence collection plan
|
|
71
|
-
|
|
72
|
-
Use existing tools/API paths:
|
|
73
|
-
|
|
74
|
-
1. `search_serp`
|
|
75
|
-
- Capture organic results, local pack when present, AI Overview detection/citations when available.
|
|
76
|
-
- Use `location` if provided.
|
|
77
|
-
2. `harvest_paa`
|
|
78
|
-
- Capture PAA tree, answers, source titles/sites/cites, forums/videos when available.
|
|
79
|
-
- Use `maxQuestions`.
|
|
80
|
-
3. Optional `extract_url`
|
|
81
|
-
- Extract target domain page if the target appears in SERP or user passes a URL later.
|
|
82
|
-
- Extract top source pages only when `includePageSnapshots` is true.
|
|
83
|
-
|
|
84
|
-
No server-side LLM is required. The workflow produces structured evidence and instructions; the calling agent can synthesize if desired.
|
|
85
|
-
|
|
86
|
-
## Artifact formats
|
|
87
|
-
|
|
88
|
-
### `evidence.json`
|
|
89
|
-
|
|
90
|
-
```ts
|
|
91
|
-
interface AgentPacketEvidence {
|
|
92
|
-
input: AgentPacketInput
|
|
93
|
-
serp: {
|
|
94
|
-
status: 'ok' | 'failed' | 'skipped'
|
|
95
|
-
organic: Array<{ position: number; title: string; url: string; domain: string; snippet?: string | null }>
|
|
96
|
-
localPack: Array<{ position: number; name: string; rating?: string | null; reviewCount?: string | null; websiteUrl?: string | null }>
|
|
97
|
-
aiOverview: { detected: boolean; text?: string | null; citations: Array<{ title?: string; url?: string; domain?: string }> }
|
|
98
|
-
}
|
|
99
|
-
paa: {
|
|
100
|
-
status: 'ok' | 'failed' | 'skipped'
|
|
101
|
-
questions: Array<{ question: string; answer?: string | null; sourceTitle?: string | null; sourceSite?: string | null; sourceUrl?: string | null }>
|
|
102
|
-
}
|
|
103
|
-
target: {
|
|
104
|
-
domain: string | null
|
|
105
|
-
organicPositions: number[]
|
|
106
|
-
citedInAiOverview: boolean
|
|
107
|
-
citedInPaa: boolean
|
|
108
|
-
localPackPositions: number[]
|
|
109
|
-
}
|
|
110
|
-
competitors: Array<{
|
|
111
|
-
domain: string
|
|
112
|
-
organicPositions: number[]
|
|
113
|
-
paaMentions: number
|
|
114
|
-
aiOverviewCitations: number
|
|
115
|
-
sourceUrls: string[]
|
|
116
|
-
}>
|
|
117
|
-
warnings: string[]
|
|
118
|
-
}
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
### `sources.csv`
|
|
122
|
-
|
|
123
|
-
Columns:
|
|
124
|
-
|
|
125
|
-
```text
|
|
126
|
-
surface,position,title,url,domain,question,answer_excerpt,is_target,is_competitor
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
### `competitors.csv`
|
|
130
|
-
|
|
131
|
-
Columns:
|
|
132
|
-
|
|
133
|
-
```text
|
|
134
|
-
domain,organic_best_position,organic_count,paa_mentions,ai_overview_citations,source_url_count
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
### `brief.md`
|
|
138
|
-
|
|
139
|
-
Sections:
|
|
140
|
-
|
|
141
|
-
```md
|
|
142
|
-
# SEO Evidence Brief
|
|
143
|
-
|
|
144
|
-
## Search Intent
|
|
145
|
-
## SERP Evidence
|
|
146
|
-
## PAA Evidence
|
|
147
|
-
## AI Overview Evidence
|
|
148
|
-
## Target Presence
|
|
149
|
-
## Competitor Sources
|
|
150
|
-
## Content Gaps
|
|
151
|
-
## Recommended Page Additions
|
|
152
|
-
```
|
|
153
|
-
|
|
154
|
-
Content should be evidence-grounded. If the workflow cannot determine a gap deterministically, phrase it as an evidence prompt:
|
|
155
|
-
|
|
156
|
-
```md
|
|
157
|
-
- Review these competitor source URLs before writing: ...
|
|
158
|
-
- The target was not found in the captured AI/PAA sources.
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
### `tasks.md`
|
|
162
|
-
|
|
163
|
-
Action checklist for an AI/code agent:
|
|
164
|
-
|
|
165
|
-
```md
|
|
166
|
-
# Agent Tasks
|
|
167
|
-
|
|
168
|
-
- [ ] Read `evidence.json`.
|
|
169
|
-
- [ ] Compare the target domain against `competitors.csv`.
|
|
170
|
-
- [ ] Draft or update page sections using only evidence from `sources.csv`.
|
|
171
|
-
- [ ] Add citations/links where appropriate.
|
|
172
|
-
- [ ] Preserve claims that can be traced to an evidence row.
|
|
173
|
-
```
|
|
174
|
-
|
|
175
|
-
### `agent-instructions.md`
|
|
176
|
-
|
|
177
|
-
Stable instructions:
|
|
178
|
-
|
|
179
|
-
```md
|
|
180
|
-
You are working from an MCP Scraper SEO evidence packet. Use `evidence.json` and CSV files as source of truth. Do not invent citations. If a recommendation is not supported by evidence, mark it as an assumption.
|
|
181
|
-
```
|
|
182
|
-
|
|
183
|
-
## HTML report
|
|
184
|
-
|
|
185
|
-
Report sections:
|
|
186
|
-
|
|
187
|
-
1. Executive summary.
|
|
188
|
-
2. Target visibility.
|
|
189
|
-
3. SERP table.
|
|
190
|
-
4. AI Overview citations.
|
|
191
|
-
5. PAA source table.
|
|
192
|
-
6. Competitor domains.
|
|
193
|
-
7. Agent task list.
|
|
194
|
-
8. Diagnostics and warnings.
|
|
195
|
-
|
|
196
|
-
## Partial-result behavior
|
|
197
|
-
|
|
198
|
-
The workflow should return partial artifacts when one surface fails.
|
|
199
|
-
|
|
200
|
-
Examples:
|
|
201
|
-
|
|
202
|
-
- SERP succeeds, PAA CAPTCHA-exhausts: write report with SERP evidence and PAA warning.
|
|
203
|
-
- PAA succeeds, SERP fails: write PAA evidence and source table.
|
|
204
|
-
- Target domain omitted: skip target-presence calculations and focus on market evidence.
|
|
205
|
-
|
|
206
|
-
## Files
|
|
207
|
-
|
|
208
|
-
New:
|
|
209
|
-
|
|
210
|
-
- `src/workflows/workflows/agent-packet.ts`
|
|
211
|
-
- `src/workflows/renderers/agent-packet-report.ts`
|
|
212
|
-
- `tests/unit/workflow-agent-packet.test.ts`
|
|
213
|
-
- `tests/fixtures/workflows/agent-packet/`
|
|
214
|
-
|
|
215
|
-
Modified:
|
|
216
|
-
|
|
217
|
-
- `src/workflows/registry.ts`
|
|
218
|
-
- `docs/specs/seo-workflow-runner-and-reports-spec.md` when implemented.
|
|
219
|
-
|
|
220
|
-
## Tests
|
|
221
|
-
|
|
222
|
-
- Fixture SERP + PAA evidence produces stable `evidence.json`.
|
|
223
|
-
- Target domain matching handles `www.` and protocol differences.
|
|
224
|
-
- CSV escaping handles commas, quotes, and newlines.
|
|
225
|
-
- Partial SERP/PAA failures still write manifest and report.
|
|
226
|
-
- `agent-instructions.md` warns against unsupported citations.
|
|
227
|
-
|
|
228
|
-
## Future MCP tool
|
|
229
|
-
|
|
230
|
-
After the CLI workflow is proven, consider an MCP tool:
|
|
231
|
-
|
|
232
|
-
```text
|
|
233
|
-
seo_agent_packet
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
Description should say it creates an evidence packet for downstream AI agents and returns local artifact paths. It should use `outputSchema` and structured content containing artifact paths, counts, and warnings.
|
|
237
|
-
|
|
@@ -1,309 +0,0 @@
|
|
|
1
|
-
# Spec — Technical SEO Audit visual demo (marketing site)
|
|
2
|
-
|
|
3
|
-
## 0. Problem & decision
|
|
4
|
-
|
|
5
|
-
`audit_site` is a Screaming-Frog-class technical audit, but the marketing site only describes it in
|
|
6
|
-
bullet text. There is **no visual** of its real output (internal/external link analysis, image audit,
|
|
7
|
-
indexability, 30+ issue types). Competitors show their crawl UI; we show prose.
|
|
8
|
-
|
|
9
|
-
**Decision:** Build a self-contained, tabbed **`AuditDemo`** React component with five tabs, each
|
|
10
|
-
rendering a realistic visual (table + bars) of one audit dimension, using sample data whose shape
|
|
11
|
-
mirrors real `audit_site` output. Render it (a) on the **Crawl & Audit feature page** (primary) and
|
|
12
|
-
(b) a condensed 1-tab teaser on the **homepage** (secondary). Plain-language labels, complete, not
|
|
13
|
-
deceptive — per `[[audience-non-technical-owner]]`.
|
|
14
|
-
|
|
15
|
-
All work is in **`public/app.jsx`** (single SPA source) → rebuilt to `public/app.js` via
|
|
16
|
-
`npm run build:ui`. No backend changes. Website-only deploy (no version bump, no npm publish).
|
|
17
|
-
|
|
18
|
-
## 1. Ground truth — what audit_site really emits (cite when writing sample data)
|
|
19
|
-
|
|
20
|
-
| Dimension | Source file | Fields the demo must mirror |
|
|
21
|
-
|---|---|---|
|
|
22
|
-
| Issues | `src/api/seo-issues.ts` (`computeIssues`, L34–66) | issue key (e.g. `title.duplicate`), `count`, severity (🔴 broken/error, 🟠 missing-title/h1/indexability/canonical, 🟡 else — see L85–87), example URLs |
|
|
23
|
-
| Internal links | `src/api/seo-link-report.ts` (`LinkReportSummary.internal`, L6–15) | `totalLinks`, `pages`, `orphans`, `brokenInternal`, `avgInlinks`, `distribution{zero,oneToTwo,threeToTen,elevenPlus}`, `topByInlinks[]{url,inlinks,outlinksInternal,outlinksExternal}` (top 20) |
|
|
24
|
-
| External links | `seo-link-report.ts` (`.external` + `ExtDomainRow` L3) | `totalLinks`, `uniqueDomains`, `domains[]{domain,links,nofollow,pages}` |
|
|
25
|
-
| Images | `src/api/image-audit.ts` (`ImageRow` L8–12, `ImageAuditSummary` L17–24) | per-row `{url,size,format,over100kb,legacyFormat}`; summary `{unique,totalSize,avgSize,over100kb,legacyFormat}` |
|
|
26
|
-
| Per-page indexability | `pages.jsonl` fields (`PageData`) | `url,status,indexable,canonicalUrl,wordCount,schemaTypes[],inlinks,crawlDepth` |
|
|
27
|
-
|
|
28
|
-
Real headline line for reference (`mcp-response-formatter.ts:768`):
|
|
29
|
-
`"{internal} internal / {external} external links · {orphans} orphans · {broken} broken · avg {n} inlinks/page · top external: {domain}"`.
|
|
30
|
-
|
|
31
|
-
## 2. New data constant — `AUDIT_DEMO`
|
|
32
|
-
|
|
33
|
-
Insert **immediately above** `function FeatureSection(` (currently near `public/app.jsx:2039`, just
|
|
34
|
-
before `var f = FEATURES[slug]...`). One module-level const, no interpolation.
|
|
35
|
-
|
|
36
|
-
```jsx
|
|
37
|
-
var AUDIT_DEMO = {
|
|
38
|
-
site: 'example.com',
|
|
39
|
-
pagesCrawled: 1284,
|
|
40
|
-
stats: [
|
|
41
|
-
{ k: 'Pages crawled', v: '1,284' },
|
|
42
|
-
{ k: 'Issues found', v: '37 types' },
|
|
43
|
-
{ k: 'Internal links', v: '18,402' },
|
|
44
|
-
{ k: 'Images', v: '512' },
|
|
45
|
-
],
|
|
46
|
-
issues: [
|
|
47
|
-
{ sev: '🔴', label: 'Broken internal links (404)', key: 'links.brokenInternal', count: 23 },
|
|
48
|
-
{ sev: '🔴', label: 'Pages returning 5xx errors', key: 'response.error5xx', count: 4 },
|
|
49
|
-
{ sev: '🟠', label: 'Missing page title', key: 'title.missing', count: 11 },
|
|
50
|
-
{ sev: '🟠', label: 'Missing H1', key: 'h1.missing', count: 19 },
|
|
51
|
-
{ sev: '🟠', label: 'Set to noindex', key: 'indexability.noindex', count: 8 },
|
|
52
|
-
{ sev: '🟠', label: 'Missing canonical tag', key: 'canonical.missing', count: 27 },
|
|
53
|
-
{ sev: '🟡', label: 'Duplicate title', key: 'title.duplicate', count: 34 },
|
|
54
|
-
{ sev: '🟡', label: 'Duplicate meta description', key: 'meta.duplicate', count: 41 },
|
|
55
|
-
{ sev: '🟡', label: 'Thin content (under 200 words)', key: 'content.thin', count: 52 },
|
|
56
|
-
{ sev: '🟡', label: 'Images missing alt text', key: 'images.missingAlt', count: 88 },
|
|
57
|
-
{ sev: '🟡', label: 'Missing schema markup', key: 'schema.missing', count: 63 },
|
|
58
|
-
{ sev: '🟡', label: 'Title longer than 60 chars', key: 'title.tooLong', count: 29 },
|
|
59
|
-
],
|
|
60
|
-
internal: {
|
|
61
|
-
totalLinks: 18402, orphans: 14, brokenInternal: 23, avgInlinks: 9.7,
|
|
62
|
-
distribution: { zero: 14, oneToTwo: 121, threeToTen: 818, elevenPlus: 331 },
|
|
63
|
-
top: [
|
|
64
|
-
{ url: '/', inlinks: 1284, out: 42 },
|
|
65
|
-
{ url: '/pricing', inlinks: 1190, out: 28 },
|
|
66
|
-
{ url: '/blog', inlinks: false ? 0 : 642, out: 35 },
|
|
67
|
-
{ url: '/features/seo-audit', inlinks: 318, out: 22 },
|
|
68
|
-
{ url: '/docs', inlinks: 291, out: 51 },
|
|
69
|
-
{ url: '/blog/best-ramen-nyc', inlinks: 47, out: 18 },
|
|
70
|
-
],
|
|
71
|
-
},
|
|
72
|
-
external: {
|
|
73
|
-
totalLinks: 1903, uniqueDomains: 142,
|
|
74
|
-
domains: [
|
|
75
|
-
{ domain: 'twitter.com', links: 311, nofollow: 311, pages: 290 },
|
|
76
|
-
{ domain: 'linkedin.com', links: 287, nofollow: 287, pages: 281 },
|
|
77
|
-
{ domain: 'github.com', links: 142, nofollow: 0, pages: 96 },
|
|
78
|
-
{ domain: 'youtube.com', links: 88, nofollow: 12, pages: 61 },
|
|
79
|
-
{ domain: 'stripe.com', links: 54, nofollow: 0, pages: 40 },
|
|
80
|
-
],
|
|
81
|
-
},
|
|
82
|
-
images: {
|
|
83
|
-
unique: 512, totalSize: '47.3 MB', avgSize: '94 KB', over100kb: 138, legacyFormat: 74,
|
|
84
|
-
heaviest: [
|
|
85
|
-
{ url: '/img/hero-bg.png', size: '2.4 MB', format: 'png', over: true, legacy: true },
|
|
86
|
-
{ url: '/img/team-photo.jpg', size: '1.1 MB', format: 'jpeg', over: true, legacy: true },
|
|
87
|
-
{ url: '/img/product-shot.png', size: '870 KB', format: 'png', over: true, legacy: true },
|
|
88
|
-
{ url: '/img/og-card.jpg', size: '512 KB', format: 'jpeg', over: true, legacy: true },
|
|
89
|
-
{ url: '/img/icon-sprite.webp', size: '210 KB', format: 'webp', over: true, legacy: false },
|
|
90
|
-
],
|
|
91
|
-
},
|
|
92
|
-
pages: [
|
|
93
|
-
{ url: '/', status: 200, indexable: true, canonical: true, words: 842, schema: 3 },
|
|
94
|
-
{ url: '/pricing', status: 200, indexable: true, canonical: true, words: 1190, schema: 2 },
|
|
95
|
-
{ url: '/blog/best-ramen-nyc', status: 200, indexable: true, canonical: true, words: 2847, schema: 4 },
|
|
96
|
-
{ url: '/old-landing', status: 301, indexable: false, canonical: false, words: 0, schema: 0 },
|
|
97
|
-
{ url: '/checkout', status: 200, indexable: false, canonical: true, words: 120, schema: 1 },
|
|
98
|
-
{ url: '/team', status: 404, indexable: false, canonical: false, words: 0, schema: 0 },
|
|
99
|
-
],
|
|
100
|
-
};
|
|
101
|
-
```
|
|
102
|
-
(Note: the `false ? 0 : 642` form avoids a literal `Math` call — keep plain literals; replace with `642`.)
|
|
103
|
-
|
|
104
|
-
## 3. New component — `AuditDemo`
|
|
105
|
-
|
|
106
|
-
Insert as a **standalone function** immediately above `function FeatureSection(`. Uses one
|
|
107
|
-
`useState` for the active tab. Reuses existing tokens (`var(--surface)`, `var(--border)`,
|
|
108
|
-
`var(--accent)`, `var(--text-muted)`, `var(--font-mono)`) and the existing `data-tab` /
|
|
109
|
-
`data-tab--active` classes already in the stylesheet (used at `app.jsx:2384`). One small reusable
|
|
110
|
-
`Bar` helper for the horizontal bars.
|
|
111
|
-
|
|
112
|
-
```jsx
|
|
113
|
-
function AuditBar({ value, max, color }) {
|
|
114
|
-
var pct = max > 0 ? Math.round((value / max) * 100) : 0;
|
|
115
|
-
return (
|
|
116
|
-
<div style={{height:7,background:'var(--bg-2)',borderRadius:99,overflow:'hidden',flex:1}}>
|
|
117
|
-
<div style={{width:pct+'%',height:'100%',background:color||'var(--accent)',borderRadius:99}}/>
|
|
118
|
-
</div>
|
|
119
|
-
);
|
|
120
|
-
}
|
|
121
|
-
|
|
122
|
-
function AuditDemo({ compact }) {
|
|
123
|
-
var TABS = compact
|
|
124
|
-
? [['issues','Issues'],['internal','Internal links'],['images','Images']]
|
|
125
|
-
: [['issues','Issues'],['internal','Internal links'],['external','External domains'],['images','Images'],['pages','Indexability']];
|
|
126
|
-
var [tab, setTab] = useState('issues');
|
|
127
|
-
var d = AUDIT_DEMO;
|
|
128
|
-
var th = { textAlign:'left', fontFamily:'var(--font-mono)', fontSize:10.5, letterSpacing:'.08em',
|
|
129
|
-
textTransform:'uppercase', color:'var(--text-subtle)', padding:'7px 10px', fontWeight:600 };
|
|
130
|
-
var td = { padding:'8px 10px', fontSize:12.5, borderTop:'1px solid var(--border)', color:'var(--text-muted)' };
|
|
131
|
-
var mono = { fontFamily:'var(--font-mono)', fontSize:12, color:'var(--text)' };
|
|
132
|
-
var distMax = Math.max(d.internal.distribution.zero, d.internal.distribution.oneToTwo,
|
|
133
|
-
d.internal.distribution.threeToTen, d.internal.distribution.elevenPlus);
|
|
134
|
-
var issueMax = Math.max.apply(null, d.issues.map(function(i){ return i.count; }));
|
|
135
|
-
|
|
136
|
-
return (
|
|
137
|
-
<div style={{border:'1px solid var(--border)',borderRadius:'var(--radius-lg)',background:'var(--surface)',overflow:'hidden'}}>
|
|
138
|
-
{/* header */}
|
|
139
|
-
<div style={{display:'flex',alignItems:'center',justifyContent:'space-between',gap:12,padding:'12px 14px',borderBottom:'1px solid var(--border)',flexWrap:'wrap'}}>
|
|
140
|
-
<div style={{fontFamily:'var(--font-mono)',fontSize:12.5,color:'var(--text-muted)'}}>
|
|
141
|
-
<span style={{color:'var(--accent)'}}>audit_site</span> {d.site} · {d.pagesCrawled.toLocaleString()} pages
|
|
142
|
-
</div>
|
|
143
|
-
<div style={{display:'flex',gap:14,flexWrap:'wrap'}}>
|
|
144
|
-
{d.stats.map(function(s){ return (
|
|
145
|
-
<div key={s.k} style={{fontSize:11.5,color:'var(--text-subtle)'}}>
|
|
146
|
-
<b style={{color:'var(--text)',fontFamily:'var(--font-mono)'}}>{s.v}</b> {s.k}
|
|
147
|
-
</div>
|
|
148
|
-
); })}
|
|
149
|
-
</div>
|
|
150
|
-
</div>
|
|
151
|
-
{/* tab bar */}
|
|
152
|
-
<div style={{display:'flex',gap:6,padding:'10px 12px',borderBottom:'1px solid var(--border)',flexWrap:'wrap'}}>
|
|
153
|
-
{TABS.map(function(t){
|
|
154
|
-
var on = tab===t[0];
|
|
155
|
-
return <button key={t[0]} className={'data-tab'+(on?' data-tab--active':'')} onClick={function(){ setTab(t[0]); }}>{t[1]}</button>;
|
|
156
|
-
})}
|
|
157
|
-
</div>
|
|
158
|
-
{/* panels */}
|
|
159
|
-
<div style={{padding:'4px 6px 10px'}}>
|
|
160
|
-
{tab==='issues' && (
|
|
161
|
-
<table style={{width:'100%',borderCollapse:'collapse'}}>
|
|
162
|
-
<thead><tr><th style={th}></th><th style={th}>Issue</th><th style={Object.assign({},th,{textAlign:'right'})}>Pages</th><th style={Object.assign({},th,{width:'34%'})}></th></tr></thead>
|
|
163
|
-
<tbody>{d.issues.map(function(i){ return (
|
|
164
|
-
<tr key={i.key}>
|
|
165
|
-
<td style={Object.assign({},td,{width:24})}>{i.sev}</td>
|
|
166
|
-
<td style={td}>{i.label} <span style={{color:'var(--text-subtle)',fontFamily:'var(--font-mono)',fontSize:10.5}}>{i.key}</span></td>
|
|
167
|
-
<td style={Object.assign({},td,mono,{textAlign:'right'})}>{i.count}</td>
|
|
168
|
-
<td style={td}><AuditBar value={i.count} max={issueMax} color={i.sev==='🔴'?'#dc2626':i.sev==='🟠'?'#ea580c':'var(--accent)'}/></td>
|
|
169
|
-
</tr>
|
|
170
|
-
); })}</tbody>
|
|
171
|
-
</table>
|
|
172
|
-
)}
|
|
173
|
-
{tab==='internal' && (
|
|
174
|
-
<div>
|
|
175
|
-
<div style={{display:'flex',gap:14,flexWrap:'wrap',padding:'8px 10px'}}>
|
|
176
|
-
{[['Total internal links',d.internal.totalLinks.toLocaleString()],['Orphan pages',d.internal.orphans],['Broken',d.internal.brokenInternal],['Avg inlinks / page',d.internal.avgInlinks]].map(function(s){
|
|
177
|
-
return <div key={s[0]} style={{fontSize:11.5,color:'var(--text-subtle)'}}><b style={{color:'var(--text)',fontFamily:'var(--font-mono)',fontSize:14,display:'block'}}>{s[1]}</b>{s[0]}</div>;
|
|
178
|
-
})}
|
|
179
|
-
</div>
|
|
180
|
-
<div style={{padding:'4px 10px 10px'}}>
|
|
181
|
-
<div style={{fontFamily:'var(--font-mono)',fontSize:10.5,letterSpacing:'.1em',textTransform:'uppercase',color:'var(--text-subtle)',margin:'4px 0 8px'}}>Inlink distribution</div>
|
|
182
|
-
{[['0 (orphan)',d.internal.distribution.zero,'#dc2626'],['1–2',d.internal.distribution.oneToTwo,'#ea580c'],['3–10',d.internal.distribution.threeToTen,'var(--accent)'],['11+',d.internal.distribution.elevenPlus,'var(--accent)']].map(function(r){
|
|
183
|
-
return <div key={r[0]} style={{display:'flex',alignItems:'center',gap:10,marginBottom:6}}>
|
|
184
|
-
<span style={{width:64,fontSize:11.5,color:'var(--text-muted)',fontFamily:'var(--font-mono)'}}>{r[0]}</span>
|
|
185
|
-
<AuditBar value={r[1]} max={distMax} color={r[2]}/>
|
|
186
|
-
<span style={{width:34,textAlign:'right',fontSize:11.5,fontFamily:'var(--font-mono)',color:'var(--text)'}}>{r[1]}</span>
|
|
187
|
-
</div>;
|
|
188
|
-
})}
|
|
189
|
-
</div>
|
|
190
|
-
<table style={{width:'100%',borderCollapse:'collapse',marginTop:6}}>
|
|
191
|
-
<thead><tr><th style={th}>Top pages by inlinks</th><th style={Object.assign({},th,{textAlign:'right'})}>Inlinks</th><th style={Object.assign({},th,{textAlign:'right'})}>Outlinks</th></tr></thead>
|
|
192
|
-
<tbody>{d.internal.top.map(function(p){ return (
|
|
193
|
-
<tr key={p.url}><td style={Object.assign({},td,mono)}>{p.url}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{p.inlinks.toLocaleString()}</td><td style={Object.assign({},td,{textAlign:'right'})}>{p.out}</td></tr>
|
|
194
|
-
); })}</tbody>
|
|
195
|
-
</table>
|
|
196
|
-
</div>
|
|
197
|
-
)}
|
|
198
|
-
{tab==='external' && (
|
|
199
|
-
<table style={{width:'100%',borderCollapse:'collapse'}}>
|
|
200
|
-
<thead><tr><th style={th}>External domain</th><th style={Object.assign({},th,{textAlign:'right'})}>Links</th><th style={Object.assign({},th,{textAlign:'right'})}>Nofollow</th><th style={Object.assign({},th,{textAlign:'right'})}>From pages</th></tr></thead>
|
|
201
|
-
<tbody>{d.external.domains.map(function(x){ return (
|
|
202
|
-
<tr key={x.domain}><td style={Object.assign({},td,mono)}>{x.domain}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{x.links}</td><td style={Object.assign({},td,{textAlign:'right'})}>{x.nofollow}</td><td style={Object.assign({},td,{textAlign:'right'})}>{x.pages}</td></tr>
|
|
203
|
-
); })}</tbody>
|
|
204
|
-
</table>
|
|
205
|
-
)}
|
|
206
|
-
{tab==='images' && (
|
|
207
|
-
<div>
|
|
208
|
-
<div style={{display:'flex',gap:14,flexWrap:'wrap',padding:'8px 10px'}}>
|
|
209
|
-
{[['Unique images',d.images.unique],['Total weight',d.images.totalSize],['Average',d.images.avgSize],['Over 100 KB',d.images.over100kb],['Legacy format',d.images.legacyFormat]].map(function(s){
|
|
210
|
-
return <div key={s[0]} style={{fontSize:11.5,color:'var(--text-subtle)'}}><b style={{color:'var(--text)',fontFamily:'var(--font-mono)',fontSize:14,display:'block'}}>{s[1]}</b>{s[0]}</div>;
|
|
211
|
-
})}
|
|
212
|
-
</div>
|
|
213
|
-
<table style={{width:'100%',borderCollapse:'collapse',marginTop:4}}>
|
|
214
|
-
<thead><tr><th style={th}>Heaviest images</th><th style={Object.assign({},th,{textAlign:'right'})}>Size</th><th style={th}>Format</th><th style={th}>Flags</th></tr></thead>
|
|
215
|
-
<tbody>{d.images.heaviest.map(function(im){ return (
|
|
216
|
-
<tr key={im.url}><td style={Object.assign({},td,mono)}>{im.url}</td><td style={Object.assign({},td,mono,{textAlign:'right'})}>{im.size}</td><td style={td}>{im.format}</td>
|
|
217
|
-
<td style={td}>{im.over && <span style={{color:'#ea580c'}}>over 100KB</span>}{im.over && im.legacy ? ' · ' : ''}{im.legacy && <span style={{color:'#dc2626'}}>legacy</span>}</td></tr>
|
|
218
|
-
); })}</tbody>
|
|
219
|
-
</table>
|
|
220
|
-
</div>
|
|
221
|
-
)}
|
|
222
|
-
{tab==='pages' && (
|
|
223
|
-
<table style={{width:'100%',borderCollapse:'collapse'}}>
|
|
224
|
-
<thead><tr><th style={th}>URL</th><th style={th}>Status</th><th style={th}>Indexable</th><th style={th}>Canonical</th><th style={Object.assign({},th,{textAlign:'right'})}>Words</th><th style={Object.assign({},th,{textAlign:'right'})}>Schema</th></tr></thead>
|
|
225
|
-
<tbody>{d.pages.map(function(p){ return (
|
|
226
|
-
<tr key={p.url}>
|
|
227
|
-
<td style={Object.assign({},td,mono)}>{p.url}</td>
|
|
228
|
-
<td style={Object.assign({},td,{color:p.status>=400?'#dc2626':p.status>=300?'#ea580c':'var(--text-muted)'})}>{p.status}</td>
|
|
229
|
-
<td style={td}>{p.indexable?'✓':'—'}</td>
|
|
230
|
-
<td style={td}>{p.canonical?'✓':'—'}</td>
|
|
231
|
-
<td style={Object.assign({},td,mono,{textAlign:'right'})}>{p.words.toLocaleString()}</td>
|
|
232
|
-
<td style={Object.assign({},td,{textAlign:'right'})}>{p.schema}</td>
|
|
233
|
-
</tr>
|
|
234
|
-
); })}</tbody>
|
|
235
|
-
</table>
|
|
236
|
-
)}
|
|
237
|
-
</div>
|
|
238
|
-
</div>
|
|
239
|
-
);
|
|
240
|
-
}
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
## 4. Wiring — render points
|
|
244
|
-
|
|
245
|
-
### 4a. Feature page (primary)
|
|
246
|
-
In `FeatureSection` (`app.jsx:2041` render), insert the demo **between** the `skill-section` block
|
|
247
|
-
(closes ~`2079`) and the `More capabilities` block (opens ~`2081`). Gate to the audit feature only:
|
|
248
|
-
|
|
249
|
-
```jsx
|
|
250
|
-
{slug === 'web' && (
|
|
251
|
-
<div style={{marginTop:48}}>
|
|
252
|
-
<div className="section-kicker">See it in action</div>
|
|
253
|
-
<h2 className="how__title" style={{margin:'6px 0 16px'}}>A full technical SEO audit, not just a crawl</h2>
|
|
254
|
-
<AuditDemo />
|
|
255
|
-
</div>
|
|
256
|
-
)}
|
|
257
|
-
```
|
|
258
|
-
`slug` is already in scope (`FeatureSection({ slug, ... })`, `var f = FEATURES[slug]`).
|
|
259
|
-
|
|
260
|
-
### 4b. Homepage teaser (secondary, optional but recommended)
|
|
261
|
-
On the landing page, after the existing data-tabs/source-tiles section, drop a compact instance:
|
|
262
|
-
```jsx
|
|
263
|
-
<div style={{maxWidth:760,margin:'28px auto 0'}}><AuditDemo compact /></div>
|
|
264
|
-
```
|
|
265
|
-
Place near the `sourceTiles` render (search `sourceTiles.map` in `app.jsx`) so it sits with the
|
|
266
|
-
other "what you get" proof. Compact mode shows Issues / Internal links / Images only.
|
|
267
|
-
|
|
268
|
-
## 5. Copy updates (same edit pass — make the claims match the visual)
|
|
269
|
-
|
|
270
|
-
1. **Feature bullets** (`FEATURES.web.bullets`, `app.jsx:1955–1958`) — split the single audit bullet
|
|
271
|
-
into the specifics the demo now proves:
|
|
272
|
-
- `'Technical SEO audit: 30+ issue checks across titles, meta, headings, indexability, schema'`
|
|
273
|
-
- `'Internal link analysis: inlinks per page, orphan detection, link distribution, broken links'`
|
|
274
|
-
- `'External link profile: top linked domains, nofollow ratio, pages linking out'`
|
|
275
|
-
- `'Image audit: weight & format per image, over-100KB and legacy-format flags'`
|
|
276
|
-
2. **Source tile** (`app.jsx:2386`) — keep, it's accurate.
|
|
277
|
-
3. **`FEATURES.web.sub`** (`app.jsx:1952`) — already good; no change.
|
|
278
|
-
|
|
279
|
-
## 6. CSS check
|
|
280
|
-
`.data-tab` / `.data-tab--active` already exist (used at `app.jsx:2384`). Confirm they style a pill
|
|
281
|
-
button; if `.data-tab` lacks base button reset, the inline usage still works because we only rely on
|
|
282
|
-
its existing visual. **No new CSS required.** If tables need zebra/scroll on mobile, wrap each
|
|
283
|
-
`<table>` in `<div style={{overflowX:'auto'}}>` (already safe at 414px since columns are few).
|
|
284
|
-
|
|
285
|
-
## 7. Build, verify, deploy (website-only)
|
|
286
|
-
|
|
287
|
-
```bash
|
|
288
|
-
cd /Users/vilovieta/Desktop/mcp-scraper
|
|
289
|
-
npm run build:ui # rebuild public/app.js from app.jsx
|
|
290
|
-
grep -c "AuditDemo" public/app.js # expect >=2 (def + render)
|
|
291
|
-
git add public/app.jsx public/app.js
|
|
292
|
-
git commit -m "feat(site): tabbed technical-SEO-audit visual demo (issues, links, images, indexability)"
|
|
293
|
-
git push origin main
|
|
294
|
-
/opt/homebrew/bin/vercel --prod --yes # deploy; no version bump, no npm publish
|
|
295
|
-
```
|
|
296
|
-
Then load `https://mcpscraper.dev/features/web` (or whatever the audit feature route is — confirm via
|
|
297
|
-
`FEATURE_ORDER` / the router) and click each tab; load the homepage for the compact teaser.
|
|
298
|
-
|
|
299
|
-
## 8. Acceptance criteria
|
|
300
|
-
- [ ] Feature page renders a 5-tab audit visual; every tab shows a populated table/bars.
|
|
301
|
-
- [ ] Numbers in `AUDIT_DEMO` are internally consistent (e.g. distribution buckets sum ≈ pages).
|
|
302
|
-
- [ ] Issue rows show severity color matching `seo-issues.ts` severity rules.
|
|
303
|
-
- [ ] Homepage shows the 3-tab compact teaser.
|
|
304
|
-
- [ ] Renders cleanly at 414px (mobile) — no horizontal overflow of the page.
|
|
305
|
-
- [ ] No backend/test changes; `npm run build:ui` clean; deployed to prod.
|
|
306
|
-
|
|
307
|
-
## 9. Out of scope / follow-ups
|
|
308
|
-
- Live "run an audit on your URL" interactive demo (would hit `/extract-site` + cost credits).
|
|
309
|
-
- Wiring the demo to real cached audit JSON instead of `AUDIT_DEMO` sample.
|