mcp-scraper 0.2.8 → 0.2.10

package/docs/specs/seo-cli-growth-roadmap-spec.md

@@ -0,0 +1,179 @@
+# SEO CLI Growth Roadmap — Product + Implementation Spec
+
+This spec packages the planned MCP Scraper growth work into buildable slices. The product goal is to make MCP Scraper the command-line and MCP data layer for AI SEO work: run live web intelligence workflows, save evidence-rich reports, and hand clean artifacts to AI agents.
+
+## Product thesis
+
+AI SEO users do not only need scraped data. They need repeatable evidence workflows that answer:
+
+- Which sources and competitors are visible in AI/Search surfaces?
+- What do Maps competitors look like across a market?
+- What do reviews say customers care about?
+- What should an AI agent do next with the evidence?
+- Can the workflow run again on a schedule?
+
+The CLI should therefore feel like a workflow machine, not a loose collection of endpoints.
+
+## Spec set
+
+| Spec | Purpose | Recommended priority |
+|---|---|---|
+| `cli-agent-wiring-spec.md` | Make MCP Scraper easy to install into Codex, Claude, and other AI-agent hosts. | P0 |
+| `seo-workflow-runner-and-reports-spec.md` | Add a human CLI workflow runner plus local HTML/CSV/JSON/Markdown report artifacts. | P0 |
+| `agent-ready-seo-packet-spec.md` | Produce a folder that an AI agent can use directly: brief, evidence, sources, tasks, report. | P1 |
+| `local-competitive-audit-spec.md` | Turn directory + Maps + review/profile hydration into a market audit workflow. | P1 |
+| `scheduled-workflows-api-spec.md` | Add hosted recurring workflow schedules, webhooks, run history, and API surfaces. | P2 |
+
+## Current tool surface to reuse
+
+Already shipped:
+
+- `maps_search` — multi-business Maps result search, up to 50 results.
+- `maps_place_intel` — profile-level Maps details, review count, category, review histogram, review topics, attributes, optional review cards.
+- `directory_workflow` — Census/city selection + city-by-city Maps searches + CSV output.
+- `rank_tracker_blueprint` — local planning tool for database schema, cron, metrics, and implementation prompts.
+- `search_serp` / `harvest_paa` — SERP, PAA, AI Overview, forums, videos, and source evidence.
+- `mcp-scraper-install` — human terminal installer card.
+- `mcp-scraper-combined` — combined MCP stdio server for web intelligence + browser tools.
+
+Important distinction: `rank_tracker_blueprint` is not a live tracker runner. It should be referenced by scheduled workflow specs, not replaced.
+
+## New CLI command family
+
+Keep existing stdio server binaries protocol-safe. Add a new human-facing CLI binary:
+
+```json
+{
+  "bin": {
+    "mcp-scraper-cli": "dist/bin/mcp-scraper-cli.js"
+  }
+}
+```
+
+Rationale: `mcp-scraper` is already the stdio MCP server. It must stay silent except for JSON-RPC. The human CLI can print progress bars, report paths, cards, and diagnostics.
+
+Primary commands:
+
+```bash
+mcp-scraper-cli doctor
+mcp-scraper-cli agent install codex
+mcp-scraper-cli agent install claude
+mcp-scraper-cli agent prompt agent-packet
+mcp-scraper-cli workflow list
+mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
+mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20
+mcp-scraper-cli report open last
+mcp-scraper-cli schedule create local-competitive-audit --weekly --webhook https://example.com/hook
+```
+
+The existing `mcp-scraper-install` stays as the branded terminal card and can point users at `mcp-scraper-cli`.
+
+## Workflow catalog
+
+P0/P1 workflows:
+
+| Workflow | What it answers | Primary outputs |
+|---|---|---|
+| `agent-packet` | What evidence and tasks should an AI agent use for this SEO problem? | agent folder, HTML report |
+| `local-competitive-audit` | Who ranks in local markets, how strong are they, and what do customers say? | report, CSVs, review insights |
+| `directory` | Which businesses are ranking across selected city markets? | CSV, market report |
+| `serp-brief` | What should a page cover based on live SERP/PAA/AI evidence? | writer brief, evidence report |
+| `answer-gap` | Which cited/ranking sources answer the query better than the target page? | gap report, task list |
+| `ai-citations` | Is the brand/domain cited in AI Overview/search evidence? | citation table, trends when scheduled |
+
+P2/P3 workflows:
+
+| Workflow | What it answers |
+|---|---|
+| `maps-rank-monitor` | How does a target GBP move across markets and keywords? |
+| `content-decay` | Which pages are stale compared with current live SERP/PAA evidence? |
+| `forum-demand` | What objections, use cases, and language are showing up in forums? |
+| `entity-audit` | Is brand/entity consistency strong across public search surfaces? |
+
+## Shared artifact contract
+
+Every workflow writes a run folder:
+
+```text
+~/Downloads/mcp-scraper/workflows/<workflow>/<timestamp>-<slug>/
+  manifest.json
+  report.html
+  summary.md
+  evidence.json
+  sources.csv
+  tasks.md
+  logs/attempts.jsonl
+  exports/*.csv
+  screenshots/*        # optional
+  replays/*            # optional
+```
+
+`manifest.json` is the stable contract:
+
+```ts
+interface WorkflowManifest {
+  workflow: string
+  runId: string
+  startedAt: string
+  completedAt: string | null
+  status: 'running' | 'succeeded' | 'partial' | 'failed'
+  input: Record<string, unknown>
+  artifacts: Array<{
+    kind: 'html_report' | 'markdown' | 'json' | 'csv' | 'log' | 'screenshot' | 'replay'
+    label: string
+    path: string
+    rows?: number
+  }>
+  counts: Record<string, number>
+  warnings: string[]
+  costs?: { estimatedMc: number; actualMc?: number }
+}
+```
+
+## Implementation phases
+
+### Phase 1 — Human CLI shell + agent wiring
+
+- Add `bin/mcp-scraper-cli.ts`.
+- Implement `doctor`, `agent install`, `agent prompt`, and `workflow list`.
+- No new paid workflow execution yet.
+- Tests: CLI argument parsing, generated config snapshots, installer card references.
+
+### Phase 2 — Workflow runner + report artifact system
+
+- Add local workflow registry and artifact writer.
+- Implement `report open last`.
+- Implement `directory` as first wrapped workflow around existing `directory_workflow`.
+- Tests: run folder creation, manifest shape, report renderer.
+
+### Phase 3 — Agent-ready SEO packet
+
+- Implement `agent-packet` workflow.
+- Use `search_serp`, `harvest_paa`, optional `extract_url`, and report writer.
+- Tests: fixture evidence to deterministic packet artifacts.
+
+### Phase 4 — Local competitive audit
+
+- Implement `local-competitive-audit`.
+- Add `review_count` to directory CSV base export.
+- Hydrate top-N competitors per city using `maps_place_intel`.
+- Generate review/category/market-strength report.
+
+### Phase 5 — Hosted schedules/API
+
+- Add workflow run and schedule database tables.
+- Add API routes for schedule CRUD, manual run, run status, report download.
+- Add webhook callbacks.
+- Keep scheduled workflows separate from `rank_tracker_blueprint`; the blueprint can generate schema/plans for custom rank trackers.
+
+## Tool-description impact
+
+When each workflow ships, update:
+
+- `README.md` install and tool sections.
+- `public/skills/mcp-scraper/skill.md`.
+- MCP tool descriptions in `src/mcp/paa-mcp-server.ts` if a workflow becomes a real MCP tool.
+- `mcp-scraper-install` newest-feature line.
+
+Avoid naming underlying browser/proxy vendors in outward-facing CLI text, tool descriptions, and error messages.
+

package/docs/specs/seo-workflow-runner-and-reports-spec.md

@@ -0,0 +1,241 @@
+# SEO Workflow Runner + Local Reports — Implementation Spec
+
+This spec introduces a human CLI workflow runner and a shared artifact/report system. It wraps existing MCP Scraper capabilities into named SEO workflows with predictable local outputs.
+
+## Goals
+
+- Run high-value SEO workflows from the CLI.
+- Save useful local artifacts every time.
+- Provide local HTML reports that are easy to inspect or share.
+- Keep workflow logic reusable by hosted API schedules later.
+- Avoid duplicating tool logic already implemented in MCP/API handlers.
+
+## Command surface
+
+```bash
+mcp-scraper-cli workflow list
+mcp-scraper-cli workflow describe local-competitive-audit
+mcp-scraper-cli workflow run directory --query roofers --state TN --min-pop 100000 --per-city 20
+mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
+mcp-scraper-cli report open last
+mcp-scraper-cli report path last
+mcp-scraper-cli report list
+```
+
+Global flags:
+
+```bash
+--output-dir <path>
+--format html,json,md,csv
+--json
+--no-open
+--api-url <url>
+--api-key <key>
+```
+
+## Workflow registry
+
+New module:
+
+```text
+src/workflows/
+  registry.ts
+  types.ts
+  artifact-writer.ts
+  report-renderer.ts
+  workflows/
+    directory.ts
+    agent-packet.ts
+    local-competitive-audit.ts
+```
+
+Types:
+
+```ts
+export interface WorkflowDefinition<TInput, TResult> {
+  id: string
+  title: string
+  description: string
+  inputSchema: z.ZodType<TInput>
+  estimate(input: TInput): Promise<WorkflowEstimate>
+  run(input: TInput, ctx: WorkflowRunContext): Promise<TResult>
+  render(result: TResult, ctx: WorkflowRunContext): Promise<WorkflowArtifacts>
+}
+
+export interface WorkflowRunContext {
+  runId: string
+  startedAt: string
+  outputDir: string
+  apiKey?: string
+  apiUrl: string
+  logger: WorkflowLogger
+  artifacts: ArtifactWriter
+  signal?: AbortSignal
+}
+```
+
+The registry is local code, not model-driven. Agents can discover workflows through `workflow list`, and MCP tools can later expose the same registry through structured content.
+
+## Artifact contract
+
+Each run creates:
+
+```text
+~/Downloads/mcp-scraper/workflows/<workflow>/<timestamp>-<slug>/
+  manifest.json
+  report.html
+  summary.md
+  evidence.json
+  tasks.md
+  exports/*.csv
+  logs/attempts.jsonl
+```
+
+`manifest.json`:
+
+```ts
+interface WorkflowManifest {
+  workflow: string
+  title: string
+  runId: string
+  status: 'running' | 'succeeded' | 'partial' | 'failed'
+  startedAt: string
+  completedAt: string | null
+  input: Record<string, unknown>
+  artifacts: Array<{
+    kind: 'html_report' | 'markdown' | 'json' | 'csv' | 'log'
+    label: string
+    path: string
+    bytes?: number
+    rows?: number
+  }>
+  warnings: string[]
+  errors: string[]
+  counts: Record<string, number>
+}
+```
+
+## HTML report renderer
+
+No frontend framework required for the first version. Generate standalone HTML with embedded CSS and static tables.
+
+Report sections:
+
+1. Header: workflow, input, run status, date, artifact links.
+2. Executive summary.
+3. Evidence tables.
+4. Competitors/sources table when available.
+5. Findings.
+6. Recommended actions.
+7. Diagnostics: attempts, partial failures, location confidence, warnings.
+
+Files:
+
+- `src/workflows/report-renderer.ts`
+- `src/workflows/report-template.ts`
+- `tests/unit/workflow-report-renderer.test.ts`
+
+## Local report index
+
+Maintain:
+
+```text
+~/Downloads/mcp-scraper/workflows/index.json
+```
+
+Shape:
+
+```ts
+interface WorkflowRunIndex {
+  runs: Array<{
+    workflow: string
+    runId: string
+    status: string
+    startedAt: string
+    reportPath: string | null
+    manifestPath: string
+    summary: string
+  }>
+}
+```
+
+Commands:
+
+```bash
+mcp-scraper-cli report list
+mcp-scraper-cli report open last
+mcp-scraper-cli report open <runId>
+mcp-scraper-cli report path <runId>
+```
+
+Opening behavior:
+
+- macOS: `open <reportPath>`.
+- Other platforms: print path.
+- `--json`: return machine-readable path only.
+
+## Initial workflows
+
+### `directory`
+
+Wrapper around existing `directory_workflow`.
+
+Input:
+
+```bash
+--query roofers
+--state TN
+--min-pop 100000
+--per-city 20
+--concurrency 5
+```
+
+Output:
+
+- Existing directory CSV.
+- HTML summary of markets, city status, result counts, top businesses.
+- `evidence.json` preserving structured `directory_workflow` response.
+
+### `agent-packet`
+
+Specified in `agent-ready-seo-packet-spec.md`.
+
+### `local-competitive-audit`
+
+Specified in `local-competitive-audit-spec.md`.
+
+## Hosted versus local execution
+
+First implementation should call hosted API endpoints when available:
+
+- `/directory/run`
+- `/maps/search`
+- `/maps/place-intel`
+- `/serp/...` and `/harvest/...` where already available.
+
+Local-only helpers are acceptable for report assembly and artifact writing.
+
+Do not directly duplicate extraction code in the CLI.
+
+## Error and partial-result behavior
+
+Workflow runs should prefer `partial` over `failed` when useful artifacts exist.
+
+Examples:
+
+- 8 of 10 cities succeed: status `partial`, report includes failed city diagnostics.
+- PAA fails but SERP succeeds: `agent-packet` still writes SERP evidence and missing PAA warning.
+- Review hydration fails for some competitors: audit keeps directory result and marks review fields unavailable.
+
+## Tests
+
+- `workflow list` contains `directory`, `agent-packet`, `local-competitive-audit`.
+- `directory` workflow with mocked API writes manifest, report, evidence JSON, CSV link.
+- Report renderer escapes HTML.
+- `report open last --json` returns the expected path without launching.
+- Partial runs preserve artifacts and warnings.
+
+## MCP description impact
+
+When the CLI workflows are stable, expose `workflow_recipe` or `workflow_plan` as a local planning MCP tool only if needed. Do not add a paid MCP workflow runner until the local CLI and hosted API schedule surfaces settle.
+

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.2.8",
+  "version": "0.2.10",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",
@@ -19,7 +19,8 @@
     "mcp-scraper": "dist/bin/mcp-stdio-server.js",
     "browser-agent": "dist/bin/browser-agent-stdio-server.js",
     "mcp-scraper-combined": "dist/bin/mcp-scraper-combined-stdio-server.js",
-    "mcp-scraper-install": "dist/bin/mcp-scraper-install.js"
+    "mcp-scraper-install": "dist/bin/mcp-scraper-install.js",
+    "mcp-scraper-cli": "dist/bin/mcp-scraper-cli.js"
   },
   "files": [
     "dist",
@@ -46,31 +47,17 @@
     "test:e2e": "playwright test"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.29.0",
-    "zod": "^3.23.0"
-  },
-  "devDependencies": {
     "@anthropic-ai/sdk": "^0.96.0",
     "@distube/ytdl-core": "^4.16.12",
     "@fal-ai/client": "^1.10.1",
     "@hono/node-server": "^2.0.2",
     "@libsql/client": "^0.17.3",
-    "@onkernel/cli": "^0.19.2",
+    "@modelcontextprotocol/sdk": "^1.29.0",
     "@onkernel/sdk": "^0.52.0",
-    "@playwright/test": "^1.60.0",
     "@resvg/resvg-js": "^2.6.2",
-    "@types/better-sqlite3": "^7.6.13",
-    "@types/jsdom": "^28.0.3",
-    "@types/node": "^20.0.0",
-    "@types/papaparse": "^5.3.0",
-    "@types/turndown": "^5.0.6",
-    "better-sqlite3": "^12.9.0",
-    "bgutils-js": "^3.2.0",
     "commander": "^12.0.0",
-    "dotenv": "^17.4.2",
     "hono": "^4.12.18",
     "inngest": "^4.4.0",
-    "jsdom": "^29.1.1",
     "p-limit": "^7.3.0",
     "papaparse": "^5.4.0",
     "playwright": "^1.44.0",
@@ -79,12 +66,26 @@
     "resend": "^6.12.3",
     "satori": "^0.26.0",
     "stripe": "^22.1.1",
+    "turndown": "^7.2.4",
+    "youtube-transcript": "^1.3.1",
+    "zod": "^3.23.0"
+  },
+  "devDependencies": {
+    "@onkernel/cli": "^0.19.2",
+    "@playwright/test": "^1.60.0",
+    "@types/better-sqlite3": "^7.6.13",
+    "@types/jsdom": "^28.0.3",
+    "@types/node": "^20.0.0",
+    "@types/papaparse": "^5.3.0",
+    "@types/turndown": "^5.0.6",
+    "better-sqlite3": "^12.9.0",
+    "bgutils-js": "^3.2.0",
+    "dotenv": "^17.4.2",
+    "jsdom": "^29.1.1",
     "tsup": "^8.0.0",
     "tsx": "^4.0.0",
-    "turndown": "^7.2.4",
     "typescript": "^5.4.0",
     "vitest": "^1.6.0",
-    "youtube-transcript": "^1.3.1",
     "youtubei.js": "^17.0.1"
   },
   "engines": {