mcp-scraper 0.2.8 → 0.2.10

package/dist/{worker-5O44YBF4.js → worker-TDJQ6TH3.js}

@@ -4,12 +4,7 @@ import {
   createHarvestAttemptRecorder,
   harvestProblemResponse,
   serializeHarvestProblem
-} from "./chunk-IQOCZGJJ.js";
-import {
-  browserServiceApiKey,
-  harvest
-} from "./chunk-CQTAKXBN.js";
-import "./chunk-M2S27J6Z.js";
+} from "./chunk-ROS67BNV.js";
 import {
   claimPendingJob,
   completeJob,
@@ -17,7 +12,12 @@ import {
   debitMc,
   failJob,
   listHarvestAttempts
-} from "./chunk-D4CJBZBY.js";
+} from "./chunk-7GCCOT3M.js";
+import {
+  browserServiceApiKey,
+  harvest
+} from "./chunk-CQTAKXBN.js";
+import "./chunk-M2S27J6Z.js";
 
 // src/api/webhook.ts
 async function deliverWebhook(url, payload, retries = 3) {
@@ -122,4 +122,4 @@ export {
   startWorker,
   tickOnce
 };
-//# sourceMappingURL=worker-5O44YBF4.js.map
+//# sourceMappingURL=worker-TDJQ6TH3.js.map

package/docs/specs/agent-ready-seo-packet-spec.md

@@ -0,0 +1,237 @@
+# Agent-Ready SEO Packet — Implementation Spec
+
+`agent-packet` turns live SEO evidence into a folder an AI agent can use without re-discovering context. It is both a CLI workflow and a product wedge: MCP Scraper becomes the evidence layer for Codex, Claude, Cursor, and other agents.
+
+## User promise
+
+```bash
+mcp-scraper-cli workflow run agent-packet \
+  --keyword "roof repair Denver" \
+  --domain example.com \
+  --location "Denver, CO"
+```
+
+Output:
+
+```text
+agent-packet/
+  report.html
+  brief.md
+  evidence.json
+  sources.csv
+  competitors.csv
+  tasks.md
+  agent-instructions.md
+  manifest.json
+```
+
+The packet should answer:
+
+- What does the live search landscape show?
+- Which sources and competitors matter?
+- What questions do people ask?
+- Is the target domain present?
+- What should an AI agent do next?
+
+## Inputs
+
+CLI flags:
+
+```bash
+--keyword <query>             # required
+--domain <domain>             # optional but recommended
+--location <city,state>       # optional
+--max-questions <n>           # default 40, max 150
+--include-serp                # default true
+--include-paa                 # default true
+--include-ai-overview         # default true through search/PAA tools
+--include-page-snapshots      # default false, cap 5
+--output-dir <path>
+--return-partial              # default true
+```
+
+Structured input:
+
+```ts
+interface AgentPacketInput {
+  keyword: string
+  domain?: string
+  location?: string
+  maxQuestions: number
+  includeSerp: boolean
+  includePaa: boolean
+  includeAiOverview: boolean
+  includePageSnapshots: boolean
+  pageSnapshotLimit: number
+  returnPartial: boolean
+}
+```
+
+## Evidence collection plan
+
+Use existing tools/API paths:
+
+1. `search_serp`
+   - Capture organic results, local pack when present, AI Overview detection/citations when available.
+   - Use `location` if provided.
+2. `harvest_paa`
+   - Capture PAA tree, answers, source titles/sites/cites, forums/videos when available.
+   - Use `maxQuestions`.
+3. Optional `extract_url`
+   - Extract target domain page if the target appears in SERP or user passes a URL later.
+   - Extract top source pages only when `includePageSnapshots` is true.
+
+No server-side LLM is required. The workflow produces structured evidence and instructions; the calling agent can synthesize if desired.
+
+## Artifact formats
+
+### `evidence.json`
+
+```ts
+interface AgentPacketEvidence {
+  input: AgentPacketInput
+  serp: {
+    status: 'ok' | 'failed' | 'skipped'
+    organic: Array<{ position: number; title: string; url: string; domain: string; snippet?: string | null }>
+    localPack: Array<{ position: number; name: string; rating?: string | null; reviewCount?: string | null; websiteUrl?: string | null }>
+    aiOverview: { detected: boolean; text?: string | null; citations: Array<{ title?: string; url?: string; domain?: string }> }
+  }
+  paa: {
+    status: 'ok' | 'failed' | 'skipped'
+    questions: Array<{ question: string; answer?: string | null; sourceTitle?: string | null; sourceSite?: string | null; sourceUrl?: string | null }>
+  }
+  target: {
+    domain: string | null
+    organicPositions: number[]
+    citedInAiOverview: boolean
+    citedInPaa: boolean
+    localPackPositions: number[]
+  }
+  competitors: Array<{
+    domain: string
+    organicPositions: number[]
+    paaMentions: number
+    aiOverviewCitations: number
+    sourceUrls: string[]
+  }>
+  warnings: string[]
+}
+```
+
+### `sources.csv`
+
+Columns:
+
+```text
+surface,position,title,url,domain,question,answer_excerpt,is_target,is_competitor
+```
+
+### `competitors.csv`
+
+Columns:
+
+```text
+domain,organic_best_position,organic_count,paa_mentions,ai_overview_citations,source_url_count
+```
+
+### `brief.md`
+
+Sections:
+
+```md
+# SEO Evidence Brief
+
+## Search Intent
+## SERP Evidence
+## PAA Evidence
+## AI Overview Evidence
+## Target Presence
+## Competitor Sources
+## Content Gaps
+## Recommended Page Additions
+```
+
+Content should be evidence-grounded. If the workflow cannot determine a gap deterministically, phrase it as an evidence prompt:
+
+```md
+- Review these competitor source URLs before writing: ...
+- The target was not found in the captured AI/PAA sources.
+```
+
+### `tasks.md`
+
+Action checklist for an AI/code agent:
+
+```md
+# Agent Tasks
+
+- [ ] Read `evidence.json`.
+- [ ] Compare the target domain against `competitors.csv`.
+- [ ] Draft or update page sections using only evidence from `sources.csv`.
+- [ ] Add citations/links where appropriate.
+- [ ] Preserve claims that can be traced to an evidence row.
+```
+
+### `agent-instructions.md`
+
+Stable instructions:
+
+```md
+You are working from an MCP Scraper SEO evidence packet. Use `evidence.json` and CSV files as source of truth. Do not invent citations. If a recommendation is not supported by evidence, mark it as an assumption.
+```
+
+## HTML report
+
+Report sections:
+
+1. Executive summary.
+2. Target visibility.
+3. SERP table.
+4. AI Overview citations.
+5. PAA source table.
+6. Competitor domains.
+7. Agent task list.
+8. Diagnostics and warnings.
+
+## Partial-result behavior
+
+The workflow should return partial artifacts when one surface fails.
+
+Examples:
+
+- SERP succeeds, PAA CAPTCHA-exhausts: write report with SERP evidence and PAA warning.
+- PAA succeeds, SERP fails: write PAA evidence and source table.
+- Target domain omitted: skip target-presence calculations and focus on market evidence.
+
+## Files
+
+New:
+
+- `src/workflows/workflows/agent-packet.ts`
+- `src/workflows/renderers/agent-packet-report.ts`
+- `tests/unit/workflow-agent-packet.test.ts`
+- `tests/fixtures/workflows/agent-packet/`
+
+Modified:
+
+- `src/workflows/registry.ts`
+- `docs/specs/seo-workflow-runner-and-reports-spec.md` when implemented.
+
+## Tests
+
+- Fixture SERP + PAA evidence produces stable `evidence.json`.
+- Target domain matching handles `www.` and protocol differences.
+- CSV escaping handles commas, quotes, and newlines.
+- Partial SERP/PAA failures still write manifest and report.
+- `agent-instructions.md` warns against unsupported citations.
+
+## Future MCP tool
+
+After the CLI workflow is proven, consider an MCP tool:
+
+```text
+seo_agent_packet
+```
+
+Description should say it creates an evidence packet for downstream AI agents and returns local artifact paths. It should use `outputSchema` and structured content containing artifact paths, counts, and warnings.
+

package/docs/specs/cli-agent-wiring-spec.md

@@ -0,0 +1,203 @@
+# CLI Agent Wiring — Implementation Spec
+
+Make MCP Scraper easy to wire into AI agents. This is the lowest-friction market-share feature: users should be able to install, verify, and prompt their agent correctly without reading a long README.
+
+## Goals
+
+- Generate correct MCP configs for common AI-agent hosts.
+- Verify local setup with a single `doctor` command.
+- Provide copyable prompts for high-value workflows.
+- Keep stdio MCP servers protocol-safe and silent.
+- Make updates obvious: users on `mcp-scraper@latest` still need to restart their MCP client.
+
+## New binary
+
+Add a human-facing CLI binary:
+
+```json
+{
+  "bin": {
+    "mcp-scraper-cli": "dist/bin/mcp-scraper-cli.js"
+  }
+}
+```
+
+Do not repurpose `mcp-scraper`; it is already a stdio MCP server. Do not print terminal UI from any stdio MCP server.
+
+## Command surface
+
+```bash
+mcp-scraper-cli doctor
+mcp-scraper-cli agent install codex
+mcp-scraper-cli agent install claude
+mcp-scraper-cli agent install claude-desktop
+mcp-scraper-cli agent prompt list
+mcp-scraper-cli agent prompt agent-packet
+mcp-scraper-cli agent prompt local-competitive-audit
+mcp-scraper-cli agent config codex --api-key sk_live_...
+mcp-scraper-cli agent config claude --api-key sk_live_...
+```
+
+Flags:
+
+```bash
+--api-key <key>          # optional; if omitted, use env placeholder
+--output <path>          # write generated config/prompt to a file
+--combined               # default true; uses mcp-scraper-combined
+--browser-tools          # default true in combined mode
+--no-color
+--json                  # machine-readable output for scripts
+```
+
+## `doctor`
+
+Checks:
+
+| Check | Pass condition | Fix text |
+|---|---|---|
+| Node version | `>=20` | Install Node 20+ |
+| npm package | `npm view mcp-scraper version` succeeds | Check npm/network |
+| local CLI version | package version available from `PACKAGE_VERSION` | Update/reinstall |
+| API key | `MCP_SCRAPER_API_KEY` set or key file exists | Set env or config |
+| hosted API | `GET /me` with key succeeds | Verify key/account |
+| output dir | writable `MCP_SCRAPER_OUTPUT_DIR` or default downloads dir | Create/fix permissions |
+| MCP command | generated `npx -y -p mcp-scraper@latest mcp-scraper-combined` | Restart MCP client |
+| browser service readiness | only check if browser tools are requested | Configure API key |
+
+Output shape for `--json`:
+
+```ts
+interface DoctorOutput {
+  ok: boolean
+  version: string
+  npmLatest: string | null
+  checks: Array<{
+    id: string
+    label: string
+    status: 'pass' | 'warn' | 'fail' | 'skip'
+    detail: string
+    fix?: string
+  }>
+  recommendedConfig: {
+    command: 'npx'
+    args: string[]
+    env: Record<string, string>
+  }
+}
+```
+
+## Agent install generators
+
+### Codex
+
+Print TOML:
+
+```toml
+[mcp_servers.mcp-scraper]
+command = "npx"
+args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"]
+env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
+```
+
+Also print the restart requirement:
+
+```text
+Restart Codex so the MCP client starts a fresh npx process.
+```
+
+### Claude Code
+
+Print command:
+
+```bash
+claude mcp add mcp-scraper --scope user \
+  --env MCP_SCRAPER_API_KEY=sk_live_your_key \
+  -- npx -y -p mcp-scraper@latest mcp-scraper-combined
+```
+
+### Claude Desktop
+
+Print JSON:
+
+```json
+{
+  "mcpServers": {
+    "mcp-scraper": {
+      "command": "npx",
+      "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
+      "env": {
+        "MCP_SCRAPER_API_KEY": "sk_live_your_key"
+      }
+    }
+  }
+}
+```
+
+Also point to the MCPB:
+
+```text
+Desktop Extension: https://mcpscraper.dev/downloads/mcp-scraper.mcpb
+```
+
+## Prompt library
+
+Store prompt templates in:
+
+```text
+src/cli/prompts/
+  agent-packet.md
+  local-competitive-audit.md
+  directory-workflow.md
+  ai-citation-monitor.md
+  serp-brief.md
+```
+
+Command:
+
+```bash
+mcp-scraper-cli agent prompt agent-packet
+```
+
+Example `agent-packet.md`:
+
+```md
+Use MCP Scraper as the evidence layer. First call search/PAA tools for the keyword and target domain. Save all source URLs and cite every recommendation to a live evidence item. Then produce: brief.md, tasks.md, and a concise HTML-ready summary.
+```
+
+Prompts should name tool intent, not implementation vendors.
+
+## Files
+
+New:
+
+- `bin/mcp-scraper-cli.ts`
+- `src/cli/agent-config.ts`
+- `src/cli/doctor.ts`
+- `src/cli/prompts.ts`
+- `src/cli/prompts/*.md`
+- `tests/unit/cli-agent-config.test.ts`
+- `tests/unit/cli-doctor.test.ts`
+- `tests/unit/cli-prompts.test.ts`
+
+Modified:
+
+- `package.json` bin and tsup entries.
+- `README.md` install section.
+- `src/install-terminal.ts` to mention `mcp-scraper-cli doctor`.
+
+## Tests
+
+- Snapshot generated Codex TOML, Claude command, Claude Desktop JSON.
+- `doctor --json` with mocked checks returns stable JSON.
+- Prompt listing and prompt rendering works.
+- No generated prompt or CLI text includes hidden vendor names.
+- `mcp-scraper-cli --help` exits 0 and does not require an API key.
+
+## Release notes
+
+Installer message candidate:
+
+```text
+Newest: agent install helpers, setup doctor, and workflow prompts for Codex and Claude.
+```
+

package/docs/specs/deferred-work-spec.md

@@ -72,3 +72,15 @@ When a decision lands, the implementation is:
 ## 7. API Forge
 
 Specified separately in `docs/specs/api-forge-spec.md`. Recommended to schedule after §1 and §2 (ops hygiene first), before §3/§4 (it produces revenue surface; progress/OAuth polish can follow demand).
+
+## 8. SEO CLI growth workflows
+
+Specified separately in `docs/specs/seo-cli-growth-roadmap-spec.md`, with buildable sub-specs for agent wiring, workflow reports, agent-ready SEO packets, local competitive audits, and scheduled workflow APIs:
+
+- `docs/specs/cli-agent-wiring-spec.md`
+- `docs/specs/seo-workflow-runner-and-reports-spec.md`
+- `docs/specs/agent-ready-seo-packet-spec.md`
+- `docs/specs/local-competitive-audit-spec.md`
+- `docs/specs/scheduled-workflows-api-spec.md`
+
+Recommended priority: ship agent wiring and local workflow reports first, then the agent packet and local competitive audit workflows, then hosted schedules/API once report artifacts are durable.