mcp-scraper 0.5.0 → 0.7.0

package/docs/specs/cli-agent-wiring-spec.md

@@ -1,206 +0,0 @@
-# CLI Agent Wiring — Implementation Spec
-
-Make MCP Scraper easy to wire into AI agents. This is the lowest-friction market-share feature: users should be able to install, verify, and prompt their agent correctly without reading a long README.
-
-## Goals
-
-- Generate correct MCP configs for common AI-agent hosts.
-- Verify local setup with a single `doctor` command.
-- Provide copyable prompts for high-value workflows.
-- Keep stdio MCP servers protocol-safe and silent.
-- Make updates obvious: users on `mcp-scraper@latest` still need to restart their MCP client.
-
-## New binary
-
-Add a human-facing CLI binary:
-
-```json
-{
-  "bin": {
-    "mcp-scraper-cli": "dist/bin/mcp-scraper-cli.js"
-  }
-}
-```
-
-Do not repurpose `mcp-scraper`; it is already a stdio MCP server. Do not print terminal UI from any stdio MCP server.
-
-## Command surface
-
-```bash
-mcp-scraper-cli doctor
-mcp-scraper-cli agent install codex
-mcp-scraper-cli agent install claude
-mcp-scraper-cli agent install claude-desktop
-mcp-scraper-cli agent prompt list
-mcp-scraper-cli agent prompt agent-packet
-mcp-scraper-cli agent prompt local-competitive-audit
-mcp-scraper-cli agent config codex --api-key sk_live_...
-mcp-scraper-cli agent config claude --api-key sk_live_...
-```
-
-Flags:
-
-```bash
---api-key <key>          # optional; if omitted, use env placeholder
---output <path>          # write generated config/prompt to a file
---combined               # default true; uses mcp-scraper-combined
---browser-tools          # default true in combined mode
---no-color
---json                  # machine-readable output for scripts
-```
-
-## `doctor`
-
-Checks:
-
-| Check | Pass condition | Fix text |
-|---|---|---|
-| Node version | `>=20` | Install Node 20+ |
-| npm package | `npm view mcp-scraper version` succeeds | Check npm/network |
-| local CLI version | package version available from `PACKAGE_VERSION` | Update/reinstall |
-| API key | `MCP_SCRAPER_API_KEY` set or key file exists | Set env or config |
-| hosted API | `GET /me` with key succeeds | Verify key/account |
-| output dir | writable `MCP_SCRAPER_OUTPUT_DIR` or default downloads dir | Create/fix permissions |
-| MCP command | generated `npx -y -p mcp-scraper@latest mcp-scraper-combined` | Restart MCP client |
-| browser service readiness | only check if browser tools are requested | Configure API key |
-
-Output shape for `--json`:
-
-```ts
-interface DoctorOutput {
-  ok: boolean
-  version: string
-  npmLatest: string | null
-  checks: Array<{
-    id: string
-    label: string
-    status: 'pass' | 'warn' | 'fail' | 'skip'
-    detail: string
-    fix?: string
-  }>
-  recommendedConfig: {
-    command: 'npx'
-    args: string[]
-    env: Record<string, string>
-  }
-}
-```
-
-## Agent install generators
-
-### Codex
-
-Print TOML:
-
-```toml
-[mcp_servers.mcp-scraper]
-command = "npx"
-args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"]
-env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
-```
-
-Also print the restart requirement:
-
-```text
-Restart Codex so the MCP client starts a fresh npx process.
-```
-
-### Claude Code
-
-Print command:
-
-```bash
-claude mcp add mcp-scraper --scope user \
-  --env MCP_SCRAPER_API_KEY=sk_live_your_key \
-  -- npx -y -p mcp-scraper@latest mcp-scraper-combined
-```
-
-### Claude Desktop
-
-Print JSON:
-
-```json
-{
-  "mcpServers": {
-    "mcp-scraper": {
-      "command": "npx",
-      "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
-      "env": {
-        "MCP_SCRAPER_API_KEY": "sk_live_your_key"
-      }
-    }
-  }
-}
-```
-
-Also point to the MCPB:
-
-```text
-Desktop Extension: https://mcpscraper.dev/downloads/mcp-scraper.mcpb
-```
-
-## Prompt library
-
-Store prompt templates in:
-
-```text
-src/cli/prompts/
-  agent-packet.md
-  local-competitive-audit.md
-  directory-workflow.md
-  map-comparison.md
-  serp-comparison.md
-  paa-expansion-brief.md
-  ai-overview-language.md
-  ai-citation-monitor.md
-  serp-brief.md
-```
-
-Command:
-
-```bash
-mcp-scraper-cli agent prompt agent-packet
-```
-
-Example `agent-packet.md`:
-
-```md
-Use MCP Scraper as the evidence layer. First call search/PAA tools for the keyword and target domain. Save all source URLs and cite every recommendation to a live evidence item. Then produce: brief.md, tasks.md, and a concise HTML-ready summary.
-```
-
-Prompts should name tool intent, not implementation vendors.
-
-## Files
-
-New:
-
-- `bin/mcp-scraper-cli.ts`
-- `src/cli/agent-config.ts`
-- `src/cli/doctor.ts`
-- `src/cli/prompts.ts`
-- `src/cli/prompts/*.md`
-- `tests/unit/cli-agent-config.test.ts`
-- `tests/unit/cli-doctor.test.ts`
-- `tests/unit/cli-prompts.test.ts`
-
-Modified:
-
-- `package.json` bin and tsup entries.
-- `README.md` install section.
-- `src/install-terminal.ts` to mention `mcp-scraper-cli doctor`.
-
-## Tests
-
-- Snapshot generated Codex TOML, Claude command, Claude Desktop JSON.
-- `doctor --json` with mocked checks returns stable JSON.
-- Prompt listing and prompt rendering works.
-- No generated prompt or CLI text includes hidden vendor names.
-- `mcp-scraper-cli --help` exits 0 and does not require an API key.
-
-## Release notes
-
-Installer message candidate:
-
-```text
-Newest: agent install helpers, setup doctor, and workflow prompts for Codex and Claude.
-```

package/docs/specs/local-competitive-audit-spec.md

@@ -1,311 +0,0 @@
-# Local Competitive Audit Workflow — Implementation Spec
-
-`local-competitive-audit` turns city selection, Maps search, and review/profile hydration into an SEO market audit. It builds on existing `directory_workflow`, `maps_search`, and `maps_place_intel`.
-
-## User promise
-
-```bash
-mcp-scraper-cli workflow run local-competitive-audit \
-  --query "roofers" \
-  --state TN \
-  --min-pop 100000 \
-  --per-city 20 \
-  --hydrate-top 5 \
-  --reviews 50 \
-  --concurrency 5
-```
-
-The workflow answers:
-
-- Which businesses rank in each city?
-- What position are they in?
-- How many reviews and what star rating?
-- What categories are competitors using?
-- What do reviews say customers praise or complain about?
-- Which cities look easiest or hardest to compete in?
-- Which competitors should the client study?
-
-## Current capabilities to reuse
-
-Existing `directory_workflow` already:
-
-- Selects US cities from Census population data.
-- Optionally enriches with US ZIP groups.
-- Runs Maps searches city-by-city.
-- Saves CSV with `source_location`, `result_position`, `business_name`, `review_stars`, `category`, address, phone, URLs, CIDs, population, ZIP groups.
-
-Existing `maps_place_intel` already:
-
-- Hydrates one known/named business.
-- Returns rating, review count, category, hours, phone, website, booking URL, review histogram, review topics, about attributes, and optional review cards.
-
-## Required base improvement
-
-Add `review_count` to Maps search and directory CSV output.
-
-Current Maps search extraction includes `reviewCount` in structured result types. Directory CSV currently writes `review_stars` but not review count. Add:
-
-```text
-review_count
-```
-
-Next to `review_stars`.
-
-Files:
-
-- `src/directory/directory-workflow.ts`
-- `src/mcp/mcp-tool-schemas.ts` descriptions/output schema if needed.
-- `src/mcp/mcp-response-formatter.ts` if table output should show count.
-- Tests that assert CSV header and row value.
-
-## Inputs
-
-CLI flags:
-
-```bash
---query <business category>       # required, e.g. roofers
---state <US state>                # required
---min-pop <number>                # default 100000
---max-cities <number>             # optional
---per-city <number>               # default 20, cap 50
---hydrate-top <number>            # default 5, cap 10
---reviews <number>                # default 50, cap 200
---concurrency <number>            # default 5
---proxy-mode location             # default
---include-reviews                 # default true when reviews > 0
---output-dir <path>
---return-partial                  # default true
-```
-
-Structured input:
-
-```ts
-interface LocalCompetitiveAuditInput {
-  query: string
-  state: string
-  minPopulation: number
-  maxCities?: number
-  maxResultsPerCity: number
-  hydrateTop: number
-  maxReviews: number
-  concurrency: number
-  proxyMode: 'location' | 'configured' | 'none'
-  returnPartial: boolean
-}
-```
-
-## Workflow steps
-
-### 1. Market seed
-
-Call `directory_workflow` with:
-
-- `query`
-- `state`
-- `minPopulation`
-- `maxResultsPerCity`
-- `concurrency`
-- `proxyMode: "location"` by default for Maps evidence; use `"configured"` only when city/ZIP-targeted residential proxy evidence should be skipped
-
-Save the raw structured result and directory CSV.
-
-### 2. Competitor selection
-
-For each successful city:
-
-- Select top `hydrateTop` businesses by `result_position`.
-- Deduplicate by CID first, then place URL, then normalized name + city.
-- Carry city/rank context forward into hydration.
-
-### 3. Profile hydration
-
-For each selected competitor:
-
-Call `maps_place_intel` with:
-
-```ts
-{
-  businessName,
-  location: sourceLocation,
-  includeReviews: maxReviews > 0,
-  maxReviews
-}
-```
-
-Concurrency:
-
-- Default 3 profile hydrations at a time.
-- Keep city Maps search concurrency separate from hydration concurrency.
-
-### 4. Review insight extraction
-
-No server-side LLM required for first version. Use structured Maps profile evidence and extractive review analysis.
-
-Inputs:
-
-- `reviewCount`
-- `reviewHistogram`
-- `reviewTopics`
-- `reviews[].stars`
-- `reviews[].text`
-- `reviews[].ownerResponse`
-- `aboutAttributes`
-
-Derived fields:
-
-```ts
-interface ReviewInsights {
-  reviewCount: string | null
-  averageRating: string | null
-  histogram: Array<{ stars: number; count: string }>
-  topics: Array<{ label: string; count: string }>
-  praiseTerms: Array<{ term: string; count: number }>
-  complaintTerms: Array<{ term: string; count: number }>
-  samplePositiveReviews: Array<{ stars: string | null; text: string }>
-  sampleNegativeReviews: Array<{ stars: string | null; text: string }>
-  ownerResponseRate: number | null
-}
-```
-
-Simple extraction rules:
-
-- Positive sample: 5-star reviews with text, longest useful snippets first.
-- Negative sample: 1-3 star reviews with text, longest useful snippets first.
-- Praise/complaint terms: n-grams from review text after stopword removal, separated by star band.
-- Owner response rate: reviews with `ownerResponse` / reviews with text.
-
-### 5. Market scoring
-
-Calculate city-level and competitor-level scores.
-
-City fields:
-
-```ts
-interface CityAuditSummary {
-  city: string
-  state: string
-  population: number | null
-  resultCount: number
-  medianReviewCount: number | null
-  medianRating: number | null
-  topThreeAverageReviewCount: number | null
-  categories: Array<{ category: string; count: number }>
-  opportunityScore: number
-  difficultyScore: number
-  notes: string[]
-}
-```
-
-Initial scoring:
-
-- Difficulty increases with top-three review count, rating, number of results with websites, and category consistency.
-- Opportunity increases when top results have low review counts, missing websites, weak ratings, sparse categories, or incomplete profiles.
-
-Do not overstate precision. Label these as heuristic scores.
-
-## Artifacts
-
-```text
-local-competitive-audit/
-  manifest.json
-  report.html
-  summary.md
-  evidence.json
-  city-summary.csv
-  competitors.csv
-  review-insights.csv
-  tasks.md
-  raw/directory-workflow.json
-  raw/maps-place-intel/*.json
-  logs/attempts.jsonl
-```
-
-### `competitors.csv`
-
-Columns:
-
-```text
-city,state,source_location,result_position,business_name,category,review_stars,review_count,phone,website_url,place_url,cid,cid_decimal,hydrated,review_topics,owner_response_rate
-```
-
-### `review-insights.csv`
-
-Columns:
-
-```text
-city,business_name,cid,review_count,average_rating,topic,praise_terms,complaint_terms,positive_sample,negative_sample
-```
-
-### `city-summary.csv`
-
-Columns:
-
-```text
-city,state,population,result_count,median_review_count,median_rating,top_three_average_review_count,top_categories,opportunity_score,difficulty_score
-```
-
-## HTML report sections
-
-1. Executive summary.
-2. City opportunity table.
-3. Top competitors by city.
-4. Review count/rating distribution.
-5. Category patterns.
-6. Review themes: praise and complaints.
-7. Weak competitor opportunities.
-8. Recommended content/GBP actions.
-9. Diagnostics: failed cities, failed profile hydrations, attempts.
-
-## Agent task output
-
-`tasks.md` should include:
-
-```md
-- [ ] Build city landing page recommendations from `city-summary.csv`.
-- [ ] Use review praise/complaint themes as customer-language evidence.
-- [ ] Identify competitors with low review count but high rank.
-- [ ] Compare target GBP category and review profile against top competitors.
-- [ ] Draft GBP/content improvements without inventing review claims.
-```
-
-## MCP tool option
-
-After CLI proof, add an MCP tool:
-
-```text
-local_competitive_audit
-```
-
-Description:
-
-```text
-Run a local SEO competitive audit by selecting city markets, searching Google Maps competitors, hydrating top profiles, collecting review/category evidence, and saving CSV/HTML artifacts. Use when the user asks who ranks locally, how strong competitors are, what reviews say, or where market opportunities exist.
-```
-
-This tool should return `structuredContent` with artifact paths and city/competitor summaries.
-
-## Credit model
-
-Estimate:
-
-- Directory seed: `cities * maps_search`.
-- Profile hydration: `hydrated_profiles * maps_place_intel`.
-- Reviews: `collected_review_cards * maps_review`.
-
-For CLI runs, print estimated costs before hosted execution unless `--yes` is passed.
-
-## Partial-result behavior
-
-- If some cities fail, still audit successful cities.
-- If hydration fails, keep base directory row and mark `hydrated=false`.
-- If reviews are unavailable, keep profile metadata and mark `reviewsStatus`.
-- If all directory cities fail, workflow fails with no audit report except diagnostics.
-
-## Tests
-
-- Directory fixture + profile fixtures produce all three CSVs.
-- `review_count` is preserved from Maps search into directory CSV.
-- Deduping by CID prevents duplicate hydration.
-- Review insight extraction is deterministic.
-- Partial hydration failure writes warning and keeps report.
-- HTML report escapes review text.