mcp-scraper 0.2.7 → 0.2.9

package/docs/specs/local-competitive-audit-spec.md

@@ -0,0 +1,312 @@
+# Local Competitive Audit Workflow — Implementation Spec
+
+`local-competitive-audit` turns city selection, Maps search, and review/profile hydration into an SEO market audit. It builds on existing `directory_workflow`, `maps_search`, and `maps_place_intel`.
+
+## User promise
+
+```bash
+mcp-scraper-cli workflow run local-competitive-audit \
+  --query "roofers" \
+  --state TN \
+  --min-pop 100000 \
+  --per-city 20 \
+  --hydrate-top 5 \
+  --reviews 50 \
+  --concurrency 5
+```
+
+The workflow answers:
+
+- Which businesses rank in each city?
+- What position are they in?
+- How many reviews and what star rating?
+- What categories are competitors using?
+- What do reviews say customers praise or complain about?
+- Which cities look easiest or hardest to compete in?
+- Which competitors should the client study?
+
+## Current capabilities to reuse
+
+Existing `directory_workflow` already:
+
+- Selects US cities from Census population data.
+- Optionally enriches with US ZIP groups.
+- Runs Maps searches city-by-city.
+- Saves CSV with `source_location`, `result_position`, `business_name`, `review_stars`, `category`, address, phone, URLs, CIDs, population, ZIP groups.
+
+Existing `maps_place_intel` already:
+
+- Hydrates one known/named business.
+- Returns rating, review count, category, hours, phone, website, booking URL, review histogram, review topics, about attributes, and optional review cards.
+
+## Required base improvement
+
+Add `review_count` to Maps search and directory CSV output.
+
+Current Maps search extraction includes `reviewCount` in structured result types. Directory CSV currently writes `review_stars` but not review count. Add:
+
+```text
+review_count
+```
+
+Next to `review_stars`.
+
+Files:
+
+- `src/directory/directory-workflow.ts`
+- `src/mcp/mcp-tool-schemas.ts` descriptions/output schema if needed.
+- `src/mcp/mcp-response-formatter.ts` if table output should show count.
+- Tests that assert CSV header and row value.
+
+## Inputs
+
+CLI flags:
+
+```bash
+--query <business category>       # required, e.g. roofers
+--state <US state>                # required
+--min-pop <number>                # default 100000
+--max-cities <number>             # optional
+--per-city <number>               # default 20, cap 50
+--hydrate-top <number>            # default 5, cap 10
+--reviews <number>                # default 50, cap 200
+--concurrency <number>            # default 5
+--proxy-mode location             # default
+--include-reviews                 # default true when reviews > 0
+--output-dir <path>
+--return-partial                  # default true
+```
+
+Structured input:
+
+```ts
+interface LocalCompetitiveAuditInput {
+  query: string
+  state: string
+  minPopulation: number
+  maxCities?: number
+  maxResultsPerCity: number
+  hydrateTop: number
+  maxReviews: number
+  concurrency: number
+  proxyMode: 'location' | 'configured' | 'none'
+  returnPartial: boolean
+}
+```
+
+## Workflow steps
+
+### 1. Market seed
+
+Call `directory_workflow` with:
+
+- `query`
+- `state`
+- `minPopulation`
+- `maxResultsPerCity`
+- `concurrency`
+- `proxyMode: "location"` by default
+
+Save the raw structured result and directory CSV.
+
+### 2. Competitor selection
+
+For each successful city:
+
+- Select top `hydrateTop` businesses by `result_position`.
+- Deduplicate by CID first, then place URL, then normalized name + city.
+- Carry city/rank context forward into hydration.
+
+### 3. Profile hydration
+
+For each selected competitor:
+
+Call `maps_place_intel` with:
+
+```ts
+{
+  businessName,
+  location: sourceLocation,
+  includeReviews: maxReviews > 0,
+  maxReviews
+}
+```
+
+Concurrency:
+
+- Default 3 profile hydrations at a time.
+- Keep city Maps search concurrency separate from hydration concurrency.
+
+### 4. Review insight extraction
+
+No server-side LLM required for first version. Use structured Maps profile evidence and extractive review analysis.
+
+Inputs:
+
+- `reviewCount`
+- `reviewHistogram`
+- `reviewTopics`
+- `reviews[].stars`
+- `reviews[].text`
+- `reviews[].ownerResponse`
+- `aboutAttributes`
+
+Derived fields:
+
+```ts
+interface ReviewInsights {
+  reviewCount: string | null
+  averageRating: string | null
+  histogram: Array<{ stars: number; count: string }>
+  topics: Array<{ label: string; count: string }>
+  praiseTerms: Array<{ term: string; count: number }>
+  complaintTerms: Array<{ term: string; count: number }>
+  samplePositiveReviews: Array<{ stars: string | null; text: string }>
+  sampleNegativeReviews: Array<{ stars: string | null; text: string }>
+  ownerResponseRate: number | null
+}
+```
+
+Simple extraction rules:
+
+- Positive sample: 5-star reviews with text, longest useful snippets first.
+- Negative sample: 1-3 star reviews with text, longest useful snippets first.
+- Praise/complaint terms: n-grams from review text after stopword removal, separated by star band.
+- Owner response rate: reviews with `ownerResponse` / reviews with text.
+
+### 5. Market scoring
+
+Calculate city-level and competitor-level scores.
+
+City fields:
+
+```ts
+interface CityAuditSummary {
+  city: string
+  state: string
+  population: number | null
+  resultCount: number
+  medianReviewCount: number | null
+  medianRating: number | null
+  topThreeAverageReviewCount: number | null
+  categories: Array<{ category: string; count: number }>
+  opportunityScore: number
+  difficultyScore: number
+  notes: string[]
+}
+```
+
+Initial scoring:
+
+- Difficulty increases with top-three review count, rating, number of results with websites, and category consistency.
+- Opportunity increases when top results have low review counts, missing websites, weak ratings, sparse categories, or incomplete profiles.
+
+Do not overstate precision. Label these as heuristic scores.
+
+## Artifacts
+
+```text
+local-competitive-audit/
+  manifest.json
+  report.html
+  summary.md
+  evidence.json
+  city-summary.csv
+  competitors.csv
+  review-insights.csv
+  tasks.md
+  raw/directory-workflow.json
+  raw/maps-place-intel/*.json
+  logs/attempts.jsonl
+```
+
+### `competitors.csv`
+
+Columns:
+
+```text
+city,state,source_location,result_position,business_name,category,review_stars,review_count,phone,website_url,place_url,cid,cid_decimal,hydrated,review_topics,owner_response_rate
+```
+
+### `review-insights.csv`
+
+Columns:
+
+```text
+city,business_name,cid,review_count,average_rating,topic,praise_terms,complaint_terms,positive_sample,negative_sample
+```
+
+### `city-summary.csv`
+
+Columns:
+
+```text
+city,state,population,result_count,median_review_count,median_rating,top_three_average_review_count,top_categories,opportunity_score,difficulty_score
+```
+
+## HTML report sections
+
+1. Executive summary.
+2. City opportunity table.
+3. Top competitors by city.
+4. Review count/rating distribution.
+5. Category patterns.
+6. Review themes: praise and complaints.
+7. Weak competitor opportunities.
+8. Recommended content/GBP actions.
+9. Diagnostics: failed cities, failed profile hydrations, attempts.
+
+## Agent task output
+
+`tasks.md` should include:
+
+```md
+- [ ] Build city landing page recommendations from `city-summary.csv`.
+- [ ] Use review praise/complaint themes as customer-language evidence.
+- [ ] Identify competitors with low review count but high rank.
+- [ ] Compare target GBP category and review profile against top competitors.
+- [ ] Draft GBP/content improvements without inventing review claims.
+```
+
+## MCP tool option
+
+After CLI proof, add an MCP tool:
+
+```text
+local_competitive_audit
+```
+
+Description:
+
+```text
+Run a local SEO competitive audit by selecting city markets, searching Google Maps competitors, hydrating top profiles, collecting review/category evidence, and saving CSV/HTML artifacts. Use when the user asks who ranks locally, how strong competitors are, what reviews say, or where market opportunities exist.
+```
+
+This tool should return `structuredContent` with artifact paths and city/competitor summaries.
+
+## Credit model
+
+Estimate:
+
+- Directory seed: `cities * maps_search`.
+- Profile hydration: `hydrated_profiles * maps_place_intel`.
+- Reviews: `collected_review_cards * maps_review`.
+
+For CLI runs, print estimated costs before hosted execution unless `--yes` is passed.
+
+## Partial-result behavior
+
+- If some cities fail, still audit successful cities.
+- If hydration fails, keep base directory row and mark `hydrated=false`.
+- If reviews are unavailable, keep profile metadata and mark `reviewsStatus`.
+- If all directory cities fail, workflow fails with no audit report except diagnostics.
+
+## Tests
+
+- Directory fixture + profile fixtures produce all three CSVs.
+- `review_count` is preserved from Maps search into directory CSV.
+- Deduping by CID prevents duplicate hydration.
+- Review insight extraction is deterministic.
+- Partial hydration failure writes warning and keeps report.
+- HTML report escapes review text.
+

package/docs/specs/scheduled-workflows-api-spec.md

@@ -0,0 +1,304 @@
+# Scheduled Workflows API — Implementation Spec
+
+Scheduled workflows turn MCP Scraper from one-off data pulls into recurring SEO intelligence. This spec adds hosted schedule CRUD, run history, webhook delivery, and report retrieval for workflow runs.
+
+## Goals
+
+- Schedule recurring workflow runs through API/CLI.
+- Persist run history and artifacts.
+- Send webhook callbacks on completion/failure.
+- Support rank-monitor style use cases without replacing `rank_tracker_blueprint`.
+- Keep local CLI and hosted scheduler using the same workflow registry.
+
+## Non-goals
+
+- Do not build a full dashboard in this phase.
+- Do not make `rank_tracker_blueprint` execute work; it remains a planning tool.
+- Do not require OAuth; API key auth is enough for this phase.
+
+## Command surface
+
+```bash
+mcp-scraper-cli schedule create local-competitive-audit \
+  --weekly \
+  --query roofers \
+  --state TN \
+  --min-pop 100000 \
+  --webhook https://example.com/mcp-scraper-hook
+
+mcp-scraper-cli schedule list
+mcp-scraper-cli schedule run <schedule-id>
+mcp-scraper-cli schedule pause <schedule-id>
+mcp-scraper-cli schedule resume <schedule-id>
+mcp-scraper-cli schedule delete <schedule-id>
+mcp-scraper-cli runs list
+mcp-scraper-cli runs status <run-id>
+mcp-scraper-cli runs download <run-id>
+```
+
+## API routes
+
+Mount:
+
+```text
+/workflows
+```
+
+Routes:
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/workflows/definitions` | List available workflows and input schemas. |
+| `POST` | `/workflows/run` | Start a manual workflow run. |
+| `GET` | `/workflows/runs/:id` | Get run status, manifest, artifact links. |
+| `GET` | `/workflows/runs/:id/artifacts/:artifactId` | Download artifact. |
+| `POST` | `/workflows/schedules` | Create schedule. |
+| `GET` | `/workflows/schedules` | List schedules for user. |
+| `PATCH` | `/workflows/schedules/:id` | Pause/resume/update. |
+| `DELETE` | `/workflows/schedules/:id` | Delete schedule. |
+| `POST` | `/workflows/schedules/:id/run` | Manual run of a schedule. |
+
+All routes use `createApiKeyAuth`.
+
+## Database schema
+
+Add to `src/api/db.ts` migrations:
+
+```sql
+CREATE TABLE IF NOT EXISTS workflow_schedules (
+  id             TEXT PRIMARY KEY,
+  user_id        INTEGER NOT NULL,
+  workflow_id    TEXT NOT NULL,
+  name           TEXT NOT NULL,
+  input_json     TEXT NOT NULL,
+  cadence        TEXT NOT NULL, -- daily | weekly | monthly | custom
+  cron_expr      TEXT,
+  timezone       TEXT NOT NULL DEFAULT 'UTC',
+  webhook_url    TEXT,
+  status         TEXT NOT NULL DEFAULT 'active', -- active | paused | deleted
+  next_run_at    TEXT,
+  last_run_at    TEXT,
+  created_at     TEXT NOT NULL,
+  updated_at     TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS workflow_runs (
+  id             TEXT PRIMARY KEY,
+  user_id        INTEGER NOT NULL,
+  schedule_id    TEXT,
+  workflow_id    TEXT NOT NULL,
+  input_json     TEXT NOT NULL,
+  status         TEXT NOT NULL DEFAULT 'queued', -- queued | running | succeeded | partial | failed | cancelled
+  manifest_json  TEXT,
+  error_message  TEXT,
+  queued_at      TEXT NOT NULL,
+  started_at     TEXT,
+  completed_at   TEXT,
+  idempotency_key TEXT
+);
+
+CREATE TABLE IF NOT EXISTS workflow_artifacts (
+  id             TEXT PRIMARY KEY,
+  run_id         TEXT NOT NULL,
+  user_id        INTEGER NOT NULL,
+  kind           TEXT NOT NULL,
+  label          TEXT NOT NULL,
+  path           TEXT NOT NULL,
+  content_type   TEXT NOT NULL,
+  bytes          INTEGER,
+  rows_count     INTEGER,
+  created_at     TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS workflow_webhook_deliveries (
+  id             TEXT PRIMARY KEY,
+  run_id         TEXT NOT NULL,
+  schedule_id    TEXT,
+  user_id        INTEGER NOT NULL,
+  webhook_url    TEXT NOT NULL,
+  status         TEXT NOT NULL, -- pending | delivered | failed
+  attempts       INTEGER NOT NULL DEFAULT 0,
+  last_error     TEXT,
+  created_at     TEXT NOT NULL,
+  delivered_at   TEXT
+);
+```
+
+Indexes:
+
+```sql
+CREATE INDEX IF NOT EXISTS idx_workflow_schedules_due ON workflow_schedules(status, next_run_at);
+CREATE INDEX IF NOT EXISTS idx_workflow_runs_user_created ON workflow_runs(user_id, queued_at);
+CREATE INDEX IF NOT EXISTS idx_workflow_artifacts_run ON workflow_artifacts(run_id);
+```
+
+## Worker model
+
+Initial implementation can use a hosted cron route:
+
+```text
+POST /workflows/cron/dispatch
+```
+
+Protected by `CRON_SECRET`.
+
+Dispatch steps:
+
+1. Find active schedules where `next_run_at <= now`.
+2. Create `workflow_runs` row with idempotency key `<scheduleId>:<scheduledTime>`.
+3. Enqueue/run workflow.
+4. Advance `next_run_at`.
+
+If no queue is available, run small workflows inline and mark long workflows as queued for later worker integration. The workflow registry should not care whether a run is local, HTTP, or queued.
+
+## Run execution
+
+`WorkflowRunService`:
+
+```ts
+interface WorkflowRunService {
+  startManualRun(userId: number, workflowId: string, input: unknown): Promise<WorkflowRun>
+  startScheduledRun(schedule: WorkflowSchedule, scheduledFor: string): Promise<WorkflowRun>
+  executeRun(runId: string): Promise<void>
+  getRun(userId: number, runId: string): Promise<WorkflowRunWithArtifacts>
+}
+```
+
+Execution rules:
+
+- Debit estimated credits before paid API work.
+- Refund failed units when workflow has per-item partial failures.
+- Persist `manifest_json` after every major phase so status pages are useful during partial failure.
+- Save artifacts through a hosted artifact writer. Local paths are not enough for hosted runs.
+
+## Artifact storage
+
+Phase 1:
+
+- Store artifacts on local/Vercel ephemeral disk only for immediate response is not durable enough.
+- Therefore hosted scheduled workflows need object storage before public rollout.
+
+Recommended minimal storage:
+
+- S3/R2-compatible bucket.
+- Store by `user_id/run_id/artifact_id`.
+- DB stores key/path and content type.
+
+Local CLI can continue writing to `~/Downloads/mcp-scraper`.
+
+## Webhooks
+
+Payload:
+
+```ts
+interface WorkflowWebhookPayload {
+  event: 'workflow.run.succeeded' | 'workflow.run.partial' | 'workflow.run.failed'
+  runId: string
+  scheduleId: string | null
+  workflowId: string
+  status: string
+  completedAt: string
+  summary: {
+    title: string
+    counts: Record<string, number>
+    warnings: string[]
+    errors: string[]
+  }
+  artifacts: Array<{
+    kind: string
+    label: string
+    downloadUrl: string
+  }>
+}
+```
+
+Security:
+
+- Sign payloads with `MCP_SCRAPER_WEBHOOK_SECRET`.
+- Header: `x-mcp-scraper-signature`.
+- Include timestamp header to prevent replay.
+
+Retries:
+
+- 3 attempts with exponential backoff.
+- Persist every attempt in `workflow_webhook_deliveries`.
+
+## Scheduling modes
+
+Supported:
+
+```ts
+type WorkflowCadence = 'daily' | 'weekly' | 'monthly' | 'custom'
+```
+
+For `custom`, require cron expression and timezone.
+
+Validation:
+
+- Minimum cadence: daily for expensive live-web workflows unless account tier allows more.
+- Cap concurrent active schedules per account tier.
+- Cap max run timeout.
+
+## Integration with `rank_tracker_blueprint`
+
+`rank_tracker_blueprint` should mention scheduled workflows as an execution option:
+
+- Use `scheduled-workflows` for hosted recurring MCP Scraper runs.
+- Use generated rank-tracker schema when the user wants their own database/dashboard.
+
+Do not make the blueprint depend on schedules.
+
+## Tests
+
+- Schedule CRUD auth and ownership tests.
+- Cron dispatch only runs due active schedules.
+- Idempotency prevents duplicate runs for same scheduled time.
+- Manual run creates run row and artifacts.
+- Webhook signing and retry tests.
+- Artifact download requires owning API key.
+- Schedule create rejects unknown workflow IDs and invalid inputs.
+
+## Rollout phases
+
+## Implementation status
+
+Implemented in the first working pass:
+
+- Database migrations and helpers for `workflow_schedules`, `workflow_runs`, `workflow_artifacts`, and `workflow_webhook_deliveries`.
+- Authenticated hosted routes for definitions, manual runs, run status, artifact downloads, schedule CRUD, manual schedule runs, and cron dispatch.
+- Cron integration from the main `/cron/tick` route.
+- CLI commands: `schedule create/list/run/pause/resume/delete` and `runs list/status/download`.
+- Signed webhook delivery when `MCP_SCRAPER_WEBHOOK_SECRET` is configured, with delivery history.
+
+Still required before broad public rollout:
+
+- Durable object storage for hosted artifacts. Current hosted artifacts are filesystem-backed, which is enough for local/server testing and immediate retrieval but not durable across serverless cold starts or instance changes.
+- More route coverage for cron idempotency, ownership on artifact downloads, and webhook retries.
+- Account-tier schedule caps and preflight cost estimation before expensive hosted runs.
+
+### Phase 1 — API definitions and manual runs
+
+- `/workflows/definitions`
+- `/workflows/run`
+- `/workflows/runs/:id`
+- Local or test artifact storage.
+
+### Phase 2 — Durable artifact storage
+
+- S3/R2 bucket.
+- Artifact download routes.
+
+### Phase 3 — Schedules and cron dispatch
+
+- Schedule CRUD.
+- Cron-protected dispatcher.
+
+### Phase 4 — Webhooks
+
+- Signed completion callbacks.
+- Retry/delivery history.
+
+### Phase 5 — CLI schedule commands
+
+- `schedule create/list/run/pause/resume/delete`.
+- `runs list/status/download`.