mcp-scraper 0.6.0 → 0.8.0

package/docs/specs/scheduled-workflows-api-spec.md

@@ -1,304 +0,0 @@
-# Scheduled Workflows API — Implementation Spec
-
-Scheduled workflows turn MCP Scraper from one-off data pulls into recurring SEO intelligence. This spec adds hosted schedule CRUD, run history, webhook delivery, and report retrieval for workflow runs.
-
-## Goals
-
-- Schedule recurring workflow runs through API/CLI.
-- Persist run history and artifacts.
-- Send webhook callbacks on completion/failure.
-- Support rank-monitor style use cases without replacing `rank_tracker_blueprint`.
-- Keep local CLI and hosted scheduler using the same workflow registry.
-
-## Non-goals
-
-- Do not build a full dashboard in this phase.
-- Do not make `rank_tracker_blueprint` execute work; it remains a planning tool.
-- Do not require OAuth; API key auth is enough for this phase.
-
-## Command surface
-
-```bash
-mcp-scraper-cli schedule create local-competitive-audit \
-  --weekly \
-  --query roofers \
-  --state TN \
-  --min-pop 100000 \
-  --webhook https://example.com/mcp-scraper-hook
-
-mcp-scraper-cli schedule list
-mcp-scraper-cli schedule run <schedule-id>
-mcp-scraper-cli schedule pause <schedule-id>
-mcp-scraper-cli schedule resume <schedule-id>
-mcp-scraper-cli schedule delete <schedule-id>
-mcp-scraper-cli runs list
-mcp-scraper-cli runs status <run-id>
-mcp-scraper-cli runs download <run-id>
-```
-
-## API routes
-
-Mount:
-
-```text
-/workflows
-```
-
-Routes:
-
-| Method | Path | Purpose |
-|---|---|---|
-| `GET` | `/workflows/definitions` | List available workflows and input schemas. |
-| `POST` | `/workflows/run` | Start a manual workflow run. |
-| `GET` | `/workflows/runs/:id` | Get run status, manifest, artifact links. |
-| `GET` | `/workflows/runs/:id/artifacts/:artifactId` | Download artifact. |
-| `POST` | `/workflows/schedules` | Create schedule. |
-| `GET` | `/workflows/schedules` | List schedules for user. |
-| `PATCH` | `/workflows/schedules/:id` | Pause/resume/update. |
-| `DELETE` | `/workflows/schedules/:id` | Delete schedule. |
-| `POST` | `/workflows/schedules/:id/run` | Manual run of a schedule. |
-
-All routes use `createApiKeyAuth`.
-
-## Database schema
-
-Add to `src/api/db.ts` migrations:
-
-```sql
-CREATE TABLE IF NOT EXISTS workflow_schedules (
-  id             TEXT PRIMARY KEY,
-  user_id        INTEGER NOT NULL,
-  workflow_id    TEXT NOT NULL,
-  name           TEXT NOT NULL,
-  input_json     TEXT NOT NULL,
-  cadence        TEXT NOT NULL, -- daily | weekly | monthly | custom
-  cron_expr      TEXT,
-  timezone       TEXT NOT NULL DEFAULT 'UTC',
-  webhook_url    TEXT,
-  status         TEXT NOT NULL DEFAULT 'active', -- active | paused | deleted
-  next_run_at    TEXT,
-  last_run_at    TEXT,
-  created_at     TEXT NOT NULL,
-  updated_at     TEXT NOT NULL
-);
-
-CREATE TABLE IF NOT EXISTS workflow_runs (
-  id             TEXT PRIMARY KEY,
-  user_id        INTEGER NOT NULL,
-  schedule_id    TEXT,
-  workflow_id    TEXT NOT NULL,
-  input_json     TEXT NOT NULL,
-  status         TEXT NOT NULL DEFAULT 'queued', -- queued | running | succeeded | partial | failed | cancelled
-  manifest_json  TEXT,
-  error_message  TEXT,
-  queued_at      TEXT NOT NULL,
-  started_at     TEXT,
-  completed_at   TEXT,
-  idempotency_key TEXT
-);
-
-CREATE TABLE IF NOT EXISTS workflow_artifacts (
-  id             TEXT PRIMARY KEY,
-  run_id         TEXT NOT NULL,
-  user_id        INTEGER NOT NULL,
-  kind           TEXT NOT NULL,
-  label          TEXT NOT NULL,
-  path           TEXT NOT NULL,
-  content_type   TEXT NOT NULL,
-  bytes          INTEGER,
-  rows_count     INTEGER,
-  created_at     TEXT NOT NULL
-);
-
-CREATE TABLE IF NOT EXISTS workflow_webhook_deliveries (
-  id             TEXT PRIMARY KEY,
-  run_id         TEXT NOT NULL,
-  schedule_id    TEXT,
-  user_id        INTEGER NOT NULL,
-  webhook_url    TEXT NOT NULL,
-  status         TEXT NOT NULL, -- pending | delivered | failed
-  attempts       INTEGER NOT NULL DEFAULT 0,
-  last_error     TEXT,
-  created_at     TEXT NOT NULL,
-  delivered_at   TEXT
-);
-```
-
-Indexes:
-
-```sql
-CREATE INDEX IF NOT EXISTS idx_workflow_schedules_due ON workflow_schedules(status, next_run_at);
-CREATE INDEX IF NOT EXISTS idx_workflow_runs_user_created ON workflow_runs(user_id, queued_at);
-CREATE INDEX IF NOT EXISTS idx_workflow_artifacts_run ON workflow_artifacts(run_id);
-```
-
-## Worker model
-
-Initial implementation can use a hosted cron route:
-
-```text
-POST /workflows/cron/dispatch
-```
-
-Protected by `CRON_SECRET`.
-
-Dispatch steps:
-
-1. Find active schedules where `next_run_at <= now`.
-2. Create `workflow_runs` row with idempotency key `<scheduleId>:<scheduledTime>`.
-3. Enqueue/run workflow.
-4. Advance `next_run_at`.
-
-If no queue is available, run small workflows inline and mark long workflows as queued for later worker integration. The workflow registry should not care whether a run is local, HTTP, or queued.
-
-## Run execution
-
-`WorkflowRunService`:
-
-```ts
-interface WorkflowRunService {
-  startManualRun(userId: number, workflowId: string, input: unknown): Promise<WorkflowRun>
-  startScheduledRun(schedule: WorkflowSchedule, scheduledFor: string): Promise<WorkflowRun>
-  executeRun(runId: string): Promise<void>
-  getRun(userId: number, runId: string): Promise<WorkflowRunWithArtifacts>
-}
-```
-
-Execution rules:
-
-- Debit estimated credits before paid API work.
-- Refund failed units when workflow has per-item partial failures.
-- Persist `manifest_json` after every major phase so status pages are useful during partial failure.
-- Save artifacts through a hosted artifact writer. Local paths are not enough for hosted runs.
-
-## Artifact storage
-
-Phase 1:
-
-- Store artifacts on local/Vercel ephemeral disk only for immediate response is not durable enough.
-- Therefore hosted scheduled workflows need object storage before public rollout.
-
-Recommended minimal storage:
-
-- S3/R2-compatible bucket.
-- Store by `user_id/run_id/artifact_id`.
-- DB stores key/path and content type.
-
-Local CLI can continue writing to `~/Downloads/mcp-scraper`.
-
-## Webhooks
-
-Payload:
-
-```ts
-interface WorkflowWebhookPayload {
-  event: 'workflow.run.succeeded' | 'workflow.run.partial' | 'workflow.run.failed'
-  runId: string
-  scheduleId: string | null
-  workflowId: string
-  status: string
-  completedAt: string
-  summary: {
-    title: string
-    counts: Record<string, number>
-    warnings: string[]
-    errors: string[]
-  }
-  artifacts: Array<{
-    kind: string
-    label: string
-    downloadUrl: string
-  }>
-}
-```
-
-Security:
-
-- Sign payloads with `MCP_SCRAPER_WEBHOOK_SECRET`.
-- Header: `x-mcp-scraper-signature`.
-- Include timestamp header to prevent replay.
-
-Retries:
-
-- 3 attempts with exponential backoff.
-- Persist every attempt in `workflow_webhook_deliveries`.
-
-## Scheduling modes
-
-Supported:
-
-```ts
-type WorkflowCadence = 'daily' | 'weekly' | 'monthly' | 'custom'
-```
-
-For `custom`, require cron expression and timezone.
-
-Validation:
-
-- Minimum cadence: daily for expensive live-web workflows unless account tier allows more.
-- Cap concurrent active schedules per account tier.
-- Cap max run timeout.
-
-## Integration with `rank_tracker_blueprint`
-
-`rank_tracker_blueprint` should mention scheduled workflows as an execution option:
-
-- Use `scheduled-workflows` for hosted recurring MCP Scraper runs.
-- Use generated rank-tracker schema when the user wants their own database/dashboard.
-
-Do not make the blueprint depend on schedules.
-
-## Tests
-
-- Schedule CRUD auth and ownership tests.
-- Cron dispatch only runs due active schedules.
-- Idempotency prevents duplicate runs for same scheduled time.
-- Manual run creates run row and artifacts.
-- Webhook signing and retry tests.
-- Artifact download requires owning API key.
-- Schedule create rejects unknown workflow IDs and invalid inputs.
-
-## Rollout phases
-
-## Implementation status
-
-Implemented in the first working pass:
-
-- Database migrations and helpers for `workflow_schedules`, `workflow_runs`, `workflow_artifacts`, and `workflow_webhook_deliveries`.
-- Authenticated hosted routes for definitions, manual runs, run status, artifact downloads, schedule CRUD, manual schedule runs, and cron dispatch.
-- Cron integration from the main `/cron/tick` route.
-- CLI commands: `schedule create/list/run/pause/resume/delete` and `runs list/status/download`.
-- Signed webhook delivery when `MCP_SCRAPER_WEBHOOK_SECRET` is configured, with delivery history.
-
-Still required before broad public rollout:
-
-- Durable object storage for hosted artifacts. Current hosted artifacts are filesystem-backed, which is enough for local/server testing and immediate retrieval but not durable across serverless cold starts or instance changes.
-- More route coverage for cron idempotency, ownership on artifact downloads, and webhook retries.
-- Account-tier schedule caps and preflight cost estimation before expensive hosted runs.
-
-### Phase 1 — API definitions and manual runs
-
-- `/workflows/definitions`
-- `/workflows/run`
-- `/workflows/runs/:id`
-- Local or test artifact storage.
-
-### Phase 2 — Durable artifact storage
-
-- S3/R2 bucket.
-- Artifact download routes.
-
-### Phase 3 — Schedules and cron dispatch
-
-- Schedule CRUD.
-- Cron-protected dispatcher.
-
-### Phase 4 — Webhooks
-
-- Signed completion callbacks.
-- Retry/delivery history.
-
-### Phase 5 — CLI schedule commands
-
-- `schedule create/list/run/pause/resume/delete`.
-- `runs list/status/download`.

package/docs/specs/seo-cli-growth-roadmap-spec.md

@@ -1,181 +0,0 @@
-# SEO CLI Growth Roadmap — Product + Implementation Spec
-
-This spec packages the planned MCP Scraper growth work into buildable slices. The product goal is to make MCP Scraper the command-line and MCP data layer for AI SEO work: run live web intelligence workflows, save evidence-rich reports, and hand clean artifacts to AI agents.
-
-## Product thesis
-
-AI SEO users do not only need scraped data. They need repeatable evidence workflows that answer:
-
-- Which sources and competitors are visible in AI/Search surfaces?
-- What do Maps competitors look like across a market?
-- What do reviews say customers care about?
-- What should an AI agent do next with the evidence?
-- Can the workflow run again on a schedule?
-
-The CLI should therefore feel like a workflow machine, not a loose collection of endpoints.
-
-## Spec set
-
-| Spec | Purpose | Recommended priority |
-|---|---|---|
-| `cli-agent-wiring-spec.md` | Make MCP Scraper easy to install into Codex, Claude, and other AI-agent hosts. | P0 |
-| `seo-workflow-runner-and-reports-spec.md` | Add a human CLI workflow runner plus local HTML/CSV/JSON/Markdown report artifacts. | P0 |
-| `agent-ready-seo-packet-spec.md` | Produce a folder that an AI agent can use directly: brief, evidence, sources, tasks, report. | P1 |
-| `local-competitive-audit-spec.md` | Turn directory + Maps + review/profile hydration into a market audit workflow. | P1 |
-| `scheduled-workflows-api-spec.md` | Add hosted recurring workflow schedules, webhooks, run history, and API surfaces. | P2 |
-
-## Current tool surface to reuse
-
-Already shipped:
-
-- `maps_search` — multi-business Maps result search, up to 50 results.
-- `maps_place_intel` — profile-level Maps details, review count, category, review histogram, review topics, attributes, optional review cards.
-- `directory_workflow` — Census/city selection + city-by-city Maps searches + CSV output.
-- `rank_tracker_blueprint` — local planning tool for database schema, cron, metrics, and implementation prompts.
-- `search_serp` / `harvest_paa` — SERP, PAA, AI Overview, forums, videos, and source evidence.
-- `mcp-scraper-install` — human terminal installer card.
-- `mcp-scraper-combined` — combined MCP stdio server for web intelligence + browser tools.
-
-Important distinction: `rank_tracker_blueprint` is not a live tracker runner. It should be referenced by scheduled workflow specs, not replaced.
-
-## New CLI command family
-
-Keep existing stdio server binaries protocol-safe. Add a new human-facing CLI binary:
-
-```json
-{
-  "bin": {
-    "mcp-scraper-cli": "dist/bin/mcp-scraper-cli.js"
-  }
-}
-```
-
-Rationale: `mcp-scraper` is already the stdio MCP server. It must stay silent except for JSON-RPC. The human CLI can print progress bars, report paths, cards, and diagnostics.
-
-Primary commands:
-
-```bash
-mcp-scraper-cli doctor
-mcp-scraper-cli agent install codex
-mcp-scraper-cli agent install claude
-mcp-scraper-cli agent prompt agent-packet
-mcp-scraper-cli workflow list
-mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
-mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20
-mcp-scraper-cli report open last
-mcp-scraper-cli schedule create local-competitive-audit --weekly --webhook https://example.com/hook
-```
-
-The existing `mcp-scraper-install` stays as the branded terminal card and can point users at `mcp-scraper-cli`.
-
-## Workflow catalog
-
-P0/P1 workflows:
-
-| Workflow | What it answers | Primary outputs |
-|---|---|---|
-| `agent-packet` | What evidence and tasks should an AI agent use for this SEO problem? | agent folder, HTML report |
-| `local-competitive-audit` | Who ranks in local markets, how strong are they, and what do customers say? | report, CSVs, review insights |
-| `directory` | Which businesses are ranking across selected city markets? | CSV, market report |
-| `map-comparison` | How do Maps competitors compare by rank, reviews, categories, websites, and profile signals? | comparison CSV, profile insights, HTML report |
-| `serp-comparison` | Why are ranking pages winning and what content gaps should the target inspect? | page comparison, heading gaps, PAA and AI Overview evidence |
-| `paa-expansion-brief` | What should a page cover based on People Also Ask expansion? | writer brief, question map, source-domain table |
-| `ai-overview-language` | How should content be phrased for AI Overview-style answer blocks and citation hooks? | language guidance, claim patterns, citation table |
-| `answer-gap` | Which cited/ranking sources answer the query better than the target page? | gap report, task list |
-| `ai-citations` | Is the brand/domain cited in AI Overview/search evidence? | citation table, trends when scheduled |
-
-P2/P3 workflows:
-
-| Workflow | What it answers |
-|---|---|
-| `maps-rank-monitor` | How does a target GBP move across markets and keywords? |
-| `content-decay` | Which pages are stale compared with current live SERP/PAA evidence? |
-| `forum-demand` | What objections, use cases, and language are showing up in forums? |
-| `entity-audit` | Is brand/entity consistency strong across public search surfaces? |
-
-## Shared artifact contract
-
-Every workflow writes a run folder:
-
-```text
-~/Downloads/mcp-scraper/workflows/<workflow>/<timestamp>-<slug>/
-  manifest.json
-  report.html
-  summary.md
-  evidence.json
-  sources.csv
-  tasks.md
-  logs/attempts.jsonl
-  exports/*.csv
-  screenshots/*        # optional
-  replays/*            # optional
-```
-
-`manifest.json` is the stable contract:
-
-```ts
-interface WorkflowManifest {
-  workflow: string
-  runId: string
-  startedAt: string
-  completedAt: string | null
-  status: 'running' | 'succeeded' | 'partial' | 'failed'
-  input: Record<string, unknown>
-  artifacts: Array<{
-    kind: 'html_report' | 'markdown' | 'json' | 'csv' | 'log' | 'screenshot' | 'replay'
-    label: string
-    path: string
-    rows?: number
-  }>
-  counts: Record<string, number>
-  warnings: string[]
-  costs?: { estimatedMc: number; actualMc?: number }
-}
-```
-
-## Implementation phases
-
-### Phase 1 — Human CLI shell + agent wiring
-
-- Add `bin/mcp-scraper-cli.ts`.
-- Implement `doctor`, `agent install`, `agent prompt`, and `workflow list`.
-- No new paid workflow execution yet.
-- Tests: CLI argument parsing, generated config snapshots, installer card references.
-
-### Phase 2 — Workflow runner + report artifact system
-
-- Add local workflow registry and artifact writer.
-- Implement `report open last`.
-- Implement `directory` as first wrapped workflow around existing `directory_workflow`.
-- Tests: run folder creation, manifest shape, report renderer.
-
-### Phase 3 — Agent-ready SEO packet
-
-- Implement `agent-packet` workflow.
-- Use `search_serp`, `harvest_paa`, optional `extract_url`, and report writer.
-- Tests: fixture evidence to deterministic packet artifacts.
-
-### Phase 4 — Local competitive audit
-
-- Implement `local-competitive-audit`.
-- Add `review_count` to directory CSV base export.
-- Hydrate top-N competitors per city using `maps_place_intel`.
-- Generate review/category/market-strength report.
-
-### Phase 5 — Hosted schedules/API
-
-- Add workflow run and schedule database tables.
-- Add API routes for schedule CRUD, manual run, run status, report download.
-- Add webhook callbacks.
-- Keep scheduled workflows separate from `rank_tracker_blueprint`; the blueprint can generate schema/plans for custom rank trackers.
-
-## Tool-description impact
-
-When each workflow ships, update:
-
-- `README.md` install and tool sections.
-- `public/skills/mcp-scraper/skill.md`.
-- MCP tool descriptions in `src/mcp/paa-mcp-server.ts` if a workflow becomes a real MCP tool.
-- `mcp-scraper-install` newest-feature line.
-
-Avoid naming underlying browser/proxy vendors in outward-facing CLI text, tool descriptions, and error messages.