mcp-scraper 0.6.0 → 0.8.0

package/docs/specs/seo-workflow-runner-and-reports-spec.md

@@ -1,241 +0,0 @@
-# SEO Workflow Runner + Local Reports — Implementation Spec
-
-This spec introduces a human CLI workflow runner and a shared artifact/report system. It wraps existing MCP Scraper capabilities into named SEO workflows with predictable local outputs.
-
-## Goals
-
-- Run high-value SEO workflows from the CLI.
-- Save useful local artifacts every time.
-- Provide local HTML reports that are easy to inspect or share.
-- Keep workflow logic reusable by hosted API schedules later.
-- Avoid duplicating tool logic already implemented in MCP/API handlers.
-
-## Command surface
-
-```bash
-mcp-scraper-cli workflow list
-mcp-scraper-cli workflow describe local-competitive-audit
-mcp-scraper-cli workflow run directory --query roofers --state TN --min-pop 100000 --per-city 20
-mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
-mcp-scraper-cli report open last
-mcp-scraper-cli report path last
-mcp-scraper-cli report list
-```
-
-Global flags:
-
-```bash
---output-dir <path>
---format html,json,md,csv
---json
---no-open
---api-url <url>
---api-key <key>
-```
-
-## Workflow registry
-
-New module:
-
-```text
-src/workflows/
-  registry.ts
-  types.ts
-  artifact-writer.ts
-  report-renderer.ts
-  workflows/
-    directory.ts
-    agent-packet.ts
-    local-competitive-audit.ts
-```
-
-Types:
-
-```ts
-export interface WorkflowDefinition<TInput, TResult> {
-  id: string
-  title: string
-  description: string
-  inputSchema: z.ZodType<TInput>
-  estimate(input: TInput): Promise<WorkflowEstimate>
-  run(input: TInput, ctx: WorkflowRunContext): Promise<TResult>
-  render(result: TResult, ctx: WorkflowRunContext): Promise<WorkflowArtifacts>
-}
-
-export interface WorkflowRunContext {
-  runId: string
-  startedAt: string
-  outputDir: string
-  apiKey?: string
-  apiUrl: string
-  logger: WorkflowLogger
-  artifacts: ArtifactWriter
-  signal?: AbortSignal
-}
-```
-
-The registry is local code, not model-driven. Agents can discover workflows through `workflow list`, and MCP tools can later expose the same registry through structured content.
-
-## Artifact contract
-
-Each run creates:
-
-```text
-~/Downloads/mcp-scraper/workflows/<workflow>/<timestamp>-<slug>/
-  manifest.json
-  report.html
-  summary.md
-  evidence.json
-  tasks.md
-  exports/*.csv
-  logs/attempts.jsonl
-```
-
-`manifest.json`:
-
-```ts
-interface WorkflowManifest {
-  workflow: string
-  title: string
-  runId: string
-  status: 'running' | 'succeeded' | 'partial' | 'failed'
-  startedAt: string
-  completedAt: string | null
-  input: Record<string, unknown>
-  artifacts: Array<{
-    kind: 'html_report' | 'markdown' | 'json' | 'csv' | 'log'
-    label: string
-    path: string
-    bytes?: number
-    rows?: number
-  }>
-  warnings: string[]
-  errors: string[]
-  counts: Record<string, number>
-}
-```
-
-## HTML report renderer
-
-No frontend framework required for the first version. Generate standalone HTML with embedded CSS and static tables.
-
-Report sections:
-
-1. Header: workflow, input, run status, date, artifact links.
-2. Executive summary.
-3. Evidence tables.
-4. Competitors/sources table when available.
-5. Findings.
-6. Recommended actions.
-7. Diagnostics: attempts, partial failures, location confidence, warnings.
-
-Files:
-
-- `src/workflows/report-renderer.ts`
-- `src/workflows/report-template.ts`
-- `tests/unit/workflow-report-renderer.test.ts`
-
-## Local report index
-
-Maintain:
-
-```text
-~/Downloads/mcp-scraper/workflows/index.json
-```
-
-Shape:
-
-```ts
-interface WorkflowRunIndex {
-  runs: Array<{
-    workflow: string
-    runId: string
-    status: string
-    startedAt: string
-    reportPath: string | null
-    manifestPath: string
-    summary: string
-  }>
-}
-```
-
-Commands:
-
-```bash
-mcp-scraper-cli report list
-mcp-scraper-cli report open last
-mcp-scraper-cli report open <runId>
-mcp-scraper-cli report path <runId>
-```
-
-Opening behavior:
-
-- macOS: `open <reportPath>`.
-- Other platforms: print path.
-- `--json`: return machine-readable path only.
-
-## Initial workflows
-
-### `directory`
-
-Wrapper around existing `directory_workflow`.
-
-Input:
-
-```bash
---query roofers
---state TN
---min-pop 100000
---per-city 20
---concurrency 5
-```
-
-Output:
-
-- Existing directory CSV.
-- HTML summary of markets, city status, result counts, top businesses.
-- `evidence.json` preserving structured `directory_workflow` response.
-
-### `agent-packet`
-
-Specified in `agent-ready-seo-packet-spec.md`.
-
-### `local-competitive-audit`
-
-Specified in `local-competitive-audit-spec.md`.
-
-## Hosted versus local execution
-
-First implementation should call hosted API endpoints when available:
-
-- `/directory/run`
-- `/maps/search`
-- `/maps/place-intel`
-- `/serp/...` and `/harvest/...` where already available.
-
-Local-only helpers are acceptable for report assembly and artifact writing.
-
-Do not directly duplicate extraction code in the CLI.
-
-## Error and partial-result behavior
-
-Workflow runs should prefer `partial` over `failed` when useful artifacts exist.
-
-Examples:
-
-- 8 of 10 cities succeed: status `partial`, report includes failed city diagnostics.
-- PAA fails but SERP succeeds: `agent-packet` still writes SERP evidence and missing PAA warning.
-- Review hydration fails for some competitors: audit keeps directory result and marks review fields unavailable.
-
-## Tests
-
-- `workflow list` contains `directory`, `agent-packet`, `local-competitive-audit`.
-- `directory` workflow with mocked API writes manifest, report, evidence JSON, CSV link.
-- Report renderer escapes HTML.
-- `report open last --json` returns the expected path without launching.
-- Partial runs preserve artifacts and warnings.
-
-## MCP description impact
-
-When the CLI workflows are stable, expose `workflow_recipe` or `workflow_plan` as a local planning MCP tool only if needed. Do not add a paid MCP workflow runner until the local CLI and hosted API schedule surfaces settle.
-