mcp-scraper 0.97.1 → 0.98.0

package/README.md

@@ -1,425 +1,29 @@
 # MCP Scraper
 
-MCP Scraper is an MCP server for live web intelligence tools backed by `https://mcpscraper.dev`.
+Authenticated clients for the hosted MCP Scraper service. The generated manifest is the authoritative current inventory across scraper, browser, manual workflow, billing, connected-service and durable Memory tools. Obtain your account API key at https://mcpscraper.dev/dashboard.
 
-The authenticated Reddit API supports search, result selection, and extraction from current Reddit pages.
-
-## Product repository
-
-This repository owns the complete private product runtime while preserving independent service and
-deployment boundaries:
-
-- `/` — primary MCP/API server and public `mcp-scraper` package;
-- `apps/worker` — private native execution of admitted video and explicit Memory jobs;
-- `apps/web` — private Next.js product and marketing application;
-- `apps/memory` — hosted Memory service plus its independently publishable DB, Tools, and stdio packages.
-
-`mcpscraper-sdk` remains the only separate public-client repository. See [`apps/README.md`](apps/README.md)
-and [`product-topology.json`](product-topology.json) for exact ownership, source receipts, release units,
-deployment roots, and verification commands.
-
-Contributors can begin with [START-HERE.md](START-HERE.md) and the [codebase edit guide](docs/architecture/codebase-file-tree.md). Current product guidance lives in `docs/product/`, reference material in `docs/reference/`, and historical designs and diagnostics in `docs/archive/`. Manual acquisition probes live in `scripts/diagnostics/`; `bin/` contains package entrypoints.
-
-## Install
-
-Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run stdio commands.
-
-MCP Scraper ships one stdio entrypoint plus human-facing helper CLIs:
-
-- `mcp-scraper` — the single MCP server with every tool: live web intelligence (SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, TikTok video transcripts, Maps, directory, rank tracker blueprint, credits) plus agent-controlled direct/no-proxy hosted browser sessions (screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile/login setup for authenticated AI visibility workflows). It is context-aware: in a human terminal it prints the branded ASCII install card; in an MCP client it runs as a protocol-clean stdio server. This is the entrypoint used by the MCPB Desktop Extension.
-- `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands.
-- `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, SEO workflow runs, and HTML reports. This command is safe to print because it is not an MCP stdio server.
-
-### Terminal installer
-
-Run the command when you want the designed terminal install experience:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper
+```sh
+export MCP_SCRAPER_API_KEY=sk_your_key
+npx mcp-scraper@latest --stdio
 ```
 
-In a human terminal, it prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. When launched by an MCP client, the same command detects non-interactive stdio and writes only valid JSON-RPC to stdout.
+Use `mcp-scraper-cli doctor`, agent installation helpers, and hosted workflow commands as before. `paa-api` is an authenticated localhost relay: send `x-api-key` with every request. It does not run a database, worker, or scraper locally.
 
-The explicit installer alias still works:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper-install
+```js
+import { McpScraperClient, harvest } from 'mcp-scraper';
+const client = new McpScraperClient({ apiKey: process.env.MCP_SCRAPER_API_KEY });
+const result = await client.callTool('credits_info');
+const questions = await harvest({ query: 'example query', maxQuestions: 5 });
 ```
 
-### Human CLI
-
-Run setup checks and generate agent wiring:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
-MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply
-npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
-npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
-```
-
-`agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y --package mcp-scraper@latest mcp-scraper`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts.
-
-The registered command uses the long `--package` flag deliberately. Claude Code's `mcp add` leaks short flags that appear after `--` back into its own option parsing, so a registered `-p` makes it reject its own `--scope`/`-s` argument with a misleading `unknown option` error. If the registration ever fails, the previous entry is captured beforehand and restored automatically.
-
-Check usage and upgrade concurrency from a normal terminal:
-
-```bash
-MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli billing concurrency info
-MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli billing concurrency checkout
-```
-
-Each account has at least 1 base concurrent operation. Each `$5/month` concurrency pack adds 2 browsers on any plan. If an MCP tool hits `concurrency_limit_exceeded`, the error includes `active`, `limit`, `upgrade_url`, and the `mcp-scraper-cli billing concurrency checkout` command so the AI can explain the next step without guessing.
-
-Run workflow reports:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow list
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20 --hydrate-top 5 --reviews 50
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run map-comparison --query roofers --location "Denver, CO" --per-city 20 --hydrate-top 5
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run serp-comparison --keyword "roof repair Denver" --domain example.com --extract-top 5
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run paa-expansion-brief --keyword "roof repair cost" --max-questions 80
-npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run ai-overview-language --keyword "best roof repair company" --domain example.com
-npx -y -p mcp-scraper@latest mcp-scraper-cli report open last
-```
-
-Workflow runs save `manifest.json`, `report.html`, CSVs, Markdown, and evidence JSON under `MCP_SCRAPER_OUTPUT_DIR/workflows` or `~/Downloads/mcp-scraper/workflows`. High-level workflow IDs are `directory`, `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`.
-
-Inspect hosted run history:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper-cli runs list
-npx -y -p mcp-scraper@latest mcp-scraper-cli runs status <run-id>
-npx -y -p mcp-scraper@latest mcp-scraper-cli runs download <run-id>
-```
-
-The hosted workflow API is mounted under `/workflows`: `GET /definitions`, `POST /run`, `GET /runs`, `GET /runs/:id`, and `GET /runs/:id/artifacts/:artifactId`. Workflows run on explicit request. Internal maintenance resumes already-admitted jobs; it does not create user schedules.
-
-## Build or enrich lead lists
-
-There are two neighboring lead journeys:
-
-- To **build a new roofing list** from a niche and city, use `workflow_run` with `workflowId: "get-leads"`, `query: "roofers"`, and a separate `location` such as `"Fort Collins, CO"`.
-- To **enrich an existing list** of named businesses or people, use `lead_list_enrich`. It preserves the supplied columns and appends public website, phone, email, social, contact-form, identity, provenance, and warning fields. For business rows, its optional `peopleDiscovery` mode can also find role-bearing owners or leadership and company-matched LinkedIn profile candidates.
-
-The shortest input path depends on what the current client can actually send:
-
-| Client journey | Input path |
-|---|---|
-| Codex or Claude Code with pasted rows | Normalize the pasted table, then call `lead_list_enrich` directly with `source.kind: "rows"`. |
-| Codex or Claude Code with a local CSV/XLSX file | Call `lead_list_upload_start`, PUT the local bytes to its signed URL, then call `lead_list_import`, `lead_list_enrich`, and `lead_list_enrich_status`. The MCP never reads the local path. |
-| ChatGPT or Claude app that exposes attachment contents | Express the attachment as mapped rows for direct enrichment, or bounded CSV text for `lead_list_import`. |
-| ChatGPT or Claude app without attachment-byte forwarding | Call `lead_list_upload_start` and give the user its upload-page link; after upload, import by `uploadId`, enrich, and poll status. |
-
-For a few pasted rows, start directly:
-
-```json
-{
-  "source": {
-    "kind": "rows",
-    "rows": [
-      { "Business": "Example Roofing", "City": "Fort Collins", "State": "CO" }
-    ]
-  },
-  "columnMap": { "name": "Business", "city": "City", "region": "State" },
-  "defaultEntityType": "business",
-  "resolveMissingUrls": true,
-  "peopleDiscovery": "owners",
-  "peopleQueryTemplates": [
-    "{business} owner of company {city} {region}",
-    "\"{business}\" founder {city} {region}"
-  ],
-  "maxPeoplePerLead": 2,
-  "outputFormats": ["csv", "xlsx"],
-  "idempotencyKey": "roofers-fort-collins-001"
-}
-```
-
-For CSV text, inspect it before any paid web work:
-
-```json
-{
-  "source": { "kind": "csv_text", "csvText": "Name,City,State\nJane Agent,Denver,CO" },
-  "idempotencyKey": "realtors-import-001"
-}
-```
-
-Review the returned headers, samples, and suggested map. Then call `lead_list_enrich` with
-`source: { "kind": "lead_list", "leadListId": "lst_..." }` and an explicit `columnMap`. Poll
-`lead_list_enrich_status` with the returned `jobId` until it returns a terminal status, then use its
-owner-scoped CSV and XLSX resource links. Reuse an idempotency key only when retrying the same semantic
-request; use a new key for a new upload, import, or enrichment job.
-
-Uploads accept CSV, TSV, and XLSX up to 25 MiB. Inline rows and CSV text are bounded to 500 KiB;
-enrichment accepts at most 250 rows. Incomplete uploads expire after 24 hours, while imported lists and
-result files are retained seven days. The generated XLSX contains `Enriched Leads` and `Run Summary`;
-input styles, macros, charts, comments, and formulas are not preserved or executed.
-
-Every discovered contact includes source provenance. Email scope is explicit (`person`, `team`,
-`office`, or `unknown`), so a brokerage inbox is not represented as a realtor's direct address. Public
-occurrence is not deliverability verification, consent, or permission to contact someone. Associated-
-people discovery defaults off. When enabled, the calling model can supply up to three ordered query
-templates with one role intent each; every role query reads AI Overview plus organic evidence, and the
-worker stops once the requested people bound is filled. It performs at most one LinkedIn result search
-per retained person and does not open LinkedIn profiles. Role evidence, AI citations, and profile-match
-evidence remain separate, and indexed public occurrences are not treated as verified current employment.
-
-### Claude Desktop MCPB
-
-Build the branded one-click bundle:
-
-```bash
-npm run build:mcpb
-```
-
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.97.1`, SHA-256 `2c98f521dd1cb860f08a6344febd9f7a4607606d9767085f899a0f834b5d88b1`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
-
-The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
-
-### Raw stdio config
-
-Claude Desktop:
-
-```json
-{
-  "mcpServers": {
-    "mcp-scraper": {
-      "command": "npx",
-      "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"],
-      "env": {
-        "MCP_SCRAPER_API_KEY": "sk_live_your_key",
-        "BROWSER_AGENT_PROFILE_NAME": "work-accounts"
-      }
-    }
-  }
-}
-```
-
-`npx -y mcp-scraper@latest` runs the same single server (the package's default bin), so existing configs get every tool, including browser tools, after they pull the latest package. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
-
-For customer AI visibility setup, use the hosted profile connection flow: run `browser_profile_connect`, give the user the returned `watch_url`, have them complete login, then poll `browser_profile_list` until the connection is authenticated. One profile can hold many logins — call `browser_profile_connect` again with the same profile and a different domain to stack accounts.
-
-Browser hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
-
-`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions.
-
-Claude Code one-command setup:
-
-```bash
-MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply
-```
-
-Then fully exit Claude Code, open a new Claude terminal, and verify:
-
-```bash
-claude mcp list
-```
-
-Manual Claude Code command:
-
-```bash
-claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest mcp-scraper
-```
-
-Codex config:
-
-```toml
-[mcp_servers.mcp-scraper]
-command = "npx"
-args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"]
-env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
-```
-
-## Tools
-
-### Two-page PAA harvests
-
-Pass this input to `harvest_paa` (or add an `idempotencyKey` for `harvest_paa_start`):
-
-```json
-{"query":"commercial truck insurance","pages":2,"maxQuestions":10}
-```
-
-The harvester attempts page 2 organic results before expanding PAA on the untouched first page. It does not harvest a second PAA graph. `pages` accepts 1 or 2 and defaults to 1; pricing is unchanged. If page 2 is missing, empty, or cannot load, available first-page evidence is retained.
-
-Check `pagination.requestedPages`, `pagination.capturedPages`, and `pagination.page2Status` in the MCP result before assuming both pages were captured. Raw REST results place the same fields under `diagnostics.pagination`. Legacy saved results may have no pagination metadata, which MCP formatters expose as `null`.
-
-### Web-intelligence tools
-
-- `harvest_paa` — expand People Also Ask on the original first result page. Optional `pages: 2` captures a second organic result page first; the default is one page.
-- `harvest_paa_start` — start the same harvest as a durable job, with the same optional `pages: 2`.
-- `search_serp`
-- `extract_url` — start a durable normal or Wayback-replayed page extraction and receive a job receipt immediately; poll `extract_url_status` with the returned job ID until it reaches a terminal state. For artifact delivery, use the returned `artifact.artifactId` with `report_artifact_read` to retrieve the full saved page in windows. Polling and reading do not start or bill another extraction. Hosted readback lasts while the completed job is retained (up to 45 days); an installed MCP client mirrors the page into its local artifact store for 24 hours from each successful status read. Wayback results omit playback chrome and can include a timestamp-matched featured image. Set `preserveMedia:true` to union static and rendered/lazy media, collapse responsive variants, attach up to `maxInlineImages` AI-readable images, and receive an owner-scoped ZIP manifest readable with `archive_read`. Branding output ranks the site logo separately from evidence-bounded proof images such as certifications, awards, memberships, partner/customer marks, and press mentions.
-- `map_site_urls`
-- `map_wayback_snapshots` — count and inventory Wayback captures across an inclusive date range without downloading page bodies. Supports exact pages, prefixes, hosts, domains, or selected URLs; reports exact versus lower-bound counts, unique URLs/content digests, monthly coverage, missing months, and optional timestamp rows.
-- `extract_site` — crawl a live site, batch one archived site snapshot from a Wayback replay URL, or pass a `wayback` plan for whole-site, single-page, or selected-page timelines across explicit months or a `from`/`to` range. Timeline ZIPs include month folders and a capture matrix.
-- `archive_read` — list files in a bounded public ZIP, read one selected UTF-8 text file in byte windows, or preserve the complete selected file in the tenant Library vault. Use it with the `bundleUrl` returned by `check_site_export`.
-- `youtube_harvest`
-- `youtube_transcribe`
-- `facebook_reels_inventory` — collect up to the requested number of public Reel URLs from a Facebook profile or Page, then pass selected URLs to `facebook_video_transcribe`. Anonymous scans return at most 60 URLs; access walls and time limits can return fewer. Partial results include a stopping reason and can be retained with `resumeUrls`. A scan costs 4 credits and is refunded when it adds no new URLs.
-- `facebook_ad_search`
-- `facebook_page_intel`
-- `facebook_ad_transcribe` — transcribe a direct Facebook ad video URL returned by `facebook_page_intel`.
-- `facebook_video_transcribe` — transcribe an organic Facebook reel, video, watch, post, or share URL, including `fb.watch` links. The tool renders the page, extracts the best matching public Facebook CDN MP4 URL, then returns transcript text, timestamped chunks, selected quality, video metadata, and the extracted MP4 URL for follow-up download.
-- `instagram_profile_content` — discover Instagram profile grid content links for a handle or profile URL, optionally through a saved hosted browser `profile` for authenticated access. Returns collected post/reel/tv URLs, profile counts, type counts, shortcodes, browser details, pagination attempts, stop reason, and limitations.
-- `instagram_media_download` — extract and download one Instagram post/reel/tv URL, optionally through a saved hosted browser `profile` for authenticated access. Returns text/caption, image URL/downloads, selected video/audio MP4 tracks, optional muxed MP4 when `ffmpeg` is available, optional transcript, and browser details.
-- `maps_search` — search Google's localized local-results list for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. Ordinary search uses the local-results feed; `websiteUrl` may be null and `profileDetailsStatus: "not_requested"` means no profile dialog was opened. Use `maps_place_intel` selectively to hydrate businesses that need complete website or profile data; this is separately billed. Set `includeServices: true` to open profiles for services and areas served where available. `maxResults` defaults to 10 and is capped at 50.
-- `maps_place_intel` — hydrate one known/named Google Maps business with profile details, entity IDs/CID, services, service areas, review aggregates/cards, and optional photos. Set `includeImages:true`, choose `imageScope:"owner"` or `"all"`, and tune `maxImages`; results include a recoverable run ID, field status, ownership evidence, and an owner-scoped ZIP readable with `archive_read`. Use `maps_place_status` to read saved results and `maps_place_resume` to explicitly retry a partial run.
-- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus localized Google business searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." Supply the business category, state, and market limits; MCP Scraper manages search transport and retry behavior internally. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
-- `workflow_list` — list higher-level workflow IDs plus AI-facing recipes for market analysis, ICP research, forum/review acquisition, brand design briefings, CRO audits, positioning briefs, content gaps, and AI search visibility audits.
-- `workflow_suggest` — route a high-level business goal to the right workflow/tool chain before spending credits.
-- `workflow_run` — run hosted workflows such as `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`; returns run metadata, summary, and artifact IDs.
-- `workflow_status` — reopen a workflow run and list its current status and artifacts.
-- `workflow_artifact_read` — read available workflow artifacts such as `evidence.json`, CSVs, Markdown briefs, and reports. Temporary file-backed artifacts may become unavailable and return 410.
-- `editorial_reading_room_guide` — load the reusable editorial workflow, 100-article content contract, image/provenance rules, or compact example before turning dense supplied material into a reading surface.
-- `create_editorial_reading_room` — render up to 100 fully authored, source-grounded articles into one self-contained mobile-first HTML reading room with contents, navigation, search, jump links, progress, text sizing, evening mode, structured card/hero images, Markdown body images, collection/article Open Graph images, and visible provenance. Static files emit collection social metadata and update article metadata in-browser; crawler-perfect article unfurls require a host that renders article-specific head tags. Hosted clients receive a private seven-day artifact; local stdio clients receive an openable file under the MCP Scraper output directory.
-- `renew_editorial_reading_room_download` — issue a fresh signed URL for an unexpired private reading-room artifact.
-- `commons_prepare_publication` / `commons_validate_publication` — check a subscriber-chosen publication subdomain and validate a claim or complete source-grounded editorial edition without writing.
-- `commons_claim_publication` — idempotently claim one globally unique `https://{name}.transparent-commons.cc` publication for the authenticated subscriber.
-- `commons_publish_editorial` — publish a validated editorial reading-room edition to the caller-owned publication and return its permanent root, archive, and edition URLs.
-- `commons_get_publication` — recover the caller-owned or public publication metadata and current edition revisions for later edits and sharing.
-- `report_artifact_read` — read owner-scoped text and JSONL artifacts through the authenticated MCP connection when a model sandbox cannot open the optional signed download URL. Continue with `nextOffset` until it is null; ZIP archives use `archive_read`.
-- `rank_tracker_workflow` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This planning tool does not spend credits.
-- `credits_info`
-
-### Connected-account tools
-
-- `gmail_search_messages`, `gmail_get_message`, and `gmail_get_attachment` — search Gmail, read one complete normalized message (including full MIME and raw EML artifacts), and retrieve external or inline attachment bytes through owner-scoped artifact pointers. Gmail content is always treated as untrusted data.
-- `gmail_prepare_selection` and `gmail_export_selection` — freeze an exact reviewed set of up to 5,000 messages for 24 hours, then export full-fidelity JSONL/EML without silently substituting the first provider page.
-- `gmail_bulk_manage_messages` and `gmail_bulk_delete_messages` — apply replay-safe label, read-state, archive, Inbox, Trash, or restore changes to one unchanged selection. Permanent delete is a separate destructive tool, requires literal confirmation plus the broader Gmail scope, and should be tested only with a purpose-created disposable message.
-- `gmail_prepare_memory_import`, `gmail_import_to_memory`, and `gmail_import_status` — preview vault and attachment routing, start or resume a checkpointed import, and inspect progress without advancing it. Source archives default to Library; relationship Communications are opt-in and require one exact existing identity. Safe attachments are retained as private files with truthful `search_ready`, `stored_not_indexed`, `skipped`, `refused`, or `failed` status.
-- `list_service_connections` — list this caller's tenant-owned Nango OAuth and official remote MCP connections, including verified provider-side account email/name when exposed, exact live reads, gated actions, permanently blocked administrative tools, credential transport, and schema-discovery metadata. Provider identity is distinct from the MCP Scraper login, and connections are never shared between customers.
-- `describe_service_connection_tool` — fetch the sanitized live MCP Tool definition for one tool listed on one tenant-owned connection, including its current callability, input schema, optional output schema, safe annotations, and schema hash. Use this before constructing provider-native arguments; provider functions stay behind the generic bridges instead of becoming dozens of permanent top-level tools.
-- `export_connected_service_data` — fetch a fresh Gmail, Google Calendar, Google Search Console, Zoom, Slack, Resend, or Meta dataset in one MCP call. Slack's `slack_channel_messages` dataset accepts a `channelId`, paginates top-level history, fetches threaded replies in bounded parallel batches, preserves file metadata, honors retry delays, and supports `allTime:true`; it never joins or changes the channel. Zoom's `zoom_transcripts` dataset resolves VTT files from recording metadata and downloads them through the authenticated connection without looping the separately rate-limited `get-meeting-transcript` function. Search Console's `search_console_performance` dataset walks accessible properties and bounded live Search Analytics pages with continuation. Small exports return inline; larger exports become private JSONL retained for seven days with exact `report_artifact_read` arguments plus an optional 15-minute human download URL. For relationship work, gather source evidence first: inspect existing People records, resolve the exact provider account, preserve provider provenance when writing a linked Communication, and never treat an export as permission to mutate the source account.
-- `export_search_console_table_data` — filter up to 50,000 Search Console historically stored Search Console rows and create a private renewable JSONL artifact without calling Google again. Get the typed `gsc_performance_*` table name from `list_service_connections`, inspect it with `table-describe`, and use the same filters with `table-query` for interactive analysis.
-- `renew_connected_data_download` — issue a fresh 15-minute signed URL for an unexpired private export artifact without pulling the provider again.
-- `read_service_connection` — run one small live read by exact allowlisted name across Nango OAuth or official remote MCP connections, including bounded Google Drive inventory, change, Doc, Sheet, and text-file tools. Do not loop it over a time range when `export_connected_service_data` supports that provider's collection.
-- `import_service_connection_to_memory` — run one exact approved read and upsert the bounded, redacted result into an existing ordinary Memory vault at a stable server-generated path. The snapshot is embedded for semantic search and explicitly marked as untrusted provider data. This is not account-wide pagination or continuous sync.
-- `call_service_connection_action` — run one exact mutation only after the account action switch is enabled. Google Drive, Search Console, Resend, GitHub, Gmail, Calendar, Slack, Zoom, Facebook, Google Analytics, X, and YouTube actions remain live-inventory, provider-policy, tenant, and audit gated; credential, OAuth-grant, and raw editor-session administration is never exposed.
-
-Connected-account charges come from the canonical rate card and actual usage receipts. OAuth connect and reconnect do not consume Credits. Separately metered scraper, browser, and media operations keep their normal rates; inspect `get_credit_costs` before paid work.
-
-GitHub uses the same provider-neutral read/action bridges: 27 live reads cover repository discovery, issues, pull requests, commits, releases, and workflows; 20 live mutations remain account- and action-gated. The 47 Nango functions and expanded reconnect scopes are deployed. The current test token remains `public_repo` until owners approve its two pending organization requests, so private-repository and workflow parity is not yet claimed.
-
-Google Drive Phase 1 exposes ten bounded reads plus the separately gated `create-drive-folder` and `copy-drive-file` actions through those same bridges with explicit account and action permissions. It is interactive tool support, not a bulk-export feature; large binary delivery and broader Drive mutations remain unreleased.
-
-Google Search Console exposes eight bounded reads and eight gated property and sitemap mutations, including API-only batches for URL inspection, Search Analytics comparison, property management, and sitemap management. Batch calls return per-item receipts, run quota-consciously, and do not require database persistence; destructive batches default to dry-run previews and require explicit confirmation. Supported direct exports and Memory imports preserve returned records and checkpoints. Query tenant-owned tables with `table-describe` and `table-query`; the server does not create recurring syncs.
-
-### Memory discovery and safe editing
-
-- Start unknown-location recall with `memory-search`. It uses Hybrid Smart RAG and returns ranked excerpts, not complete notes.
-- Read strong candidates with `memory-get` before relying on, summarizing, linking, or editing them.
-- Use `memory-list` for an exhaustive inventory. Pass `allVaults:true` for every note and folder across the whole entitled Memory account; omit it for one logical vault. Account-wide results include aggregate totals and complete per-vault note/folder groups, so a default-vault count is never mistaken for the account total.
-- Use `list-memory-tags` for the complete account-wide canonical tag vocabulary, aliases, usage counts, and per-vault distribution.
-- `memory-put` replaces the complete note body. For edits, read the note, merge the requested change into the full body, write with `baseRevision`, then read back and verify it.
-
-### Browser-agent tools
-
-- `serp_identity_create` — create an account-scoped Google search identity that binds one saved browser profile to one reusable static ISP exit, opens `google.com`, and returns a takeover URL so the user can clear consent or CAPTCHA before automation reuses it.
-- `serp_identity_list` — list the caller's named search identities and their readiness and last-used state without exposing provider resource IDs.
-- `serp_identity_delete` — delete a named search identity and its hosted profile and static ISP proxy.
-- `browser_open` — open a direct/no-proxy hosted browser session, or pass `serp_identity` to load that identity's saved profile and fixed exit. Returns a human `watch_url`.
-- `browser_profile_connect` — save a site login to a profile and return a user takeover `watch_url` on mcpscraper.dev. The user signs in through that URL, then `browser_profile_list` verifies it is authenticated. One profile holds many logins — call again with the same profile and a different domain to stack accounts. Use this again whenever a login disconnects.
-- `browser_profile_list` — list every login saved in a profile with its auth status, account, and note. Poll it after `browser_profile_connect` until a login reads AUTHENTICATED, then launch with `browser_open`.
-- `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
-- `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
-- `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
-- `browser_goto`
-- `browser_click`
-- `browser_type`
-- `browser_scroll`
-- `browser_press`
-- `browser_replay_start` — start an MP4 replay. Returns `replay_id`, `view_url`, and `download_url` when available.
-- `browser_replay_stop` — stop a replay. Returns the final `view_url` and `download_url`.
-- `browser_list_replays` — list replay videos for a session.
-- `browser_replay_download` — download and save the replay MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
-- `browser_replay_mark` — while recording, locate a DOM target and return a replay-timed annotation object.
-- `browser_replay_annotate` — download a replay MP4, render timed boxes, circles, underlines, arrows, and labels using annotation objects from `browser_replay_mark` or exact bounds from `browser_locate`, and save a new annotated MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
-- `browser_capture_fanout` — capture ChatGPT/Claude AI-search fan-out from an open logged-in hosted session. Every client receives the complete structured capture inline. Installed stdio/MCPB clients can use `export=true` for durable `fanout.json`, query/source/citation/domain/snippet CSVs, TSV, and `report.html` under `MCP_SCRAPER_OUTPUT_DIR/fanout`; hosted OAuth clients receive `exports: null` and use the inline data.
-- `browser_close`
-- `browser_list_sessions`
-
-For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`.
-
-For `search_serp`, callers provide a query and can reuse an `idempotencyKey` after an uncertain response. Light mode returns organic positions, URLs, titles, and descriptions. Unfiltered one-page full mode adds available same-page SERP features, including local results, discussions, videos, AI features, and on-page questions. Both modes default to one page; request `pages: 2` explicitly for a second page. Set `recency: "week"` or `recency: "month"` to apply Google's past-week or past-month filter. Filtered searches use Option 2 Browser API as the primary provider and return organic results only. A one-page filtered search costs 35 Credits. With `pages: 2`, the browser scrolls to Google's pagination control and clicks the native Page 2 link; two delivered pages cost 70 Credits. Unfiltered light searches use up to two primary-provider attempts followed by one SERP API fallback, with no browser recovery. Unknown execution outcomes stop retries. Both primary attempts retain the requested page count; the API fallback delivers one page and reports partial coverage for a two-page request. Unfiltered one-page full searches use the SERP API directly to preserve requested feature semantics. Full mode does not open result URLs, expand PAA questions, or fetch Maps business profiles. Unfiltered light mode costs 20 Credits per delivered page, or 35 Credits when a backup supplies it; full mode costs 35 Credits per delivered page. Two-page searches return organic results only. Location, language, device, and legacy optional-module fields remain accepted for compatibility but do not change ordinary searches. Other Google SERP and Maps tools retain their own regional inputs. MCP Scraper owns transport selection and bounded retries internally; implementation controls and receipts are not part of the public tool contract.
-
-The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both sections through one MCP server.
-
-All MCP tools return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step, plus readable text content for compatibility. Runtime `tools/list` omits output schemas so strict clients can register the complete catalog; the generated developer manifest retains every canonical output schema for validation and typed SDK generation. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
-
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The generated manifest is the authoritative current inventory across scraper, browser, manual workflow, billing, connected-service and durable Memory tools. Personal Assistant tools and routes are retired from active builds; their source history and persisted data remain recoverable from the archived pre-retirement branch. The scraper-side inventory includes complete Gmail selection, message, attachment, export, bulk-action, and Memory-import workflows; durable SERP, PAA, and single-page extraction starts and status; rendered site-content similarity; governed Local Sourcebook and Transparent Commons workflows; direct site-export reads; Editorial Reading Room and News Publisher templates; and production X-Ray setup, analytics, seven-model attribution, structured post-purchase surveys, reported impact, truthful view-evidence status, CRM policy and receipt, campaign, export, and report tools. Provider setup remains absent until its authorization, ingestion, reconciliation, privacy, canary, cleanup, and deployment receipts are complete. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
-
-For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone Memory service, not a second customer setup path. Existing direct Memory credentials remain compatible for one release, but all new customer setup uses the root endpoint and `MCP_SCRAPER_API_KEY`.
-
-The public REST registry in `src/api/public-rest-contract.ts` owns Maps search and directory run/status request and response schemas, headers, descriptions, and paths. Run `npm run contracts:rest` after changing one of those operations; `npm run contracts:rest:check` fails when the versioned artifact or the web docs projection is stale. The SDK imports `contracts/public-rest.v1.json`, projects it over its curated OpenAPI base, and regenerates Node and Python models together. Other public REST operations remain in that curated base until migrated.
-
-## Resources
-
-The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files).
-
-## Environment
-
-- `MCP_SCRAPER_API_KEY` is required.
-- `MCP_SCRAPER_BASE_URL` is optional and defaults to `https://mcpscraper.dev`.
-- `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
-- `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
-- `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
-- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `Option 1_BROWSER_PROFILE_NAME`, `Option 1_PROFILE_NAME`.
-- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `Option 1_BROWSER_PROFILE_SAVE_CHANGES`, `Option 1_PROFILE_SAVE_CHANGES`.
-
-Hosted operators can isolate authorization state in a dedicated Turso/libSQL database without changing the public MCP tool catalog or API-key authentication. The secured store uses atomic authorization-code exchange and refresh rotation, keyed secret lookup, bounded encrypted replay receipts, authority epochs, and fail-closed maintenance behavior. Production migration and rollback are controlled data moves, not ordinary mode flips; see [MCP OAuth operations](docs/operations/mcp-oauth-runbook.md). Existing client setup and reconnect behavior are unchanged in Phase 1.
-
-Every web intelligence tool call made through `mcp-scraper` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out captures are always returned inline; only installed stdio/MCPB clients write optional `export=true` files under `MCP_SCRAPER_OUTPUT_DIR/fanout`, returning relative paths. Hosted clients always receive `exports: null`.
-
-## Updating Existing Installs
-
-Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. Stdio MCP changes require publishing a new npm package version and restarting the MCP client. Running MCP server processes do not hot-update, and tool names/descriptions are loaded when the server process starts.
-
-Recommended config for update-friendly installs:
-
-```bash
-npx -y -p mcp-scraper@latest mcp-scraper
-```
-
-This is context-aware: in a normal terminal it prints the visible installer and ASCII card; in an MCP client it runs as the silent stdio server. Use `--stdio` or `MCP_SCRAPER_FORCE_STDIO=1` if you need to force server mode from a terminal. `npx -y mcp-scraper@latest` (the package default bin) runs the same server.
-
-If a user configured `mcp-scraper@0.2.0`, installed globally with `npm install -g mcp-scraper`, or installed it as a project dependency, they will stay on that version until they update the config or reinstall:
-
-```bash
-npm update -g mcp-scraper
-npm install mcp-scraper@latest
-```
-
-Users who do not update can keep using the tools their installed package already advertises, but they will not see newly added stdio tools, schemas, or AI-facing descriptions. For example, a client running an older package cannot call `rank_tracker_workflow`, `directory_workflow`, connected-service actions, or browser tools through stdio even if the hosted API already supports adjacent endpoints.
-
-## Branded One-Click Installs
-
-Raw `npx` MCP server installs are command/config based. `mcp-scraper` is context-aware: terminal TTY prints onboarding text and the ASCII card; MCP-client stdio pipes stay protocol-clean. Do not print marketing text to stdout from an active MCP stdio session; stdout is reserved for JSON-RPC protocol messages.
-
-For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. The repository builds one MCPB bundle with a generated icon, `manifest.json`, one self-contained server file, and `user_config` fields for API-key setup, API URL, and output folder. The build rejects `node_modules`, source maps, oversized archives, and unexpected file growth.
-
-```bash
-npm run build:mcpb
-```
-
-The bundle runs `mcp-scraper` internally, so the user installs `MCP Scraper` once and gets web-intelligence tools plus live browser tools in one MCP server.
-
-## Development
-
-Production deployments track `codex/mcp-ownership-tree` for the server, web app, Memory and native worker. Their Vercel configurations admit this branch and disable automatic deployments from `main`. Before deploying server schema changes, run `npm run release:server:predeploy` against the production database. Customer request capture additionally requires `MCP_ANALYTICS_CAPTURE=1`; missing historical attribution stays unknown.
-
-See the [codebase layout and edit guide](docs/architecture/codebase-file-tree.md) for the populated tree, feature homes and compatibility entrypoints.
+## Migration from versions before 0.98
 
-Edit behavior in its domain under `src/`: `paa`, `google-search`, `reddit`, `maps`, `web-extraction`, `transcripts`, `site-audit`, `analytics`, `personal-assistant`, `twilio`, `gmail`, `integrations`, `commons`, `leads`, and the other named features.
+All execution runs on the hosted service. Local browser, network-routing, database, and model-credential options are no longer supported. `harvest` returns the public MCP result, including complete-result delivery handles, rather than a local scraping result. Use those handles to retrieve retained content without starting another scrape. Use the official `mcpscraper-sdk` for fully typed API clients. Local video generator classes and undocumented direct `dist` imports are no longer exported.
 
-- `src/api/server.ts` assembles HTTP routes; handlers live in `src/api/routes/`.
-- `src/mcp/features/<feature>/` holds each feature's tool registrations, schemas, and formatting. `src/mcp/paa-mcp-server.ts` assembles the main server. Common MCP delivery and errors live in `src/mcp/shared/`.
-- Domain repositories keep their queries beside the behavior they serve. `src/platform/` owns shared database connections, core schema migrations, files, and HTTP plumbing.
-- `src/runtime/` holds shared execution, concurrency, maintenance, and provider-cost machinery. `src/browser/` and `src/providers/` hold vendor adapters.
-- `src/artifacts/` and `src/delivery/` own stored content, expiry, downloads, and delivery.
-- `bin/` and `api/` are terminal and hosted entrypoints; `apps/`, `packages/`, `ops/`, and `nango-integrations/` retain their separate product and provider code.
+The hosted API still enforces your account access, operation limits and credit settlement. Clients do not retry ambiguous paid requests automatically: inspect the original job before retrying.
 
-Use `npm run typecheck`, `npm test`, and `npm run build` for ordinary changes. Tests requiring databases or live providers belong in `tests/integration/` or `tests/live/`.
+Each `$5/month` concurrency pack adds 2 browsers.
 
+Result files are retained by the hosted service; use returned artifact handles to read or download complete results. The desktop extension no longer offers local output-folder or default browser-profile settings; supply supported profile arguments when calling hosted browser tools.
 
-- [MCP Tool Quality Spec](docs/mcp-tool-quality-spec.md) defines the shipping bar for model-facing tool names, descriptions, schemas, structured outputs, errors, packaging, and deployment.
+Documentation: https://mcpscraper.dev/docs

package/dist/bin/api-server.js

@@ -1,3 +1,4 @@
 #!/usr/bin/env node
-import{readFileSync as s}from"fs";function c(){try{for(let r of s(".env","utf8").split(`
-`)){let o=r.indexOf("=");if(o<1||r.trimStart().startsWith("#"))continue;let e=r.slice(0,o).trim();process.env[e]||(process.env[e]=r.slice(o+1).trim())}}catch{}}c();async function a(){let[{serve:r},{app:o},{startWorker:e},{migrate:i}]=await Promise.all([import("@hono/node-server"),import("../server-EYMJROPX.js"),import("../worker-AWJF7UYR.js"),import("../core-schema-Y47YFKH2.js")]),n=parseInt(process.env.PORT??"3001");try{if(await i(),process.env.ANALYTICS_DATABASE_URL){let{migrateAnalytics:t}=await import("../migrations-3AKEC5QE.js");await t()}e(),r({fetch:o.fetch,port:n},t=>{console.log(`[server] http://localhost:${t.port}`),console.log(`[server] admin auth: ${process.env.ADMIN_KEY?"configured":"not configured"}`)})}catch(t){console.error("[startup] server preflight failed",t instanceof Error?t.name:"unknown_error"),process.exit(1)}}a();
+import{createServer as h}from"node:http";import{Readable as g}from"node:stream";function y(m=process.env.MCP_SCRAPER_BASE_URL||"https://mcpscraper.dev",u=fetch){let o=new URL(m);if(!["http:","https:"].includes(o.protocol)||o.username||o.password)throw new Error("Invalid upstream endpoint");return h(async(n,e)=>{let a=n.headers["x-api-key"];if(typeof a!="string"||!a.trim()){e.writeHead(401,{"content-type":"application/json"}),e.end('{"error":"x-api-key is required"}');return}let c;try{c=new URL(n.url||"/",o)}catch{e.writeHead(400),e.end();return}if(c.origin!==o.origin){e.writeHead(400),e.end();return}try{let s=[],d=0;for await(let r of n){let t=Buffer.from(r);if(d+=t.length,d>20*1024*1024){e.writeHead(413),e.end();return}s.push(t)}let p={"x-api-key":a};for(let r of["content-type","accept","idempotency-key","x-idempotency-key","mcp-session-id","mcp-protocol-version","mcp-method","last-event-id"]){let t=n.headers[r];typeof t=="string"&&(p[r]=t)}let l=new AbortController;e.on("close",()=>l.abort());let i=await u(c,{method:n.method,headers:p,body:s.length?Buffer.concat(s):void 0,redirect:"manual",signal:l.signal}),f={};for(let r of["content-type","content-disposition","retry-after","idempotency-key","x-maps-run-id","mcp-session-id","mcp-protocol-version","location"]){let t=i.headers.get(r);t&&(f[r]=t)}e.writeHead(i.status,f),i.body?g.fromWeb(i.body).pipe(e):e.end()}catch{e.headersSent||e.writeHead(502,{"content-type":"application/json"}),e.end('{"error":"MCP Scraper upstream transport failed; inspect the original job before retrying"}')}})}process.argv.includes("--help")?process.stdout.write(`paa-api: authenticated localhost relay to MCP Scraper. Use x-api-key on every request. PORT defaults to3001.
+`):y().listen(Number(process.env.PORT||3001),"127.0.0.1",()=>process.stderr.write(`MCP Scraper API relay listening on localhost
+`));