mcp-scraper 0.3.14 → 0.3.15

package/docs/tool-catalog-spec.md

@@ -0,0 +1,388 @@
+# Tool Catalog Spec — Firecrawl-style categorized tool surface
+
+Engineering blueprint for a DB-backed **tool catalog** that drives the frontend's categorized
+tool picker (Discover / Extract / Crawl / Special), the per-tool **source dropdowns**
+(Search, Transcribe), **mode toggles** (Scrape scope, Browser Mode), and the dynamic **param forms**.
+
+The catalog is the single source of truth the frontend reads via `GET /catalog`. It is seeded
+idempotently from a canonical TS definition on startup, so the DB is the runtime store and code is
+the authored source. Underlying execution still routes to the existing MCP tools / REST endpoints —
+the catalog only describes *how to present and assemble a request*.
+
+---
+
+## 0. Current-state anchors (verified this codebase)
+
+- MCP tool metadata already lives in `src/mcp/mcp-tool-manifest.ts` (name, surfaces, sample prompts,
+  format examples). The catalog is a *presentation* layer on top; it references tool names but adds
+  grouping, sources, modes, and form fields the manifest doesn't model.
+- Input schemas: `src/mcp/mcp-tool-schemas.ts` (Zod). Param definitions in the catalog should mirror
+  these (same keys, bounds, enums) so the form posts valid bodies.
+- REST routes: `src/api/server.ts` (e.g. `POST /extract-site`, `POST /map-urls`, `POST /extract-url`,
+  `/youtube/*`, `/maps/*`, `/instagram/*`). Catalog rows store the route each tool/source hits.
+- DB: Turso/libSQL via `src/api/db.ts` `migrate()` (CREATE TABLE IF NOT EXISTS, TEXT/INTEGER,
+  `datetime('now')`). Billing ops in `src/api/rates.ts` (`LedgerOperation`, `MC_COSTS`).
+
+---
+
+## 1. Information architecture
+
+Three render primitives, chosen per tool by `catalog_tools.kind`:
+
+| kind | Frontend control | Examples |
+|---|---|---|
+| `simple` | param form only | Map, Directory, Place Intel |
+| `composite_source` | **source dropdown** (multi-select) + per-source params | Search, Transcribe |
+| `modal` | **mode toggle groups** + param form | Scrape (scope + browser_mode) |
+
+Groups (top nav, Firecrawl's DISCOVER/EXTRACT/CRAWL plus our SPECIAL):
+
+| group key | label | tools |
+|---|---|---|
+| `discover` | Discover | search |
+| `extract` | Extract | scrape, transcribe |
+| `crawl` | Crawl | map, crawl |
+| `special` | Special | directory, seo_analysis, deep_research, place_intel, page_intel, rank_tracker, ai_visibility |
+
+---
+
+## 2. DB schema — full DDL (add to `src/api/db.ts` `migrate()`)
+
+All TEXT PKs are stable string keys (not autoincrement) so seed data is idempotent and referenceable.
+
+```sql
+CREATE TABLE IF NOT EXISTS catalog_groups (
+  key           TEXT PRIMARY KEY,          -- 'discover' | 'extract' | 'crawl' | 'special'
+  label         TEXT NOT NULL,
+  icon          TEXT,                       -- lucide icon name
+  display_order INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE TABLE IF NOT EXISTS catalog_tools (
+  key             TEXT PRIMARY KEY,         -- 'search' | 'scrape' | 'crawl' | 'map' | 'transcribe' | ...
+  group_key       TEXT NOT NULL REFERENCES catalog_groups(key),
+  label           TEXT NOT NULL,
+  tagline         TEXT,                     -- one-liner under the card
+  description     TEXT,                     -- full help text
+  icon            TEXT,
+  kind            TEXT NOT NULL,            -- 'simple' | 'composite_source' | 'modal'
+  is_special      INTEGER NOT NULL DEFAULT 0,
+  is_workflow     INTEGER NOT NULL DEFAULT 0,  -- backed by a skill/workflow vs a direct tool/route
+  default_endpoint TEXT,                    -- REST route for simple/modal tools (null for composite_source — see sources)
+  default_mcp_name TEXT,                    -- MCP tool name for simple/modal tools
+  billing_op      TEXT,                     -- LedgerOperation key (rates.ts) for cost display
+  cost_hint_mc    INTEGER,                  -- per-unit MC estimate for the UI
+  enabled         INTEGER NOT NULL DEFAULT 1,
+  display_order   INTEGER NOT NULL DEFAULT 0,
+  created_at      TEXT NOT NULL DEFAULT (datetime('now')),
+  updated_at      TEXT NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE TABLE IF NOT EXISTS catalog_sources (
+  id              TEXT PRIMARY KEY,         -- '<tool>.<source>' e.g. 'search.web'
+  tool_key        TEXT NOT NULL REFERENCES catalog_tools(key),
+  key             TEXT NOT NULL,            -- 'web' | 'maps' | 'fb_ads' | 'youtube' | 'fb_video' ...
+  label           TEXT NOT NULL,
+  icon            TEXT,
+  endpoint        TEXT NOT NULL,            -- REST route this source posts to
+  mcp_name        TEXT NOT NULL,            -- MCP tool name this source maps to
+  billing_op      TEXT,
+  cost_hint_mc    INTEGER,
+  default_selected INTEGER NOT NULL DEFAULT 0,
+  multi_select    INTEGER NOT NULL DEFAULT 1,  -- may be combined with other sources
+  display_order   INTEGER NOT NULL DEFAULT 0,
+  UNIQUE (tool_key, key)
+);
+
+CREATE TABLE IF NOT EXISTS catalog_modes (
+  id              TEXT PRIMARY KEY,         -- '<tool>.<group>.<option>' e.g. 'scrape.scope.bulk'
+  tool_key        TEXT NOT NULL REFERENCES catalog_tools(key),
+  group_key       TEXT NOT NULL,            -- 'scope' | 'browser_mode'
+  group_label     TEXT NOT NULL,
+  option_key      TEXT NOT NULL,            -- 'single' | 'bulk' | 'default' | 'chrome_profile'
+  option_label    TEXT NOT NULL,
+  is_default      INTEGER NOT NULL DEFAULT 0,
+  endpoint_override TEXT,                   -- if this option changes the target route
+  mcp_name_override TEXT,
+  param_overrides TEXT,                     -- JSON object merged into the request body when selected
+  display_order   INTEGER NOT NULL DEFAULT 0,
+  UNIQUE (tool_key, group_key, option_key)
+);
+
+CREATE TABLE IF NOT EXISTS catalog_params (
+  id              TEXT PRIMARY KEY,         -- '<tool>.<source|*>.<param>'
+  tool_key        TEXT NOT NULL REFERENCES catalog_tools(key),
+  source_key      TEXT,                     -- null = applies to all; else only when that source/mode active
+  key             TEXT NOT NULL,            -- the actual API body field name
+  label           TEXT NOT NULL,
+  type            TEXT NOT NULL,            -- 'string'|'number'|'boolean'|'enum'|'multi_enum'|'url'
+  required        INTEGER NOT NULL DEFAULT 0,
+  default_value   TEXT,                     -- JSON-encoded
+  min_value       REAL,
+  max_value       REAL,
+  enum_values     TEXT,                     -- JSON array of {value,label}
+  placeholder     TEXT,
+  help            TEXT,
+  depends_on      TEXT,                     -- another param key
+  depends_value   TEXT,                     -- show this param only when depends_on == depends_value
+  advanced        INTEGER NOT NULL DEFAULT 0,  -- behind an "Advanced" disclosure
+  display_order   INTEGER NOT NULL DEFAULT 0,
+  UNIQUE (tool_key, source_key, key)
+);
+
+CREATE TABLE IF NOT EXISTS catalog_examples (
+  id            TEXT PRIMARY KEY,
+  tool_key      TEXT NOT NULL REFERENCES catalog_tools(key),
+  text          TEXT NOT NULL,
+  display_order INTEGER NOT NULL DEFAULT 0
+);
+```
+
+Indexes:
+```sql
+CREATE INDEX IF NOT EXISTS catalog_tools_group   ON catalog_tools(group_key);
+CREATE INDEX IF NOT EXISTS catalog_sources_tool  ON catalog_sources(tool_key);
+CREATE INDEX IF NOT EXISTS catalog_modes_tool    ON catalog_modes(tool_key);
+CREATE INDEX IF NOT EXISTS catalog_params_tool   ON catalog_params(tool_key);
+CREATE INDEX IF NOT EXISTS catalog_examples_tool ON catalog_examples(tool_key);
+```
+
+---
+
+## 3. Complete catalog mapping (the seed data — every tool, source, mode, param)
+
+Authored in `src/api/catalog-seed.ts` as typed constants, written via `seedCatalog()` (Section 5).
+
+### 3.1 Groups
+```
+discover  "Discover"  order 0
+extract   "Extract"   order 1
+crawl     "Crawl"     order 2
+special   "Special"   order 3
+```
+
+### 3.2 Tools
+
+Endpoints and billing_op VERIFIED against `http-mcp-tool-executor.ts` and `rates.ts LedgerOperation`:
+
+| key | group | label | kind | endpoint · mcp | special | workflow | billing_op |
+|---|---|---|---|---|---|---|---|
+| `search` | discover | Search | composite_source | (per source) | 0 | 0 | (per source) |
+| `scrape` | extract | Scrape | modal | (per mode) | 0 | 0 | `EXTRACT_URL` / `EXTRACT_SITE` |
+| `transcribe` | extract | Transcribe | composite_source | (per source) | 0 | 0 | (per source) |
+| `map` | crawl | Map | simple | `/map-urls` · `map_site_urls` | 0 | 0 | `URL_MAP` ✅(was MAP_SITE) |
+| `crawl` | crawl | Crawl | simple | `/extract-site` · `extract_site` | 0 | 0 | `EXTRACT_SITE` |
+| `directory` | special | Directory | simple | `/directory/run` ✅(was /directory) · `directory_workflow` | 1 | 1 | ⚠️ no dedicated op — bills per-city via `MAPS_SEARCH`; verify in `/directory/run` handler |
+| `seo_analysis` | special | SEO Technical Analysis | simple | `/extract-site` (background) · `extract_site` | 1 | 1 | `EXTRACT_SITE` |
+| `deep_research` | special | Deep Research | simple | `/workflows/run` · `workflow_run` | 1 | 1 | ⚠️ no `WORKFLOW` op — confirm how `/workflows/run` bills |
+| `place_intel` | special | Place Intel | simple | `/maps/place` ✅(was /maps/place-intel) · `maps_place_intel` | 1 | 0 | `MAPS_PLACE` |
+| `page_intel` | special | Page Intel | simple | `/facebook/page-intel` · `facebook_page_intel` | 1 | 0 | ⚠️ no `FB_PAGE` op — verify which op `/facebook/page-intel` debits |
+| `rank_tracker` | special | Rank Tracker | simple | ⚠️ NO REST route in executor · `rank_tracker_blueprint` | 1 | 1 | ⚠️ unknown — see §9 |
+| `ai_visibility` | special | AI Visibility (Fan-out) | simple | ⚠️ no simple REST route — driven by `browser_*` agent · `browser_capture_fanout` | 1 | 0 | `BROWSER_SESSION` ✅(was BROWSER_AGENT) |
+
+### 3.3 Sources
+
+**search** (multi-select, default = web) — routes VERIFIED against `http-mcp-tool-executor.ts`:
+| key | label | endpoint | mcp_name | billing_op | default |
+|---|---|---|---|---|---|
+| `web` | Web | `/harvest/sync` (body `serpOnly:true`) | `search_serp` | `SERP` | ✓ |
+| `web_detailed` | Web — SERP + PAA detail | `/harvest/sync` | `harvest_paa` | `PAA` | |
+| `maps` | Maps | `/maps/search` | `maps_search` | `MAPS_SEARCH` | |
+| `fb_ads` | Facebook Ads | `/facebook/search` | `facebook_ad_search` | `FB_SEARCH` | |
+| `youtube` | YouTube | `/youtube/harvest` | `youtube_harvest` | `YT_CHANNEL` | |
+| `instagram` | Instagram | `/instagram/profile-content` | `instagram_profile_content` | `INSTAGRAM_PROFILE` | |
+
+> Note: `search_serp` and `harvest_paa` BOTH hit `/harvest/sync` — `search_serp` adds `serpOnly:true`.
+> The catalog source rows carry that override in `catalog_sources` (add a `body_overrides TEXT` column,
+> see §9) rather than via separate endpoints.
+
+**transcribe** (single-select, default = youtube) — routes VERIFIED:
+| key | label | endpoint | mcp_name | billing_op | default |
+|---|---|---|---|---|---|
+| `youtube` | YouTube Video | `/youtube/transcribe` | `youtube_transcribe` | `TRANSCRIPTION` | ✓ |
+| `fb_video` | Facebook Video | `/facebook/video-transcribe` | `facebook_video_transcribe` | `TRANSCRIPTION` | |
+| `fb_ad` | Facebook Ad | `/facebook/transcribe` | `facebook_ad_transcribe` | `FB_TRANSCRIBE` | |
+| `instagram` | Instagram Reel | `/instagram/media-download` | `instagram_media_download` | `INSTAGRAM_MEDIA` | |
+
+### 3.4 Modes (scrape)
+
+group `scope` (default single):
+| option | label | endpoint_override | mcp_name_override | param_overrides |
+|---|---|---|---|---|
+| `single` | Headless Single | `/extract-url` | `extract_url` | `{}` |
+| `bulk` | Headless Bulk | `/extract-site` | `extract_site` | `{ "rotateProxies": true }` |
+
+group `browser_mode` (default default):
+| option | label | param_overrides |
+|---|---|---|
+| `default` | Default | `{}` |
+| `chrome_profile` | Use Chrome Profile | `{ "browserProfile": true }` |
+
+> `browserProfile` is a new body flag the scrape route would honor to route through a saved hosted
+> profile (the `browser_profile_*` system). Wire in a follow-up; the catalog can ship the toggle disabled.
+
+### 3.5 Params (per tool/source) — keys MUST match the Zod schemas in `mcp-tool-schemas.ts`
+
+**search / web** (`search_serp`): `query`(string,req), `location`(string), `gl`(string),
+`maxResults`(number,1..100,default 10).
+**search / web_detailed** (`harvest_paa`): `query`(string,req), `maxQuestions`(number,1..200,default 30), `location`(string).
+**search / maps** (`maps_search`): `query`(string,req), `location`(string,req), `maxResults`(number,1..20,default 10), `includeReviews`(boolean).
+**search / fb_ads** (`facebook_ad_search`): `query`(string,req), `country`(string,default US), `maxResults`(number,1..20,default 10).
+**search / youtube** (`youtube_harvest`): mode `enum[search,channel]`, `query`(string), `channelHandle`(string), `maxVideos`(number,1..500,default 50).
+**search / instagram** (`instagram_profile_content`): `handle|url`(string,req), `maxItems`(number,1..2000,default 50), `maxScrolls`(number,0..250,default 10).
+
+**transcribe / ***: `url`(url,req) for all four. Add `includeTranscript`(boolean,default true) +
+`mediaTypes`(multi_enum[image,video,audio]) for instagram.
+
+**scrape (all)**: `url`(url,req), `formats`(multi_enum[markdown,links,json,images,branding]),
+`screenshot`(boolean,advanced), `downloadMedia`(boolean,advanced).
+**scrape / scope=bulk only** (`source_key='scope:bulk'`): `maxPages`(number,1..10000,default 100),
+`rotateProxyEvery`(number,1..100,default 30,advanced), `background`(boolean).
+
+**map** (`map_site_urls`): `url`(url,req), `maxUrls`(number,1..10000,default 100).
+**crawl** (`extract_site`): `url`(url,req), `maxPages`(number,1..10000,default 100),
+`formats`(multi_enum...), `rotateProxies`(boolean,default true), `background`(boolean,default true),
+`rotateProxyEvery`(number,advanced).
+
+**directory** (`directory_workflow`): `query`(string,req), `state`(string,req), `minPopulation`(number,default 100000),
+`maxResultsPerCity`(number,1..50,default 50,advanced), `concurrency`(number,advanced).
+**seo_analysis** (`extract_site` background): `url`(url,req), `maxPages`(number,default 1000),
+`formats` preset `["markdown","links","json","images"]` (hidden/locked).
+**deep_research**: `question`(string,req), `depth`(enum[quick,standard,deep],default standard).
+**place_intel** (`maps_place_intel`): `businessName`(string,req), `location`(string,req), `includeReviews`(boolean), `maxReviews`(number,1..500,default 50).
+**page_intel** (`facebook_page_intel`): `url|pageId`(string,req).
+**rank_tracker** (`rank_tracker_blueprint`): `domain`(string,req), `keywords`(array<string>), `location`(string).
+**ai_visibility** (`browser_capture_fanout`): `prompt`(string), `session_id`(string), `first_party_domain`(string), `export`(boolean).
+
+### 3.6 Examples (sample prompts)
+- search: "Top restaurants in San Francisco"
+- scrape: "https://steenshoney.com/"
+- crawl: "https://steenshoney.com/ (full site SEO crawl)"
+- transcribe: a Facebook/YouTube video URL
+- directory: "roofers in TN, minPopulation 100000"
+
+---
+
+## 4. Repository — `src/api/tool-catalog-repository.ts`
+
+```ts
+export interface CatalogParam { key: string; label: string; type: string; required: boolean;
+  default?: unknown; min?: number; max?: number; enumValues?: Array<{value:string;label:string}>;
+  placeholder?: string; help?: string; dependsOn?: string; dependsValue?: string; advanced: boolean; sourceKey: string | null }
+export interface CatalogSource { key: string; label: string; icon?: string; endpoint: string; mcpName: string;
+  defaultSelected: boolean; multiSelect: boolean; costHintMc?: number }
+export interface CatalogModeOption { groupKey: string; groupLabel: string; optionKey: string; optionLabel: string;
+  isDefault: boolean; endpointOverride?: string; mcpNameOverride?: string; paramOverrides?: Record<string,unknown> }
+export interface CatalogTool { key: string; groupKey: string; label: string; tagline?: string; description?: string;
+  icon?: string; kind: 'simple'|'composite_source'|'modal'; isSpecial: boolean; isWorkflow: boolean;
+  defaultEndpoint?: string; defaultMcpName?: string; billingOp?: string; costHintMc?: number;
+  sources: CatalogSource[]; modes: CatalogModeOption[]; params: CatalogParam[]; examples: string[] }
+export interface CatalogGroup { key: string; label: string; icon?: string; tools: CatalogTool[] }
+
+export async function getCatalog(): Promise<CatalogGroup[]>  // one assembled tree, ordered
+export async function seedCatalog(): Promise<void>            // idempotent upsert from catalog-seed.ts
+```
+
+`getCatalog()`: 5 SELECTs (groups, tools, sources, modes, params, examples) ordered by `display_order`,
+assembled in memory into the nested tree. Single round-trip-ish; cache in-process for 60s (catalog is static).
+
+---
+
+## 5. Seeding — `src/api/catalog-seed.ts` + wire into `migrate()`
+
+- `catalog-seed.ts` exports `CATALOG_SEED` (typed literal mirroring Section 3).
+- `seedCatalog()` performs `INSERT ... ON CONFLICT(key/id) DO UPDATE` (libSQL `INSERT OR REPLACE`)
+  for every row, so redeploys converge the DB to the authored definition. Rows removed from the seed
+  are pruned with `DELETE FROM <table> WHERE key NOT IN (...)`.
+- Call order in `db.ts`: after `migrate()` creates the tables, call `await seedCatalog()` once at boot
+  (in `bin/api-server.ts` startup, after `migrate()`), behind `if (process.env.SEED_CATALOG !== 'false')`.
+
+---
+
+## 6. REST endpoint — `src/api/server.ts`
+
+```
+GET /catalog        -> 200 { groups: CatalogGroup[] }   (public; no auth; cache-control 60s)
+```
+- Add `app.get('/catalog', async (c) => c.json({ groups: await getCatalog() }))`.
+- No vendor leakage risk (static metadata) but still run labels/descriptions through the existing
+  output path conventions if any dynamic field is ever added.
+
+Frontend contract (the exact JSON the picker consumes):
+```jsonc
+{ "groups": [
+  { "key":"discover","label":"Discover","icon":"search","tools":[
+    { "key":"search","label":"Search","kind":"composite_source","isSpecial":false,
+      "sources":[{"key":"web","label":"Web","endpoint":"/search-serp","mcpName":"search_serp","defaultSelected":true,"multiSelect":true}, ...],
+      "modes":[], "params":[{"key":"query","label":"Query","type":"string","required":true,"sourceKey":null}, ...],
+      "examples":["Top restaurants in San Francisco"] } ]},
+  ... ] }
+```
+
+Frontend behavior:
+- Render groups as the top nav (Discover/Extract/Crawl/Special). `isSpecial` tools live under the
+  **Special** group already by `group_key='special'`; the flag is redundant insurance for filtering.
+- `composite_source` → render the source dropdown (multi/single per `multiSelect`); union the params of
+  selected sources (params with matching `sourceKey`) plus shared params (`sourceKey=null`).
+- `modal` → render mode toggle groups; on selection, merge `paramOverrides` into the body and switch
+  endpoint per `endpointOverride`.
+- `simple` → render `params` form; POST to `defaultEndpoint`.
+- "Get code" button: serialize the assembled request (endpoint + body) to a curl/SDK snippet.
+
+---
+
+## 7. Build order
+1. **Schema** — add the 6 tables + indexes to `db.ts migrate()`. (P1)
+2. **Seed authoring** — `catalog-seed.ts` with the full Section-3 mapping. (P1)
+3. **Repository + seed routine** — `tool-catalog-repository.ts`; call `seedCatalog()` at boot. (P1)
+4. **Endpoint** — `GET /catalog`. (P1)
+5. **Billing ops** — add any missing `LedgerOperation` keys referenced by `billing_op`. (P1, gating)
+6. **Frontend picker** — consume `/catalog`, render the three kinds. (P2)
+7. **Mode wiring follow-ups** — `browserProfile` flag on the scrape route; `seo_analysis`/`deep_research`
+   workflow runners if not already endpoints. (P2/P3)
+
+## 8. Non-goals (v1)
+Auth-gated per-plan tool visibility, usage analytics on catalog, A/B of layouts, i18n of labels,
+and the actual `browserProfile` / workflow-runner execution wiring (catalog ships them as
+described-but-some-disabled until the routes exist).
+
+---
+
+## 9. Verification notes (self-checked against the codebase)
+
+Routes verified against `src/mcp/http-mcp-tool-executor.ts`; ledger ops against
+`src/api/rates.ts LedgerOperation`. Corrections already applied above:
+
+**Route fixes (my first draft was wrong):**
+- `search_serp` → **`/harvest/sync` with body `serpOnly:true`** (not `/search-serp` — that route doesn't exist).
+- `harvest_paa` → **`/harvest/sync`** (not `/harvest-paa`). Both search sources share this route.
+- `facebook_ad_search` → **`/facebook/search`** (not `/facebook/ad-search`).
+- `facebook_ad_transcribe` → **`/facebook/transcribe`** (not `/facebook/ad-transcribe`).
+- `maps_place_intel` → **`/maps/place`** (not `/maps/place-intel`).
+- `directory_workflow` → **`/directory/run`** (the executor route; there is also a separate top-level `/directory` page route — do not confuse).
+
+**Billing-op fixes:**
+- `map` → **`URL_MAP`** (not the invented `MAP_SITE`).
+- `ai_visibility` → **`BROWSER_SESSION`** (not `BROWSER_AGENT`).
+- Verified-correct as-is: `EXTRACT_URL`, `EXTRACT_SITE`, `MAPS_PLACE`, `MAPS_SEARCH`, `SERP`, `PAA`,
+  `FB_SEARCH`, `FB_TRANSCRIBE`, `YT_CHANNEL`, `TRANSCRIPTION`, `INSTAGRAM_PROFILE`, `INSTAGRAM_MEDIA`.
+
+**Unresolved — needs a human/code check before seeding (do NOT guess):**
+1. **`rank_tracker_blueprint` has NO route in `http-mcp-tool-executor.ts`.** It's a registered MCP tool
+   but isn't wired through the HTTP executor — it may be served by a different server (`paa-mcp-server`),
+   be skill-only, or unimplemented over REST. Confirm its execution path before adding it to the catalog;
+   otherwise ship it disabled.
+2. **No dedicated ledger ops for `directory`, `deep_research`/workflows, `page_intel`.** Directory likely
+   bills per-city through `MAPS_SEARCH`; `/workflows/run` and `/facebook/page-intel` billing must be read
+   from their handlers. Set `billing_op` only after confirming, or leave null and compute cost server-side.
+3. **`ai_visibility` / `browser_capture_fanout` is not a simple POST** — it requires an open browser
+   session (`session_id` from `browser_open`). It can't be a one-shot form like the others; model it as a
+   guided flow or omit from v1.
+
+**Schema addendum from verification:** add `body_overrides TEXT` (JSON) to `catalog_sources` so the
+`web` source can carry `{"serpOnly":true}` against the shared `/harvest/sync` route. (The §2 DDL should
+include this column.)
+
+**Confidence:** the four core verbs (Search, Scrape, Map, Crawl) + Transcribe + Place/Page Intel are
+fully route- and billing-verified and safe to build. The three flagged Special tools
+(rank_tracker, deep_research, ai_visibility) need route/billing confirmation first.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.3.14",
+  "version": "0.3.15",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",
@@ -58,6 +58,7 @@
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@onkernel/sdk": "^0.52.0",
     "@resvg/resvg-js": "^2.6.2",
+    "@vercel/blob": "^2.4.1",
     "commander": "^12.0.0",
     "hono": "^4.12.18",
     "inngest": "^4.4.0",

package/dist/chunk-3QHZPR4U.js

@@ -1,7 +0,0 @@
-// src/version.ts
-var PACKAGE_VERSION = "0.3.14";
-
-export {
-  PACKAGE_VERSION
-};
-//# sourceMappingURL=chunk-3QHZPR4U.js.map

package/dist/chunk-3QHZPR4U.js.map

@@ -1 +0,0 @@
-{"version":3,"sources":["../src/version.ts"],"sourcesContent":["export const PACKAGE_VERSION = '0.3.14'\n"],"mappings":";AAAO,IAAM,kBAAkB;","names":[]}