mcp-scraper 0.3.25 → 0.3.26

package/dist/bin/api-server.cjs

@@ -10872,16 +10872,6 @@ var init_catalog_seed = __esm({
               isSpecial: true,
               tagline: "Generate a rank-tracker blueprint",
               params: [{ key: "domain", label: "Domain", type: "string", required: true }]
-            },
-            {
-              key: "deep_research_workflow",
-              label: "Deep Research",
-              kind: "simple",
-              endpoint: "/workflows/run",
-              mcpName: "workflow_run",
-              isSpecial: true,
-              tagline: "Multi-source, fact-checked research report",
-              params: [{ key: "question", label: "Question", type: "string", required: true }]
             }
           ]
         }
@@ -24879,7 +24869,7 @@ var PACKAGE_VERSION;
 var init_version = __esm({
   "src/version.ts"() {
     "use strict";
-    PACKAGE_VERSION = "0.3.25";
+    PACKAGE_VERSION = "0.3.26";
   }
 });
 
@@ -24889,33 +24879,73 @@ var init_server_instructions = __esm({
   "src/mcp/server-instructions.ts"() {
     "use strict";
     SERVER_INSTRUCTIONS = `
-This server scrapes and analyzes web, social, search, maps, and site data. Pick a tool by NAME using
-this routing map, then load its schema before calling.
-
-ROUTING
-- One web page -> extract_url. Whole site (crawl + SEO report) -> extract_site. Just the URL list -> map_site_urls.
-- Web search (organic SERP) -> search_serp. Full SERP + People-Also-Ask / AI-Overview detail -> harvest_paa.
-- Google Maps: find places -> maps_search; one place deep-dive + reviews -> maps_place_intel.
-- YouTube: find or list videos -> youtube_harvest; transcribe one video -> youtube_transcribe.
-- Facebook: find ads -> facebook_ad_search; transcribe an ad -> facebook_ad_transcribe; transcribe a
-  video -> facebook_video_transcribe; page profile/intel -> facebook_page_intel.
-- Instagram: profile inventory -> instagram_profile_content; one post or reel -> instagram_media_download.
-- Workflows (multi-step): local directory build -> directory_workflow; rank blueprint -> rank_tracker_workflow;
-  AI-answer citation fan-out (AEO) -> query_fanout_workflow; deep research -> deep_research_workflow.
-- Anything without a dedicated tool (e.g. Reddit, arbitrary logged-in sites) -> the browser_* agent
-  (browser_open, then navigate/read).
-- Logged-in sites (ChatGPT, Claude, Reddit, any account): save the login first -> browser_profile_connect
-  (returns an mcpscraper.dev sign-in link for the user; one profile holds MANY logins, call it again with
-  the same profile + a new domain to add accounts) -> poll browser_profile_list until AUTHENTICATED ->
-  browser_open with that profile. browser_profile_list also shows what a profile is connected to.
-
-NOTES
-- Bulk / full-site crawls: call extract_site with rotateProxies:true for blocked or rate-limited sites.
-  It discovers URLs and returns a saved folder/artifact plus a summary, not the full content inline.
-- Tools that open a browser session or transcribe media cost credits and take longer. Prefer the
-  cheapest tool that answers the question (plain search/extract before browser agents).
-- Large results are saved to disk or an artifact and returned as a summary plus a path; read the path
-  for full detail rather than expecting the whole payload inline.
+# MCP Scraper
+
+Scrapes and analyzes web, search, maps, social, and site data. **Pick a tool by NAME from the map
+below, then load its schema before calling.** Where a tool needs a value another tool produces, the
+seam is noted so you can chain them.
+
+## Search & research
+- Organic Google results, rankings, local pack -> **search_serp**.
+- Full SERP plus People-Also-Ask and AI-Overview detail -> **harvest_paa**.
+- \`search_serp\` returns \`organicResults[].url\` (often including reddit.com threads) \u2014 feed those into
+  \`extract_url\` or \`reddit_thread\`.
+
+## Pages & sites
+- One page -> **extract_url** (takes a url).
+- Whole site, crawl + SEO report -> **extract_site** (takes a url).
+- Just the URL list/inventory -> **map_site_urls** (takes a url).
+- \`map_site_urls\` returns urls you can feed straight into \`extract_url\`.
+
+## Google Maps
+- Find multiple places/competitors/prospects -> **maps_search** (returns \`results[]\` with name,
+  placeUrl, cid).
+- One business deep-dive + reviews -> **maps_place_intel** (takes businessName + location, NOT an id \u2014
+  call it directly with a name from a \`maps_search\` result or from the user).
+
+## YouTube
+- Find or list videos -> **youtube_harvest** (returns \`videos[].videoId\`).
+- Transcribe one video -> **youtube_transcribe** (takes a videoId from \`youtube_harvest\`, or a url).
+
+## Facebook
+- Find advertisers -> **facebook_ad_search** (returns \`advertisers[]\` with pageId, libraryId).
+- One advertiser's ads -> **facebook_page_intel** (takes pageId, libraryId, or query; returns
+  \`ads[].videoUrl\`).
+- Transcribe an ad video -> **facebook_ad_transcribe** (takes a videoUrl from \`facebook_page_intel\`).
+- Transcribe an organic reel/video/post -> **facebook_video_transcribe** (takes the Facebook url directly).
+
+## Instagram
+- Profile inventory -> **instagram_profile_content** (returns post/reel/tv urls).
+- One post or reel -> **instagram_media_download** (takes a url from \`instagram_profile_content\`, or a
+  url the user gives).
+
+## Reddit
+- A reddit.com thread/post URL -> **reddit_thread**. Returns the post plus its comment tree and handles
+  Reddit's bot wall itself (no login needed). Find threads first with \`search_serp\`.
+
+## Other sites & logins (browser agent)
+For an arbitrary site or a logged-in dashboard with no dedicated tool, use the browser_* agent. **First
+decide whether the site needs a login:**
+- **Needs a login** (ChatGPT, Claude, any account) -> save it first: **browser_profile_connect** returns
+  an mcpscraper.dev sign-in link for the user; one profile holds MANY logins (call it again with the same
+  profile + a new domain to add accounts). Poll **browser_profile_list** until AUTHENTICATED, then
+  **browser_open** with that profile.
+- **No login** -> **browser_open**, then navigate/read.
+
+## Workflows
+Multi-step orchestrations \u2014 prefer these over hand-chaining primitives when the whole job is one of these:
+- Build a local business directory for a niche across a state/region -> **directory_workflow**.
+- Stand up a rank-tracking blueprint (database + cron + ingestion plan) -> **rank_tracker_workflow**.
+- Find which sources an AI answer cites for a query (AEO) -> **query_fanout_workflow** (open a browser
+  session on chatgpt.com or claude.ai FIRST, then run it against that session).
+
+## Notes
+- Bulk / full-site crawls: call \`extract_site\` with \`rotateProxies:true\` for blocked or rate-limited
+  sites. It returns a saved folder/artifact plus a summary, not the full content inline.
+- Browser sessions and media transcription cost credits and take longer \u2014 prefer the cheapest tool that
+  answers the question (plain search/extract before browser agents).
+- Large results are saved to disk or an artifact and returned as a summary plus a path; read the path for
+  full detail rather than expecting the whole payload inline.
 `.trim();
   }
 });