mcp-scraper 0.3.25 → 0.3.26

package/docs/spec-server-instructions.md

@@ -0,0 +1,94 @@
+# Spec — server-instructions.ts routing-map rewrite
+
+Implements the approved improvements from the MCP-design audit + reviewer feedback. **One file changes:** `src/mcp/server-instructions.ts` (the `SERVER_INSTRUCTIONS` string — the only content loaded into the model's context before it fetches any tool schema). No other source files. Requires a build + Vercel deploy (remote-MCP gets it immediately) + npm republish (the string ships in the stdio package).
+
+## Approved improvements (the full list)
+1. **Add `reddit_thread` to the map; stop routing Reddit to the browser.** Current line 15 says "Anything without a dedicated tool (e.g. Reddit…) -> browser_* agent" — **stale**, we shipped `reddit_thread` in 0.3.25. A model reading the current map will drive a browser for Reddit instead of calling the dedicated tool.
+2. **Make discovery→action ID seams explicit and schema-accurate** (reviewer #1). Encode, per chain, what the discovery tool *returns* and what the action tool *takes* — using the real params, not inferred ones.
+3. **Workflow triggers + boundary** (reviewer #2): plain-language "when to reach for a workflow vs hand-chaining," and state `query_fanout_workflow`'s browser prerequisite.
+4. **Reorder the no-dedicated-tool branch to check login FIRST** (reviewer #3), so a logged-in site isn't short-circuited into `browser_open` without `browser_profile_connect`.
+5. **Keep** the `rotateProxies` NOTE (reviewer #4 — defensible cross-cutting hint).
+
+Not in this spec (explicitly out): annotations (audited → correct as-is, not a bug), server namespace rename (breaking, deferred decision), tool-count tiering (soft, not needed). `deep_research_workflow` removal already shipped.
+
+## Verified schema facts backing each seam (source of truth)
+| Discovery tool | Returns (output schema) | Action tool | Takes (input schema) | Seam |
+|---|---|---|---|---|
+| `map_site_urls` / `search_serp` | `urls` / `organicResults[].url` | `extract_url` | `url` | url → url |
+| `search_serp` | `organicResults[].url` (incl. reddit.com) | `reddit_thread` | `url` | reddit url → reddit_thread |
+| `maps_search` | `results[].{name, placeUrl, cid}` | `maps_place_intel` | **`businessName` + `location`** (NOT an id) | name → name; **independent, callable directly** |
+| `youtube_harvest` | `videos[].videoId` | `youtube_transcribe` | `videoId` OR `url` | videoId → videoId |
+| `facebook_ad_search` | `advertisers[].{pageId, libraryId}` | `facebook_page_intel` | `pageId` / `libraryId` / `query` | id → id |
+| `facebook_page_intel` | `ads[].videoUrl` | `facebook_ad_transcribe` | `videoUrl` | videoUrl → videoUrl |
+| (organic FB url) | — | `facebook_video_transcribe` | `url` (FB reel/video/post) | url direct |
+| `instagram_profile_content` | `post`/`reel`/`tv` url arrays | `instagram_media_download` | `url` | url → url |
+| (must precede) `browser_open` on chatgpt.com/claude.ai | — | `query_fanout_workflow` | runs against the open AI session | browser-session prerequisite |
+
+`facebook_ad_transcribe` does NOT consume anything from `facebook_ad_search` — the videoUrl comes from `facebook_page_intel`. `maps_place_intel` does NOT consume a `place_id` — it takes a business name + location.
+
+## Exact change
+Replace the entire `SERVER_INSTRUCTIONS` template literal in `src/mcp/server-instructions.ts` (lines 1–29) with:
+
+```ts
+export const SERVER_INSTRUCTIONS = `
+This server scrapes and analyzes web, social, search, maps, and site data. Pick a tool by NAME using
+this routing map, then load its schema before calling. Where a tool needs a value another tool
+produces, the seam is noted so you can chain them.
+
+ROUTING
+- One web page -> extract_url (takes a url). Whole site (crawl + SEO report) -> extract_site (takes a
+  url). Just the URL list -> map_site_urls (takes a url). map_site_urls and search_serp return urls you
+  can feed straight into extract_url.
+- Web search (organic SERP) -> search_serp. Full SERP + People-Also-Ask / AI-Overview detail ->
+  harvest_paa. search_serp returns organicResults[].url (often including reddit.com threads) to feed
+  extract_url or reddit_thread.
+- Google Maps: find places -> maps_search (returns results[] with name, placeUrl, cid). One place
+  deep-dive + reviews -> maps_place_intel (takes businessName + location, NOT an id — call it directly
+  with a name from a maps_search result or from the user).
+- YouTube: find or list videos -> youtube_harvest (returns videos[].videoId). Transcribe one ->
+  youtube_transcribe (takes a videoId from youtube_harvest, or a url).
+- Facebook: find advertisers -> facebook_ad_search (returns advertisers[] with pageId, libraryId). Get
+  one advertiser's ads -> facebook_page_intel (takes pageId, libraryId, or query; returns ads[].videoUrl).
+  Transcribe an ad video -> facebook_ad_transcribe (takes a videoUrl from facebook_page_intel).
+  Transcribe an organic Facebook reel/video/post -> facebook_video_transcribe (takes the Facebook url
+  directly).
+- Instagram: profile inventory -> instagram_profile_content (returns post/reel/tv urls). One post or
+  reel -> instagram_media_download (takes a url from instagram_profile_content, or a url the user gives).
+- Reddit: a reddit.com thread/post URL -> reddit_thread. It returns the post plus its comment tree and
+  handles Reddit's bot wall itself (no login needed). Find threads first with search_serp.
+- No dedicated tool (an arbitrary site, or a logged-in dashboard) -> the browser_* agent. FIRST decide
+  whether the site needs a login:
+  - Needs a login (ChatGPT, Claude, any account) -> save it first: browser_profile_connect returns an
+    mcpscraper.dev sign-in link for the user; one profile holds MANY logins (call it again with the same
+    profile + a new domain to add accounts). Poll browser_profile_list until AUTHENTICATED, then
+    browser_open with that profile. browser_profile_list also shows what a profile is connected to.
+  - No login -> browser_open, then navigate/read.
+
+WORKFLOWS (multi-step orchestrations — prefer these over hand-chaining primitives when the whole job is one of these)
+- Build a local business directory for a niche across a state/region -> directory_workflow.
+- Stand up a rank-tracking blueprint (database + cron + ingestion plan) -> rank_tracker_workflow.
+- Find which sources an AI answer cites for a query (AEO) -> query_fanout_workflow (open a browser
+  session on chatgpt.com or claude.ai FIRST, then run it against that session).
+
+NOTES
+- Bulk / full-site crawls: call extract_site with rotateProxies:true for blocked or rate-limited sites.
+  It discovers URLs and returns a saved folder/artifact plus a summary, not the full content inline.
+- Tools that open a browser session or transcribe media cost credits and take longer. Prefer the
+  cheapest tool that answers the question (plain search/extract before browser agents).
+- Large results are saved to disk or an artifact and returned as a summary plus a path; read the path
+  for full detail rather than expecting the whole payload inline.
+\`.trim()
+```
+
+## Build / deploy / release
+1. Edit the one file above.
+2. `npm run build` (typecheck + bundle; regenerates rate catalog — no functional change here).
+3. `vercel deploy --prod --yes` → remote-MCP clients get the new routing map immediately.
+4. **Bump 0.3.25 → 0.3.26** in the 3 version files (`package.json`, `package-lock.json`, `src/version.ts`) and `npm publish` — `SERVER_INSTRUCTIONS` is compiled into the stdio package, so npm-installed clients need the new version to see the corrected map.
+5. Commit on `main` (now the source of truth): `fix(mcp): routing map — add reddit_thread, explicit chaining seams, workflow triggers, login-first fallback`.
+
+## Verification
+- `grep -c reddit_thread src/mcp/server-instructions.ts` → ≥1 (was 0).
+- `grep -c "without a dedicated tool" src/mcp/server-instructions.ts` → 0 (old Reddit-to-browser line gone).
+- After deploy, the live `/mcp` server-instructions block reflects the new map (inspect via an MCP client's server info, or the SERVER_INSTRUCTIONS in the published tarball).
+- Behavioral smoke (manual, in an MCP client): "get the comments on <reddit thread url>" routes to `reddit_thread`, not `browser_open`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.3.25",
+  "version": "0.3.26",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",

package/dist/chunk-WPHAMNT7.js

@@ -1,7 +0,0 @@
-// src/version.ts
-var PACKAGE_VERSION = "0.3.25";
-
-export {
-  PACKAGE_VERSION
-};
-//# sourceMappingURL=chunk-WPHAMNT7.js.map

package/dist/chunk-WPHAMNT7.js.map

@@ -1 +0,0 @@
-{"version":3,"sources":["../src/version.ts"],"sourcesContent":["export const PACKAGE_VERSION = '0.3.25'\n"],"mappings":";AAAO,IAAM,kBAAkB;","names":[]}