mcp-scraper 0.3.25 → 0.3.27

package/dist/{worker-UC6D2756.js → worker-NV4GTIRG.js}

@@ -4,10 +4,10 @@ import {
   createHarvestAttemptRecorder,
   harvestProblemResponse,
   serializeHarvestProblem
-} from "./chunk-WYCER2HW.js";
+} from "./chunk-H74L743B.js";
 import {
   harvest
-} from "./chunk-G7PQ64LM.js";
+} from "./chunk-QSSH4RCX.js";
 import {
   browserServiceApiKey
 } from "./chunk-3PRO376E.js";
@@ -125,4 +125,4 @@ export {
   startWorker,
   tickOnce
 };
-//# sourceMappingURL=worker-UC6D2756.js.map
+//# sourceMappingURL=worker-NV4GTIRG.js.map

package/docs/spec-kernel-computer-controls.md

@@ -0,0 +1,166 @@
+# Spec — Tighten Kernel Computer Controls usage (human-like input on the hosted browser)
+
+## Correction to the original premise
+An earlier assessment claimed "we drive the Kernel browser over CDP with raw Playwright — we never use the Computer Controls API." **That was wrong** (a bad grep — searched raw HTTP paths + `page.mouse`, missed the SDK method names). Ground truth in `src/services/browser-agent/browser-agent-service.ts`:
+
+| Action | Implementation today | API used |
+|---|---|---|
+| screenshot | `k.browsers.computer.captureScreenshot` | ✅ Computer Controls |
+| click | `k.browsers.computer.clickMouse` | ✅ Computer Controls |
+| type | `k.browsers.computer.typeText` | ✅ Computer Controls |
+| scroll | `k.browsers.computer.scroll` | ✅ Computer Controls |
+| press | `k.browsers.computer.pressKey` | ✅ Computer Controls |
+| moveMouse | `k.browsers.computer.moveMouse({smooth:true})` | ✅ defined… but never called |
+| goto / read / locate / captureFanout | `playwrightChromium.connectOverCDP` | Playwright/CDP (DOM access — required, kept) |
+
+So the interactive agent's **input + screenshot already follow Kernel's API**. The DOM-read path uses Playwright-over-CDP because Computer Controls cannot read the DOM / element coordinates — that path stays (and is the same free Playwright code used off-Kernel).
+
+## Dual-path principle (the rule, stated)
+- **On Kernel (hosted browser):** drive *input* (move/click/type/scroll/press/screenshot) through **Computer Controls** (`k.browsers.computer.*`) — native OS input, no CDP input fingerprint, bezier curves, inter-keystroke delays. Read the DOM via Playwright-over-CDP (no Computer Controls equivalent).
+- **Off Kernel (free local/HTTP path, `BrowserDriver` non-hosted, future local mode):** keep Playwright. It's free and there's no Computer Controls layer to use.
+
+This spec changes only the Kernel input path. It does not touch Playwright scraping (`BrowserDriver`, reddit/serp/maps/fb deterministic tools) — their bottleneck is IP/challenge, not input cadence, and they barely click.
+
+## The two real gaps (verified)
+1. **Clicks teleport.** `click()` calls `clickMouse(x,y)` with no preceding move. `moveMouse({smooth:true})` exists (bezier) but is **called from nowhere** (`grep` confirms one definition, zero call sites). So every click is an instant cursor jump to the target — a behavioral bot signal Kernel's docs flag ("identical cursor paths, rapid mouse movements"). The bezier capability is built and unused.
+2. **Typing has no human cadence by default.** `typeText(id, text, delayMs?)` only passes `delay` if the *caller* supplies `body.delay` (`browser-agent-routes.ts:486`). The model never sends it → `delay` is `undefined` → instant uniform typing ("uniform typing speeds" — another flagged signal).
+
+## Changes
+
+### 1. `src/services/browser-agent/browser-agent-service.ts`
+
+**Add a constant + jitter helper near the top of the action section (after `client()`):**
+```ts
+const HUMAN_TYPE_DELAY_MS = 55
+function jitter(base: number, spread: number): number {
+  return Math.max(1, Math.round(base + (Math.random() * 2 - 1) * spread))
+}
+```
+
+**`click()` — move (bezier) to the target before clicking.** Replace the body:
+```ts
+export async function click(
+  runtimeSessionId: string,
+  x: number,
+  y: number,
+  opts: { button?: 'left' | 'right' | 'middle'; numClicks?: number } = {},
+): Promise<void> {
+  const k = client()
+  await k.browsers.computer.moveMouse(runtimeSessionId, { x, y, smooth: true })
+  await k.browsers.computer.clickMouse(runtimeSessionId, {
+    x,
+    y,
+    button: opts.button ?? 'left',
+    ...(opts.numClicks ? { num_clicks: opts.numClicks } : {}),
+  })
+}
+```
+(Optional optimization — collapse the two round-trips into one with `computer.batch` if the installed SDK exposes it: `await k.browsers.computer.batch(runtimeSessionId, { actions: [{ type: 'move_mouse', x, y, smooth: true }, { type: 'click_mouse', x, y, button: opts.button ?? 'left', num_clicks: opts.numClicks ?? 1 }] })`. Verify the SDK method/shape before using; if unavailable, keep the two sequential calls above.)
+
+**`typeText()` — default a jittered human delay when the caller gives none:**
+```ts
+export async function typeText(runtimeSessionId: string, text: string, delayMs?: number): Promise<void> {
+  const k = client()
+  const delay = typeof delayMs === 'number' ? delayMs : jitter(HUMAN_TYPE_DELAY_MS, 30)
+  await k.browsers.computer.typeText(runtimeSessionId, { text, delay })
+}
+```
+
+No route changes required — `browser-agent-routes.ts` already forwards `x/y`, `button`, `num_clicks`, and an optional `delay`; the new behavior is purely in the service defaults. The MCP tool schemas/descriptions need no change (the human-like behavior is automatic; callers still just pass x/y/text).
+
+### 2. No change to
+- `goto` / `readPage` / `locatePageTargets` / `captureFanout` — DOM access over CDP is required and free-path-shared. Leave as-is.
+- `BrowserDriver` and all deterministic scrapers (reddit, serp, maps, fb, instagram, youtube) — Playwright stays.
+- The free/local browser path — Playwright stays.
+
+## Verification
+1. `grep -n "moveMouse" src/services/browser-agent/browser-agent-service.ts` → now **2** hits (definition + the new call in `click`).
+2. `npm run build` clean.
+3. Deploy prod; open a session and drive it:
+   - `POST /agent/sessions {url:"https://example.com"}` → `POST /agent/sessions/:id/screenshot` (note an element's x/y) → `POST /agent/sessions/:id/click {x,y}` → screenshot again: the click lands (move+click path works).
+   - `POST /agent/sessions/:id/type {text:"hello world"}` (no `delay`) succeeds — typing now carries the default jittered cadence server-side.
+   - `DELETE /agent/sessions/:id`.
+4. Behavioral check on a detection-sensitive target (manual): drive a known bot-walled login form; clicks should approach via curve and typing should be paced (observable in the live-view / replay).
+
+## Release
+- One file changed (`browser-agent-service.ts`). The behavior ships server-side → live on Vercel deploy; **no npm republish required** (the stdio package is a thin client; tool schemas/descriptions are unchanged).
+- Commit on `main`: `feat(browser): human-like input on Kernel — bezier move-before-click + default typing cadence`.
+
+---
+
+# Phase 2 — Move the DOM-read path to the Playwright Execution API (in-VM + Patchright)
+
+## Why (revised from "defer")
+The read path (`goto`, `readPage`, `locatePageTargets`, `captureFanout`) uses `playwrightChromium.connectOverCDP(cdpWsUrl)` from our serverless function — **outside** the Kernel VM. Kernel's recommended-practices table explicitly says *"Prefer our Playwright Execution API or Computer Controls API over self-hosted Playwright/Puppeteer,"* and the bot-detection guide states the Execution API *"runs in the same VM as the browser, ensuring headers, user-agent strings, and environment match. Kernel automatically applies Patchright to remove automation fingerprints, including headless indicators."* So external CDP is the residual detection signal, and this is the documented fix — not optional polish.
+
+## The API
+`POST /browsers/{id}/playwright/execute` — body `{ code: string, timeout_sec?: 1..300 }`. The `code` runs **in-VM** inside a function with `page`, `context`, `browser` in scope; `return` a value. Response `{ success, result, error, stdout, stderr }`. SDK call (verify exact name against installed `@onkernel/sdk`): `k.browsers.playwright.execute(runtimeSessionId, { code, timeout_sec })`.
+
+## Mechanic
+Our read functions already do their work via `page.evaluate(FN)` over a `connectOverCDP` page. The migration is **mechanical**: take the existing in-page extraction JS (the function bodies passed to `page.evaluate`) and ship them as the `code` string to `execute`, returning the same shape — but now running in-VM with Patchright instead of over external CDP.
+
+### `src/services/browser-agent/browser-agent-service.ts`
+Add a helper:
+```ts
+async function runInVm<T>(runtimeSessionId: string, code: string, timeoutSec = 30): Promise<T> {
+  const k = client()
+  const res = await k.browsers.playwright.execute(runtimeSessionId, { code, timeout_sec: timeoutSec })
+  if (!res.success) throw new Error(res.error || 'playwright/execute failed')
+  return res.result as T
+}
+```
+Then rewrite each read function to call `runInVm` with its existing extraction logic as a code string instead of `connectOverCDP`:
+- **`goto(runtimeSessionId, url)`** → `code`: `` await page.goto(${JSON.stringify(url)}, { waitUntil: 'domcontentloaded', timeout: 45000 }); return { url: page.url(), title: await page.title() }; ``  (signature changes from `cdpWsUrl` to `runtimeSessionId`).
+- **`readPage`** → move the current element/text-collection `page.evaluate` body into the `code` string; `return` the same `{ url, title, text, elements }` object.
+- **`locatePageTargets`** → same: the selector/text-bounds logic becomes the `code` body, `return` the bounds array.
+- **`captureFanout`** → port its capture logic; if it relies on response listeners over CDP, evaluate whether it can run in-VM or must stay on CDP (the fan-out hook listens to network events — this one may legitimately stay on CDP; verify before migrating).
+
+### Routes (`src/api/browser-agent-routes.ts`)
+The read/locate/goto routes currently pass `row.cdp_ws_url` to the service. Change those call sites to pass `row.runtime_session_id`. No other route logic changes.
+
+### Keep CDP only where required
+If `captureFanout` needs live network-event listeners (CDP-only), leave it on `connectOverCDP`. Everything else moves in-VM.
+
+## Dual-path still holds
+- **On Kernel:** input → Computer Controls (Phase 1); DOM read → Playwright Execution API (Phase 2). Zero external CDP except any network-listener edge case.
+- **Off Kernel (free/local):** unchanged local Playwright.
+
+## Verification (Phase 2)
+1. `grep -c connectOverCDP src/services/browser-agent/browser-agent-service.ts` → drops to 0 (or 1 if captureFanout stays).
+2. Build clean.
+3. Drive a session: `goto` → `read` → `locate` return the same shapes as before (regression check against current output).
+4. On a detection-sensitive page, confirm reads no longer attach an external CDP client (the session shows only in-VM execution + Computer Controls input).
+
+## Release (Phase 2)
+- Same file (+ route call-site edits). Server-side → ships on deploy; no npm republish.
+- Separate commit from Phase 1 so each is independently revertible: `feat(browser): read DOM via Kernel Playwright Execution API (in-VM, Patchright) instead of external CDP`.
+
+## Sequencing
+Ship **Phase 1 first** (tiny, pure win), validate, then **Phase 2** (mechanical but touches 3–4 functions + their routes; needs a regression pass on read/locate output shape). Don't bundle — Phase 2 has real surface area.
+
+## Still out of scope
+- Migrating the **deterministic scrapers** (`BrowserDriver` reddit/serp/maps/fb) from `connectOverCDP` to Playwright Execution API. Same benefit would apply, but their bottleneck was IP/challenge (solved) and they're a larger blast radius. Flag as Phase 3 if detection on those tools resurfaces.
+- **Batch beyond move+click** — only if a concrete need appears.
+
+---
+
+# Other Kernel APIs — assessment + findings (2026-06-30)
+
+Reviewed the full Kernel API reference (browsers, playwright, computer-controls, filesystem, processes). Findings:
+
+## TESTED & REJECTED — browser curl for Reddit
+**Hypothesis:** `client.browsers.curl(id, {url})` (`POST /browsers/{id}/curl`) sends an HTTP request through Chrome's stack (TLS fingerprint + cookies + proxy), so curling `old.reddit/...json` through a residential-proxied browser would return the comment tree cheaply, no render.
+**Result: FAILED.** Both residential-proxied and no-proxy curl returned **HTTP 403 / Reddit block page** (no JSON). Reason: curl does NOT execute JavaScript, and Reddit's wall is a **JS challenge** that only a *rendered* browser solves. `reddit_thread` therefore MUST stay headful-render (the render is what beats the challenge; the ~$0.0009 cost is justified). **Do not re-attempt curl for Reddit.**
+**Curl is still valid** for genuine JSON/REST APIs with no JS challenge (inherits the browser's authenticated session + residential IP). Keep as a back-pocket tier; not a current build item.
+
+## REJECTED — do NOT run ffmpeg/media in the Kernel VM (cost-wrong)
+Initially flagged process-exec/filesystem for in-VM ffmpeg muxing. **Retracted — it's the wrong place cost-wise.** A Kernel browser VM bills $0.0001333/s (headful); muxing is pure CPU work unrelated to browsing, so running it there is the most expensive option. And we don't need it for product value:
+- **Transcription already runs outside Kernel** via fal.ai wizper, which takes an `audio_url` and fetches it itself (`media-transcription.ts`) — no download, no ffmpeg, no Kernel. Cheap and correct.
+- **Muxing** (`VideoMixer.ts`) is an optional convenience that gracefully degrades to returning separate track URLs when ffmpeg is absent.
+- The `instagram_media` ENOENT was the **Vercel serverless read-only filesystem** (`mkdir ~/Downloads/...`), NOT a Kernel gap. Fix = write to `/tmp` or return track URLs; if muxing must be reliable, run ffmpeg on a cheap non-serverless worker — never in Kernel.
+**Conclusion: keep all media transcode/transcription OUTSIDE Kernel.** Process-exec/filesystem APIs have no compelling MCP Scraper use today.
+
+## LOW / skip unless needed
+- computer-controls extras: `drag` (slider captchas), `clipboard` read/write, `set_cursor`, `get_cursor_position` — situational agent polish.
+- browsers: create/get/list/delete/update (already used), ad-hoc extension upload (niche), load-extensions.
+- filesystem watch/stream, process async/PTY/signals — only for long-running in-VM jobs.

package/docs/spec-server-instructions.md

@@ -0,0 +1,94 @@
+# Spec — server-instructions.ts routing-map rewrite
+
+Implements the approved improvements from the MCP-design audit + reviewer feedback. **One file changes:** `src/mcp/server-instructions.ts` (the `SERVER_INSTRUCTIONS` string — the only content loaded into the model's context before it fetches any tool schema). No other source files. Requires a build + Vercel deploy (remote-MCP gets it immediately) + npm republish (the string ships in the stdio package).
+
+## Approved improvements (the full list)
+1. **Add `reddit_thread` to the map; stop routing Reddit to the browser.** Current line 15 says "Anything without a dedicated tool (e.g. Reddit…) -> browser_* agent" — **stale**, we shipped `reddit_thread` in 0.3.25. A model reading the current map will drive a browser for Reddit instead of calling the dedicated tool.
+2. **Make discovery→action ID seams explicit and schema-accurate** (reviewer #1). Encode, per chain, what the discovery tool *returns* and what the action tool *takes* — using the real params, not inferred ones.
+3. **Workflow triggers + boundary** (reviewer #2): plain-language "when to reach for a workflow vs hand-chaining," and state `query_fanout_workflow`'s browser prerequisite.
+4. **Reorder the no-dedicated-tool branch to check login FIRST** (reviewer #3), so a logged-in site isn't short-circuited into `browser_open` without `browser_profile_connect`.
+5. **Keep** the `rotateProxies` NOTE (reviewer #4 — defensible cross-cutting hint).
+
+Not in this spec (explicitly out): annotations (audited → correct as-is, not a bug), server namespace rename (breaking, deferred decision), tool-count tiering (soft, not needed). `deep_research_workflow` removal already shipped.
+
+## Verified schema facts backing each seam (source of truth)
+| Discovery tool | Returns (output schema) | Action tool | Takes (input schema) | Seam |
+|---|---|---|---|---|
+| `map_site_urls` / `search_serp` | `urls` / `organicResults[].url` | `extract_url` | `url` | url → url |
+| `search_serp` | `organicResults[].url` (incl. reddit.com) | `reddit_thread` | `url` | reddit url → reddit_thread |
+| `maps_search` | `results[].{name, placeUrl, cid}` | `maps_place_intel` | **`businessName` + `location`** (NOT an id) | name → name; **independent, callable directly** |
+| `youtube_harvest` | `videos[].videoId` | `youtube_transcribe` | `videoId` OR `url` | videoId → videoId |
+| `facebook_ad_search` | `advertisers[].{pageId, libraryId}` | `facebook_page_intel` | `pageId` / `libraryId` / `query` | id → id |
+| `facebook_page_intel` | `ads[].videoUrl` | `facebook_ad_transcribe` | `videoUrl` | videoUrl → videoUrl |
+| (organic FB url) | — | `facebook_video_transcribe` | `url` (FB reel/video/post) | url direct |
+| `instagram_profile_content` | `post`/`reel`/`tv` url arrays | `instagram_media_download` | `url` | url → url |
+| (must precede) `browser_open` on chatgpt.com/claude.ai | — | `query_fanout_workflow` | runs against the open AI session | browser-session prerequisite |
+
+`facebook_ad_transcribe` does NOT consume anything from `facebook_ad_search` — the videoUrl comes from `facebook_page_intel`. `maps_place_intel` does NOT consume a `place_id` — it takes a business name + location.
+
+## Exact change
+Replace the entire `SERVER_INSTRUCTIONS` template literal in `src/mcp/server-instructions.ts` (lines 1–29) with:
+
+```ts
+export const SERVER_INSTRUCTIONS = `
+This server scrapes and analyzes web, social, search, maps, and site data. Pick a tool by NAME using
+this routing map, then load its schema before calling. Where a tool needs a value another tool
+produces, the seam is noted so you can chain them.
+
+ROUTING
+- One web page -> extract_url (takes a url). Whole site (crawl + SEO report) -> extract_site (takes a
+  url). Just the URL list -> map_site_urls (takes a url). map_site_urls and search_serp return urls you
+  can feed straight into extract_url.
+- Web search (organic SERP) -> search_serp. Full SERP + People-Also-Ask / AI-Overview detail ->
+  harvest_paa. search_serp returns organicResults[].url (often including reddit.com threads) to feed
+  extract_url or reddit_thread.
+- Google Maps: find places -> maps_search (returns results[] with name, placeUrl, cid). One place
+  deep-dive + reviews -> maps_place_intel (takes businessName + location, NOT an id — call it directly
+  with a name from a maps_search result or from the user).
+- YouTube: find or list videos -> youtube_harvest (returns videos[].videoId). Transcribe one ->
+  youtube_transcribe (takes a videoId from youtube_harvest, or a url).
+- Facebook: find advertisers -> facebook_ad_search (returns advertisers[] with pageId, libraryId). Get
+  one advertiser's ads -> facebook_page_intel (takes pageId, libraryId, or query; returns ads[].videoUrl).
+  Transcribe an ad video -> facebook_ad_transcribe (takes a videoUrl from facebook_page_intel).
+  Transcribe an organic Facebook reel/video/post -> facebook_video_transcribe (takes the Facebook url
+  directly).
+- Instagram: profile inventory -> instagram_profile_content (returns post/reel/tv urls). One post or
+  reel -> instagram_media_download (takes a url from instagram_profile_content, or a url the user gives).
+- Reddit: a reddit.com thread/post URL -> reddit_thread. It returns the post plus its comment tree and
+  handles Reddit's bot wall itself (no login needed). Find threads first with search_serp.
+- No dedicated tool (an arbitrary site, or a logged-in dashboard) -> the browser_* agent. FIRST decide
+  whether the site needs a login:
+  - Needs a login (ChatGPT, Claude, any account) -> save it first: browser_profile_connect returns an
+    mcpscraper.dev sign-in link for the user; one profile holds MANY logins (call it again with the same
+    profile + a new domain to add accounts). Poll browser_profile_list until AUTHENTICATED, then
+    browser_open with that profile. browser_profile_list also shows what a profile is connected to.
+  - No login -> browser_open, then navigate/read.
+
+WORKFLOWS (multi-step orchestrations — prefer these over hand-chaining primitives when the whole job is one of these)
+- Build a local business directory for a niche across a state/region -> directory_workflow.
+- Stand up a rank-tracking blueprint (database + cron + ingestion plan) -> rank_tracker_workflow.
+- Find which sources an AI answer cites for a query (AEO) -> query_fanout_workflow (open a browser
+  session on chatgpt.com or claude.ai FIRST, then run it against that session).
+
+NOTES
+- Bulk / full-site crawls: call extract_site with rotateProxies:true for blocked or rate-limited sites.
+  It discovers URLs and returns a saved folder/artifact plus a summary, not the full content inline.
+- Tools that open a browser session or transcribe media cost credits and take longer. Prefer the
+  cheapest tool that answers the question (plain search/extract before browser agents).
+- Large results are saved to disk or an artifact and returned as a summary plus a path; read the path
+  for full detail rather than expecting the whole payload inline.
+\`.trim()
+```
+
+## Build / deploy / release
+1. Edit the one file above.
+2. `npm run build` (typecheck + bundle; regenerates rate catalog — no functional change here).
+3. `vercel deploy --prod --yes` → remote-MCP clients get the new routing map immediately.
+4. **Bump 0.3.25 → 0.3.26** in the 3 version files (`package.json`, `package-lock.json`, `src/version.ts`) and `npm publish` — `SERVER_INSTRUCTIONS` is compiled into the stdio package, so npm-installed clients need the new version to see the corrected map.
+5. Commit on `main` (now the source of truth): `fix(mcp): routing map — add reddit_thread, explicit chaining seams, workflow triggers, login-first fallback`.
+
+## Verification
+- `grep -c reddit_thread src/mcp/server-instructions.ts` → ≥1 (was 0).
+- `grep -c "without a dedicated tool" src/mcp/server-instructions.ts` → 0 (old Reddit-to-browser line gone).
+- After deploy, the live `/mcp` server-instructions block reflects the new map (inspect via an MCP client's server info, or the SERVER_INSTRUCTIONS in the published tarball).
+- Behavioral smoke (manual, in an MCP client): "get the comments on <reddit thread url>" routes to `reddit_thread`, not `browser_open`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.3.25",
+  "version": "0.3.27",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",