scrapeless-mcp-server 0.4.9 → 0.6.0

package/README.md

@@ -9,6 +9,7 @@ Built on the open MCP standard, Scrapeless MCP Server seamlessly connects models
 - **Google services integration** (Search, Trends)
 - **Browser automation** for page-level navigation and interaction
 - **Scrape** dynamic, JS-heavy sites—export as HTML, Markdown, or screenshots
+- **Crawl** entire websites by following links and capture each page in multiple formats
 
 Whether you're building an AI research assistant, a coding copilot, or autonomous web agents, this server provides the dynamic context and real-world data your workflows need—**without getting blocked**.
 
@@ -100,13 +101,12 @@ Scrapeless MCP Server supports both **Stdio** and **Streamable HTTP** transport
 
 Customize browser session behavior with optional parameters. These can be set via environment variables (for Stdio) or HTTP headers (for Streamable HTTP):
 
-| Stdio (Env Var)         | Streamable HTTP (HTTP Header) | Description                                                  |
-| ----------------------- | ----------------------------- | ------------------------------------------------------------ |
-| BROWSER_PROFILE_ID      | x-browser-profile-id          | Specifies a reusable browser profile ID for session continuity. |
-| BROWSER_PROFILE_PERSIST | x-browser-profile-persist     | Enables persistent storage for cookies, local storage, etc.  |
+| Stdio (Env Var)         | Streamable HTTP (HTTP Header) | Description                                                                                                                  |
+| ----------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| BROWSER_PROFILE_ID      | x-browser-profile-id          | Specifies a reusable browser profile ID for session continuity.                                                              |
+| BROWSER_PROFILE_PERSIST | x-browser-profile-persist     | Enables persistent storage for cookies, local storage, etc.                                                                  |
 | BROWSER_SESSION_TTL     | x-browser-session-ttl         | Defines the **maximum session timeout** in seconds. The session will automatically expire after this duration of inactivity. |
 
-
 ## Integration with Claude Desktop
 
 1. Open **Claude Desktop**
@@ -129,26 +129,32 @@ Customize browser session behavior with optional parameters. These can be set vi
 
 ## Supported MCP Tools
 
-| Name               | Description                                       |
-| ------------------ | ------------------------------------------------- |
-| google_search      | Universal information search engine.              |
-| google_trends      | Get trending search data from Google Trends.      |
-| browser_goto       | Navigate browser to a specified URL.              |
-| browser_go_back    | Go back one step in browser history.              |
-| browser_go_forward | Go forward one step in browser history.           |
-| browser_click      | Click a specific element on the page.             |
-| browser_type       | Type text into a specified input field.           |
-| browser_press_key  | Simulate a key press.                             |
-| browser_wait_for   | Wait for a specific page element to appear.       |
-| browser_wait       | Pause execution for a fixed duration.             |
-| browser_screenshot | Capture a screenshot of the current page.         |
-| browser_get_html   | Get the full HTML of the current page.            |
-| browser_get_text   | Get all visible text from the current page.       |
-| browser_scroll     | Scroll to the bottom of the page.                 |
-| browser_scroll_to  | Scroll a specific element into view.              |
-| scrape_html        | Scrape a URL and return its full HTML content.    |
-| scrape_markdown    | Scrape a URL and return its content as Markdown.  |
-| scrape_screenshot  | Capture a high-quality screenshot of any webpage. |
+| Name               | Description                                                    |
+| ------------------ | -------------------------------------------------------------- |
+| google_search      | Universal information search engine.                           |
+| google_trends      | Get trending search data from Google Trends.                   |
+| browser_create     | Create or reuse a cloud browser session using Scrapeless.      |
+| browser_close      | Closes the current session by disconnecting the cloud browser. |
+| browser_goto       | Navigate browser to a specified URL.                           |
+| browser_go_back    | Go back one step in browser history.                           |
+| browser_go_forward | Go forward one step in browser history.                        |
+| browser_click      | Click a specific element on the page.                          |
+| browser_type       | Type text into a specified input field.                        |
+| browser_press_key  | Simulate a key press.                                          |
+| browser_wait_for   | Wait for a specific page element to appear.                    |
+| browser_wait       | Pause execution for a fixed duration.                          |
+| browser_screenshot | Capture a screenshot of the current page.                      |
+| browser_get_html   | Get the full HTML of the current page.                         |
+| browser_get_text   | Get all visible text from the current page.                    |
+| browser_scroll     | Scroll to the bottom of the page.                              |
+| browser_scroll_to  | Scroll a specific element into view.                           |
+| scrape_html        | Scrape a URL and return its full HTML content.                 |
+| scrape_markdown    | Scrape a URL and return its content as Markdown.               |
+| scrape_screenshot  | Capture a high-quality screenshot of any webpage.              |
+| crawl_start        | Start an asynchronous crawl job from a base URL and return its job id. |
+| crawl_cancel       | Cancel an in-progress crawl job by its id.                     |
+| crawl_result       | Poll a crawl job by its id until it completes and return the crawled data. |
+| llm_chat_scraper   | Create an LLM Chat Scraper task for ChatGPT, Gemini, Perplexity, Copilot, Google AI Mode, Google AI Overview, Grok, or Alexa. |
 
 ## Security Best Practices
 

package/build/server.js

@@ -27,7 +27,7 @@ export const initMcpTools = (server, headers, apiKey) => {
     };
     // tools registration
     Object.values(toolsList).forEach((tool) => {
-        server.tool(tool.name, tool.description, tool.inputSchema, (params) => tool.handle(params, getScrapelessClient()));
+        server.tool(tool.name, tool.description, tool.inputSchema, (params) => tool.handle(params, getScrapelessClient(), headers));
     });
     const context = ContextManager.getInstance().getContext(apiKey ?? API_KEY);
     Object.values(browserTools).forEach((tool) => {

package/build/tools/crawl/api.js

@@ -0,0 +1,26 @@
+import axios from "axios";
+import { API_KEY, BASE_URL, API_KEY_NAME } from "../../config.js";
+/**
+ * Build an axios instance targeting the Scrapeless v2 crawler API.
+ *
+ * The `x-api-token` is resolved from the per-request client when available
+ * (so multi-tenant HTTP mode keeps using the caller's key) and falls back to
+ * the `SCRAPELESS_KEY` environment variable.
+ */
+export function getCrawlApi(client, headers) {
+    // Priority: explicit request header (HTTP multi-tenant) -> key carried by the
+    // per-request client -> SCRAPELESS_KEY env (stdio / fallback).
+    const apiKey = headers?.[API_KEY_NAME] ||
+        client?.scrapingCrawl?.crawl?.apiKey ||
+        API_KEY ||
+        "";
+    return axios.create({
+        baseURL: BASE_URL,
+        headers: {
+            "Content-Type": "application/json",
+            [API_KEY_NAME]: apiKey,
+        },
+        timeout: 30000,
+    });
+}
+export const CRAWL_ENDPOINT = "/api/v2/crawler/crawl";

package/build/tools/crawl/crawlCancel.js

@@ -0,0 +1,19 @@
+import { defineTool, wrapMcpResponse } from "../utils.js";
+import { getCrawlApi, CRAWL_ENDPOINT } from "./api.js";
+import z from "zod";
+export const crawlCancel = defineTool({
+    name: "crawl_cancel",
+    description: `Cancel an in-progress crawl job by its id (the id returned by crawl_start). Returns the cancelled status.`,
+    inputSchema: {
+        id: z
+            .string()
+            .describe("The crawl job id returned when the job was created."),
+    },
+    handle: async (params, client, headers) => {
+        return wrapMcpResponse(async () => {
+            const api = getCrawlApi(client, headers);
+            const { data } = await api.delete(`${CRAWL_ENDPOINT}/${params.id}`);
+            return data;
+        });
+    },
+});

package/build/tools/crawl/crawlResult.js

@@ -0,0 +1,72 @@
+import { defineTool } from "../utils.js";
+import { getCrawlApi, CRAWL_ENDPOINT } from "./api.js";
+import z from "zod";
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+const TERMINAL_STATUSES = ["completed", "failed", "cancelled"];
+export const crawlResult = defineTool({
+    name: "crawl_result",
+    description: `Fetch the result of a crawl job by its id (the id returned by crawl_start).
+    Polls the job status until it reaches a terminal state (completed / failed / cancelled) or the timeout is reached.
+    If the timeout is reached before the job finishes, the latest status is returned along with the job id so you can retry later.`,
+    inputSchema: {
+        id: z
+            .string()
+            .describe("The crawl job id returned when the job was created."),
+        timeout: z
+            .number()
+            .optional()
+            .describe("Maximum time to wait in seconds before returning the latest result. Defaults to 300 (5 minutes)."),
+        pollInterval: z
+            .number()
+            .optional()
+            .describe("Interval between status checks in seconds. Defaults to 5 seconds."),
+    },
+    handle: async (params, client, headers) => {
+        const { id } = params;
+        const timeoutMs = (params.timeout ?? 300) * 1000;
+        const pollIntervalMs = (params.pollInterval ?? 5) * 1000;
+        const api = getCrawlApi(client, headers);
+        const deadline = Date.now() + timeoutMs;
+        try {
+            let data;
+            while (true) {
+                const response = await api.get(`${CRAWL_ENDPOINT}/${id}`);
+                data = response.data;
+                if (data && TERMINAL_STATUSES.includes(data.status)) {
+                    return {
+                        content: [
+                            {
+                                type: "text",
+                                text: `Response:\n\n${JSON.stringify(data)}`,
+                            },
+                        ],
+                    };
+                }
+                if (Date.now() + pollIntervalMs >= deadline) {
+                    return {
+                        content: [
+                            {
+                                type: "text",
+                                text: `The crawl job did not finish within the timeout. ` +
+                                    `Latest status is "${data?.status}". You can call crawl_result again ` +
+                                    `with this job id to keep waiting, or crawl_cancel to cancel it.\n\n` +
+                                    `Job ID: ${id}\n\nLatest response:\n\n${JSON.stringify(data)}`,
+                            },
+                        ],
+                    };
+                }
+                await sleep(pollIntervalMs);
+            }
+        }
+        catch (error) {
+            return {
+                content: [
+                    {
+                        type: "text",
+                        text: `Failed to fetch crawl result for job ${id}. Error: ${error.message}`,
+                    },
+                ],
+            };
+        }
+    },
+});

package/build/tools/crawl/crawlStart.js

@@ -0,0 +1,138 @@
+import { defineTool, wrapMcpResponse } from "../utils.js";
+import { getCrawlApi, CRAWL_ENDPOINT } from "./api.js";
+import z from "zod";
+const formatEnum = z.enum([
+    "markdown",
+    "html",
+    "rawHtml",
+    "links",
+    "screenshot",
+    "screenshot@fullPage",
+    "json",
+]);
+const scrapeOptionsSchema = z
+    .object({
+    formats: z
+        .array(formatEnum)
+        .optional()
+        .describe("Formats to include in the output. Defaults to ['markdown']."),
+    onlyMainContent: z
+        .boolean()
+        .optional()
+        .describe("Only return the main content of the page excluding headers, navs, footers, etc."),
+    includeTags: z
+        .array(z.string())
+        .optional()
+        .describe("Tags to include in the output."),
+    excludeTags: z
+        .array(z.string())
+        .optional()
+        .describe("Tags to exclude from the output."),
+    headers: z
+        .record(z.any())
+        .optional()
+        .describe("Headers to send with the request. Can be used to send cookies, user-agent, etc."),
+    waitFor: z
+        .number()
+        .optional()
+        .describe("Delay in milliseconds before fetching the content, allowing the page sufficient time to load."),
+    timeout: z
+        .number()
+        .optional()
+        .describe("Timeout in milliseconds for the request."),
+})
+    .describe("Options that control how each page is scraped.");
+const browserOptionsSchema = z
+    .object({
+    sessionName: z
+        .string()
+        .optional()
+        .describe("A name for your session to facilitate searching and viewing in the historical session list."),
+    sessionTTL: z
+        .string()
+        .optional()
+        .describe("Session duration in seconds. Defaults to 180s, customizable between 60s and 900s."),
+    sessionRecording: z
+        .string()
+        .optional()
+        .describe("Whether to enable session recording. Defaults to false."),
+    proxyCountry: z
+        .string()
+        .optional()
+        .describe("Target country/region for the proxy as a country code (e.g. US, GB, ANY)."),
+    proxyURL: z
+        .string()
+        .optional()
+        .describe("Custom proxy URL, e.g. http://user:pass@ip:port. If set, all other proxy_* parameters are ignored."),
+    fingerprint: z
+        .string()
+        .optional()
+        .describe("Custom browser fingerprint configuration."),
+})
+    .describe("Options that control the underlying scraping browser session.");
+export const crawlStart = defineTool({
+    name: "crawl_start",
+    description: `Start an asynchronous crawl job. Crawls a website starting from a base URL, following links according to the provided options, and captures page content in various formats (markdown, html, links, screenshot, etc.).
+    Returns a job id that can be used with crawl_result to fetch results and crawl_cancel to cancel the job.
+    Only 'url' is required; all other parameters are optional.`,
+    inputSchema: {
+        url: z.string().url().describe("The base URL to start crawling from."),
+        limit: z
+            .number()
+            .optional()
+            .describe("Maximum number of pages to crawl. Default limit is 10000."),
+        excludePaths: z
+            .array(z.string())
+            .optional()
+            .describe("URL pathname regex patterns that exclude matching URLs from the crawl."),
+        includePaths: z
+            .array(z.string())
+            .optional()
+            .describe("URL pathname regex patterns that include matching URLs in the crawl."),
+        maxDepth: z
+            .number()
+            .optional()
+            .describe("Maximum depth to crawl relative to the base URL (max number of slashes in the pathname)."),
+        maxDiscoveryDepth: z
+            .number()
+            .optional()
+            .describe("Maximum depth to crawl based on discovery order."),
+        ignoreSitemap: z
+            .boolean()
+            .optional()
+            .describe("Ignore the website sitemap when crawling."),
+        ignoreQueryParameters: z
+            .boolean()
+            .optional()
+            .describe("Do not re-scrape the same path with different (or none) query parameters."),
+        deduplicateSimilarURLs: z
+            .boolean()
+            .optional()
+            .describe("Controls whether similar URLs should be deduplicated."),
+        regexOnFullURL: z
+            .boolean()
+            .optional()
+            .describe("Controls whether the include/exclude regex should be applied to the full URL."),
+        allowBackwardLinks: z
+            .boolean()
+            .optional()
+            .describe("Allow the crawler to follow links that are not part of the URL hierarchy you specify."),
+        allowExternalLinks: z
+            .boolean()
+            .optional()
+            .describe("Allow the crawler to follow links to external websites."),
+        delay: z
+            .number()
+            .optional()
+            .describe("Delay in seconds between scrapes. This helps respect website rate limits."),
+        scrapeOptions: scrapeOptionsSchema.optional(),
+        browserOptions: browserOptionsSchema.optional(),
+    },
+    handle: async (params, client, headers) => {
+        return wrapMcpResponse(async () => {
+            const api = getCrawlApi(client, headers);
+            const { data } = await api.post(CRAWL_ENDPOINT, params);
+            return data;
+        });
+    },
+});

package/build/tools/index.js

@@ -3,3 +3,7 @@ export * from "./deepserp/googleTrends.js";
 export * from "./universal/scrapeHtml.js";
 export * from "./universal/scrapeMarkdown.js";
 export * from "./universal/scrapeScreenshot.js";
+export * from "./crawl/crawlStart.js";
+export * from "./crawl/crawlCancel.js";
+export * from "./crawl/crawlResult.js";
+export * from "./llm_chat_scraper/llmChatScraper.js";

package/build/tools/llm_chat_scraper/api.js

@@ -0,0 +1,27 @@
+import axios from "axios";
+import { API_KEY, BASE_URL, API_KEY_NAME } from "../../config.js";
+/**
+ * Build an axios instance targeting the Scrapeless v2 scraper task API.
+ *
+ * The `x-api-token` is resolved from the per-request client when available
+ * (so multi-tenant HTTP mode keeps using the caller's key) and falls back to
+ * the `SCRAPELESS_KEY` environment variable.
+ */
+export function getLlmChatScraperApi(client, headers) {
+    // Priority: explicit request header (HTTP multi-tenant) -> key carried by the
+    // per-request client -> SCRAPELESS_KEY env (stdio / fallback).
+    const apiKey = headers?.[API_KEY_NAME] ||
+        client?.scraping?.apiKey ||
+        API_KEY ||
+        "";
+    return axios.create({
+        baseURL: BASE_URL,
+        headers: {
+            "Content-Type": "application/json",
+            [API_KEY_NAME]: apiKey,
+        },
+        timeout: 30000,
+    });
+}
+export const LLM_CHAT_SCRAPER_REQUEST_ENDPOINT = "/api/v2/scraper/request";
+export const LLM_CHAT_SCRAPER_RESULT_ENDPOINT = "/api/v2/scraper/result";

package/build/tools/llm_chat_scraper/llmChatScraper.js

@@ -0,0 +1,634 @@
+import axios from "axios";
+import z from "zod";
+import { BASE_URL } from "../../config.js";
+import { defineTool } from "../utils.js";
+import { getLlmChatScraperApi, LLM_CHAT_SCRAPER_REQUEST_ENDPOINT, LLM_CHAT_SCRAPER_RESULT_ENDPOINT, } from "./api.js";
+const POLL_INTERVAL_MS = 5000;
+const DEFAULT_TIMEOUT_SECONDS = 180;
+const MIN_TIMEOUT_SECONDS = 60;
+const MAX_TIMEOUT_SECONDS = 600;
+const ACTOR_OPTIONS = [
+    "scraper.chatgpt",
+    "scraper.gemini",
+    "scraper.perplexity",
+    "scraper.copilot",
+    "scraper.aimode",
+    "scraper.overview",
+    "scraper.grok",
+    "scraper.alexa",
+];
+const MODE_OPTIONS = [
+    "search",
+    "smart",
+    "chat",
+    "reasoning",
+    "study",
+    "MODEL_MODE_FAST",
+    "MODEL_MODE_EXPERT",
+    "MODEL_MODE_AUTO",
+];
+const ACTOR_CONFIG = {
+    "scraper.chatgpt": {
+        displayName: "ChatGPT",
+        supportsWebSearch: true,
+        supportsShopping: true,
+    },
+    "scraper.gemini": {
+        displayName: "Gemini",
+    },
+    "scraper.perplexity": {
+        displayName: "Perplexity",
+        supportsWebSearch: true,
+    },
+    "scraper.copilot": {
+        displayName: "Microsoft Copilot",
+        supportsMode: true,
+        requiresMode: true,
+    },
+    "scraper.aimode": {
+        displayName: "Google AI Mode",
+        supportsShopping: true,
+        supportsGoogleLocation: true,
+    },
+    "scraper.overview": {
+        displayName: "Google AI Overview",
+        supportsShopping: true,
+        supportsGoogleLocation: true,
+    },
+    "scraper.grok": {
+        displayName: "Grok",
+        supportsMode: true,
+        requiresMode: true,
+    },
+    "scraper.alexa": {
+        displayName: "Alexa",
+    },
+};
+const RESULT_FIELDS_BY_ACTOR = {
+    "scraper.chatgpt": [
+        "prompt",
+        "result_text",
+        "model",
+        "web_search",
+        "links",
+        "search_result",
+        "content_references",
+        "products",
+        "ads",
+        "map",
+        "search_model_queries",
+    ],
+    "scraper.gemini": [
+        "result_text",
+        "prompt",
+        "citations",
+        "related_queries",
+    ],
+    "scraper.perplexity": [
+        "prompt",
+        "result_text",
+        "related_prompt",
+        "web_results",
+        "media_items",
+    ],
+    "scraper.copilot": [
+        "result_text",
+        "prompt",
+        "mode",
+        "links",
+        "citations",
+    ],
+    "scraper.aimode": [
+        "result_text",
+        "result_md",
+        "result_html",
+        "raw_url",
+        "citations",
+        "search_result",
+        "products",
+    ],
+    "scraper.overview": [
+        "content",
+        "rawtext",
+        "metadata",
+        "is_overview_shopping",
+        "products",
+        "source",
+        "web_source",
+        "ads",
+    ],
+    "scraper.grok": [
+        "conversation",
+        "create_time",
+        "follow_up_suggestions",
+        "full_response",
+        "tool_usages",
+        "user_model",
+        "user_query",
+        "web_search_results",
+        "footnotes",
+        "x_search_results",
+    ],
+    "scraper.alexa": [
+        "user_text",
+        "md_text",
+        "raw_text",
+        "completed",
+        "answer_fragment_uri",
+        "answer_revision",
+        "dialog_request_id",
+        "endpoint_id",
+        "fragment_count",
+        "conversation",
+        "directives",
+        "references",
+        "sources",
+        "suggestions",
+        "products",
+    ],
+};
+const llmChatScraperSchema = z.object({
+    prompt: z
+        .string()
+        .min(1)
+        .describe("Question or prompt to send to the selected LLM Chat Scraper actor."),
+    actor: z
+        .enum(ACTOR_OPTIONS)
+        .describe("Scrapeless actor to run. Choose one explicitly: scraper.chatgpt, scraper.gemini, scraper.perplexity, scraper.copilot, scraper.aimode, scraper.overview, scraper.grok, or scraper.alexa."),
+    country: z
+        .string()
+        .optional()
+        .default("US")
+        .describe("Country or region code used by the actor. Defaults to US."),
+    web_search: z
+        .boolean()
+        .optional()
+        .describe("Enable web search for scraper.chatgpt or scraper.perplexity when explicitly provided."),
+    shopping: z
+        .boolean()
+        .optional()
+        .describe("Fetch shopping/product data for scraper.chatgpt, scraper.aimode, or scraper.overview when explicitly provided."),
+    mode: z
+        .enum(MODE_OPTIONS)
+        .optional()
+        .describe("Required for scraper.copilot or scraper.grok. Copilot: search, smart, chat, reasoning, study. Grok: MODEL_MODE_FAST, MODEL_MODE_EXPERT, MODEL_MODE_AUTO."),
+    location: z
+        .string()
+        .optional()
+        .describe("Google canonical location for Google AI Mode or Google AI Overview. Mutually exclusive with uule."),
+    uule: z
+        .string()
+        .optional()
+        .describe("Pre-encoded Google UULE for Google AI Mode or Google AI Overview. Mutually exclusive with location."),
+    webhook: z
+        .object({
+        url: z
+            .string()
+            .url()
+            .describe("Webhook URL Scrapeless calls when the task completes."),
+    })
+        .optional()
+        .describe("Optional webhook object passed to the Scrapeless task request."),
+    timeout_seconds: z
+        .number()
+        .int()
+        .min(MIN_TIMEOUT_SECONDS)
+        .max(MAX_TIMEOUT_SECONDS)
+        .optional()
+        .default(DEFAULT_TIMEOUT_SECONDS)
+        .describe("Maximum polling time in seconds. Defaults to 180. Minimum is 60 and maximum is 600."),
+});
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+function textResponse(data) {
+    return {
+        content: [
+            {
+                type: "text",
+                text: `Response:\n\n${JSON.stringify(data)}`,
+            },
+        ],
+    };
+}
+function normalizeCountry(country) {
+    const normalized = (country ?? "US").trim().toUpperCase();
+    return normalized || "US";
+}
+function validateActorParams(params) {
+    const config = ACTOR_CONFIG[params.actor];
+    if (params.location && params.uule) {
+        return {
+            code: "invalid_input",
+            message: "location and uule are mutually exclusive.",
+        };
+    }
+    if (params.web_search !== undefined && !config.supportsWebSearch) {
+        return {
+            code: "invalid_input",
+            message: `web_search is not a supported option for ${params.actor}.`,
+        };
+    }
+    if (params.shopping !== undefined && !config.supportsShopping) {
+        return {
+            code: "invalid_input",
+            message: `shopping is not a supported top-level option for ${params.actor}.`,
+        };
+    }
+    if ((params.location || params.uule) && !config.supportsGoogleLocation) {
+        return {
+            code: "invalid_input",
+            message: `location and uule are only supported for scraper.aimode and scraper.overview.`,
+        };
+    }
+    if (params.mode !== undefined && !config.supportsMode) {
+        return {
+            code: "invalid_input",
+            message: `mode is only supported for scraper.copilot and scraper.grok.`,
+        };
+    }
+    if (params.actor === "scraper.copilot" && params.mode !== undefined) {
+        const allowedModes = ["search", "smart", "chat", "reasoning", "study"];
+        if (!allowedModes.includes(params.mode)) {
+            return {
+                code: "invalid_input",
+                message: `mode for scraper.copilot must be one of: ${allowedModes.join(", ")}.`,
+            };
+        }
+    }
+    if (params.actor === "scraper.grok" && params.mode !== undefined) {
+        const allowedModes = [
+            "MODEL_MODE_FAST",
+            "MODEL_MODE_EXPERT",
+            "MODEL_MODE_AUTO",
+        ];
+        if (!allowedModes.includes(params.mode)) {
+            return {
+                code: "invalid_input",
+                message: `mode for scraper.grok must be one of: ${allowedModes.join(", ")}.`,
+            };
+        }
+    }
+    if (config.requiresMode &&
+        params.mode === undefined) {
+        return {
+            code: "invalid_input",
+            message: `mode is required for ${params.actor}.`,
+        };
+    }
+    return undefined;
+}
+function buildActorInput(params, country) {
+    const actorInput = {
+        prompt: params.prompt,
+        country,
+    };
+    if (params.web_search !== undefined) {
+        actorInput.web_search = params.web_search;
+    }
+    if (params.shopping !== undefined) {
+        actorInput.shopping = params.shopping;
+    }
+    if (params.mode !== undefined) {
+        actorInput.mode = params.mode;
+    }
+    if (params.location)
+        actorInput.location = params.location;
+    if (params.uule)
+        actorInput.uule = params.uule;
+    return actorInput;
+}
+function isRecord(value) {
+    return Boolean(value) && typeof value === "object" && !Array.isArray(value);
+}
+function getStringField(source, field) {
+    if (!isRecord(source))
+        return undefined;
+    const value = source[field];
+    return typeof value === "string" && value.length > 0 ? value : undefined;
+}
+function extractTaskId(createResponse) {
+    return getStringField(createResponse, "task_id");
+}
+function extractRequestId(createResponse) {
+    return getStringField(createResponse, "request_id");
+}
+function extractTaskStatus(resultResponse) {
+    return getStringField(resultResponse, "status")?.toLowerCase();
+}
+function extractTaskMessage(resultResponse) {
+    return getStringField(resultResponse, "message");
+}
+function extractTaskResult(resultResponse) {
+    return isRecord(resultResponse) ? resultResponse.task_result : undefined;
+}
+function pickActorResult(actor, result) {
+    if (!isRecord(result))
+        return result;
+    return RESULT_FIELDS_BY_ACTOR[actor].reduce((actorResult, field) => {
+        if (field in result)
+            actorResult[field] = result[field];
+        return actorResult;
+    }, {});
+}
+function formatPollHttpError(statusCode, responseData) {
+    return {
+        status_code: statusCode,
+        code: "api_error",
+        message: getStringField(responseData, "message") ||
+            getStringField(responseData, "error") ||
+            `LLM chat scraper result request failed with HTTP ${statusCode}.`,
+        response: responseData,
+    };
+}
+function formatTaskResultError(code, message, response) {
+    return {
+        code,
+        message,
+        response,
+    };
+}
+async function pollTask(api, taskId, timeoutSeconds) {
+    const startedAtMs = Date.now();
+    const startedAt = new Date(startedAtMs).toISOString();
+    const deadline = startedAtMs + timeoutSeconds * 1000;
+    let latestResult;
+    let latestStatus = "processing";
+    let latestHttpStatus;
+    while (true) {
+        const response = await api.get(`${LLM_CHAT_SCRAPER_RESULT_ENDPOINT}/${encodeURIComponent(taskId)}`, {
+            validateStatus: () => true,
+        });
+        latestResult = response.data;
+        latestHttpStatus = response.status;
+        if (response.status === 429) {
+            latestStatus = "rate_limited_retrying";
+        }
+        else if (response.status < 200 || response.status >= 300) {
+            const finishedAtMs = Date.now();
+            return {
+                taskId,
+                status: `http_${response.status}`,
+                terminal: true,
+                ok: false,
+                timedOut: false,
+                startedAt,
+                finishedAt: new Date(finishedAtMs).toISOString(),
+                elapsedMs: finishedAtMs - startedAtMs,
+                latestResult,
+                httpStatus: response.status,
+                error: formatPollHttpError(response.status, response.data),
+            };
+        }
+        else {
+            const taskStatus = extractTaskStatus(response.data);
+            if (taskStatus === "success") {
+                const taskResult = extractTaskResult(response.data);
+                const finishedAtMs = Date.now();
+                if (!isRecord(taskResult)) {
+                    return {
+                        taskId,
+                        status: "invalid_response",
+                        terminal: true,
+                        ok: false,
+                        timedOut: false,
+                        startedAt,
+                        finishedAt: new Date(finishedAtMs).toISOString(),
+                        elapsedMs: finishedAtMs - startedAtMs,
+                        latestResult,
+                        httpStatus: response.status,
+                        error: formatTaskResultError("invalid_response", "Scrapeless result response status is success but task_result is missing or is not an object.", response.data),
+                    };
+                }
+                return {
+                    taskId,
+                    status: taskStatus,
+                    terminal: true,
+                    ok: true,
+                    timedOut: false,
+                    startedAt,
+                    finishedAt: new Date(finishedAtMs).toISOString(),
+                    elapsedMs: finishedAtMs - startedAtMs,
+                    latestResult,
+                    taskResult,
+                    httpStatus: response.status,
+                };
+            }
+            if (taskStatus === "failed") {
+                const finishedAtMs = Date.now();
+                return {
+                    taskId,
+                    status: taskStatus,
+                    terminal: true,
+                    ok: false,
+                    timedOut: false,
+                    startedAt,
+                    finishedAt: new Date(finishedAtMs).toISOString(),
+                    elapsedMs: finishedAtMs - startedAtMs,
+                    latestResult,
+                    httpStatus: response.status,
+                    error: formatTaskResultError("task_failed", extractTaskMessage(response.data) || "LLM chat scraper task failed.", response.data),
+                };
+            }
+            if (taskStatus === "pending" || taskStatus === "running") {
+                latestStatus = taskStatus;
+            }
+            else {
+                const finishedAtMs = Date.now();
+                return {
+                    taskId,
+                    status: "invalid_response",
+                    terminal: true,
+                    ok: false,
+                    timedOut: false,
+                    startedAt,
+                    finishedAt: new Date(finishedAtMs).toISOString(),
+                    elapsedMs: finishedAtMs - startedAtMs,
+                    latestResult,
+                    httpStatus: response.status,
+                    error: formatTaskResultError("invalid_response", "Scrapeless result response did not include a valid status: pending, running, success, or failed.", response.data),
+                };
+            }
+        }
+        const remainingMs = deadline - Date.now();
+        if (remainingMs <= 0) {
+            const finishedAtMs = Date.now();
+            return {
+                taskId,
+                status: latestStatus,
+                terminal: false,
+                ok: false,
+                timedOut: true,
+                startedAt,
+                finishedAt: new Date(finishedAtMs).toISOString(),
+                elapsedMs: finishedAtMs - startedAtMs,
+                latestResult,
+                httpStatus: latestHttpStatus,
+            };
+        }
+        await sleep(Math.min(POLL_INTERVAL_MS, remainingMs));
+    }
+}
+function timeoutHelp(taskId) {
+    return {
+        message: "The LLM chat scraper task did not finish before the timeout. Fetch the result manually with the Scrapeless API.",
+        manual_result_request: {
+            method: "GET",
+            url: `${BASE_URL}${LLM_CHAT_SCRAPER_RESULT_ENDPOINT}/${taskId}`,
+            headers: {
+                "x-api-token": "YOUR_SCRAPELESS_KEY",
+            },
+        },
+    };
+}
+function formatApiError(error) {
+    if (axios.isAxiosError(error)) {
+        const statusCode = error.response?.status;
+        const responseData = error.response?.data;
+        return {
+            status_code: statusCode,
+            code: statusCode === 429 ? "rate_limited" : "api_error",
+            message: getStringField(responseData, "message") ||
+                getStringField(responseData, "error") ||
+                error.message,
+            response: responseData,
+        };
+    }
+    return {
+        code: "unknown_error",
+        message: error.message,
+    };
+}
+export const llmChatScraper = defineTool({
+    name: "llm_chat_scraper",
+    description: `Create an LLM Chat Scraper task for an explicit Scrapeless actor, then poll every 5 seconds until the answer is ready or the timeout is reached.
+    Supports ChatGPT, Gemini, Perplexity, Copilot, Google AI Mode, Google AI Overview, Grok, and Alexa.
+    Defaults to a 3 minute timeout. The timeout can be set from 60 to 600 seconds.
+    On timeout, returns the task_id and instructions for manually fetching the result.`,
+    inputSchema: llmChatScraperSchema.shape,
+    handle: async (rawParams, client, headers) => {
+        const params = llmChatScraperSchema.parse(rawParams);
+        const country = normalizeCountry(params.country);
+        const actorConfig = ACTOR_CONFIG[params.actor];
+        const startedAtMs = Date.now();
+        const startedAt = new Date(startedAtMs).toISOString();
+        const validationError = validateActorParams(params);
+        if (validationError) {
+            return textResponse({
+                status: "failed",
+                actor: params.actor,
+                actor_display_name: actorConfig.displayName,
+                error: validationError,
+            });
+        }
+        const actorInput = buildActorInput(params, country);
+        const taskRequest = {
+            actor: params.actor,
+            input: actorInput,
+        };
+        if (params.webhook) {
+            taskRequest.webhook = params.webhook;
+        }
+        let taskId;
+        let requestId;
+        let createResponseBody;
+        try {
+            const api = getLlmChatScraperApi(client, headers);
+            const createResponse = await api.post(LLM_CHAT_SCRAPER_REQUEST_ENDPOINT, taskRequest);
+            createResponseBody = createResponse.data;
+            taskId = extractTaskId(createResponseBody);
+            requestId = extractRequestId(createResponseBody);
+            if (!taskId) {
+                return textResponse({
+                    status: "failed",
+                    actor: params.actor,
+                    actor_display_name: actorConfig.displayName,
+                    request_id: requestId,
+                    error: {
+                        code: "missing_task_id",
+                        message: "Scrapeless did not return a task id from /api/v2/scraper/request.",
+                    },
+                    create_response: createResponseBody,
+                });
+            }
+            const pollResult = await pollTask(api, taskId, params.timeout_seconds);
+            const finishedAtMs = Date.now();
+            if (pollResult.timedOut) {
+                return textResponse({
+                    status: "timeout",
+                    task_id: taskId,
+                    request_id: requestId,
+                    actor: params.actor,
+                    actor_display_name: actorConfig.displayName,
+                    timeout_seconds: params.timeout_seconds,
+                    poll_interval_seconds: POLL_INTERVAL_MS / 1000,
+                    execution_time: {
+                        started_at: startedAt,
+                        finished_at: new Date(finishedAtMs).toISOString(),
+                        elapsed_ms: finishedAtMs - startedAtMs,
+                    },
+                    latest_status: pollResult.status,
+                    latest_http_status: pollResult.httpStatus,
+                    latest_result: pollResult.latestResult,
+                    help: timeoutHelp(taskId),
+                    create_response: createResponseBody,
+                });
+            }
+            if (!pollResult.ok) {
+                return textResponse({
+                    status: "failed",
+                    task_id: taskId,
+                    request_id: requestId,
+                    actor: params.actor,
+                    actor_display_name: actorConfig.displayName,
+                    timeout_seconds: params.timeout_seconds,
+                    poll_interval_seconds: POLL_INTERVAL_MS / 1000,
+                    execution_time: {
+                        started_at: startedAt,
+                        finished_at: new Date(finishedAtMs).toISOString(),
+                        elapsed_ms: finishedAtMs - startedAtMs,
+                    },
+                    task_status: pollResult.status,
+                    result_http_status: pollResult.httpStatus,
+                    error: pollResult.error,
+                    latest_result: pollResult.latestResult,
+                    create_response: createResponseBody,
+                });
+            }
+            return textResponse({
+                status: "completed",
+                task_id: taskId,
+                request_id: requestId,
+                actor: params.actor,
+                actor_display_name: actorConfig.displayName,
+                timeout_seconds: params.timeout_seconds,
+                poll_interval_seconds: POLL_INTERVAL_MS / 1000,
+                execution_time: {
+                    started_at: startedAt,
+                    finished_at: new Date(finishedAtMs).toISOString(),
+                    elapsed_ms: finishedAtMs - startedAtMs,
+                },
+                task_status: pollResult.status,
+                result_http_status: pollResult.httpStatus,
+                result: pickActorResult(params.actor, pollResult.taskResult),
+                create_response: createResponseBody,
+            });
+        }
+        catch (error) {
+            const finishedAtMs = Date.now();
+            return textResponse({
+                status: "failed",
+                task_id: taskId,
+                request_id: requestId,
+                actor: params.actor,
+                actor_display_name: actorConfig.displayName,
+                execution_time: {
+                    started_at: startedAt,
+                    finished_at: new Date(finishedAtMs).toISOString(),
+                    elapsed_ms: finishedAtMs - startedAtMs,
+                },
+                error: formatApiError(error),
+                help: taskId ? timeoutHelp(taskId) : undefined,
+                create_response: createResponseBody,
+            });
+        }
+    },
+});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "scrapeless-mcp-server",
-  "version": "0.4.9",
+  "version": "0.6.0",
   "main": "index.js",
   "type": "module",
   "bin": {
@@ -23,7 +23,9 @@
     "cursor",
     "claude",
     "ai",
-    "model-context-protocol"
+    "model-context-protocol",
+    "crawl",
+    "llm-chat-scraper"
   ],
   "author": "",
   "license": "ISC",