@scavio/mcp-server 0.20.0 → 0.21.0

package/README.md

@@ -3,9 +3,9 @@
 ![GitHub Repo stars](https://img.shields.io/github/stars/scavio-ai/scavio-mcp?style=social)
 ![License](https://img.shields.io/github/license/scavio-ai/scavio-mcp)
 
-[Scavio](https://scavio.dev?utm_source=npm-mcp&utm_medium=package&utm_campaign=readme) is a unified [Web Search API](https://scavio.dev/docs/search-api?utm_source=npm-mcp&utm_medium=package&utm_campaign=readme) and MCP server that gives AI agents web search, page extraction, and structured data from e-commerce, social, travel, jobs, real-estate, app-store, ad-library and company-filing sources. 229 tools across 36 platforms plus Extract, one API key.
+[Scavio](https://scavio.dev?utm_source=npm-mcp&utm_medium=package&utm_campaign=readme) is a unified [Web Search API](https://scavio.dev/docs/search-api?utm_source=npm-mcp&utm_medium=package&utm_campaign=readme) and MCP server that gives AI agents web search, page extraction, and structured data from e-commerce, social, travel, jobs, real-estate, app-store, ad-library and company-filing sources. 255 tools across 36 platforms plus Extract, map, crawl and batch, one API key.
 
-**The 22 platforms added in 0.13.0 are opt-in.** Registering all 248 tools puts 234KB of tool definitions — roughly 58k tokens — into every session before you type anything. So the default is the platform set 0.12.x already had, plus Extract: 126 tools, 96KB. Upgrading never removes a tool you were using. Everything else is one env var away: see [Choosing which tools load](#choosing-which-tools-load).
+**The 22 platforms added in 0.13.0 are opt-in.** Registering all 255 tools puts 243KB of tool definitions — roughly 61k tokens — into every session before you type anything. So the default is the platform set 0.12.x already had, plus Extract: 133 tools, 105KB. Upgrading never removes a tool you were using. Everything else is one env var away: see [Choosing which tools load](#choosing-which-tools-load).
 
 ## Remote MCP Server
 
@@ -207,8 +207,8 @@ Add to settings (`Cmd+,`):
 
 ## Choosing which tools load
 
-Scavio exposes 248 tools. Loading all of them costs 234KB of `tools/list` —
-roughly 58k tokens of context in every session before the user has said anything,
+Scavio exposes 255 tools. Loading all of them costs 243KB of `tools/list` —
+roughly 61k tokens of context in every session before the user has said anything,
 and more tools than some clients will accept at all. So the server registers a
 curated subset by default and lets you pick the rest. The sizes below are
 measured, not estimated: `npm run toolslist` reproduces them.
@@ -217,10 +217,10 @@ measured, not estimated: `npm run toolslist` reproduces them.
 
 | Value | Registers |
 |-------|-----------|
-| *(unset)* or `default` | Every platform 0.12.x had, plus `extract` — **126 tools, 96KB** |
-| `all` | every platform — **248 tools, 234KB** |
-| `extract,sec,g2` | just those platforms — 11 tools, 12KB |
-| `default,zillow,metaads` | the default set plus two more — 132 tools, 103KB |
+| *(unset)* or `default` | Every platform 0.12.x had, plus `extract` — **133 tools, 105KB** |
+| `all` | every platform — **255 tools, 243KB** |
+| `extract,sec,g2` | just those platforms — 18 tools, 21KB |
+| `default,zillow,metaads` | the default set plus two more — 139 tools, 113KB |
 | `none` | no platform tools, only `get_usage` |
 
 Keys are matched case- and punctuation-insensitively, so `meta-ads`, `metaads`
@@ -279,8 +279,8 @@ https://mcp.scavio.dev/mcp?platforms=all
 
 Nothing to do. The default set is exactly the 10 platforms 0.12.x registered,
 plus Extract — every tool you were using still loads. Those platforms keep gaining
-tools (Walmart grew from 2 to 7; TikTok, YouTube, Reddit and X gained 19 in 0.20.0),
-which is why the default count is now 126.
+tools (Walmart grew from 2 to 7; TikTok, YouTube, Reddit and X gained 19 in 0.20.0;
+Extract gained map, crawl and batch in 0.21.0), which is why the default count is now 133.
 
 The 22 platforms added in 0.13.0 are opt-in. Add them additively:
 
@@ -306,14 +306,50 @@ are free.
 | Tool | Description |
 |------|-------------|
 | `extract_url` | Read any web page and return it as clean Markdown, plain text, or raw HTML |
+| `map_website` | List a site's page URLs from its sitemaps and start page, path-filtered, with `lastmod` when known |
+| `crawl_website` | Crawl a site from one URL and extract every same-site page; waits up to `wait_seconds` and returns results, or the job id |
+| `get_crawl_status` | A crawl's status, page counts, credits used and page results, paged by cursor; can wait for completion |
+| `cancel_crawl` | Stop a running crawl; nothing more is billed, results so far stay readable |
+| `batch_extract` | Extract a list of URLs in one job, results in input order; waits up to `wait_seconds` |
+| `get_batch_status` | A batch's status and results, paged by cursor; can wait for completion |
+| `cancel_batch` | Stop a running batch |
 
 Extract is tier-priced by the `mode` parameter, not flat: `normal` (the default)
-and `advanced` (full JavaScript rendering) cost 1 credit, `ultra` (for pages that
-block the other two) costs 2. Start at `normal` and escalate only when the
+and `advanced` (renders JavaScript) cost 1 credit, `ultra` (for the hardest pages)
+costs 2. Start at `normal` and escalate only when the
 content comes back empty or blocked. You are billed only on a successful
 extraction — a dead link, a bot wall or a timeout costs nothing. There is no
 pagination: the whole page comes back in one call.
 
+**Map, crawl and batch** (new in 0.21.0) are the multi-page operations:
+
+- **Pricing.** `map_website` costs 1 credit. Crawl and batch bill per
+  *successful* page at Extract's price for the tier (normal 1, advanced 1,
+  ultra 2); failed and skipped pages are free, and status and cancel calls are
+  free. Credits are taken page by page, and a job stops cleanly with
+  `stopped_reason: "insufficient_credits"` if the balance runs out.
+- **Plan limits** (max pages per crawl / active jobs / URLs per batch): Free
+  10 / 1 / 10, PAYG 100 / 1 / 50, Project 500 / 2 / 100, Bootstrap 2,000 / 3 / 500,
+  Startup 5,000 / 5 / 1,000, Growth 10,000 / 10 / 1,000. Starting a job past the
+  active-job limit returns a 429 that names the running jobs.
+- **Behaviour.** Robots.txt is always obeyed for map and crawl. A crawl stays on
+  the start URL's site (subdomains optional) and visits pages under the start
+  path first. Results are kept for 3 days; a job runs for at most 6 hours.
+- **Built for tool-call time limits.** MCP clients usually cancel a tool call
+  after about 60 seconds, so a crawl is a job, not one long call.
+  `crawl_website` / `batch_extract` start the job and wait up to `wait_seconds`
+  (default 30, max 50): a small job comes back finished with its first page of
+  results; a large one returns the job id (and `webhook_secret`, shown only
+  once, if you set `webhook_url`). `get_crawl_status` / `get_batch_status` take
+  the same `wait_seconds` to long-poll, and `cursor` to page.
+- **Output stays small.** Status calls return 10 results per call by default
+  (`limit` up to 100) and clip each page's content to `max_content_chars`
+  (default 4,000); clipped pages carry `content_clipped: true` next to the full
+  `content_length`. `map_website` asks for 200 links when you set no `limit`
+  (the API default is 1,000). Every response that has more to read carries a
+  `_hint` naming the exact next call. Repeat a start call with the same
+  `idempotency_key` and you get the original job back instead of a second one.
+
 ### [Google Search API](https://scavio.dev/docs/search-api?utm_source=npm-mcp&utm_medium=package&utm_campaign=readme)
 
 | Tool | Description |

package/dist/lib/client.d.ts

@@ -10,5 +10,6 @@ export declare class ScavioClient {
     constructor(apiKey: string);
     private request;
     get<T>(path: string): Promise<T>;
-    post<T>(path: string, body?: unknown): Promise<T>;
+    post<T>(path: string, body?: unknown, headers?: Record<string, string>): Promise<T>;
+    delete<T>(path: string): Promise<T>;
 }

package/dist/lib/client.js

@@ -14,7 +14,7 @@ export class ScavioClient {
         this.baseUrl = env.SCAVIO_API_BASE_URL;
         this.apiKey = apiKey;
     }
-    async request(method, path, body) {
+    async request(method, path, body, extraHeaders) {
         const response = await fetch(`${this.baseUrl}${path}`, {
             method,
             headers: {
@@ -22,6 +22,7 @@ export class ScavioClient {
                 "Authorization": `Bearer ${this.apiKey}`,
                 "User-Agent": USER_AGENT,
                 "X-Client-Source": CLIENT_SOURCE,
+                ...extraHeaders,
             },
             ...(body !== undefined ? { body: JSON.stringify(body) } : {}),
         });
@@ -29,24 +30,30 @@ export class ScavioClient {
             const text = await response.text();
             let message;
             let billingUrl;
+            let body;
             try {
                 const json = JSON.parse(text);
                 message = json.error ?? json.message ?? text;
                 if (typeof json.billing_url === "string")
                     billingUrl = json.billing_url;
+                if (json && typeof json === "object" && !Array.isArray(json))
+                    body = json;
             }
             catch {
                 message = text;
             }
-            throw new ApiError(response.status, message, undefined, billingUrl);
+            throw new ApiError(response.status, message, undefined, billingUrl, body);
         }
         return response.json();
     }
     async get(path) {
         return this.request("GET", path);
     }
-    async post(path, body) {
-        return this.request("POST", path, body);
+    async post(path, body, headers) {
+        return this.request("POST", path, body, headers);
+    }
+    async delete(path) {
+        return this.request("DELETE", path);
     }
 }
 //# sourceMappingURL=client.js.map

package/dist/lib/client.js.map

@@ -1 +1 @@
-{"version":3,"file":"client.js","sourceRoot":"","sources":["../../src/lib/client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAC/B,OAAO,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AACvC,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AAE9C,MAAM,CAAC,MAAM,UAAU,GAAG,cAAc,cAAc,EAAE,CAAC;AAEzD;;;GAGG;AACH,MAAM,CAAC,MAAM,aAAa,GAAG,GAAG,CAAC,SAAS,KAAK,MAAM,CAAC,CAAC,CAAC,mBAAmB,CAAC,CAAC,CAAC,YAAY,CAAC;AAE3F,MAAM,OAAO,YAAY;IACf,OAAO,CAAS;IAChB,MAAM,CAAS;IAEvB,YAAY,MAAc;QACxB,IAAI,CAAC,OAAO,GAAG,GAAG,CAAC,mBAAmB,CAAC;QACvC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACvB,CAAC;IAEO,KAAK,CAAC,OAAO,CAAI,MAAc,EAAE,IAAY,EAAE,IAAc;QACnE,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,CAAC,OAAO,GAAG,IAAI,EAAE,EAAE;YACrD,MAAM;YACN,OAAO,EAAE;gBACP,cAAc,EAAE,kBAAkB;gBAClC,eAAe,EAAE,UAAU,IAAI,CAAC,MAAM,EAAE;gBACxC,YAAY,EAAE,UAAU;gBACxB,iBAAiB,EAAE,aAAa;aACjC;YACD,GAAG,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAC9D,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;YACjB,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YACnC,IAAI,OAAe,CAAC;YACpB,IAAI,UAA8B,CAAC;YACnC,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;gBAC9B,OAAO,GAAG,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC;gBAC7C,IAAI,OAAO,IAAI,CAAC,WAAW,KAAK,QAAQ;oBAAE,UAAU,GAAG,IAAI,CAAC,WAAW,CAAC;YAC1E,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,GAAG,IAAI,CAAC;YACjB,CAAC;YACD,MAAM,IAAI,QAAQ,CAAC,QAAQ,CAAC,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;QACtE,CAAC;QAED,OAAO,QAAQ,CAAC,IAAI,EAAgB,CAAC;IACvC,CAAC;IAED,KAAK,CAAC,GAAG,CAAI,IAAY;QACvB,OAAO,IAAI,CAAC,OAAO,CAAI,KAAK,EAAE,IAAI,CAAC,CAAC;IACtC,CAAC;IAED,KAAK,CAAC,IAAI,CAAI,IAAY,EAAE,IAAc;QACxC,OAAO,IAAI,CAAC,OAAO,CAAI,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;IAC7C,CAAC;CACF"}
+{"version":3,"file":"client.js","sourceRoot":"","sources":["../../src/lib/client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAC/B,OAAO,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AACvC,OAAO,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AAE9C,MAAM,CAAC,MAAM,UAAU,GAAG,cAAc,cAAc,EAAE,CAAC;AAEzD;;;GAGG;AACH,MAAM,CAAC,MAAM,aAAa,GAAG,GAAG,CAAC,SAAS,KAAK,MAAM,CAAC,CAAC,CAAC,mBAAmB,CAAC,CAAC,CAAC,YAAY,CAAC;AAE3F,MAAM,OAAO,YAAY;IACf,OAAO,CAAS;IAChB,MAAM,CAAS;IAEvB,YAAY,MAAc;QACxB,IAAI,CAAC,OAAO,GAAG,GAAG,CAAC,mBAAmB,CAAC;QACvC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACvB,CAAC;IAEO,KAAK,CAAC,OAAO,CAAI,MAAc,EAAE,IAAY,EAAE,IAAc,EAAE,YAAqC;QAC1G,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,CAAC,OAAO,GAAG,IAAI,EAAE,EAAE;YACrD,MAAM;YACN,OAAO,EAAE;gBACP,cAAc,EAAE,kBAAkB;gBAClC,eAAe,EAAE,UAAU,IAAI,CAAC,MAAM,EAAE;gBACxC,YAAY,EAAE,UAAU;gBACxB,iBAAiB,EAAE,aAAa;gBAChC,GAAG,YAAY;aAChB;YACD,GAAG,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAC9D,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;YACjB,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YACnC,IAAI,OAAe,CAAC;YACpB,IAAI,UAA8B,CAAC;YACnC,IAAI,IAAyC,CAAC;YAC9C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;gBAC9B,OAAO,GAAG,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC;gBAC7C,IAAI,OAAO,IAAI,CAAC,WAAW,KAAK,QAAQ;oBAAE,UAAU,GAAG,IAAI,CAAC,WAAW,CAAC;gBACxE,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;oBAAE,IAAI,GAAG,IAAI,CAAC;YAC5E,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,GAAG,IAAI,CAAC;YACjB,CAAC;YACD,MAAM,IAAI,QAAQ,CAAC,QAAQ,CAAC,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,IAAI,CAAC,CAAC;QAC5E,CAAC;QAED,OAAO,QAAQ,CAAC,IAAI,EAAgB,CAAC;IACvC,CAAC;IAED,KAAK,CAAC,GAAG,CAAI,IAAY;QACvB,OAAO,IAAI,CAAC,OAAO,CAAI,KAAK,EAAE,IAAI,CAAC,CAAC;IACtC,CAAC;IAED,KAAK,CAAC,IAAI,CAAI,IAAY,EAAE,IAAc,EAAE,OAAgC;QAC1E,OAAO,IAAI,CAAC,OAAO,CAAI,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,OAAO,CAAC,CAAC;IACtD,CAAC;IAED,KAAK,CAAC,MAAM,CAAI,IAAY;QAC1B,OAAO,IAAI,CAAC,OAAO,CAAI,QAAQ,EAAE,IAAI,CAAC,CAAC;IACzC,CAAC;CACF"}

package/dist/lib/errors.d.ts

@@ -3,7 +3,11 @@ export declare class ApiError extends Error {
     readonly code?: string | undefined;
     /** Where to top up, when the API sent one (402 responses do). */
     readonly billingUrl?: string | undefined;
+    /** The parsed JSON error body, when there was one. */
+    readonly body?: Record<string, unknown> | undefined;
     constructor(status: number, message: string, code?: string | undefined, 
     /** Where to top up, when the API sent one (402 responses do). */
-    billingUrl?: string | undefined);
+    billingUrl?: string | undefined, 
+    /** The parsed JSON error body, when there was one. */
+    body?: Record<string, unknown> | undefined);
 }

package/dist/lib/errors.js

@@ -2,13 +2,17 @@ export class ApiError extends Error {
     status;
     code;
     billingUrl;
+    body;
     constructor(status, message, code, 
     /** Where to top up, when the API sent one (402 responses do). */
-    billingUrl) {
+    billingUrl, 
+    /** The parsed JSON error body, when there was one. */
+    body) {
         super(message);
         this.status = status;
         this.code = code;
         this.billingUrl = billingUrl;
+        this.body = body;
         this.name = "ApiError";
     }
 }

package/dist/lib/errors.js.map

@@ -1 +1 @@
-{"version":3,"file":"errors.js","sourceRoot":"","sources":["../../src/lib/errors.ts"],"names":[],"mappings":"AAAA,MAAM,OAAO,QAAS,SAAQ,KAAK;IAEf;IAEA;IAEA;IALlB,YACkB,MAAc,EAC9B,OAAe,EACC,IAAa;IAC7B,iEAAiE;IACjD,UAAmB;QAEnC,KAAK,CAAC,OAAO,CAAC,CAAC;QANC,WAAM,GAAN,MAAM,CAAQ;QAEd,SAAI,GAAJ,IAAI,CAAS;QAEb,eAAU,GAAV,UAAU,CAAS;QAGnC,IAAI,CAAC,IAAI,GAAG,UAAU,CAAC;IACzB,CAAC;CACF"}
+{"version":3,"file":"errors.js","sourceRoot":"","sources":["../../src/lib/errors.ts"],"names":[],"mappings":"AAAA,MAAM,OAAO,QAAS,SAAQ,KAAK;IAEf;IAEA;IAEA;IAEA;IAPlB,YACkB,MAAc,EAC9B,OAAe,EACC,IAAa;IAC7B,iEAAiE;IACjD,UAAmB;IACnC,sDAAsD;IACtC,IAA8B;QAE9C,KAAK,CAAC,OAAO,CAAC,CAAC;QARC,WAAM,GAAN,MAAM,CAAQ;QAEd,SAAI,GAAJ,IAAI,CAAS;QAEb,eAAU,GAAV,UAAU,CAAS;QAEnB,SAAI,GAAJ,IAAI,CAA0B;QAG9C,IAAI,CAAC,IAAI,GAAG,UAAU,CAAC;IACzB,CAAC;CACF"}

package/dist/lib/tool-error.js

@@ -17,8 +17,16 @@ import { ApiError } from "./errors.js";
  */
 export function handleApiError(err) {
     if (err instanceof ApiError) {
-        if (err.status === 429)
-            return { isError: true, content: [{ type: "text", text: "Rate limited. Wait and retry." }] };
+        // The API's 429 message says which limit was hit (per-plan concurrency, or
+        // the active crawl/batch job cap, which names the running jobs to wait on).
+        if (err.status === 429) {
+            const b = err.body ?? {};
+            const parts = [err.message, typeof b.message === "string" ? b.message : ""].filter((p) => p && p !== "Too Many Requests");
+            if (Array.isArray(b.running_jobs) && b.running_jobs.length)
+                parts.push(`Running jobs: ${b.running_jobs.join(", ")}`);
+            const detail = parts.map((p) => ` ${/[.!?]$/.test(p) ? p : `${p}.`}`).join("");
+            return { isError: true, content: [{ type: "text", text: `Rate limited (429).${detail} Wait and retry.` }] };
+        }
         // The API's 402 body carries billing_url; the bare "Insufficient credits"
         // string used to be all the model saw, so it could not tell the user where
         // to go. Surface the link, and say that nothing needs reconfiguring.

package/dist/lib/tool-error.js.map

@@ -1 +1 @@
-{"version":3,"file":"tool-error.js","sourceRoot":"","sources":["../../src/lib/tool-error.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,oCAAoC,CAAC;AACzE,OAAO,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AAEvC;;;;;;;;;;;;;;GAcG;AACH,MAAM,UAAU,cAAc,CAAC,GAAY;IACzC,IAAI,GAAG,YAAY,QAAQ,EAAE,CAAC;QAC5B,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,+BAA+B,EAAE,CAAC,EAAE,CAAC;QACrH,0EAA0E;QAC1E,2EAA2E;QAC3E,qEAAqE;QACrE,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,GAAG,CAAC,UAAU,IAAI,sCAAsC,CAAC;YACrE,OAAO;gBACL,OAAO,EAAE,IAAI;gBACb,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,qDAAqD,GAAG,mEAAmE,EAAE,CAAC;aAC/J,CAAC;QACJ,CAAC;QACD,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG;YAAE,MAAM,IAAI,QAAQ,CAAC,SAAS,CAAC,aAAa,EAAE,mDAAmD,CAAC,CAAC;QACzH,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,qBAAqB,GAAG,CAAC,MAAM,MAAM,GAAG,CAAC,OAAO,EAAE,EAAE,CAAC,EAAE,CAAC;IAClH,CAAC;IACD,MAAM,IAAI,QAAQ,CAAC,SAAS,CAAC,aAAa,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;AAC3D,CAAC"}
+{"version":3,"file":"tool-error.js","sourceRoot":"","sources":["../../src/lib/tool-error.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,oCAAoC,CAAC;AACzE,OAAO,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AAEvC;;;;;;;;;;;;;;GAcG;AACH,MAAM,UAAU,cAAc,CAAC,GAAY;IACzC,IAAI,GAAG,YAAY,QAAQ,EAAE,CAAC;QAC5B,2EAA2E;QAC3E,4EAA4E;QAC5E,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YACvB,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,IAAI,EAAE,CAAC;YACzB,MAAM,KAAK,GAAG,CAAC,GAAG,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,CAAC,KAAK,mBAAmB,CAAC,CAAC;YAC1H,IAAI,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,YAAY,CAAC,MAAM;gBAAE,KAAK,CAAC,IAAI,CAAC,iBAAiB,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;YACrH,MAAM,MAAM,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;YAC/E,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,sBAAsB,MAAM,kBAAkB,EAAE,CAAC,EAAE,CAAC;QAC9G,CAAC;QACD,0EAA0E;QAC1E,2EAA2E;QAC3E,qEAAqE;QACrE,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,GAAG,CAAC,UAAU,IAAI,sCAAsC,CAAC;YACrE,OAAO;gBACL,OAAO,EAAE,IAAI;gBACb,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,qDAAqD,GAAG,mEAAmE,EAAE,CAAC;aAC/J,CAAC;QACJ,CAAC;QACD,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG;YAAE,MAAM,IAAI,QAAQ,CAAC,SAAS,CAAC,aAAa,EAAE,mDAAmD,CAAC,CAAC;QACzH,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,qBAAqB,GAAG,CAAC,MAAM,MAAM,GAAG,CAAC,OAAO,EAAE,EAAE,CAAC,EAAE,CAAC;IAClH,CAAC;IACD,MAAM,IAAI,QAAQ,CAAC,SAAS,CAAC,aAAa,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;AAC3D,CAAC"}

package/dist/lib/web-jobs.d.ts

@@ -0,0 +1,35 @@
+/**
+ * MCP-side shaping for website map, crawl and batch (the Extract family's
+ * multi-page operations). The REST API is untouched; this only decides how much
+ * of a response lands in a model's context and what the model is told to do
+ * next.
+ *
+ * - map: a site can list 5,000 URLs. When the caller sets no `limit` the tool
+ *   asks for MCP_MAP_DEFAULT_LIMIT, and a truncated list carries a hint.
+ * - crawl/batch status: a result page can hold up to 100 pages of up to 2 MB
+ *   each. The tool asks for MCP_RESULTS_DEFAULT_LIMIT results per call and
+ *   clips each page's content to `max_content_chars`, flagging the clip, so a
+ *   status call stays inside a context window. `next_cursor` always survives.
+ *
+ * Deterministic and offline. Exported for scripts/trim-check.mjs.
+ */
+import { type JsonValue } from "./trim-response.js";
+export declare const MCP_MAP_DEFAULT_LIMIT = 200;
+export declare const MCP_RESULTS_DEFAULT_LIMIT = 10;
+export declare const MCP_DEFAULT_CONTENT_CHARS = 4000;
+/** MCP clients commonly cancel a tool call at 60 s; stay well inside it. */
+export declare const MAX_WAIT_SECONDS = 50;
+export declare const CRAWL_DEFAULT_WAIT_SECONDS = 30;
+export declare const POLL_INTERVAL_MS = 3000;
+export declare const TERMINAL_STATUSES: Set<string>;
+/** Shape a map response: add a hint when the link list was cut. */
+export declare function shapeMap(raw: unknown): JsonValue;
+export interface JobShapeOptions {
+    type: "crawl" | "batch";
+    maxContentChars?: number;
+}
+/**
+ * Shape a crawl/batch status response: clip page content, then drop nulls, then
+ * tell the model what to call next (more results, or wait for a running job).
+ */
+export declare function shapeJobStatus(raw: unknown, opts: JobShapeOptions): JsonValue;

package/dist/lib/web-jobs.js

@@ -0,0 +1,75 @@
+/**
+ * MCP-side shaping for website map, crawl and batch (the Extract family's
+ * multi-page operations). The REST API is untouched; this only decides how much
+ * of a response lands in a model's context and what the model is told to do
+ * next.
+ *
+ * - map: a site can list 5,000 URLs. When the caller sets no `limit` the tool
+ *   asks for MCP_MAP_DEFAULT_LIMIT, and a truncated list carries a hint.
+ * - crawl/batch status: a result page can hold up to 100 pages of up to 2 MB
+ *   each. The tool asks for MCP_RESULTS_DEFAULT_LIMIT results per call and
+ *   clips each page's content to `max_content_chars`, flagging the clip, so a
+ *   status call stays inside a context window. `next_cursor` always survives.
+ *
+ * Deterministic and offline. Exported for scripts/trim-check.mjs.
+ */
+import { shapeResponse } from "./trim-response.js";
+export const MCP_MAP_DEFAULT_LIMIT = 200;
+export const MCP_RESULTS_DEFAULT_LIMIT = 10;
+export const MCP_DEFAULT_CONTENT_CHARS = 4000;
+/** MCP clients commonly cancel a tool call at 60 s; stay well inside it. */
+export const MAX_WAIT_SECONDS = 50;
+export const CRAWL_DEFAULT_WAIT_SECONDS = 30;
+export const POLL_INTERVAL_MS = 3000;
+export const TERMINAL_STATUSES = new Set(["completed", "failed", "cancelled"]);
+function isObject(v) {
+    return typeof v === "object" && v !== null && !Array.isArray(v);
+}
+/** Shape a map response: add a hint when the link list was cut. */
+export function shapeMap(raw) {
+    const out = shapeResponse(raw);
+    if (isObject(out) && isObject(out.data) && out.data.truncated === true) {
+        const d = out.data;
+        return {
+            ...out,
+            _hint: `${d.returned} of ${d.total_found} URLs returned. Raise limit (up to 5,000, or your plan's cap) or narrow with search / include_paths / exclude_paths.`,
+        };
+    }
+    return out;
+}
+/**
+ * Shape a crawl/batch status response: clip page content, then drop nulls, then
+ * tell the model what to call next (more results, or wait for a running job).
+ */
+export function shapeJobStatus(raw, opts) {
+    const max = opts.maxContentChars ?? MCP_DEFAULT_CONTENT_CHARS;
+    const copy = structuredClone(raw ?? {});
+    let clipped = 0;
+    if (isObject(copy) && isObject(copy.data) && Array.isArray(copy.data.results)) {
+        for (const r of copy.data.results) {
+            if (isObject(r) && typeof r.content === "string" && r.content.length > max) {
+                r.content = r.content.slice(0, max);
+                r.content_clipped = true;
+                clipped++;
+            }
+        }
+    }
+    const out = shapeResponse(copy);
+    if (!isObject(out) || !isObject(out.data))
+        return out;
+    const d = out.data;
+    const tool = `get_${opts.type}_status`;
+    const hints = [];
+    const status = typeof d.status === "string" ? d.status : "";
+    if (status && !TERMINAL_STATUSES.has(status)) {
+        hints.push(`The ${opts.type} is ${status}: call ${tool} with id="${d.id}" and wait_seconds (up to ${MAX_WAIT_SECONDS}) to wait for it.`);
+    }
+    if (typeof d.next_cursor === "string") {
+        hints.push(`More results: call ${tool} with id="${d.id}" and cursor="${d.next_cursor}".`);
+    }
+    if (clipped > 0) {
+        hints.push(`${clipped} page(s) clipped to ${max} characters (content_clipped: true; content_length is the full size). Raise max_content_chars, or filter with status, to read more.`);
+    }
+    return hints.length ? { ...out, _hint: hints.join(" ") } : out;
+}
+//# sourceMappingURL=web-jobs.js.map

package/dist/lib/web-jobs.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"web-jobs.js","sourceRoot":"","sources":["../../src/lib/web-jobs.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AACH,OAAO,EAAE,aAAa,EAAmC,MAAM,oBAAoB,CAAC;AAEpF,MAAM,CAAC,MAAM,qBAAqB,GAAG,GAAG,CAAC;AACzC,MAAM,CAAC,MAAM,yBAAyB,GAAG,EAAE,CAAC;AAC5C,MAAM,CAAC,MAAM,yBAAyB,GAAG,IAAI,CAAC;AAC9C,4EAA4E;AAC5E,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,0BAA0B,GAAG,EAAE,CAAC;AAC7C,MAAM,CAAC,MAAM,gBAAgB,GAAG,IAAI,CAAC;AAErC,MAAM,CAAC,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC,CAAC,WAAW,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC;AAE/E,SAAS,QAAQ,CAAC,CAAU;IAC1B,OAAO,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;AAClE,CAAC;AAED,mEAAmE;AACnE,MAAM,UAAU,QAAQ,CAAC,GAAY;IACnC,MAAM,GAAG,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;IAC/B,IAAI,QAAQ,CAAC,GAAG,CAAC,IAAI,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,GAAG,CAAC,IAAI,CAAC,SAAS,KAAK,IAAI,EAAE,CAAC;QACvE,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,CAAC;QACnB,OAAO;YACL,GAAG,GAAG;YACN,KAAK,EAAE,GAAG,CAAC,CAAC,QAAQ,OAAO,CAAC,CAAC,WAAW,sHAAsH;SAC/J,CAAC;IACJ,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAOD;;;GAGG;AACH,MAAM,UAAU,cAAc,CAAC,GAAY,EAAE,IAAqB;IAChE,MAAM,GAAG,GAAG,IAAI,CAAC,eAAe,IAAI,yBAAyB,CAAC;IAC9D,MAAM,IAAI,GAAG,eAAe,CAAC,GAAG,IAAI,EAAE,CAAc,CAAC;IACrD,IAAI,OAAO,GAAG,CAAC,CAAC;IAChB,IAAI,QAAQ,CAAC,IAAI,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;QAC9E,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClC,IAAI,QAAQ,CAAC,CAAC,CAAC,IAAI,OAAO,CAAC,CAAC,OAAO,KAAK,QAAQ,IAAI,CAAC,CAAC,OAAO,CAAC,MAAM,GAAG,GAAG,EAAE,CAAC;gBAC3E,CAAC,CAAC,OAAO,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;gBACpC,CAAC,CAAC,eAAe,GAAG,IAAI,CAAC;gBACzB,OAAO,EAAE,CAAC;YACZ,CAAC;QACH,CAAC;IACH,CAAC;IACD,MAAM,GAAG,GAAG,aAAa,CAAC,IAAI,CAAC,CAAC;IAChC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;QAAE,OAAO,GAAG,CAAC;IAEtD,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,CAAC;IACnB,MAAM,IAAI,GAAG,OAAO,IAAI,CAAC,IAAI,SAAS,CAAC;IACvC,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,MAAM,MAAM,GAAG,OAAO,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC;IAC5D,IAAI,MAAM,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;QAC7C,KAAK,CAAC,IAAI,CAAC,OAAO,IAAI,CAAC,IAAI,OAAO,MAAM,UAAU,IAAI,aAAa,CAAC,CAAC,EAAE,6BAA6B,gBAAgB,mBAAmB,CAAC,CAAC;IAC3I,CAAC;IACD,IAAI,OAAO,CAAC,CAAC,WAAW,KAAK,QAAQ,EAAE,CAAC;QACtC,KAAK,CAAC,IAAI,CAAC,sBAAsB,IAAI,aAAa,CAAC,CAAC,EAAE,iBAAiB,CAAC,CAAC,WAAW,IAAI,CAAC,CAAC;IAC5F,CAAC;IACD,IAAI,OAAO,GAAG,CAAC,EAAE,CAAC;QAChB,KAAK,CAAC,IAAI,CAAC,GAAG,OAAO,uBAAuB,GAAG,qIAAqI,CAAC,CAAC;IACxL,CAAC;IACD,OAAO,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,GAAG,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;AACjE,CAAC"}

package/dist/tools/extract.js

@@ -1,6 +1,103 @@
 import { z } from "zod";
 import { handleApiError } from "../lib/tool-error.js";
-import { trimResponse } from "../lib/trim-response.js";
+import { shapeResponse, trimResponse } from "../lib/trim-response.js";
+import { CRAWL_DEFAULT_WAIT_SECONDS, MAX_WAIT_SECONDS, MCP_DEFAULT_CONTENT_CHARS, MCP_MAP_DEFAULT_LIMIT, MCP_RESULTS_DEFAULT_LIMIT, POLL_INTERVAL_MS, TERMINAL_STATUSES, shapeJobStatus, shapeMap, } from "../lib/web-jobs.js";
+// Schemas mirror backend/src/routes/extract-web.ts field for field: same names,
+// enums, ranges and optionality, and no .default() -- an omitted field is
+// omitted on the wire and the API's own default applies. The MCP-only knobs
+// (wait_seconds, max_content_chars) are stripped before the request.
+const MAX_GLOBS = 50;
+const MAX_GLOB_LENGTH = 200;
+const PRICING = "Per successful page: normal 1cr, advanced 1cr, ultra 2cr; failed/skipped pages free. Obeys robots.txt.";
+const mode = z.enum(["normal", "advanced", "ultra"]).optional()
+    .describe("PRICE-BEARING. 'normal'=1cr (default), 'advanced'=renders JavaScript 1cr, 'ultra'=hardest pages 2cr.");
+const format = z.enum(["html", "markdown", "text"]).optional()
+    .describe("Page format: 'markdown' (default), 'text', or 'html'.");
+const globs = (what) => z.array(z.string().max(MAX_GLOB_LENGTH)).max(MAX_GLOBS).optional()
+    .describe(`Path globs to ${what} (* one segment, ** any depth), e.g. ["/blog/**"].`);
+const webhookUrl = z.string().max(2048).optional()
+    .describe("Optional https URL that gets a signed POST when the job ends; secret returned once as webhook_secret.");
+const idempotencyKey = z.string().max(255).optional()
+    .describe("Optional; the same key within 24h returns the original job.");
+const jobId = z.string().min(1).max(64).describe("Job id from the start call.");
+const statusQuery = {
+    cursor: z.string().max(20).optional().describe("next_cursor from the previous call."),
+    limit: z.number().int().min(1).max(100).optional()
+        .describe(`Results per call (default ${MCP_RESULTS_DEFAULT_LIMIT} here).`),
+    status: z.enum(["succeeded", "failed", "skipped"]).optional().describe("Only results with this status."),
+    include_content: z.boolean().optional().describe("false = results without content. Default true."),
+    max_content_chars: z.number().int().min(200).max(200000).optional()
+        .describe(`Clip each page's content to this many chars (default ${MCP_DEFAULT_CONTENT_CHARS}).`),
+};
+const waitSeconds = (dflt) => z.number().int().min(0).max(MAX_WAIT_SECONDS).optional()
+    .describe(`Seconds to wait for the job to finish (default ${dflt}).`);
+function statusPath(type, id, q) {
+    const p = new URLSearchParams();
+    if (q.cursor !== undefined)
+        p.set("cursor", q.cursor);
+    p.set("limit", String(q.limit ?? MCP_RESULTS_DEFAULT_LIMIT));
+    if (q.status !== undefined)
+        p.set("status", q.status);
+    if (q.include_content !== undefined)
+        p.set("include_content", String(q.include_content));
+    return `/api/v1/extract/${type}/${encodeURIComponent(id)}?${p.toString()}`;
+}
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+/** Poll (content-free) until the job is terminal or the wait runs out. */
+async function waitFor(client, type, id, seconds) {
+    const deadline = Date.now() + seconds * 1000;
+    for (;;) {
+        const s = await client.get(statusPath(type, id, { limit: 1, include_content: false }));
+        const progress = { status: s.data?.status, pages: s.data?.pages, credits_used: s.data?.credits_used };
+        if (TERMINAL_STATUSES.has(progress.status ?? ""))
+            return { done: true, progress };
+        const left = deadline - Date.now();
+        if (left <= 0)
+            return { done: false, progress };
+        await sleep(Math.min(POLL_INTERVAL_MS, left));
+    }
+}
+function text(value) {
+    return { content: [{ type: "text", text: JSON.stringify(value) }] };
+}
+function strip(params, keys) {
+    const out = {};
+    for (const [k, v] of Object.entries(params))
+        if (!keys.includes(k) && v !== undefined)
+            out[k] = v;
+    return out;
+}
+/**
+ * Start a crawl/batch, then wait up to wait_seconds. Finished in time: the
+ * first page of results. Still running: the job as created (id, status_url,
+ * webhook_secret, which is shown only once) plus a hint to poll.
+ */
+async function startJob(client, type, params, defaultWait) {
+    const { wait_seconds, max_content_chars, idempotency_key } = params;
+    const body = strip(params, ["wait_seconds", "max_content_chars", "idempotency_key"]);
+    const headers = idempotency_key ? { "Idempotency-Key": idempotency_key } : undefined;
+    const created = await client.post(`/api/v1/extract/${type}`, body, headers);
+    const id = created.data.id;
+    const wait = wait_seconds ?? defaultWait;
+    const waited = wait > 0 ? await waitFor(client, type, id, wait) : null;
+    if (waited?.done) {
+        const status = await client.get(statusPath(type, id, {}));
+        const shaped = shapeJobStatus(status, { type, maxContentChars: max_content_chars });
+        return text(created.data.webhook_secret ? { ...shaped, webhook_secret: created.data.webhook_secret } : shaped);
+    }
+    return text({
+        ...shapeResponse(created),
+        ...(waited ? { progress: shapeResponse(waited.progress) } : {}),
+        _hint: `Job ${id} is still running. Call get_${type}_status with id="${id}" (add wait_seconds up to ${MAX_WAIT_SECONDS} to wait) for progress and results.`,
+    });
+}
+async function jobStatus(client, type, params) {
+    const { id, wait_seconds, max_content_chars, ...q } = params;
+    if (wait_seconds && wait_seconds > 0)
+        await waitFor(client, type, id, wait_seconds);
+    const status = await client.get(statusPath(type, id, q));
+    return text(shapeJobStatus(status, { type, maxContentChars: max_content_chars }));
+}
 export function registerExtractTool(server, getClient) {
     server.tool("extract_url", `Read any web page as Markdown, text, or HTML. Tier-priced: normal=1cr, advanced(JS render)=1cr, ultra(hardest pages)=2cr. Start normal, escalate on empty/blocked. Only billed on success.`, {
         url: z.string().min(1).max(2048)
@@ -18,5 +115,99 @@ export function registerExtractTool(server, getClient) {
             return handleApiError(err);
         }
     });
+    server.tool("map_website", `List a site's page URLs from its sitemaps and start page (deduplicated, path-filtered, with lastmod when known). Obeys robots.txt. 1 credit (normal). Use to pick pages for batch_extract.`, {
+        url: z.string().min(1).max(2048).describe("Site to map, e.g. https://example.com."),
+        search: z.string().max(200).optional().describe("Only URLs containing every word of this text (case-insensitive)."),
+        limit: z.number().int().min(1).max(5000).optional()
+            .describe(`Max links (default ${MCP_MAP_DEFAULT_LIMIT} here). Plan caps: Free 500, PAYG 1,000, paid 5,000.`),
+        sitemap: z.enum(["include", "only", "skip"]).optional()
+            .describe("'include' (default) sitemaps + page links; 'only' sitemaps; 'skip' page links."),
+        include_subdomains: z.boolean().optional().describe("Include subdomains. Default false."),
+        include_paths: globs("keep"),
+        exclude_paths: globs("drop"),
+        ignore_query_parameters: z.boolean().optional().describe("Merge URLs differing only by query. Default true."),
+        mode,
+    }, async (params) => {
+        try {
+            const body = { ...strip(params, []), limit: params.limit ?? MCP_MAP_DEFAULT_LIMIT };
+            const data = await getClient().post("/api/v1/extract/map", body);
+            return text(shapeMap(data));
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("crawl_website", `Crawl a site from one URL and extract its same-site pages as Markdown/text/HTML. Starts a job and waits up to wait_seconds: returns results if done, else the job id for get_crawl_status. ${PRICING} Max pages: Free 10, PAYG 100, Project 500, Bootstrap 2k, Startup 5k, Growth 10k. Results kept 3 days.`, {
+        url: z.string().min(1).max(2048).describe("Start URL; pages under its path are crawled first."),
+        max_pages: z.number().int().min(1).max(10000).optional().describe("Max pages (default 100, capped by plan)."),
+        max_depth: z.number().int().min(0).max(10).optional().describe("Link depth to follow (default 3; 0 = start page only)."),
+        include_paths: globs("crawl"),
+        exclude_paths: globs("skip"),
+        allow_subdomains: z.boolean().optional().describe("Include subdomains. Default false."),
+        ignore_query_parameters: z.boolean().optional().describe("Merge URLs differing only by query. Default true."),
+        sitemap: z.enum(["include", "skip", "only"]).optional()
+            .describe("'include' (default) also seeds from the sitemap; 'skip' links only; 'only' sitemap pages, no link following."),
+        format,
+        mode,
+        webhook_url: webhookUrl,
+        idempotency_key: idempotencyKey,
+        wait_seconds: waitSeconds(CRAWL_DEFAULT_WAIT_SECONDS),
+        max_content_chars: statusQuery.max_content_chars,
+    }, async (params) => {
+        try {
+            return await startJob(getClient(), "crawl", params, CRAWL_DEFAULT_WAIT_SECONDS);
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("get_crawl_status", `A crawl's status, page counts, credits used and page results (paged by cursor). Can wait for it to finish. Free.`, { id: jobId, ...statusQuery, wait_seconds: waitSeconds(0) }, async (params) => {
+        try {
+            return await jobStatus(getClient(), "crawl", params);
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("cancel_crawl", `Stop a running crawl; nothing more is billed and results so far stay readable. 409 if already finished. Free.`, { id: jobId }, async ({ id }) => {
+        try {
+            return trimResponse(await getClient().delete(`/api/v1/extract/crawl/${encodeURIComponent(id)}`));
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("batch_extract", `Extract a list of URLs in one job, each as Markdown/text/HTML, results in input order; invalid URLs are reported, never billed. Waits up to wait_seconds: returns results if done, else the job id for get_batch_status. ${PRICING} Max URLs: Free 10, PAYG 50, Project 100, Bootstrap 500, Startup/Growth 1k.`, {
+        urls: z.array(z.string().max(2048)).min(1).max(1000).describe("Pages to extract (duplicates removed)."),
+        format,
+        mode,
+        webhook_url: webhookUrl,
+        idempotency_key: idempotencyKey,
+        wait_seconds: waitSeconds(CRAWL_DEFAULT_WAIT_SECONDS),
+        max_content_chars: statusQuery.max_content_chars,
+    }, async (params) => {
+        try {
+            return await startJob(getClient(), "batch", params, CRAWL_DEFAULT_WAIT_SECONDS);
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("get_batch_status", `A batch's status, page counts, credits used and results in input order (paged by cursor). Can wait for it to finish. Free.`, { id: jobId, ...statusQuery, wait_seconds: waitSeconds(0) }, async (params) => {
+        try {
+            return await jobStatus(getClient(), "batch", params);
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
+    server.tool("cancel_batch", `Stop a running batch; nothing more is billed and results so far stay readable. 409 if already finished. Free.`, { id: jobId }, async ({ id }) => {
+        try {
+            return trimResponse(await getClient().delete(`/api/v1/extract/batch/${encodeURIComponent(id)}`));
+        }
+        catch (err) {
+            return handleApiError(err);
+        }
+    });
 }
 //# sourceMappingURL=extract.js.map

package/dist/tools/extract.js.map

@@ -1 +1 @@
-{"version":3,"file":"extract.js","sourceRoot":"","sources":["../../src/tools/extract.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAGxB,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAEvD,MAAM,UAAU,mBAAmB,CAAC,MAAiB,EAAE,SAA6B;IAClF,MAAM,CAAC,IAAI,CACT,aAAa,EACb,4LAA4L,EAC5L;QACE,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC;aAC7B,QAAQ,CAAC,uEAAuE,CAAC;QACpF,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,UAAU,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE;aACpD,QAAQ,CAAC,0CAA0C,CAAC;QACvD,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,QAAQ,EAAE;aACrD,QAAQ,CAAC,6FAA6F,CAAC;KAC3G,EACD,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,SAAS,EAAE,CAAC,IAAI,CAAC,iBAAiB,EAAE,MAAM,CAAC,CAAC;YAC/D,OAAO,YAAY,CAAC,IAAI,CAAC,CAAC;QAC5B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;AACJ,CAAC"}
+{"version":3,"file":"extract.js","sourceRoot":"","sources":["../../src/tools/extract.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAGxB,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAAE,aAAa,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AACtE,OAAO,EACL,0BAA0B,EAC1B,gBAAgB,EAChB,yBAAyB,EACzB,qBAAqB,EACrB,yBAAyB,EACzB,gBAAgB,EAChB,iBAAiB,EACjB,cAAc,EACd,QAAQ,GACT,MAAM,oBAAoB,CAAC;AAE5B,gFAAgF;AAChF,0EAA0E;AAC1E,4EAA4E;AAC5E,qEAAqE;AAErE,MAAM,SAAS,GAAG,EAAE,CAAC;AACrB,MAAM,eAAe,GAAG,GAAG,CAAC;AAE5B,MAAM,OAAO,GAAG,wGAAwG,CAAC;AAEzH,MAAM,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,QAAQ,EAAE;KAC5D,QAAQ,CAAC,sGAAsG,CAAC,CAAC;AACpH,MAAM,MAAM,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,UAAU,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE;KAC3D,QAAQ,CAAC,uDAAuD,CAAC,CAAC;AACrE,MAAM,KAAK,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,QAAQ,EAAE;KAC/F,QAAQ,CAAC,iBAAiB,IAAI,oDAAoD,CAAC,CAAC;AACvF,MAAM,UAAU,GAAG,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE;KAC/C,QAAQ,CAAC,uGAAuG,CAAC,CAAC;AACrH,MAAM,cAAc,GAAG,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE;KAClD,QAAQ,CAAC,6DAA6D,CAAC,CAAC;AAC3E,MAAM,KAAK,GAAG,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,CAAC,6BAA6B,CAAC,CAAC;AAChF,MAAM,WAAW,GAAG;IAClB,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,qCAAqC,CAAC;IACrF,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE;SAC/C,QAAQ,CAAC,6BAA6B,yBAAyB,SAAS,CAAC;IAC5E,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,WAAW,EAAE,QAAQ,EAAE,SAAS,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,gCAAgC,CAAC;IACxG,eAAe,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,gDAAgD,CAAC;IAClG,iBAAiB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,QAAQ,EAAE;SAChE,QAAQ,CAAC,wDAAwD,yBAAyB,IAAI,CAAC;CACnG,CAAC;AACF,MAAM,WAAW,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,gBAAgB,CAAC,CAAC,QAAQ,EAAE;KAC3F,QAAQ,CAAC,kDAAkD,IAAI,IAAI,CAAC,CAAC;AAIxE,SAAS,UAAU,CAAC,IAAuB,EAAE,EAAU,EAAE,CAAQ;IAC/D,MAAM,CAAC,GAAG,IAAI,eAAe,EAAE,CAAC;IAChC,IAAI,CAAC,CAAC,MAAM,KAAK,SAAS;QAAE,CAAC,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC;IACtD,CAAC,CAAC,GAAG,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC,KAAK,IAAI,yBAAyB,CAAC,CAAC,CAAC;IAC7D,IAAI,CAAC,CAAC,MAAM,KAAK,SAAS;QAAE,CAAC,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC;IACtD,IAAI,CAAC,CAAC,eAAe,KAAK,SAAS;QAAE,CAAC,CAAC,GAAG,CAAC,iBAAiB,EAAE,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC;IACzF,OAAO,mBAAmB,IAAI,IAAI,kBAAkB,CAAC,EAAE,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC;AAC7E,CAAC;AAED,MAAM,KAAK,GAAG,CAAC,EAAU,EAAE,EAAE,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC;AAIpE,0EAA0E;AAC1E,KAAK,UAAU,OAAO,CAAC,MAAoB,EAAE,IAAuB,EAAE,EAAU,EAAE,OAAe;IAC/F,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,GAAG,IAAI,CAAC;IAC7C,SAAS,CAAC;QACR,MAAM,CAAC,GAAG,MAAM,MAAM,CAAC,GAAG,CAAsB,UAAU,CAAC,IAAI,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,CAAC,EAAE,eAAe,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QAC5G,MAAM,QAAQ,GAAa,EAAE,MAAM,EAAE,CAAC,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,YAAY,EAAE,CAAC,CAAC,IAAI,EAAE,YAAY,EAAE,CAAC;QAChH,IAAI,iBAAiB,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,IAAI,EAAE,CAAC;YAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC;QAClF,MAAM,IAAI,GAAG,QAAQ,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACnC,IAAI,IAAI,IAAI,CAAC;YAAE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,QAAQ,EAAE,CAAC;QAChD,MAAM,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,gBAAgB,EAAE,IAAI,CAAC,CAAC,CAAC;IAChD,CAAC;AACH,CAAC;AAED,SAAS,IAAI,CAAC,KAAc;IAC1B,OAAO,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,MAAe,EAAE,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;AAC/E,CAAC;AAED,SAAS,KAAK,CAAoC,MAAS,EAAE,IAAc;IACzE,MAAM,GAAG,GAA4B,EAAE,CAAC;IACxC,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC;QAAE,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,SAAS;YAAE,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;IAClG,OAAO,GAAG,CAAC;AACb,CAAC;AAED;;;;GAIG;AACH,KAAK,UAAU,QAAQ,CACrB,MAAoB,EACpB,IAAuB,EACvB,MAA+B,EAC/B,WAAmB;IAEnB,MAAM,EAAE,YAAY,EAAE,iBAAiB,EAAE,eAAe,EAAE,GAAG,MAE5D,CAAC;IACF,MAAM,IAAI,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,cAAc,EAAE,mBAAmB,EAAE,iBAAiB,CAAC,CAAC,CAAC;IACrF,MAAM,OAAO,GAAG,eAAe,CAAC,CAAC,CAAC,EAAE,iBAAiB,EAAE,eAAe,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IACrF,MAAM,OAAO,GAAG,MAAM,MAAM,CAAC,IAAI,CAAoD,mBAAmB,IAAI,EAAE,EAAE,IAAI,EAAE,OAAO,CAAC,CAAC;IAC/H,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;IAC3B,MAAM,IAAI,GAAG,YAAY,IAAI,WAAW,CAAC;IACzC,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,OAAO,CAAC,MAAM,EAAE,IAAI,EAAE,EAAE,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IACvE,IAAI,MAAM,EAAE,IAAI,EAAE,CAAC;QACjB,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,GAAG,CAAC,UAAU,CAAC,IAAI,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC;QAC1D,MAAM,MAAM,GAAG,cAAc,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,iBAAiB,EAAE,CAA4B,CAAC;QAC/G,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,CAAC,EAAE,GAAG,MAAM,EAAE,cAAc,EAAE,OAAO,CAAC,IAAI,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;IACjH,CAAC;IACD,OAAO,IAAI,CAAC;QACV,GAAI,aAAa,CAAC,OAAO,CAA6B;QACtD,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,aAAa,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC/D,KAAK,EAAE,OAAO,EAAE,+BAA+B,IAAI,oBAAoB,EAAE,6BAA6B,gBAAgB,qCAAqC;KAC5J,CAAC,CAAC;AACL,CAAC;AAED,KAAK,UAAU,SAAS,CAAC,MAAoB,EAAE,IAAuB,EAAE,MAA+B;IACrG,MAAM,EAAE,EAAE,EAAE,YAAY,EAAE,iBAAiB,EAAE,GAAG,CAAC,EAAE,GAAG,MAErD,CAAC;IACF,IAAI,YAAY,IAAI,YAAY,GAAG,CAAC;QAAE,MAAM,OAAO,CAAC,MAAM,EAAE,IAAI,EAAE,EAAE,EAAE,YAAY,CAAC,CAAC;IACpF,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,GAAG,CAAC,UAAU,CAAC,IAAI,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC;IACzD,OAAO,IAAI,CAAC,cAAc,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,iBAAiB,EAAE,CAAC,CAAC,CAAC;AACpF,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,MAAiB,EAAE,SAA6B;IAClF,MAAM,CAAC,IAAI,CACT,aAAa,EACb,4LAA4L,EAC5L;QACE,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC;aAC7B,QAAQ,CAAC,uEAAuE,CAAC;QACpF,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,UAAU,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE;aACpD,QAAQ,CAAC,0CAA0C,CAAC;QACvD,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,QAAQ,EAAE;aACrD,QAAQ,CAAC,6FAA6F,CAAC;KAC3G,EACD,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,SAAS,EAAE,CAAC,IAAI,CAAC,iBAAiB,EAAE,MAAM,CAAC,CAAC;YAC/D,OAAO,YAAY,CAAC,IAAI,CAAC,CAAC;QAC5B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,aAAa,EACb,4LAA4L,EAC5L;QACE,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,CAAC,wCAAwC,CAAC;QACnF,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,kEAAkE,CAAC;QACnH,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE;aAChD,QAAQ,CAAC,sBAAsB,qBAAqB,sDAAsD,CAAC;QAC9G,OAAO,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE;aACpD,QAAQ,CAAC,gFAAgF,CAAC;QAC7F,kBAAkB,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,oCAAoC,CAAC;QACzF,aAAa,EAAE,KAAK,CAAC,MAAM,CAAC;QAC5B,aAAa,EAAE,KAAK,CAAC,MAAM,CAAC;QAC5B,uBAAuB,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mDAAmD,CAAC;QAC7G,IAAI;KACL,EACD,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,EAAE,GAAG,KAAK,CAAC,MAAM,EAAE,EAAE,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,qBAAqB,EAAE,CAAC;YACpF,MAAM,IAAI,GAAG,MAAM,SAAS,EAAE,CAAC,IAAI,CAAC,qBAAqB,EAAE,IAAI,CAAC,CAAC;YACjE,OAAO,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;QAC9B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,eAAe,EACf,8LAA8L,OAAO,wGAAwG,EAC7S;QACE,GAAG,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,CAAC,oDAAoD,CAAC;QAC/F,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,0CAA0C,CAAC;QAC7G,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,wDAAwD,CAAC;QACxH,aAAa,EAAE,KAAK,CAAC,OAAO,CAAC;QAC7B,aAAa,EAAE,KAAK,CAAC,MAAM,CAAC;QAC5B,gBAAgB,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,oCAAoC,CAAC;QACvF,uBAAuB,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,mDAAmD,CAAC;QAC7G,OAAO,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,EAAE;aACpD,QAAQ,CAAC,8GAA8G,CAAC;QAC3H,MAAM;QACN,IAAI;QACJ,WAAW,EAAE,UAAU;QACvB,eAAe,EAAE,cAAc;QAC/B,YAAY,EAAE,WAAW,CAAC,0BAA0B,CAAC;QACrD,iBAAiB,EAAE,WAAW,CAAC,iBAAiB;KACjD,EACD,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,MAAM,QAAQ,CAAC,SAAS,EAAE,EAAE,OAAO,EAAE,MAAM,EAAE,0BAA0B,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,kBAAkB,EAClB,kHAAkH,EAClH,EAAE,EAAE,EAAE,KAAK,EAAE,GAAG,WAAW,EAAE,YAAY,EAAE,WAAW,CAAC,CAAC,CAAC,EAAE,EAC3D,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,MAAM,SAAS,CAAC,SAAS,EAAE,EAAE,OAAO,EAAE,MAAM,CAAC,CAAC;QACvD,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,cAAc,EACd,+GAA+G,EAC/G,EAAE,EAAE,EAAE,KAAK,EAAE,EACb,KAAK,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,YAAY,CAAC,MAAM,SAAS,EAAE,CAAC,MAAM,CAAC,yBAAyB,kBAAkB,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC;QACnG,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,eAAe,EACf,4NAA4N,OAAO,6EAA6E,EAChT;QACE,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,CAAC,wCAAwC,CAAC;QACvG,MAAM;QACN,IAAI;QACJ,WAAW,EAAE,UAAU;QACvB,eAAe,EAAE,cAAc;QAC/B,YAAY,EAAE,WAAW,CAAC,0BAA0B,CAAC;QACrD,iBAAiB,EAAE,WAAW,CAAC,iBAAiB;KACjD,EACD,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,MAAM,QAAQ,CAAC,SAAS,EAAE,EAAE,OAAO,EAAE,MAAM,EAAE,0BAA0B,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,kBAAkB,EAClB,4HAA4H,EAC5H,EAAE,EAAE,EAAE,KAAK,EAAE,GAAG,WAAW,EAAE,YAAY,EAAE,WAAW,CAAC,CAAC,CAAC,EAAE,EAC3D,KAAK,EAAE,MAAM,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,MAAM,SAAS,CAAC,SAAS,EAAE,EAAE,OAAO,EAAE,MAAM,CAAC,CAAC;QACvD,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;IAEF,MAAM,CAAC,IAAI,CACT,cAAc,EACd,+GAA+G,EAC/G,EAAE,EAAE,EAAE,KAAK,EAAE,EACb,KAAK,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE;QACf,IAAI,CAAC;YACH,OAAO,YAAY,CAAC,MAAM,SAAS,EAAE,CAAC,MAAM,CAAC,yBAAyB,kBAAkB,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC;QACnG,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC,CACF,CAAC;AACJ,CAAC"}

package/dist/tools/index.d.ts

@@ -4,12 +4,13 @@ export declare const PLATFORM_KEYS: string[];
 /**
  * What registers when SCAVIO_PLATFORMS is unset.
  *
- * Registering everything is not an option any more. The full surface is 248
- * tools, which serialises to 234KB of tools/list (measured with
+ * Registering everything is not an option any more. The full surface is 255
+ * tools, which serialises to 243KB of tools/list (measured with
  * `npm run toolslist`, not estimated) — roughly 58k tokens pushed into the
  * context of EVERY session before the user has typed anything, and past the
- * hard tool-count ceiling some clients enforce. The default set below is 126
- * tools, 96KB (0.20.0: TikTok, YouTube, Reddit and X gained 19 tools).
+ * hard tool-count ceiling some clients enforce. The default set below is 133
+ * tools, 105KB (0.20.0: TikTok, YouTube, Reddit and X gained 19 tools; 0.21.0:
+ * Extract gained map, crawl and batch, 7 tools).
  *
  * The default is therefore exactly what 0.12.x registered, plus extract:
  * upgrading must never silently remove a tool someone already depends on. A
@@ -26,7 +27,7 @@ export declare const DEFAULT_PLATFORMS: string[];
  * Turn a SCAVIO_PLATFORMS value into the canonical platform keys to register.
  *
  * Accepts a comma-separated list of platform keys, plus three keywords:
- *   all      — every platform (248 tools)
+ *   all      — every platform (255 tools)
  *   default  — expands to DEFAULT_PLATFORMS, so `default,sec,g2` is additive
  *   none     — no platform tools at all, leaving only get_usage
  *

package/dist/tools/index.js

@@ -105,12 +105,13 @@ export const PLATFORM_KEYS = Object.keys(PLATFORMS);
 /**
  * What registers when SCAVIO_PLATFORMS is unset.
  *
- * Registering everything is not an option any more. The full surface is 248
- * tools, which serialises to 234KB of tools/list (measured with
+ * Registering everything is not an option any more. The full surface is 255
+ * tools, which serialises to 243KB of tools/list (measured with
  * `npm run toolslist`, not estimated) — roughly 58k tokens pushed into the
  * context of EVERY session before the user has typed anything, and past the
- * hard tool-count ceiling some clients enforce. The default set below is 126
- * tools, 96KB (0.20.0: TikTok, YouTube, Reddit and X gained 19 tools).
+ * hard tool-count ceiling some clients enforce. The default set below is 133
+ * tools, 105KB (0.20.0: TikTok, YouTube, Reddit and X gained 19 tools; 0.21.0:
+ * Extract gained map, crawl and batch, 7 tools).
  *
  * The default is therefore exactly what 0.12.x registered, plus extract:
  * upgrading must never silently remove a tool someone already depends on. A
@@ -162,7 +163,7 @@ function warnOnce(message) {
  * Turn a SCAVIO_PLATFORMS value into the canonical platform keys to register.
  *
  * Accepts a comma-separated list of platform keys, plus three keywords:
- *   all      — every platform (248 tools)
+ *   all      — every platform (255 tools)
  *   default  — expands to DEFAULT_PLATFORMS, so `default,sec,g2` is additive
  *   none     — no platform tools at all, leaving only get_usage
  *
@@ -239,9 +240,20 @@ const READ_ONLY_ANNOTATIONS = {
     destructiveHint: false,
     openWorldHint: true,
 };
+/**
+ * The exceptions to "reads and changes nothing": starting a crawl or batch
+ * creates a server-side job (not idempotent unless idempotency_key is set), and
+ * cancelling one stops it for good. Still no change to any external site.
+ */
+const ANNOTATION_OVERRIDES = {
+    crawl_website: { readOnlyHint: false, destructiveHint: false, idempotentHint: false },
+    batch_extract: { readOnlyHint: false, destructiveHint: false, idempotentHint: false },
+    cancel_crawl: { readOnlyHint: false, destructiveHint: true, idempotentHint: true },
+    cancel_batch: { readOnlyHint: false, destructiveHint: true, idempotentHint: true },
+};
 /**
  * Wrap the server so every tool() call is annotated at registration, instead
- * of editing 248 call sites. A tool added later picks this up automatically.
+ * of editing 255 call sites. A tool added later picks this up automatically.
  */
 function annotating(server) {
     return new Proxy(server, {
@@ -251,7 +263,7 @@ function annotating(server) {
             return (name, ...rest) => {
                 const registered = Reflect.apply(target.tool, target, [name, ...rest]);
                 const title = toolTitle(name);
-                registered.update({ title, annotations: { title, ...READ_ONLY_ANNOTATIONS } });
+                registered.update({ title, annotations: { title, ...READ_ONLY_ANNOTATIONS, ...ANNOTATION_OVERRIDES[name] } });
                 return registered;
             };
         },

package/dist/tools/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/tools/index.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,GAAG,EAAE,MAAM,eAAe,CAAC;AAEpC,OAAO,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,cAAc,EAAE,MAAM,QAAQ,CAAC;AACxC,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,WAAW,CAAC;AAC9C,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,yBAAyB,EAAE,MAAM,mBAAmB,CAAC;AAC9D,OAAO,EAAE,uBAAuB,EAAE,MAAM,kBAAkB,CAAC;AAC3D,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,wBAAwB,EAAE,MAAM,kBAAkB,CAAC;AAC5D,OAAO,EAAE,iBAAiB,EAAE,MAAM,WAAW,CAAC;AAC9C,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAC;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC1C,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAC;AAC1D,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAC5C,OAAO,EAAE,2BAA2B,EAAE,MAAM,qBAAqB,CAAC;AAClE,OAAO,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAI/C;;;;;;;;;;;;GAYG;AACH,MAAM,SAAS,GAA+B;IAC5C,OAAO;IACP,OAAO,EAAE,mBAAmB;IAC5B,SAAS;IACT,MAAM,EAAE,mBAAmB;IAC3B,mBAAmB;IACnB,OAAO,EAAE,oBAAoB;IAC7B,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,MAAM,EAAE,mBAAmB;IAC3B,CAAC,EAAE,cAAc;IACjB,QAAQ,EAAE,qBAAqB;IAC/B,OAAO,EAAE,oBAAoB;IAC7B,QAAQ,EAAE,qBAAqB;IAC/B,0BAA0B;IAC1B,MAAM,EAAE,mBAAmB;IAC3B,OAAO,EAAE,oBAAoB;IAC7B,IAAI,EAAE,iBAAiB;IACvB,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,MAAM,EAAE,mBAAmB;IAC3B,YAAY,EAAE,yBAAyB;IACvC,aAAa,EAAE,uBAAuB;IACtC,mBAAmB;IACnB,OAAO,EAAE,oBAAoB;IAC7B,MAAM,EAAE,mBAAmB;IAC3B,WAAW,EAAE,wBAAwB;IACrC,IAAI,EAAE,iBAAiB;IACvB,cAAc;IACd,MAAM,EAAE,mBAAmB;IAC3B,MAAM,EAAE,mBAAmB;IAC3B,OAAO,EAAE,oBAAoB;IAC7B,SAAS,EAAE,sBAAsB;IACjC,yBAAyB;IACzB,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,aAAa;IACb,QAAQ,EAAE,qBAAqB;IAC/B,UAAU,EAAE,uBAAuB;IACnC,mBAAmB;IACnB,EAAE,EAAE,eAAe;IACnB,QAAQ,EAAE,qBAAqB;IAC/B,mBAAmB;IACnB,UAAU,EAAE,uBAAuB;IACnC,eAAe;IACf,SAAS,EAAE,sBAAsB;IACjC,OAAO,EAAE,oBAAoB;IAC7B,gCAAgC;IAChC,GAAG,EAAE,gBAAgB;IACrB,cAAc,EAAE,2BAA2B;CAC5C,CAAC;AAEF,MAAM,CAAC,MAAM,aAAa,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;AAEpD;;;;;;;;;;;;;;;;;;;GAmBG;AACH,MAAM,CAAC,MAAM,iBAAiB,GAAG;IAC/B,SAAS;IACT,QAAQ;IACR,SAAS;IACT,QAAQ;IACR,SAAS;IACT,QAAQ;IACR,QAAQ;IACR,aAAa;IACb,WAAW;IACX,GAAG;IACH,UAAU;CACX,CAAC;AAEF,kEAAkE;AAClE,MAAM,OAAO,GAA2B;IACtC,OAAO,EAAE,GAAG;IACZ,aAAa,EAAE,SAAS;IACxB,WAAW,EAAE,SAAS;IACtB,qBAAqB,EAAE,WAAW;CACnC,CAAC;AAEF,SAAS,SAAS,CAAC,KAAa;IAC9B,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,YAAY,EAAE,EAAE,CAAC,CAAC;AAC9D,CAAC;AAED,MAAM,uBAAuB,GAAG,IAAI,GAAG,EAAkB,CAAC;AAC1D,KAAK,MAAM,GAAG,IAAI,aAAa;IAAE,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC;AAClF,KAAK,MAAM,CAAC,KAAK,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC;IAAE,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,GAAG,CAAC,CAAC;AAEvG,MAAM,MAAM,GAAG,IAAI,GAAG,EAAU,CAAC;AACjC,SAAS,QAAQ,CAAC,OAAe;IAC/B,IAAI,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC;QAAE,OAAO;IAChC,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACpB,+CAA+C;IAC/C,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;AACzB,CAAC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,UAAU,gBAAgB,CAAC,GAA8B;IAC7D,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE;QAAE,OAAO,CAAC,GAAG,iBAAiB,CAAC,CAAC;IAE1F,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAU,CAAC;IACnC,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,YAAY,GAAG,KAAK,CAAC;IAEzB,KAAK,MAAM,KAAK,IAAI,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;QACnC,MAAM,GAAG,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,GAAG,KAAK,EAAE;YAAE,SAAS;QACzB,IAAI,GAAG,KAAK,KAAK,IAAI,GAAG,KAAK,GAAG;YAAE,OAAO,CAAC,GAAG,aAAa,CAAC,CAAC;QAC5D,IAAI,GAAG,KAAK,MAAM,EAAE,CAAC;YACnB,YAAY,GAAG,IAAI,CAAC;YACpB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtB,KAAK,MAAM,CAAC,IAAI,iBAAiB;gBAAE,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;YACnD,SAAS;QACX,CAAC;QACD,MAAM,SAAS,GAAG,uBAAuB,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACnD,IAAI,SAAS;YAAE,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;;YAClC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IAClC,CAAC;IAED,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvB,QAAQ,CACN,kDAAkD,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,QAAQ,KAAK,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI;YACpH,oBAAoB,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,cAAc,CAC7D,CAAC;IACJ,CAAC;IAED,IAAI,QAAQ,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QACxB,IAAI,YAAY;YAAE,OAAO,EAAE,CAAC;QAC5B,QAAQ,CAAC,6FAA6F,iBAAiB,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACvI,OAAO,CAAC,GAAG,iBAAiB,CAAC,CAAC;IAChC,CAAC;IAED,OAAO,aAAa,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;AAC1D,CAAC;AAED,sEAAsE;AACtE,MAAM,WAAW,GAA2B;IAC1C,MAAM,EAAE,QAAQ;IAChB,OAAO,EAAE,SAAS;IAClB,QAAQ,EAAE,UAAU;IACpB,IAAI,EAAE,MAAM;IACZ,GAAG,EAAE,KAAK;IACV,EAAE,EAAE,IAAI;IACR,EAAE,EAAE,IAAI;IACR,GAAG,EAAE,KAAK;CACX,CAAC;AAEF,0DAA0D;AAC1D,MAAM,UAAU,SAAS,CAAC,IAAY;IACpC,OAAO,IAAI;SACR,KAAK,CAAC,GAAG,CAAC;SACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,WAAW,EAAE,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;SACpE,IAAI,CAAC,GAAG,CAAC,CAAC;AACf,CAAC;AAED;;;;;GAKG;AACH,MAAM,qBAAqB,GAAG;IAC5B,YAAY,EAAE,IAAI;IAClB,eAAe,EAAE,KAAK;IACtB,aAAa,EAAE,IAAI;CACX,CAAC;AAEX;;;GAGG;AACH,SAAS,UAAU,CAAC,MAAiB;IACnC,OAAO,IAAI,KAAK,CAAC,MAAM,EAAE;QACvB,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,QAAQ;YACxB,IAAI,IAAI,KAAK,MAAM;gBAAE,OAAO,OAAO,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,QAAQ,CAAC,CAAC;YAChE,OAAO,CAAC,IAAY,EAAE,GAAG,IAAe,EAAE,EAAE;gBAC1C,MAAM,UAAU,GAAkC,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,CAAC,CAAC;gBACtG,MAAM,KAAK,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;gBAC9B,UAAU,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE,WAAW,EAAE,EAAE,KAAK,EAAE,GAAG,qBAAqB,EAAE,EAAE,CAAC,CAAC;gBAC/E,OAAO,UAAU,CAAC;YACpB,CAAC,CAAC;QACJ,CAAC;KACF,CAAC,CAAC;AACL,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,gBAAgB,CAC9B,MAAiB,EACjB,SAA6B,EAC7B,eAAmC,GAAG,CAAC,gBAAgB;IAEvD,MAAM,SAAS,GAAG,gBAAgB,CAAC,YAAY,CAAC,CAAC;IACjD,MAAM,SAAS,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;IACrC,KAAK,MAAM,GAAG,IAAI,SAAS;QAAE,SAAS,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;IAClE,iBAAiB,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;IACxC,OAAO,SAAS,CAAC;AACnB,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/tools/index.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,GAAG,EAAE,MAAM,eAAe,CAAC;AAEpC,OAAO,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,cAAc,EAAE,MAAM,QAAQ,CAAC;AACxC,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,WAAW,CAAC;AAC9C,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,yBAAyB,EAAE,MAAM,mBAAmB,CAAC;AAC9D,OAAO,EAAE,uBAAuB,EAAE,MAAM,kBAAkB,CAAC;AAC3D,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,wBAAwB,EAAE,MAAM,kBAAkB,CAAC;AAC5D,OAAO,EAAE,iBAAiB,EAAE,MAAM,WAAW,CAAC;AAC9C,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAC;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC1C,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAC;AAC1D,OAAO,EAAE,sBAAsB,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,oBAAoB,EAAE,MAAM,cAAc,CAAC;AACpD,OAAO,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAC5C,OAAO,EAAE,2BAA2B,EAAE,MAAM,qBAAqB,CAAC;AAClE,OAAO,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAI/C;;;;;;;;;;;;GAYG;AACH,MAAM,SAAS,GAA+B;IAC5C,OAAO;IACP,OAAO,EAAE,mBAAmB;IAC5B,SAAS;IACT,MAAM,EAAE,mBAAmB;IAC3B,mBAAmB;IACnB,OAAO,EAAE,oBAAoB;IAC7B,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,MAAM,EAAE,mBAAmB;IAC3B,CAAC,EAAE,cAAc;IACjB,QAAQ,EAAE,qBAAqB;IAC/B,OAAO,EAAE,oBAAoB;IAC7B,QAAQ,EAAE,qBAAqB;IAC/B,0BAA0B;IAC1B,MAAM,EAAE,mBAAmB;IAC3B,OAAO,EAAE,oBAAoB;IAC7B,IAAI,EAAE,iBAAiB;IACvB,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,MAAM,EAAE,mBAAmB;IAC3B,YAAY,EAAE,yBAAyB;IACvC,aAAa,EAAE,uBAAuB;IACtC,mBAAmB;IACnB,OAAO,EAAE,oBAAoB;IAC7B,MAAM,EAAE,mBAAmB;IAC3B,WAAW,EAAE,wBAAwB;IACrC,IAAI,EAAE,iBAAiB;IACvB,cAAc;IACd,MAAM,EAAE,mBAAmB;IAC3B,MAAM,EAAE,mBAAmB;IAC3B,OAAO,EAAE,oBAAoB;IAC7B,SAAS,EAAE,sBAAsB;IACjC,yBAAyB;IACzB,MAAM,EAAE,mBAAmB;IAC3B,SAAS,EAAE,sBAAsB;IACjC,aAAa;IACb,QAAQ,EAAE,qBAAqB;IAC/B,UAAU,EAAE,uBAAuB;IACnC,mBAAmB;IACnB,EAAE,EAAE,eAAe;IACnB,QAAQ,EAAE,qBAAqB;IAC/B,mBAAmB;IACnB,UAAU,EAAE,uBAAuB;IACnC,eAAe;IACf,SAAS,EAAE,sBAAsB;IACjC,OAAO,EAAE,oBAAoB;IAC7B,gCAAgC;IAChC,GAAG,EAAE,gBAAgB;IACrB,cAAc,EAAE,2BAA2B;CAC5C,CAAC;AAEF,MAAM,CAAC,MAAM,aAAa,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;AAEpD;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,MAAM,CAAC,MAAM,iBAAiB,GAAG;IAC/B,SAAS;IACT,QAAQ;IACR,SAAS;IACT,QAAQ;IACR,SAAS;IACT,QAAQ;IACR,QAAQ;IACR,aAAa;IACb,WAAW;IACX,GAAG;IACH,UAAU;CACX,CAAC;AAEF,kEAAkE;AAClE,MAAM,OAAO,GAA2B;IACtC,OAAO,EAAE,GAAG;IACZ,aAAa,EAAE,SAAS;IACxB,WAAW,EAAE,SAAS;IACtB,qBAAqB,EAAE,WAAW;CACnC,CAAC;AAEF,SAAS,SAAS,CAAC,KAAa;IAC9B,OAAO,KAAK,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,YAAY,EAAE,EAAE,CAAC,CAAC;AAC9D,CAAC;AAED,MAAM,uBAAuB,GAAG,IAAI,GAAG,EAAkB,CAAC;AAC1D,KAAK,MAAM,GAAG,IAAI,aAAa;IAAE,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC;AAClF,KAAK,MAAM,CAAC,KAAK,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC;IAAE,uBAAuB,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,GAAG,CAAC,CAAC;AAEvG,MAAM,MAAM,GAAG,IAAI,GAAG,EAAU,CAAC;AACjC,SAAS,QAAQ,CAAC,OAAe;IAC/B,IAAI,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC;QAAE,OAAO;IAChC,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACpB,+CAA+C;IAC/C,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;AACzB,CAAC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,UAAU,gBAAgB,CAAC,GAA8B;IAC7D,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,CAAC,IAAI,EAAE,KAAK,EAAE;QAAE,OAAO,CAAC,GAAG,iBAAiB,CAAC,CAAC;IAE1F,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAU,CAAC;IACnC,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,YAAY,GAAG,KAAK,CAAC;IAEzB,KAAK,MAAM,KAAK,IAAI,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;QACnC,MAAM,GAAG,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC;QAC7B,IAAI,GAAG,KAAK,EAAE;YAAE,SAAS;QACzB,IAAI,GAAG,KAAK,KAAK,IAAI,GAAG,KAAK,GAAG;YAAE,OAAO,CAAC,GAAG,aAAa,CAAC,CAAC;QAC5D,IAAI,GAAG,KAAK,MAAM,EAAE,CAAC;YACnB,YAAY,GAAG,IAAI,CAAC;YACpB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtB,KAAK,MAAM,CAAC,IAAI,iBAAiB;gBAAE,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;YACnD,SAAS;QACX,CAAC;QACD,MAAM,SAAS,GAAG,uBAAuB,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACnD,IAAI,SAAS;YAAE,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;;YAClC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IAClC,CAAC;IAED,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvB,QAAQ,CACN,kDAAkD,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,QAAQ,KAAK,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI;YACpH,oBAAoB,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,cAAc,CAC7D,CAAC;IACJ,CAAC;IAED,IAAI,QAAQ,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QACxB,IAAI,YAAY;YAAE,OAAO,EAAE,CAAC;QAC5B,QAAQ,CAAC,6FAA6F,iBAAiB,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACvI,OAAO,CAAC,GAAG,iBAAiB,CAAC,CAAC;IAChC,CAAC;IAED,OAAO,aAAa,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;AAC1D,CAAC;AAED,sEAAsE;AACtE,MAAM,WAAW,GAA2B;IAC1C,MAAM,EAAE,QAAQ;IAChB,OAAO,EAAE,SAAS;IAClB,QAAQ,EAAE,UAAU;IACpB,IAAI,EAAE,MAAM;IACZ,GAAG,EAAE,KAAK;IACV,EAAE,EAAE,IAAI;IACR,EAAE,EAAE,IAAI;IACR,GAAG,EAAE,KAAK;CACX,CAAC;AAEF,0DAA0D;AAC1D,MAAM,UAAU,SAAS,CAAC,IAAY;IACpC,OAAO,IAAI;SACR,KAAK,CAAC,GAAG,CAAC;SACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,WAAW,EAAE,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;SACpE,IAAI,CAAC,GAAG,CAAC,CAAC;AACf,CAAC;AAED;;;;;GAKG;AACH,MAAM,qBAAqB,GAAG;IAC5B,YAAY,EAAE,IAAI;IAClB,eAAe,EAAE,KAAK;IACtB,aAAa,EAAE,IAAI;CACX,CAAC;AAEX;;;;GAIG;AACH,MAAM,oBAAoB,GAAoG;IAC5H,aAAa,EAAE,EAAE,YAAY,EAAE,KAAK,EAAE,eAAe,EAAE,KAAK,EAAE,cAAc,EAAE,KAAK,EAAE;IACrF,aAAa,EAAE,EAAE,YAAY,EAAE,KAAK,EAAE,eAAe,EAAE,KAAK,EAAE,cAAc,EAAE,KAAK,EAAE;IACrF,YAAY,EAAE,EAAE,YAAY,EAAE,KAAK,EAAE,eAAe,EAAE,IAAI,EAAE,cAAc,EAAE,IAAI,EAAE;IAClF,YAAY,EAAE,EAAE,YAAY,EAAE,KAAK,EAAE,eAAe,EAAE,IAAI,EAAE,cAAc,EAAE,IAAI,EAAE;CACnF,CAAC;AAEF;;;GAGG;AACH,SAAS,UAAU,CAAC,MAAiB;IACnC,OAAO,IAAI,KAAK,CAAC,MAAM,EAAE;QACvB,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,QAAQ;YACxB,IAAI,IAAI,KAAK,MAAM;gBAAE,OAAO,OAAO,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,QAAQ,CAAC,CAAC;YAChE,OAAO,CAAC,IAAY,EAAE,GAAG,IAAe,EAAE,EAAE;gBAC1C,MAAM,UAAU,GAAkC,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,CAAC,IAAI,EAAE,GAAG,IAAI,CAAC,CAAC,CAAC;gBACtG,MAAM,KAAK,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;gBAC9B,UAAU,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE,WAAW,EAAE,EAAE,KAAK,EAAE,GAAG,qBAAqB,EAAE,GAAG,oBAAoB,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC,CAAC;gBAC9G,OAAO,UAAU,CAAC;YACpB,CAAC,CAAC;QACJ,CAAC;KACF,CAAC,CAAC;AACL,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,gBAAgB,CAC9B,MAAiB,EACjB,SAA6B,EAC7B,eAAmC,GAAG,CAAC,gBAAgB;IAEvD,MAAM,SAAS,GAAG,gBAAgB,CAAC,YAAY,CAAC,CAAC;IACjD,MAAM,SAAS,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;IACrC,KAAK,MAAM,GAAG,IAAI,SAAS;QAAE,SAAS,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;IAClE,iBAAiB,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;IACxC,OAAO,SAAS,CAAC;AACnB,CAAC"}

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "@scavio/mcp-server",
-  "version": "0.20.0",
+  "version": "0.21.0",
   "mcpName": "io.github.scavio-ai/scavio-mcp",
-  "description": "MCP server connecting AI agents to Scavio: web search, page extraction, and structured e-commerce, social, travel, jobs, real-estate, app-store, ad-library and company-filing data. 248 tools across 36 platforms plus Extract, one API key.",
+  "description": "MCP server connecting AI agents to Scavio: web search, page extraction, and structured e-commerce, social, travel, jobs, real-estate, app-store, ad-library and company-filing data. 255 tools across 36 platforms plus Extract, site map, crawl and batch extraction, one API key.",
   "type": "module",
   "license": "MIT",
   "author": "Scavio",
@@ -43,6 +43,9 @@
     "sec-filings",
     "web-scraping",
     "url-extraction",
+    "web-crawler",
+    "website-crawler",
+    "sitemap",
     "tavily-alternative",
     "serpapi-alternative"
   ],