composio-scavio 0.4.1 → 0.5.1

package/README.md

@@ -2,7 +2,11 @@
 
 [Scavio](https://scavio.dev) real-time search tools for [Composio](https://composio.dev) (TypeScript).
 
-Scavio is a single Search API over 31 platforms -- Google, YouTube, Amazon, Walmart, eBay, Target, Home Depot, Reddit, TikTok, TikTok Shop, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, the Apple App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads Transparency and the Meta Ad Library -- plus `extract`, which reads any URL as Markdown, plain text or raw HTML. This package exposes **189 tools, one per live endpoint** as a Composio custom toolkit so your agents can pull structured, up-to-date results across any Composio-supported framework.
+Scavio is a single Search API over 31 platforms -- Google, YouTube, Amazon, Walmart, eBay, Target, Home Depot, Reddit, TikTok, TikTok Shop, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, the Apple App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads Transparency and the Meta Ad Library -- plus `extract`, which reads any URL as Markdown, plain text or raw HTML. This package exposes **194 tools** -- one per live endpoint plus five website tools that map, crawl and batch-extract whole sites -- as a Composio custom toolkit so your agents can pull structured, up-to-date results across any Composio-supported framework.
+
+> **New in 0.5.0: website map, crawl and batch.** Five opt-in tools behind
+> `enableCrawl` -- see [Map, crawl and batch](#map-crawl-and-batch). Needs
+> `scavio` >= 0.23.0; the default tool set is unchanged.
 
 > **New in 0.4.0: 21 more platforms, and they are opt-in.** Registering all 189
 > tools at once buries the handful an agent actually wants, so the verticals
@@ -12,7 +16,7 @@ Scavio is a single Search API over 31 platforms -- Google, YouTube, Amazon, Walm
 > everything. Nothing that worked in 0.3.0 changes, except that Walmart grew from
 > 2 tools to 7.
 
-> **Walmart was rebuilt (breaking).** Walmart now runs on a scrape.do rebuild:
+> **Walmart was rebuilt (breaking):**
 > `device`, `delivery_zip` and `store_id` never existed on it and are gone,
 > `start_page` is replaced by `page`, and five endpoints beyond search/product
 > are exposed for the first time (`WALMART_REVIEWS`, `WALMART_CATEGORY`,
@@ -48,7 +52,7 @@ Scavio is a single Search API over 31 platforms -- Google, YouTube, Amazon, Walm
 npm install composio-scavio
 ```
 
-`@composio/core`, `scavio` (>= 0.15.0) and `zod` are dependencies. Get a Scavio API key from the [Scavio Dashboard](https://dashboard.scavio.dev) (50 one-time signup credits, no credit card; the free plan does not refill monthly).
+`@composio/core`, `scavio` (>= 0.23.0) and `zod` are dependencies. Get a Scavio API key from the [Scavio Dashboard](https://dashboard.scavio.dev) (50 one-time signup credits, no credit card; the free plan does not refill monthly).
 
 ## Usage
 
@@ -111,7 +115,7 @@ All tools are grouped under the `SCAVIO` custom toolkit; agent-facing slugs are
 | LinkedIn | `enableLinkedin` | 9 | person, person about, person posts, company, company posts, search jobs, job, post, post comments |
 | Extract (any URL) | `enableExtract` | 1 | extract |
 
-### Opt-in (85 tools)
+### Opt-in (90 tools)
 
 | Platform | Flag | Tools | Endpoints |
 |---|---|---|---|
@@ -136,7 +140,8 @@ All tools are grouped under the `SCAVIO` custom toolkit; agent-facing slugs are
 | Capterra | `enableCapterra` | 3 | search, product, reviews |
 | Google Ads Transparency | `enableGoogleAds` | 3 | advertisers, search, creative |
 | Meta Ad Library | `enableMetaAds` | 3 | search, advertiser, ad |
-| **Total** | | **189** | |
+| Website map, crawl, batch | `enableCrawl` | 5 | map website, crawl website, crawl status, batch extract, batch status |
+| **Total** | | **194** | |
 
 Two endpoints are deliberately not exposed:
 
@@ -147,6 +152,61 @@ Two endpoints are deliberately not exposed:
 
 Six tools exist only to turn a name you have into the id everything else on that platform is keyed by. Call them first, or the rest of the platform is unreachable: `SEC_LOOKUP` (ticker to CIK), `GLASSDOOR_COMPANIES` (name to `employer_id`), `TRIPADVISOR_LOCATIONS` (name to `geo_id` + `location_id`), `GOOGLE_ADS_ADVERTISERS` (brand to `advertiser_id`), `COMPANIES_HOUSE_SEARCH` (name to `company_number`) and `KUAISHOU_USER_RESOLVE` (share link to `user_id`).
 
+## Map, crawl and batch
+
+Five opt-in tools for whole sites, behind `enableCrawl` (off by default, because a
+crawl bills per page):
+
+| Tool | What it does | Credits |
+|---|---|---|
+| `LOCAL_SCAVIO_MAP_WEBSITE` | List a site's page URLs from its sitemaps and start page, deduplicated and filtered by path or search words | 1 (normal mode; advanced up to 2, ultra up to 10) |
+| `LOCAL_SCAVIO_CRAWL_WEBSITE` | Crawl a site from one URL; every same-site page as Markdown, text or HTML | per successful page |
+| `LOCAL_SCAVIO_CRAWL_STATUS` | A crawl's progress and page results, cursor-paged | free |
+| `LOCAL_SCAVIO_BATCH_EXTRACT` | Extract a list of URLs (up to 1,000, plan-limited) in one job | per successful page |
+| `LOCAL_SCAVIO_BATCH_STATUS` | A batch's progress and page results, cursor-paged | free |
+
+Map answers in one call. Crawl and batch run as background jobs: the start tool
+posts the job and then **waits up to `wait_seconds` (default 45, max 300)** for
+it to finish, returning the job's status with its first page of results (10 by
+default, `limit` up to 100) either way. A job still `queued` or `running` when
+the wait ends keeps going server-side: read the rest later with the matching
+status tool, the returned `data.id` and `next_cursor`. That keeps an agent turn
+bounded while a large crawl (up to 6 hours) runs on.
+
+```ts
+const scavio = buildScavioToolkit({ enableCrawl: true });
+const session = await composio.create("user_1", {
+  experimental: { customToolkits: [scavio] },
+});
+
+const links = await session.execute("LOCAL_SCAVIO_MAP_WEBSITE", { url: "https://hono.dev/docs", limit: 50 });
+const job = await session.execute("LOCAL_SCAVIO_CRAWL_WEBSITE", {
+  url: "https://hono.dev/docs",
+  max_pages: 25,
+  include_paths: ["/docs/**"],
+});
+// -> { data: { id, status: "completed" | "running", pages, credits_used, results, next_cursor } }
+// Page on, or pick up a job that outlived the wait (free):
+await session.execute("LOCAL_SCAVIO_CRAWL_STATUS", { id: "<data.id>", cursor: "<data.next_cursor>" });
+```
+
+`pollIntervalMs` (default 3000) sets how often a waiting tool checks its job.
+
+**Pricing.** Crawl and batch bill **per successful page** at Extract's price for
+the mode: `normal` (the standard fetch) and `advanced` (renders JavaScript-built
+pages) 1 credit, `ultra` (the hardest pages) 2. Failed and skipped pages are
+free, and if your balance runs out the job stops cleanly with
+`stopped_reason: "insufficient_credits"`. Reading a job is free. Crawl and map
+obey the site's robots.txt, and results are kept for 3 days.
+
+**Plan limits** (pages per crawl / active jobs / URLs per batch): Free 10/1/10,
+PAYG 100/1/50, Project 500/2/100, Bootstrap 2,000/3/500, Startup 5,000/5/1,000,
+Growth 10,000/10/1,000.
+
+Cancellation, webhooks (signed `Scavio-Signature`) and idempotency keys exist in
+the `scavio` SDK (`extract.cancelCrawl`, `webhook_url`, `idempotencyKey`) and the
+[REST API](https://scavio.dev/docs/extract), but are deliberately not agent tools.
+
 ## Credits
 
 Every tool states its cost in its description. Most endpoints cost 1 credit.
@@ -161,11 +221,11 @@ Every tool states its cost in its description. Most endpoints cost 1 credit.
 | eBay, Target, Zillow, Booking.com, Airbnb, Glassdoor, App Store, SEC EDGAR, Redfin, Companies House, Google Ads, Meta Ads | 1 |
 | Home Depot, Tripadvisor, Indeed, Yelp, Google Play, Capterra | 2 |
 | G2 | 5 |
-| YouTube | 1, except search 2, shorts 2, streams 3, transcript 8 |
+| YouTube | 1, except search 2, shorts 2, streams 3, transcript 5 |
 | Instagram | 10, except post 8, comment replies 8, user posts 2 |
 | LinkedIn | 1 for person, person about, company, post; 10 for person posts, company posts, search jobs, post comments; **30 for job** (the most expensive endpoint in the API) |
 
-**Four surfaces are body-priced** -- their cost is a function of the request, not a constant for the route, so no single number can be quoted:
+**These surfaces are body-priced** -- their cost is a function of the request, not a constant for the route, so no single number can be quoted:
 
 | Surface | Price |
 |---|---|
@@ -173,6 +233,8 @@ Every tool states its cost in its description. Most endpoints cost 1 credit.
 | Threads | 2 credits addressed by `user_id`, 4 by `username` (the handle has to be resolved through people search first). `THREADS_POST`, `_POST_COMMENTS` and `_SEARCH_USERS` have no username form and are always 2. |
 | Kuaishou | Priced **per endpoint**, never per platform: 1 (`USER_POSTS`, `USER_LIVE`, `USER_RESOLVE`, `VIDEO_COMMENTS`, `COMMENT_REPLIES`, `TAG_FEED`, `TRENDING`), 2 (`VIDEO`), 10 (`PROFILE` and the four searches, per page), 40 (`VIDEOS_BATCH`). |
 | Extract | Tier-priced by `mode`: `normal` and `advanced` 1 credit, `ultra` 2. Only a successful extraction is billed -- a dead link, bot wall or timeout costs nothing. |
+| Map website | 1 credit on `normal`; up to 2 on `advanced`, up to 10 on `ultra`, priced by what the call needed. |
+| Crawl website, batch extract | **Per successful page** at Extract's price for the mode (1 / 1 / 2). Failed and skipped pages are free; `CRAWL_STATUS` and `BATCH_STATUS` cost nothing. |
 
 See [scavio.dev/docs](https://scavio.dev/docs).
 

package/dist/index.cjs

@@ -44,6 +44,53 @@ var device = ostr("Device to emulate: 'desktop' or 'mobile'.");
 var currency = ostr("Currency code (ISO 4217, e.g. 'USD').");
 var checkInDate = str("Check-in date (YYYY-MM-DD).");
 var checkOutDate = str("Check-out date (YYYY-MM-DD).");
+var JOB_POLL_INTERVAL_MS = 3e3;
+var JOB_WAIT_DEFAULT_SECONDS = 45;
+var JOB_WAIT_MAX_SECONDS = 300;
+var JOB_RESULTS_DEFAULT_LIMIT = 10;
+var JOB_TERMINAL = /* @__PURE__ */ new Set(["completed", "failed", "cancelled"]);
+var sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+async function waitForJob(status, id, waitSeconds, pollMs, query) {
+  const deadline = Date.now() + Math.max(0, Math.min(waitSeconds, JOB_WAIT_MAX_SECONDS)) * 1e3;
+  for (; ; ) {
+    const remaining = deadline - Date.now();
+    if (remaining <= 0) break;
+    await sleep(Math.min(pollMs, remaining));
+    const probe = await status(id, { limit: 1, include_content: false });
+    if (JOB_TERMINAL.has(String(probe?.data?.status))) break;
+  }
+  return await status(id, query);
+}
+async function startAndWait(start, status, input, pollMs) {
+  const { wait_seconds, limit, include_content, ...body } = input;
+  const created = await start(body);
+  const id = created?.data?.id;
+  const wait = wait_seconds ?? JOB_WAIT_DEFAULT_SECONDS;
+  if (!id || wait <= 0) return created;
+  const query = { limit: limit ?? JOB_RESULTS_DEFAULT_LIMIT };
+  if (include_content !== void 0) query.include_content = include_content;
+  return waitForJob(status, id, wait, pollMs, query);
+}
+var JOB_MODE = ostr(
+  "Fetch tier, and the price-bearing parameter: 'normal' is the standard fetch, 'advanced' renders JavaScript-built pages, 'ultra' reaches the hardest pages. Server default 'normal'."
+);
+var JOB_PATHS = "Path globs (max 20). '*' matches within one path segment, '**' across segments; a pattern without wildcards matches that path and everything under it. Example: ['/blog/**'].";
+var JOB_PRICE = "Billed per SUCCESSFUL page at Extract's price for the mode: 'normal' and 'advanced' 1 credit, 'ultra' 2 credits. Failed and skipped pages are free, and if the balance runs out the job stops cleanly.";
+var JOB_FORMAT = ostr("Format for each page, as on extract: 'html', 'markdown' or 'text' (server default 'markdown').");
+var JOB_WAIT = onum(
+  "Seconds to wait for the job to finish before returning, 0-300 (default 45). 0 returns the job id at once. A job still running when the wait ends keeps going; read it later with its status tool."
+);
+var JOB_LIMIT = onum(
+  "Results per response, 1-100 (default 10, to keep page content inside the context window); page on with next_cursor."
+);
+var JOB_INCLUDE_CONTENT = obool("Set false to list results without their page content (default true).");
+var JOB_STATUS_PARAMS = import_zod.z.object({
+  id: str("Job id returned when the job was started (data.id)."),
+  cursor: ostr("next_cursor from the previous response, to read the next page."),
+  limit: JOB_LIMIT,
+  status: ostr("Only results with this status: 'succeeded', 'failed' or 'skipped'."),
+  include_content: JOB_INCLUDE_CONTENT
+});
 function buildScavioToolkit(options = {}) {
   const {
     apiKey,
@@ -79,6 +126,8 @@ function buildScavioToolkit(options = {}) {
     enableCapterra = false,
     enableGoogleAds = false,
     enableMetaAds = false,
+    enableCrawl = false,
+    pollIntervalMs = JOB_POLL_INTERVAL_MS,
     all = false
   } = options;
   const client = new import_scavio.Scavio({ apiKey: apiKey ?? process.env.SCAVIO_API_KEY });
@@ -532,7 +581,7 @@ function buildScavioToolkit(options = {}) {
       tool(
         "YOUTUBE_TRANSCRIPT",
         "Scavio YouTube Transcript",
-        "Fetch the transcript or timed captions for a YouTube video. Costs 8 credits.",
+        "Fetch the transcript or timed captions for a YouTube video. Costs 5 credits.",
         import_zod.z.object({
           video_id: str("YouTube video id or a full watch URL."),
           language: ostr("Caption language code (default 'en')."),
@@ -2428,15 +2477,101 @@ function buildScavioToolkit(options = {}) {
         import_zod.z.object({
           url: str("Page to read (1-2048 characters). http(s) only; a bare host is upgraded to https, and loopback, private, link-local and metadata hosts are rejected with a 400."),
           format: ostr("Output format: 'html' is the raw page, 'markdown' a readability extraction, 'text' that markdown flattened to plain text (server default 'markdown')."),
-          mode: ostr("Fetch tier, and the price-bearing parameter: 'normal' plain datacenter fetch (1 credit), 'advanced' full browser render (1 credit), 'ultra' the hardest-target tier (2 credits). Server default 'normal'.")
+          mode: ostr("Fetch tier, and the price-bearing parameter: 'normal' is the standard fetch (1 credit), 'advanced' renders JavaScript-built pages (1 credit), 'ultra' reaches the hardest pages (2 credits). Server default 'normal'.")
         }),
         (i) => client.extract(i)
       )
     );
   }
+  if (all || enableCrawl) {
+    const crawlStatus = (id, q) => client.extract.crawlStatus(id, q);
+    const batchStatus = (id, q) => client.extract.batchStatus(id, q);
+    const readStatus = (status) => async ({ id, limit, ...q }) => await status(id, {
+      ...q,
+      limit: limit ?? JOB_RESULTS_DEFAULT_LIMIT
+    });
+    tools.push(
+      tool(
+        "MAP_WEBSITE",
+        "Scavio Map Website",
+        "List a website's page URLs, gathered from its sitemaps and its start page, deduplicated, filtered by path or search words and limited: { url, links: [{ url, source, lastmod }], total_found, returned, truncated }. Pages the site's robots.txt disallows are left out. Use it to find the right pages before extracting or crawling. Costs 1 credit in normal mode; advanced costs up to 2 credits and ultra up to 10, priced by what the call needed.",
+        import_zod.z.object({
+          url: str("The site to map. A bare host is upgraded to https."),
+          search: ostr("Only return URLs containing every word of this text (case-insensitive)."),
+          limit: onum("Maximum links to return, 1-5000. Default 1000 (or the plan's limit if lower: Free 500, PAYG 1,000, paid plans 5,000)."),
+          sitemap: ostr("'include' combines the site's sitemaps with the start page's links (default); 'only' uses sitemaps alone; 'skip' uses page links alone."),
+          include_subdomains: obool("Also return URLs on subdomains of the site (default false)."),
+          include_paths: oarrstr("Keep only URLs matching these. " + JOB_PATHS),
+          exclude_paths: oarrstr("Drop URLs matching these. " + JOB_PATHS),
+          ignore_query_parameters: obool("Treat URLs that differ only by query string as one URL (default true)."),
+          mode: JOB_MODE
+        }),
+        async (i) => await client.extract.map(i)
+      ),
+      tool(
+        "CRAWL_WEBSITE",
+        "Scavio Crawl Website",
+        "Crawl a website from one start URL: same-site pages are discovered and extracted breadth-first within the page, depth and path limits, each returned as Markdown, text or HTML. Pages the site's robots.txt disallows are skipped. Waits up to wait_seconds (default 45) and returns { id, status, pages, credits_used, results, next_cursor }; if status is still queued or running, call LOCAL_SCAVIO_CRAWL_STATUS with the id later. " + JOB_PRICE,
+        import_zod.z.object({
+          url: str("Where the crawl starts. Only pages on the same site are crawled."),
+          max_pages: onum("Maximum pages to fetch. Default 100. Plan limits: Free 10, PAYG 100, Project 500, Bootstrap 2,000, Startup 5,000, Growth 10,000."),
+          max_depth: onum("How many links away from the start page to follow, 0-10 (default 3). 0 fetches only the start page."),
+          include_paths: oarrstr("Crawl only paths matching these. " + JOB_PATHS),
+          exclude_paths: oarrstr("Skip paths matching these. " + JOB_PATHS),
+          allow_subdomains: obool("Also crawl subdomains of the site (default false)."),
+          ignore_query_parameters: obool("Treat URLs that differ only by query string as one page (default true)."),
+          sitemap: ostr("'include' also starts from the site's sitemap (default); 'skip' follows links only; 'only' fetches sitemap pages without following links."),
+          format: JOB_FORMAT,
+          mode: JOB_MODE,
+          wait_seconds: JOB_WAIT,
+          limit: JOB_LIMIT,
+          include_content: JOB_INCLUDE_CONTENT
+        }),
+        (i) => startAndWait(
+          (body) => client.extract.crawl(body),
+          crawlStatus,
+          i,
+          pollIntervalMs
+        )
+      ),
+      tool(
+        "CRAWL_STATUS",
+        "Scavio Crawl Status",
+        "Read a crawl's progress and its page results by job id, paged by cursor: { status, stopped_reason, pages, credits_used, results: [{ url, status, content }], next_cursor }. Results are kept for 3 days after the crawl finishes. Free: reading a job costs no credits.",
+        JOB_STATUS_PARAMS,
+        readStatus(crawlStatus)
+      ),
+      tool(
+        "BATCH_EXTRACT",
+        "Scavio Batch Extract",
+        "Extract a list of URLs (up to 1,000, plan-limited) in one job, each returned as Markdown, text or HTML exactly like LOCAL_SCAVIO_EXTRACT. Waits up to wait_seconds (default 45) and returns { id, status, pages, credits_used, results, next_cursor }; if status is still queued or running, call LOCAL_SCAVIO_BATCH_STATUS with the id later. " + JOB_PRICE,
+        import_zod.z.object({
+          urls: arrstr("The pages to extract (duplicates removed). Plan limits: Free 10, PAYG 50, Project 100, Bootstrap 500, Startup and Growth 1,000."),
+          format: JOB_FORMAT,
+          mode: JOB_MODE,
+          wait_seconds: JOB_WAIT,
+          limit: JOB_LIMIT,
+          include_content: JOB_INCLUDE_CONTENT
+        }),
+        (i) => startAndWait(
+          (body) => client.extract.batch(body),
+          batchStatus,
+          i,
+          pollIntervalMs
+        )
+      ),
+      tool(
+        "BATCH_STATUS",
+        "Scavio Batch Status",
+        "Read a batch extract's progress and its page results by job id, paged by cursor: { status, pages, credits_used, results: [{ url, status, content }], next_cursor }. Results are kept for 3 days after the batch finishes. Free: reading a job costs no credits.",
+        JOB_STATUS_PARAMS,
+        readStatus(batchStatus)
+      )
+    );
+  }
   return (0, import_core.experimental_createToolkit)("SCAVIO", {
     name: "Scavio",
-    description: "Real-time structured search over 31 platforms - Google (SERP, AI Mode, Maps, Shopping, Flights, Hotels, News, Trends), YouTube, Amazon, Walmart, eBay, Target, Home Depot, Reddit, TikTok, TikTok Shop, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, the Apple App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads Transparency and the Meta Ad Library - plus extract, which reads any URL as Markdown, plain text or raw HTML.",
+    description: "Real-time structured search over 31 platforms - Google (SERP, AI Mode, Maps, Shopping, Flights, Hotels, News, Trends), YouTube, Amazon, Walmart, eBay, Target, Home Depot, Reddit, TikTok, TikTok Shop, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, the Apple App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads Transparency and the Meta Ad Library - plus extract, which reads any URL as Markdown, plain text or raw HTML, and website map, crawl and batch extract.",
     tools
   });
 }