webscraping-ai-cli 1.0.2 → 1.2.0

package/.claude-plugin/marketplace.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-marketplace.json",
   "name": "webscraping-ai",
-  "description": "Official WebScraping.AI plugins for Claude Code — scrape and AI-extract data from any URL.",
+  "description": "Official WebScraping.AI plugins for Claude Code — scrape and AI-extract data from any URL, get parsed Google search (SERP) results, and structured data for supported sites.",
   "owner": {
     "name": "WebScraping.AI",
     "url": "https://webscraping.ai"
@@ -10,10 +10,20 @@
     {
       "name": "webscraping-ai",
       "source": "./plugins/webscraping-ai",
-      "description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — via the WebScraping.AI CLI.",
-      "version": "1.0.0",
+      "description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — parsed Google search results, and structured data for supported sites (YouTube, TikTok, X, LinkedIn, Instagram, Reddit, …), via the WebScraping.AI CLI.",
+      "version": "1.2.0",
       "category": "web-scraping",
-      "keywords": ["web-scraping", "html", "text-extraction", "ai-extraction", "scraping", "cli"]
+      "keywords": [
+        "web-scraping",
+        "html",
+        "text-extraction",
+        "ai-extraction",
+        "scraping",
+        "search",
+        "serp",
+        "google-search",
+        "cli"
+      ]
     }
   ]
 }

package/CHANGELOG.md

@@ -5,6 +5,46 @@ All notable changes to `webscraping-ai-cli` are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.2.0 — 2026-09-25
+
+### Added
+
+- `data <url>` subcommand wrapping the new `/data` endpoint: structured JSON for a page on a supported site (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit; more are added server-side). Flags: `--country` (proxy country, `us` by default), `--transcript` (YouTube videos only; if the transcript fetch itself fails, the whole request fails with a 500, exit 7, not charged), `--transcript-language`, and a repeatable `--param key=value` passed through as-is for site-specific parameters. `-` reads one URL per line from stdin (stops at the first failing item, like every command); `-o` and `--pretty`/`--no-pretty` work as on `serp`. The scrape flags (`--js`, `--proxy`, `--headers`, …) don't apply and aren't registered on it. 15 credits per request.
+- The URL is sent exactly as given and never checked against a list of sites. An unsupported URL or page type returns a 400 that is not charged (exit 3); its message lists what is supported. A blank URL exits 2 before any request.
+- `--param` rejects, with exit 2 before any request: a missing `=` or empty key, a repeated key, `api_key`, `url`, `from_cli`, `__proto__`, and keys that have their own flag (`country`, `transcript`, `transcript_language` — the message names the flag).
+- Bundled agent skill documents when to reach for `data` before scraping a supported site yourself.
+- Smoke script adds a YouTube `data` call (checks `parse_status` `ok`, provider `youtube`, a non-empty `data.title`) and an example.com call that must exit 3 with a server 400 whose message contains `Unsupported URL` (~46 credits per sweep).
+
+### Changed
+
+- Requires the `webscraping-ai` SDK release after 4.1.0 (adds `data()` and the `params` pass-through on `serp()`/`data()`).
+- `from_cli` is now sent on `serp` and `data`, through the SDK's `params` pass-through (closes the known gap: SDK 4.1.0's `serp()` dropped it).
+
+### Fixed
+
+- `-o/--output` in stdin batch mode is truncated when the batch starts, so an empty stdin or a failing first item no longer leaves the previous run's content in the file.
+
+## 1.1.0 — 2026-09-25
+
+### Added
+
+- `serp <query>` subcommand (alias `search`) wrapping the new `/serp` endpoint: parsed Google search results as JSON. Flags: `--engine` (`google`), `--gl`, `--hl`, `--page <n>`. Multi-word queries don't need quoting; `-` reads one query per line from stdin. The scrape flags (`--js`, `--proxy`, `--country`, …) don't apply and aren't registered on it. Flat 15 credits per search.
+- Bundled agent skill documents when to reach for `serp` (query-first research) instead of scraping a search URL.
+
+### Changed
+
+- Requires `webscraping-ai` SDK `^4.1.0` (adds `serp()`).
+- `serp --page` is validated strictly (digits only, >= 1): `1.5`, `2abc` and `0` are rejected with exit code 2 before stdin is read or the key is resolved (the server also rejects them with a 400, not billed; checking client-side saves the round trip). Pages are 1–100: the server rejects a `page` above 100 with a 400.
+- `serp` stdin batches skip only blank lines, so `#hashtag` queries are searched rather than dropped as comments. URL commands still treat `#` lines as comments.
+- `serp` sends the query as typed (whitespace-only queries are still rejected). `from_cli` is not sent on `serp`: the SDK forwards only the SERP parameters.
+
+### Fixed
+
+- `--pretty` / `--no-pretty` no longer fail with "option '--no-pretty' cannot be used with option '--no-pretty'" (a self-conflict on every scrape command and `serp`).
+- `-o/--output` in stdin batch mode kept only the last result; the file is now truncated once and each result appended.
+- Error messages printed to stderr redact the API key and any `api_key=...` pattern (an HTML error body echoing the request URL used to print the key).
+- Smoke script (`npm run smoke`) now checks results, not just exit codes: `serp` needs non-empty `organic_results` and a matching `search_parameters.q`, `selected-multiple` needs a non-empty inner array; signal-killed runs fail; page commands run with `--no-js --proxy datacenter` (~31 credits per sweep); FAIL lines redact the key.
+
 ## 1.0.2 — 2026-07-17
 
 ### Changed

package/README.md

@@ -45,29 +45,43 @@ webscraping-ai selected <url> --selector <css>           # One CSS area
 webscraping-ai selected-multiple <url> --selector a --selector b ...
 webscraping-ai ask <url> --question "..."  # AI question about the page
 webscraping-ai extract <url> --fields '{"title":"...","price":"..."}'
+webscraping-ai serp <query> [--gl us] [--hl en] [--page 1]  # Parsed Google results (alias: search)
+webscraping-ai data <url> [--country us] [--transcript]      # Structured JSON for a supported site
 webscraping-ai account                     # Remaining credits / quota
 ```
 
+`serp` (alias `search`) is query-shaped, not URL-shaped: it takes `--engine` (`google`, the default), `--gl` (country), `--hl` (language), `--page` (1-based, 10 results per page) plus `--api-key`/`--output`/`--pretty`, and none of the scrape flags below. It prints the JSON result (`search_parameters`, `search_information`, `organic_results`, `related_searches`, `pagination`). Flat 15 credits per search; failed searches are not charged. `--page` must be an integer >= 1 (pages are 1–100; the server rejects values above 100 with a 400); anything else is a usage error (exit 2) before any request. Multi-word queries don't need quoting; `-` reads one query per line from stdin, skipping only blank lines (a leading `#` is kept as part of the query, e.g. `#coffee`). The CLI's `from_cli` analytics flag is sent on `serp` through the SDK's `params` pass-through.
+
+`data <url>` returns structured JSON for a public page on a supported site — for example a YouTube video, TikTok profile, X post, LinkedIn company, Instagram reel or Reddit thread. Pass the page's normal URL; the site and page kind are detected server-side and echoed in `request_parameters` (`provider`, `type`), next to `parse_status` (`ok`, `parse_failed` or `not_found`) and `data` (shape depends on the site; `null` when nothing parsed). More sites are added on the server over time, so the CLI never checks the URL itself and sends it exactly as given. An unsupported URL or page type returns a 400 that is not charged (exit 3); its message lists what is supported. 15 credits per request, including `parse_failed`/`not_found` results; failed fetches are not charged. None of the scrape flags below apply; `data` has its own:
+
+- `--country <code>`: two-letter country code of the proxy used to fetch the page, `us` by default.
+- `--transcript`: YouTube videos only. Also fetch the video's transcript into `data.transcript`. It's null when no matching captions are available. If the transcript fetch itself fails, the whole request fails with a 500 (exit 7) and is not charged.
+- `--transcript-language <code>`: caption language to pick, e.g. `en` or `de`. Without it, English is preferred, then the first available track. If the video has no captions in that language, `data.transcript` is null.
+- `--param key=value` (repeatable): site-specific parameters added after this CLI version, sent as-is. A repeated key, `api_key`, `url`, `from_cli`, `__proto__`, or a key that has its own flag (`country`, `transcript`, `transcript_language` — use the flag) is a usage error (exit 2).
+- `--api-key`, `--output`, `--pretty`/`--no-pretty`.
+
+`-` reads one URL per line from stdin (blank and `#` lines skipped). Like every URL command, a batch **stops at the first failing item**: earlier results are already written, the remaining URLs are not requested, and the exit code is that error's. With `-o FILE` the file is truncated when the batch starts, then each result is appended. A blank URL or a bad `--param` is a usage error (exit 2) before any request.
+
 Common flags shared by every scrape command:
 
-| Flag                  | Purpose                                                  |
-| --------------------- | -------------------------------------------------------- |
-| `--js` / `--no-js`    | Toggle JavaScript rendering (default: on)                |
-| `--proxy TYPE`        | `datacenter` (default), `residential`, `stealth`         |
-| `--country CC`        | Two-letter proxy country code                            |
-| `--device TYPE`       | `desktop`, `mobile`, `tablet`                            |
-| `--timeout MS`        | Total page-retrieval timeout (max 30000)                 |
-| `--js-timeout MS`     | JS rendering time after page load (max 20000)            |
-| `--wait-for CSS`      | CSS selector to wait for                                 |
-| `--headers JSON`      | HTTP headers to forward; inline JSON or `@file.json`     |
-| `--custom-proxy URL`  | `http://user:pass@host:port`                             |
-| `--js-script CODE`    | Custom JS to evaluate on the target page                 |
-| `--error-on-404`      | Treat 404 responses as errors                            |
-| `--error-on-redirect` | Treat redirects as errors                                |
-| `--output FILE`       | Write to FILE instead of stdout                          |
-| `--pretty / --no-pretty` | Force pretty / single-line JSON                       |
-
-`<url>` may be `-` to read URLs from stdin (one per line; lines starting with `#` are ignored).
+| Flag                     | Purpose                                              |
+| ------------------------ | ---------------------------------------------------- |
+| `--js` / `--no-js`       | Toggle JavaScript rendering (default: on)            |
+| `--proxy TYPE`           | `datacenter` (default), `residential`, `stealth`     |
+| `--country CC`           | Two-letter proxy country code                        |
+| `--device TYPE`          | `desktop`, `mobile`, `tablet`                        |
+| `--timeout MS`           | Total page-retrieval timeout (max 30000)             |
+| `--js-timeout MS`        | JS rendering time after page load (max 20000)        |
+| `--wait-for CSS`         | CSS selector to wait for                             |
+| `--headers JSON`         | HTTP headers to forward; inline JSON or `@file.json` |
+| `--custom-proxy URL`     | `http://user:pass@host:port`                         |
+| `--js-script CODE`       | Custom JS to evaluate on the target page             |
+| `--error-on-404`         | Treat 404 responses as errors                        |
+| `--error-on-redirect`    | Treat redirects as errors                            |
+| `--output FILE`          | Write to FILE instead of stdout                      |
+| `--pretty / --no-pretty` | Force pretty / single-line JSON                      |
+
+`<url>` may be `-` to read URLs from stdin (one per line; lines starting with `#` are ignored). With `--output FILE`, a stdin batch writes every result to FILE, one after another (the file is truncated once at the start).
 
 ## Install the AI agent skill
 
@@ -88,18 +102,18 @@ You can also install the skill via the Claude Code [Plugin Marketplace](https://
 
 ## Exit codes
 
-| Code | Meaning                                          |
-| ---- | ------------------------------------------------ |
-| 0    | Success                                          |
-| 2    | Usage error (commander)                          |
-| 3    | 400 Bad Request from the API                     |
-| 4    | 403 Authentication error                         |
-| 5    | 402 Payment Required (out of credits)            |
-| 6    | 429 Rate limit                                   |
-| 7    | 500 Server error (or wrapped target-page error)  |
-| 8    | Timeout (504 or local AbortController)           |
-| 9    | Connection / DNS failure                         |
-| 1    | Anything else                                    |
+| Code | Meaning                                         |
+| ---- | ----------------------------------------------- |
+| 0    | Success                                         |
+| 2    | Usage error (commander)                         |
+| 3    | 400 Bad Request from the API                    |
+| 4    | 403 Authentication error                        |
+| 5    | 402 Payment Required (out of credits)           |
+| 6    | 429 Rate limit                                  |
+| 7    | 500 Server error (or wrapped target-page error) |
+| 8    | Timeout (504 or local AbortController)          |
+| 9    | Connection / DNS failure                        |
+| 1    | Anything else                                   |
 
 Scripts can branch on these without parsing stderr.
 
@@ -118,6 +132,18 @@ cat urls.txt | webscraping-ai extract - \
   --fields '{"title":"Product title","price":"Current price"}' \
   --output products.ndjson
 
+# Google results for a query, page 2, from Germany in German
+webscraping-ai serp coffee machines --gl de --hl de --page 2
+
+# Just the organic result links
+webscraping-ai search "best espresso grinder" | jq -r '.organic_results[].link'
+
+# Structured data for a YouTube video, with its transcript
+webscraping-ai data 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' --transcript | jq '.data.title'
+
+# Structured data for a list of profile/post URLs, one JSON result per line
+cat social-urls.txt | webscraping-ai data - --no-pretty -o social.ndjson
+
 # Headers from a file
 webscraping-ai html https://example.com --headers @./headers.json
 ```