webscraping_ai 4.2.0 → 4.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +15 -4
- data/README.md +59 -11
- data/lib/webscraping_ai/client.rb +52 -5
- data/lib/webscraping_ai/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bd048e4bbb3a0bec22e8dac6616be20895e1c78761773295f164746985468261
|
|
4
|
+
data.tar.gz: 97972326da495b82684af9a58c95f9d0c3220d1b02559b54b1211381d5e16727
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 26d0348149011fcdb606717b7e4c8c178e9f21f544c0f720842678411c900dbecc673ad3046f0e0847704e818bdd6cee4aa9135e9c1e657bddd4e0cd109a1b82
|
|
7
|
+
data.tar.gz: f02a194c0a72c510f06da7ef20d0aa6019e89caf272494349a4b3a89ad0ed3f9c6a53ef1ed768f55d9edc6b40ceaef735080da5fb7fb5832df18b749b8b3fa84
|
data/CHANGELOG.md
CHANGED
|
@@ -2,11 +2,22 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file. This project follows [Semantic Versioning](https://semver.org/).
|
|
4
4
|
|
|
5
|
+
## 4.3.0 — 2026-10-24
|
|
6
|
+
### Added
|
|
7
|
+
|
|
8
|
+
- `Client#serp` gains `pages:`, `location:`, `uule:`, `lat:`, `lon:` and `radius:`. Every search runs from one fixed location (`location`, `lat` + `lon`, or `uule`, otherwise the capital of the `gl` country), so repeats and later pages are consistent; `gl` defaults to the location's country. `pages` (1–5) fetches consecutive pages as one de-duplicated list, billed per results page returned (see https://webscraping.ai/docs#serp). Raises `ArgumentError` when `pages` or `radius` is not an Integer >= 1, or when `lat`/`lon` is not a finite number. `lat`/`lon` are sent as plain decimals rounded to 7 places (never exponent notation). Ranges and conflicting location options are left to the server's 400 (not charged). New response keys are documented in the README (`search_parameters.location_used`, `organic_results[].page`, `pagination.pages_returned`, ...).
|
|
9
|
+
- `Client#serp_locations(q = nil, limit: nil)` for the new `GET /serp/locations` endpoint — returns an Array of matching locations (`id`, `canonical_name`, `country_code`, `target_type`, `reach`, `gps`, ...), most populous first; pass `canonical_name` or `id` as `serp`'s `location:`. Free, and the endpoint needs no API key. Raises `ArgumentError` when `limit` is not an Integer >= 1.
|
|
10
|
+
- `bin/smoke.rb` checks a two-page SERP from Austin and `serp_locations("austin", limit: 3)`.
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- Docs: stop stating credit prices (they're set server-side and change); link to https://webscraping.ai/docs pricing instead.
|
|
15
|
+
|
|
5
16
|
## 4.2.0 — 2026-09-25
|
|
6
17
|
### Added
|
|
7
18
|
|
|
8
|
-
- `Client#data(url, country: nil, transcript: nil, transcript_language: nil, **params)` for the `GET /data` endpoint — structured JSON for a page on a supported site (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit) as a Hash with `request_parameters`, `parse_status` and `data`.
|
|
9
|
-
- `bin/smoke.rb` checks `/data` on a YouTube video and that `https://example.com/` gets the server's 400 "Unsupported URL" error
|
|
19
|
+
- `Client#data(url, country: nil, transcript: nil, transcript_language: nil, **params)` for the `GET /data` endpoint — structured JSON for a page on a supported site (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit) as a Hash with `request_parameters`, `parse_status` and `data`. Priced per site (see https://webscraping.ai/docs#data). The client does not check the URL's site: new sites are added server-side, and an unsupported URL or page type returns a 400 that is not charged (`BadRequestError`); its message lists what is supported. Raises `ArgumentError` when `url` is blank or not a String. Extra keyword arguments are sent as-is as query params (String, Integer, Float or boolean; Floats are sent as plain decimals). `api_key`, `url`, `country`, `transcript` and `transcript_language` as extra params raise `ArgumentError` (use the named options).
|
|
20
|
+
- `bin/smoke.rb` checks `/data` on a YouTube video and that `https://example.com/` gets the server's 400 "Unsupported URL" error.
|
|
10
21
|
|
|
11
22
|
### Fixed
|
|
12
23
|
|
|
@@ -16,8 +27,8 @@ All notable changes to this project will be documented in this file. This projec
|
|
|
16
27
|
|
|
17
28
|
### Added
|
|
18
29
|
|
|
19
|
-
- `Client#serp(q:, engine: nil, gl: nil, hl: nil, page: nil)` for the new `GET /serp` endpoint — parsed Google search results (organic results, related searches, pagination) as a Hash.
|
|
20
|
-
- `bin/smoke.rb` live smoke script: asserts result shapes (not just the absence of exceptions), runs page tools with `js: false` on datacenter proxies
|
|
30
|
+
- `Client#serp(q:, engine: nil, gl: nil, hl: nil, page: nil)` for the new `GET /serp` endpoint — parsed Google search results (organic results, related searches, pagination) as a Hash. Priced per search (see https://webscraping.ai/docs#serp). Raises `ArgumentError` when `q` is blank or not a String, or when `page` is not an Integer >= 1 (the server also rejects it with a 400, not billed; checking client-side saves the round trip). Pages are 1–100: the server rejects a `page` above 100 with a 400.
|
|
31
|
+
- `bin/smoke.rb` live smoke script: asserts result shapes (not just the absence of exceptions), runs page tools with `js: false` on datacenter proxies to keep the sweep cheap, and redacts the API key from failure output.
|
|
21
32
|
|
|
22
33
|
### Fixed
|
|
23
34
|
|
data/README.md
CHANGED
|
@@ -26,8 +26,8 @@ Requires Ruby 3.1+.
|
|
|
26
26
|
|
|
27
27
|
## Quick start
|
|
28
28
|
|
|
29
|
-
[Sign up](https://webscraping.ai/auth/sign_up) to get an API key —
|
|
30
|
-
trial
|
|
29
|
+
[Sign up](https://webscraping.ai/auth/sign_up) to get an API key — a free
|
|
30
|
+
trial, no credit card required. Your key lives in the
|
|
31
31
|
[dashboard](https://webscraping.ai/dashboard).
|
|
32
32
|
|
|
33
33
|
```ruby
|
|
@@ -64,6 +64,12 @@ data = client.fields(
|
|
|
64
64
|
results = client.serp(q: "coffee machines", gl: "us", hl: "en", page: 1)
|
|
65
65
|
results["organic_results"].first["link"]
|
|
66
66
|
|
|
67
|
+
# Two pages of results searched from Austin, as one de-duplicated list
|
|
68
|
+
results = client.serp(q: "coffee machines", location: "Austin, TX", pages: 2)
|
|
69
|
+
|
|
70
|
+
# Location names for serp's location: (free)
|
|
71
|
+
client.serp_locations("austin").first["canonical_name"] # => "Austin,Texas,United States"
|
|
72
|
+
|
|
67
73
|
# Structured data for a page on a supported site (YouTube, TikTok, X, LinkedIn, Instagram, Reddit, ...)
|
|
68
74
|
video = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
|
|
69
75
|
video["data"]["title"]
|
|
@@ -132,15 +138,28 @@ Returns: `String` for HTML/text responses, `Hash`/`Array` for JSON responses.
|
|
|
132
138
|
### SERP (`#serp`)
|
|
133
139
|
|
|
134
140
|
`#serp` is query-shaped rather than URL-shaped, so none of the page-fetch options above apply.
|
|
135
|
-
It returns the parsed search results as a `Hash`.
|
|
141
|
+
It returns the parsed search results as a `Hash`. Billed per results page returned (see [pricing](https://webscraping.ai/docs#serp)); failed searches are not charged.
|
|
142
|
+
|
|
143
|
+
Every search runs from one fixed location: `location`, `lat` + `lon`, or `uule`, otherwise the capital of
|
|
144
|
+
the `gl` country (for the few countries outside the location list, Google places the search by the exit IP),
|
|
145
|
+
so repeats and later pages are consistent. `gl` defaults to the location's country.
|
|
146
|
+
`pages` (1–5) fetches consecutive pages, starting at `page`, as one de-duplicated list, billed per page returned.
|
|
136
147
|
|
|
137
148
|
| Option | Type | Default | Description |
|
|
138
149
|
| --- | --- | --- | --- |
|
|
139
150
|
| `q` | `String` | — | Search query (required; blank or non-String raises `ArgumentError`) |
|
|
140
151
|
| `engine` | `String` | `"google"` | Search engine; currently only `google` |
|
|
141
|
-
| `gl` | `String` | `"us"` | Two-letter country code for the search |
|
|
152
|
+
| `gl` | `String` | `"us"` | Two-letter country code for the search. Defaults to the country of `location`, `lat`/`lon` or `uule`, otherwise `us` |
|
|
142
153
|
| `hl` | `String` | `"en"` | Two-letter language code for the results |
|
|
143
154
|
| `page` | `Integer` | `1` | Results page number (10 results per page). Must be an `Integer` >= 1, otherwise `ArgumentError`; the server rejects values above 100 with a 400 (not billed) |
|
|
155
|
+
| `pages` | `Integer` | `1` | Number of consecutive results pages to fetch in one call, starting at `page`, from 1 to 5. Duplicates across pages are removed, `position` counts through the combined list and each result gets a `page` key. Must be an `Integer` >= 1, otherwise `ArgumentError` |
|
|
156
|
+
| `location` | `String` | — | Where the search runs from: a name such as `"Austin,Texas,United States"` or `"Austin, TX"`, or an `id` from `#serp_locations`. Large cities also match local names, e.g. `"Москва"`. Sent as-is; an unknown name returns a 400 (not billed). Cannot be combined with `uule`, `lat` or `lon` |
|
|
157
|
+
| `uule` | `String` | — | Google-encoded location (`w+CAIQICI…` or `a+…`). Cannot be combined with `location`, `lat` or `lon` |
|
|
158
|
+
| `lat` | `Float` | — | Latitude of the search origin in decimal degrees (-90 to 90). Requires `lon`. Must be a finite number, otherwise `ArgumentError`; sent as a plain decimal rounded to 7 places |
|
|
159
|
+
| `lon` | `Float` | — | Longitude of the search origin in decimal degrees (-180 to 180). Requires `lat`. Same rules as `lat` |
|
|
160
|
+
| `radius` | `Integer` | — | Radius in meters around `location` or `lat`/`lon`, from 1 to 1000. Must be an `Integer` >= 1, otherwise `ArgumentError` |
|
|
161
|
+
|
|
162
|
+
Ranges and which location options can be combined are checked by the server (a 400, not billed).
|
|
144
163
|
|
|
145
164
|
```ruby
|
|
146
165
|
results = client.serp(q: "coffee machines", gl: "gb", page: 2)
|
|
@@ -149,20 +168,49 @@ results["organic_results"].each do |r|
|
|
|
149
168
|
puts "#{r["position"]}. #{r["title"]} — #{r["link"]}"
|
|
150
169
|
end
|
|
151
170
|
results["pagination"] # => { "current" => 2, "next" => 3 }
|
|
171
|
+
|
|
172
|
+
results = client.serp(q: "coffee machines", location: "Austin, TX", pages: 2)
|
|
173
|
+
results["search_parameters"]["location_used"] # => "Austin,Texas,United States"
|
|
174
|
+
results["organic_results"].last["page"] # => 2 (position counts through both pages)
|
|
175
|
+
results["pagination"]["pages_returned"] # => 2
|
|
152
176
|
```
|
|
153
177
|
|
|
154
|
-
Response keys: `search_parameters` (`engine`, `q`, `gl`, `hl`, `page`
|
|
178
|
+
Response keys: `search_parameters` (`engine`, `q`, `gl`, `hl`, `page`; `location_used`, `lat` and `lon`, usually
|
|
179
|
+
present, which are the `gl` country's capital when no location is given; `location_requested` and `radius` when given;
|
|
180
|
+
`pages` only above 1), `search_information`
|
|
155
181
|
(`query_displayed`, `organic_results_state`, optional `showing_results_for` and `total_results`),
|
|
156
|
-
`organic_results` (`position` — 1-based within the page
|
|
157
|
-
|
|
158
|
-
|
|
182
|
+
`organic_results` (`position` — 1-based within the page, or through the combined list with `pages` above 1 —
|
|
183
|
+
`title`, `link`, `domain`, `displayed_link`, optional `snippet` and `date`, and `page` with `pages` above 1),
|
|
184
|
+
optional `related_searches` (`query`), and `pagination` (`current`, optional `next`, and `pages_returned` with
|
|
185
|
+
`pages` above 1). Optional keys are absent when Google does not show them.
|
|
186
|
+
|
|
187
|
+
### SERP locations (`#serp_locations`)
|
|
188
|
+
|
|
189
|
+
`#serp_locations(q = nil, limit: nil)` looks up names for `#serp`'s `location:` option. It is free and the
|
|
190
|
+
endpoint needs no API key (the client still sends yours). It returns an `Array` of up to 10 location `Hash`es,
|
|
191
|
+
most populous first. `q` is part of an English location name (without it, the most populous locations are
|
|
192
|
+
returned); `limit` must be an `Integer` >= 1 (the server allows up to 10).
|
|
193
|
+
|
|
194
|
+
```ruby
|
|
195
|
+
locations = client.serp_locations("austin", limit: 3)
|
|
196
|
+
locations.first
|
|
197
|
+
# => { "id" => "1026201", "google_id" => 1026201, "google_parent_id" => 21176, "name" => "Austin",
|
|
198
|
+
# "canonical_name" => "Austin,Texas,United States", "country_code" => "US", "target_type" => "City",
|
|
199
|
+
# "reach" => 974447, "gps" => [-97.7431, 30.2672] }
|
|
200
|
+
|
|
201
|
+
client.serp(q: "coffee machines", location: locations.first["canonical_name"]) # or its "id"
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
`gps` is `[longitude, latitude]`, `target_type` is an open set of strings (e.g. `City`, `State`, `Country`),
|
|
205
|
+
and `google_parent_id` is `nil` for countries.
|
|
159
206
|
|
|
160
207
|
### Structured data (`#data`)
|
|
161
208
|
|
|
162
209
|
`#data(url, country: nil, transcript: nil, transcript_language: nil, **params)` returns structured JSON
|
|
163
210
|
for a public page on a supported site as a `Hash`. Pass the page's normal URL; the site (`provider`) and
|
|
164
|
-
page kind (`type`) are detected from it.
|
|
165
|
-
(`parse_status` `"parse_failed"`) or no longer exist (`"not_found"`);
|
|
211
|
+
page kind (`type`) are detected from it. Priced per site (see [pricing](https://webscraping.ai/docs#data)),
|
|
212
|
+
including pages that parse empty (`parse_status` `"parse_failed"`) or no longer exist (`"not_found"`);
|
|
213
|
+
unsupported URLs and failed fetches are not charged.
|
|
166
214
|
None of the page-fetch options above apply.
|
|
167
215
|
|
|
168
216
|
Supported sites today include, for example, YouTube (video/channel/playlist), TikTok (video/profile),
|
|
@@ -233,7 +281,7 @@ bundle exec rubocop
|
|
|
233
281
|
|
|
234
282
|
## Smoke testing
|
|
235
283
|
|
|
236
|
-
`bin/smoke.rb` hits every endpoint once against the live API, loading the gem from `lib/` so it tests the working tree. It is not part of the spec suite and
|
|
284
|
+
`bin/smoke.rb` hits every endpoint once against the live API, loading the gem from `lib/` so it tests the working tree. It is not part of the spec suite and uses real credits on every run (see [pricing](https://webscraping.ai/docs)): the four page calls run with `js: false` and `proxy: "datacenter"` to keep them cheap, there are two AI calls (`question`, `fields`), two SERP calls (one plain, one two-page search from Austin, billed per page returned) and one `/data` call (a YouTube video). `serp_locations` is free. A second `/data` call on `https://example.com/` must come back as the server's free 400, proving there is no client-side site filter. Each case checks the result shape as well as exceptions (e.g. SERP must return organic results for the right query, the Austin SERP must report an Austin `location_used`, `serp_locations` must find Austin, `/data` must parse a title, `selected_multiple` must match something), and failure messages redact the API key.
|
|
237
285
|
|
|
238
286
|
```bash
|
|
239
287
|
WEBSCRAPING_AI_API_KEY=... bundle exec rake smoke
|
|
@@ -69,19 +69,47 @@ module WebScrapingAI
|
|
|
69
69
|
|
|
70
70
|
# GET /serp — returns parsed search engine results for `q` as a Hash
|
|
71
71
|
# (search_parameters, search_information, organic_results, related_searches, pagination).
|
|
72
|
-
# Query-shaped: none of the page-fetch options apply.
|
|
72
|
+
# Query-shaped: none of the page-fetch options apply. Billed per results page returned
|
|
73
|
+
# (see https://webscraping.ai/docs#serp); failed searches are not charged.
|
|
73
74
|
# `q` must be a non-blank String (sent as-is, untrimmed). `page`, when given, must be an
|
|
74
75
|
# Integer >= 1; the server rejects values above 100 with a 400 (not billed).
|
|
75
|
-
|
|
76
|
+
#
|
|
77
|
+
# Every search runs from one fixed location: `location` (a name such as "Austin, TX" or
|
|
78
|
+
# "Austin,Texas,United States", or an id from #serp_locations), `lat` + `lon`, or `uule`, otherwise
|
|
79
|
+
# the capital of the `gl` country, so repeats and later pages are consistent. `gl` defaults to the
|
|
80
|
+
# location's country. `pages` (1-5) fetches consecutive pages starting at `page` as one de-duplicated
|
|
81
|
+
# list: `position` counts through the combined list and each result gets a `page` key.
|
|
82
|
+
#
|
|
83
|
+
# `pages` and `radius` (meters, 1-1000), when given, must be Integers >= 1; `lat` and `lon` must be
|
|
84
|
+
# finite numbers and are sent as plain decimals. Ranges and which location params can be combined are
|
|
85
|
+
# checked by the server (a 400, not billed).
|
|
86
|
+
def serp(q:, engine: nil, gl: nil, hl: nil, page: nil, pages: nil, location: nil, uule: nil, lat: nil,
|
|
87
|
+
lon: nil, radius: nil)
|
|
76
88
|
raise ArgumentError, "q is required" if q.nil? || (q.is_a?(String) && q.strip.empty?)
|
|
77
89
|
raise ArgumentError, "q must be a String" unless q.is_a?(String)
|
|
78
|
-
raise ArgumentError, "page must be an Integer >= 1" unless page.nil? || (page.is_a?(Integer) && page >= 1)
|
|
79
90
|
|
|
80
|
-
|
|
91
|
+
check_positive_integer(:page, page)
|
|
92
|
+
check_positive_integer(:pages, pages)
|
|
93
|
+
check_positive_integer(:radius, radius)
|
|
94
|
+
|
|
95
|
+
get("/serp", q: q, engine: engine, gl: gl, hl: hl, page: page, pages: pages, location: location,
|
|
96
|
+
uule: uule, lat: coordinate(:lat, lat), lon: coordinate(:lon, lon), radius: radius)
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# GET /serp/locations — returns an Array of location Hashes matching `q` (part of an English
|
|
100
|
+
# location name), most populous first: id, google_id, google_parent_id (nil for countries), name,
|
|
101
|
+
# canonical_name, country_code, target_type, reach and gps ([longitude, latitude]). Pass
|
|
102
|
+
# `canonical_name` (or `id`) as #serp's `location:`. Without `q`, the most populous locations are
|
|
103
|
+
# returned. `limit`, when given, must be an Integer >= 1; the server allows up to 10. Free.
|
|
104
|
+
def serp_locations(q = nil, limit: nil)
|
|
105
|
+
check_positive_integer(:limit, limit)
|
|
106
|
+
|
|
107
|
+
get("/serp/locations", q: q, limit: limit)
|
|
81
108
|
end
|
|
82
109
|
|
|
83
110
|
# GET /data — returns structured JSON for a page on a supported site as a Hash
|
|
84
|
-
# (request_parameters: url/provider/type, parse_status, data).
|
|
111
|
+
# (request_parameters: url/provider/type, parse_status, data). Priced per site
|
|
112
|
+
# (see https://webscraping.ai/docs#data); unsupported URLs and failed fetches are not charged.
|
|
85
113
|
#
|
|
86
114
|
# The client never checks which site `url` belongs to: supported sites (e.g. YouTube, TikTok,
|
|
87
115
|
# X/Twitter, LinkedIn, Instagram, Reddit) are added server-side. An unsupported URL or page type
|
|
@@ -150,6 +178,25 @@ module WebScrapingAI
|
|
|
150
178
|
end
|
|
151
179
|
end
|
|
152
180
|
|
|
181
|
+
def check_positive_integer(name, value)
|
|
182
|
+
return if value.nil? || (value.is_a?(Integer) && value >= 1)
|
|
183
|
+
|
|
184
|
+
raise ArgumentError, "#{name} must be an Integer >= 1"
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# The server only accepts plain decimals (`-?\d+(\.\d+)?`) for lat/lon, so never send Ruby's
|
|
188
|
+
# exponent notation (1.0e-07): round to 7 fractional digits and trim trailing zeros.
|
|
189
|
+
def coordinate(name, value)
|
|
190
|
+
return nil if value.nil?
|
|
191
|
+
raise ArgumentError, "#{name} must be a finite number" unless value.is_a?(Numeric) && value.real? && value.finite?
|
|
192
|
+
|
|
193
|
+
text = format("%.7f", value).sub(/0+\z/, "").delete_suffix(".")
|
|
194
|
+
# BigDecimal("1e400") is finite but formats as "Inf".
|
|
195
|
+
raise ArgumentError, "#{name} must be a finite number" unless text.match?(/\A-?\d+(\.\d+)?\z/)
|
|
196
|
+
|
|
197
|
+
text == "-0" ? "0" : text
|
|
198
|
+
end
|
|
199
|
+
|
|
153
200
|
# Float#to_s switches to exponent notation (1.0e+20, 1.5e-07); send plain decimals instead.
|
|
154
201
|
def plain_float(value)
|
|
155
202
|
text = value.to_s
|