scrapeunblocker 0.9.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +15 -0
- data/README.md +42 -32
- data/lib/scrapeunblocker/client.rb +10 -4
- data/lib/scrapeunblocker/plugins.rb +3814 -0
- data/lib/scrapeunblocker/version.rb +1 -1
- data/lib/scrapeunblocker.rb +1 -0
- metadata +5 -3
|
@@ -0,0 +1,3814 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Generated by headquarters/scripts/sdk-plugins/generate.py from the public OpenAPI spec - do not edit by hand.
|
|
4
|
+
|
|
5
|
+
module ScrapeUnblocker
|
|
6
|
+
# Plugin endpoints as methods; Client includes this module. Each method
|
|
7
|
+
# returns the parsed JSON body. Options left nil are not sent.
|
|
8
|
+
module Plugins
|
|
9
|
+
# Call any plugin endpoint by path, e.g.
|
|
10
|
+
# plugin("/marketplace/target-search", query: "tv"). Parameters go in as
|
|
11
|
+
# the API names them (snake_case). Use it for an endpoint this SDK version
|
|
12
|
+
# has no method for yet.
|
|
13
|
+
def plugin(path, **params)
|
|
14
|
+
post_json(path, params)
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
# Scrape Amazon Best Sellers lists.
|
|
18
|
+
#
|
|
19
|
+
# Returns an Amazon ranked list for a category on any regional marketplace: Best Sellers
|
|
20
|
+
# (default), New Releases, Movers & Shakers, Most Wished For or Gift Ideas (`list_type`).
|
|
21
|
+
# The response carries the `category` (`name`, `slug`, `nodeId`, `url`, `parents`,
|
|
22
|
+
# `subcategories` - each with a url you can pass back as `category`) and up to `max_results`
|
|
23
|
+
# ranked `items`: `rank`, `asin`, `title`, `url`, `image`, `price` + `currency`,
|
|
24
|
+
# `offersCount`, `rating`, `reviewsCount`, and on Movers & Shakers `salesRank`,
|
|
25
|
+
# `previousSalesRank` and `salesRankChangePct`. Pass `category` as a department slug
|
|
26
|
+
# (`electronics`), slug plus browse-node id (`electronics/172541`) or a pasted Amazon list
|
|
27
|
+
# url; leave it empty for the all-departments root (department list, no items). A list holds
|
|
28
|
+
# up to 100 items over two pages of 50, and each page fetched is billed as one request. When
|
|
29
|
+
# Amazon has no list for a category it says so, and that text is returned in `message` with
|
|
30
|
+
# an empty `items` array. An item whose card Amazon did not load (it loads ranks 31-50 of
|
|
31
|
+
# each page lazily) comes back with only `rank`, `asin` and `url`; `itemsWithDetails` counts
|
|
32
|
+
# the complete ones. `proxy_country` defaults to the marketplace's home country (amazon.com
|
|
33
|
+
# -> US) so prices are in the store's currency.
|
|
34
|
+
#
|
|
35
|
+
# @param category [String] Which category: a department slug ('electronics', 'books', 'toys-
|
|
36
|
+
# and-games'), a slug plus browse-node id ('electronics/172541' = Headphones), or any Amazon
|
|
37
|
+
# Best Sellers / New Releases / Movers & Shakers / Most Wished For / Gift Ideas url (its
|
|
38
|
+
# marketplace and list type are then used). Empty = the all-departments root, which returns
|
|
39
|
+
# the department list (`category.subcategories`) and no items.
|
|
40
|
+
# @param list_type [String] Which ranked list: bestsellers, new_releases,
|
|
41
|
+
# movers_and_shakers, most_wished_for, gift_ideas. Ignored when `category` is a list url.
|
|
42
|
+
# API default: bestsellers.
|
|
43
|
+
# @param marketplace [String] Regional Amazon site. Ignored when `category` is a url. One
|
|
44
|
+
# of: amazon.com, amazon.co.uk, amazon.de, amazon.fr, amazon.it, amazon.es, amazon.nl,
|
|
45
|
+
# amazon.se, amazon.pl, amazon.com.be, amazon.com.tr, amazon.ca, amazon.com.mx,
|
|
46
|
+
# amazon.com.br, amazon.co.jp, amazon.in, amazon.ae, amazon.sa, amazon.sg, amazon.com.au.
|
|
47
|
+
# API default: amazon.com.
|
|
48
|
+
# @param max_results [Integer] How many ranked items to return (1-100). Amazon lists hold up
|
|
49
|
+
# to 100 items, 50 per page; each page fetched is billed as one request. Amazon's page HTML
|
|
50
|
+
# carries full details for the first 30 of each page; the rest load while scrolling, so
|
|
51
|
+
# asking for them adds a browser render (slower, ~10-25 s). API default: 30.
|
|
52
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
53
|
+
# 'DE'). Defaults to the marketplace's home country so prices come back in the store's
|
|
54
|
+
# currency.
|
|
55
|
+
# Calls POST /marketplace/amazon-bestsellers.
|
|
56
|
+
def amazon_bestsellers(category = nil, list_type: nil, marketplace: nil, max_results: nil, proxy_country: nil)
|
|
57
|
+
post_json("/marketplace/amazon-bestsellers", {
|
|
58
|
+
category: category,
|
|
59
|
+
list_type: list_type,
|
|
60
|
+
marketplace: marketplace,
|
|
61
|
+
max_results: max_results,
|
|
62
|
+
proxy_country: proxy_country,
|
|
63
|
+
})
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Scrape the reviews on an Amazon product page.
|
|
67
|
+
#
|
|
68
|
+
# Returns the review data an Amazon product page shows to a signed-out visitor, as JSON: the
|
|
69
|
+
# overall `rating`, total `reviewCount` (global ratings), the 5..1-star `histogram` (percent
|
|
70
|
+
# per star), the AI `customersSay` summary (overall text plus per-aspect `sentiment`,
|
|
71
|
+
# `mentions`, `positive`, `negative`, `summary` and the quoted review `excerpts` Amazon
|
|
72
|
+
# shows for the aspect) and the top `reviews` (each with `id`, `url`, `author`, `rating`,
|
|
73
|
+
# `title`, `body`, `date`, `country`, `fromOtherCountry`, `verified`, `helpfulVotes`,
|
|
74
|
+
# `variant` and `images`). Identify the product with `asin` (plus `marketplace`) or a
|
|
75
|
+
# product `url`. Scope: Amazon only shows its full review list to signed-in customers, so
|
|
76
|
+
# this endpoint returns the public slice on the product page - typically 5 to 13 top
|
|
77
|
+
# reviews, local ones first and then reviews from other countries - not a complete review
|
|
78
|
+
# export. When Amazon hides the top reviews behind a sign-in prompt (common on amazon.com),
|
|
79
|
+
# `reviewsGated` is `true` and `reviews` is empty; the rating, histogram and customers-say
|
|
80
|
+
# data, including its review excerpts, are still returned. Every field is always present and
|
|
81
|
+
# is `null` when the page does not show it. `proxy_country` defaults to the marketplace's
|
|
82
|
+
# home country; the proxy that served the request is reported as `exitCountry` (its country)
|
|
83
|
+
# and `exitProvider` (its type: `isp`, `residential` or `datacenter`).
|
|
84
|
+
#
|
|
85
|
+
# @param asin [String] Amazon product id (10-char ASIN, e.g. 'B07PXGQC1Q'; books use their
|
|
86
|
+
# ISBN-10). Give either `asin` (with `marketplace`) or a product `url`.
|
|
87
|
+
# @param url [String] Full Amazon product url, e.g. 'https://www.amazon.de/dp/B0D1XD1ZV3'.
|
|
88
|
+
# The ASIN and marketplace are read from it, so `asin`/`marketplace` are not needed when a
|
|
89
|
+
# url is given.
|
|
90
|
+
# @param marketplace [String] Regional Amazon site the ASIN belongs to. Reviews are read
|
|
91
|
+
# from that store. One of: amazon.com, amazon.co.uk, amazon.de, amazon.fr, amazon.it,
|
|
92
|
+
# amazon.es, amazon.nl, amazon.se, amazon.pl, amazon.com.be, amazon.com.tr, amazon.ca,
|
|
93
|
+
# amazon.com.mx, amazon.com.br, amazon.co.jp, amazon.in, amazon.ae, amazon.sa, amazon.sg,
|
|
94
|
+
# amazon.com.au. API default: amazon.com.
|
|
95
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
96
|
+
# 'DE'). Defaults to the marketplace's home country. The reviews come from the marketplace
|
|
97
|
+
# itself, so any proxy country returns the same reviews.
|
|
98
|
+
# Calls POST /marketplace/amazon-reviews.
|
|
99
|
+
def amazon_reviews(asin: nil, url: nil, marketplace: nil, proxy_country: nil)
|
|
100
|
+
post_json("/marketplace/amazon-reviews", {
|
|
101
|
+
asin: asin,
|
|
102
|
+
url: url,
|
|
103
|
+
marketplace: marketplace,
|
|
104
|
+
proxy_country: proxy_country,
|
|
105
|
+
})
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# Scrape Walmart product search.
|
|
109
|
+
#
|
|
110
|
+
# Searches Walmart for `query` and returns the products as JSON: `name`, `price` and
|
|
111
|
+
# `wasPrice`, `rating`, `reviewCount`, `brand`, `seller`, `availability`, `image` and a
|
|
112
|
+
# product `url`. Sort with `sort`, filter with `min_price`/`max_price`, and collect up to
|
|
113
|
+
# `max_results` across pages.
|
|
114
|
+
#
|
|
115
|
+
# @param query [String] What to search, e.g. 'laptop' or 'coffee maker'.
|
|
116
|
+
# @param sort [String] Ordering: best_match, price_low, price_high, best_seller,
|
|
117
|
+
# rating_high, new. API default: best_match.
|
|
118
|
+
# @param min_price [Integer] Lowest price to include (USD).
|
|
119
|
+
# @param max_price [Integer] Highest price to include (USD).
|
|
120
|
+
# @param page [Integer] 1-based page to start from (~40 per page). API default: 1.
|
|
121
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 40.
|
|
122
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
123
|
+
# 'DE'). Defaults to US.
|
|
124
|
+
# Calls POST /marketplace/walmart-search.
|
|
125
|
+
def walmart_search(query, sort: nil, min_price: nil, max_price: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
126
|
+
post_json("/marketplace/walmart-search", {
|
|
127
|
+
query: query,
|
|
128
|
+
sort: sort,
|
|
129
|
+
min_price: min_price,
|
|
130
|
+
max_price: max_price,
|
|
131
|
+
page: page,
|
|
132
|
+
max_results: max_results,
|
|
133
|
+
proxy_country: proxy_country,
|
|
134
|
+
})
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
# Scrape Target product search.
|
|
138
|
+
#
|
|
139
|
+
# Searches Target for `query` and returns the products as JSON: `tcin`, `title`, `brand`,
|
|
140
|
+
# `price` and `formattedPrice`, `currency`, `image` and a product `url`. Sort with `sort`,
|
|
141
|
+
# filter with `min_price`/`max_price`, and collect up to `max_results` across pages. Data
|
|
142
|
+
# comes from Target's Redsky API, since target.com itself is a client-rendered shell.
|
|
143
|
+
#
|
|
144
|
+
# @param query [String] What to search, e.g. 'coffee maker' or 'tv'.
|
|
145
|
+
# @param sort [String] Ordering: relevance, newest, price_low, price_high. API default:
|
|
146
|
+
# relevance.
|
|
147
|
+
# @param min_price [Integer] Lowest price to include (USD).
|
|
148
|
+
# @param max_price [Integer] Highest price to include (USD).
|
|
149
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 24.
|
|
150
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
151
|
+
# 'DE'). Leave blank to let the API pick.
|
|
152
|
+
# Calls POST /marketplace/target-search.
|
|
153
|
+
def target_search(query, sort: nil, min_price: nil, max_price: nil, max_results: nil, proxy_country: nil)
|
|
154
|
+
post_json("/marketplace/target-search", {
|
|
155
|
+
query: query,
|
|
156
|
+
sort: sort,
|
|
157
|
+
min_price: min_price,
|
|
158
|
+
max_price: max_price,
|
|
159
|
+
max_results: max_results,
|
|
160
|
+
proxy_country: proxy_country,
|
|
161
|
+
})
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
# Scrape Best Buy product search.
|
|
165
|
+
#
|
|
166
|
+
# Searches Best Buy (US) for `query` and returns the products as JSON: `sku`, `title`,
|
|
167
|
+
# `brand`, `model`, `price`, `originalPrice` (when on sale), `currency`, `rating`,
|
|
168
|
+
# `reviewCount`, `image` and a product `url`, plus Best Buy's `totalResults`. Sort with
|
|
169
|
+
# `sort`, filter with `min_price`/`max_price`, and collect up to `max_results` across pages
|
|
170
|
+
# starting at `page`.
|
|
171
|
+
#
|
|
172
|
+
# @param query [String] What to search, e.g. 'laptop' or 'noise cancelling headphones'.
|
|
173
|
+
# @param sort [String] Ordering: relevance, best_selling, price_low, price_high, rating,
|
|
174
|
+
# newest, discount. API default: relevance.
|
|
175
|
+
# @param min_price [Integer] Lowest price to include (USD).
|
|
176
|
+
# @param max_price [Integer] Highest price to include (USD).
|
|
177
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 24.
|
|
178
|
+
# @param page [Integer] 1-based results page to start from (48 per page, up to 40). API
|
|
179
|
+
# default: 1.
|
|
180
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
181
|
+
# 'DE'). Leave blank to let the API pick.
|
|
182
|
+
# Calls POST /marketplace/bestbuy-search.
|
|
183
|
+
def bestbuy_search(query, sort: nil, min_price: nil, max_price: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
184
|
+
post_json("/marketplace/bestbuy-search", {
|
|
185
|
+
query: query,
|
|
186
|
+
sort: sort,
|
|
187
|
+
min_price: min_price,
|
|
188
|
+
max_price: max_price,
|
|
189
|
+
max_results: max_results,
|
|
190
|
+
page: page,
|
|
191
|
+
proxy_country: proxy_country,
|
|
192
|
+
})
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
# Scrape Home Depot product search.
|
|
196
|
+
#
|
|
197
|
+
# Searches Home Depot for `query` and returns the products as JSON: `itemId`, `title`,
|
|
198
|
+
# `brand`, `price`, `originalPrice`, `currency`, `rating`, `reviewCount`, `image`, a product
|
|
199
|
+
# `url` and `modelNumber`. Sort with `sort`, filter with `min_price`/`max_price`, and
|
|
200
|
+
# collect up to `max_results` across pages.
|
|
201
|
+
#
|
|
202
|
+
# @param query [String] What to search, e.g. 'cordless drill' or 'interior paint'.
|
|
203
|
+
# @param sort [String] Ordering: relevance, price_low, price_high, top_rated, best_seller.
|
|
204
|
+
# API default: relevance.
|
|
205
|
+
# @param min_price [Integer] Lowest price to include (USD).
|
|
206
|
+
# @param max_price [Integer] Highest price to include (USD).
|
|
207
|
+
# @param page [Integer] 1-based page to start from (~24 per page). API default: 1.
|
|
208
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 24.
|
|
209
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
210
|
+
# 'DE'). Leave blank to let the API pick.
|
|
211
|
+
# Calls POST /marketplace/homedepot-search.
|
|
212
|
+
def homedepot_search(query, sort: nil, min_price: nil, max_price: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
213
|
+
post_json("/marketplace/homedepot-search", {
|
|
214
|
+
query: query,
|
|
215
|
+
sort: sort,
|
|
216
|
+
min_price: min_price,
|
|
217
|
+
max_price: max_price,
|
|
218
|
+
page: page,
|
|
219
|
+
max_results: max_results,
|
|
220
|
+
proxy_country: proxy_country,
|
|
221
|
+
})
|
|
222
|
+
end
|
|
223
|
+
|
|
224
|
+
# Scrape Etsy product search.
|
|
225
|
+
#
|
|
226
|
+
# Searches Etsy for `query` and returns the products as JSON: `title`, `price` + `currency`
|
|
227
|
+
# (ISO code) + `currencySymbol`, `shop`, `image`, a listing `url` and `sponsored`. Paid ad
|
|
228
|
+
# placements, which Etsy mixes in regardless of `sort`, are left out unless
|
|
229
|
+
# `include_sponsored=true`. Sort with `sort`, filter with `min_price`/`max_price`, and
|
|
230
|
+
# collect up to `max_results` across pages. Prices are in USD by default.
|
|
231
|
+
#
|
|
232
|
+
# @param query [String] What to search, e.g. 'handmade jewelry'.
|
|
233
|
+
# @param sort [String] Ordering: relevancy, price_asc, price_desc, newest, top_reviews. API
|
|
234
|
+
# default: relevancy.
|
|
235
|
+
# @param min_price [Integer] Lowest price to include.
|
|
236
|
+
# @param max_price [Integer] Highest price to include.
|
|
237
|
+
# @param page [Integer] 1-based page to start from. API default: 1.
|
|
238
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 48.
|
|
239
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
240
|
+
# 'DE'). Defaults to US (USD prices).
|
|
241
|
+
# @param include_sponsored [Boolean] Also return Etsy's paid ad placements (`sponsored:
|
|
242
|
+
# true`). Off by default: ads ignore the search order and `sort`. API default: false.
|
|
243
|
+
# Calls POST /marketplace/etsy-search.
|
|
244
|
+
def etsy_search(query, sort: nil, min_price: nil, max_price: nil, page: nil, max_results: nil, proxy_country: nil, include_sponsored: nil)
|
|
245
|
+
post_json("/marketplace/etsy-search", {
|
|
246
|
+
query: query,
|
|
247
|
+
sort: sort,
|
|
248
|
+
min_price: min_price,
|
|
249
|
+
max_price: max_price,
|
|
250
|
+
page: page,
|
|
251
|
+
max_results: max_results,
|
|
252
|
+
proxy_country: proxy_country,
|
|
253
|
+
include_sponsored: include_sponsored,
|
|
254
|
+
})
|
|
255
|
+
end
|
|
256
|
+
|
|
257
|
+
# Scrape SHEIN product search.
|
|
258
|
+
#
|
|
259
|
+
# Searches SHEIN (US store) for `query` and returns the products as JSON: `id`, `sku`,
|
|
260
|
+
# `title`, `url`, `price`, `originalPrice`, `currency`, `discountPercent`, `rating`,
|
|
261
|
+
# `reviewCount`, `category`, `image`, `soldOut` and `storeCode`. Order with `sort`, filter
|
|
262
|
+
# with `min_price`/`max_price`. Returns SHEIN's first results page (up to 20 products);
|
|
263
|
+
# SHEIN does not serve deeper pages to automated clients.
|
|
264
|
+
#
|
|
265
|
+
# @param query [String] What to search, e.g. 'summer dress' or 'phone case'.
|
|
266
|
+
# @param sort [String] Ordering: relevance, price_low, price_high. API default: relevance.
|
|
267
|
+
# @param min_price [Integer] Lowest price to include (USD).
|
|
268
|
+
# @param max_price [Integer] Highest price to include (USD).
|
|
269
|
+
# @param max_results [Integer] How many products to return (at most 20: SHEIN serves only
|
|
270
|
+
# its first results page). API default: 20.
|
|
271
|
+
# @param page [Integer] Must be 1: SHEIN challenges every deeper results page. API default:
|
|
272
|
+
# 1.
|
|
273
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
274
|
+
# 'DE'). Optional.
|
|
275
|
+
# Calls POST /marketplace/shein-search.
|
|
276
|
+
def shein_search(query, sort: nil, min_price: nil, max_price: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
277
|
+
post_json("/marketplace/shein-search", {
|
|
278
|
+
query: query,
|
|
279
|
+
sort: sort,
|
|
280
|
+
min_price: min_price,
|
|
281
|
+
max_price: max_price,
|
|
282
|
+
max_results: max_results,
|
|
283
|
+
page: page,
|
|
284
|
+
proxy_country: proxy_country,
|
|
285
|
+
})
|
|
286
|
+
end
|
|
287
|
+
|
|
288
|
+
# Scrape Zalando product search.
|
|
289
|
+
#
|
|
290
|
+
# Searches Zalando for `query` and returns the products as JSON: `sku`, `brand`, `name`,
|
|
291
|
+
# `price`, `currency`, `image` and a product `url`. Pick the storefront with `domain` and
|
|
292
|
+
# collect up to `max_results` across pages.
|
|
293
|
+
#
|
|
294
|
+
# @param query [String] What to search, e.g. 'nike sneakers' or 'winter jacket'.
|
|
295
|
+
# @param domain [String] Zalando storefront: en.zalando.de, www.zalando.co.uk,
|
|
296
|
+
# www.zalando.fr, www.zalando.it, www.zalando.es, www.zalando.nl, www.zalando.pl,
|
|
297
|
+
# www.zalando.de. API default: en.zalando.de.
|
|
298
|
+
# @param page [Integer] 1-based page to start from (~48 per page). API default: 1.
|
|
299
|
+
# @param max_results [Integer] How many products to collect across pages. API default: 48.
|
|
300
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
301
|
+
# 'DE'). Defaults to an EU country.
|
|
302
|
+
# Calls POST /marketplace/zalando-search.
|
|
303
|
+
def zalando_search(query, domain: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
304
|
+
post_json("/marketplace/zalando-search", {
|
|
305
|
+
query: query,
|
|
306
|
+
domain: domain,
|
|
307
|
+
page: page,
|
|
308
|
+
max_results: max_results,
|
|
309
|
+
proxy_country: proxy_country,
|
|
310
|
+
})
|
|
311
|
+
end
|
|
312
|
+
|
|
313
|
+
# Search Temu by keyword.
|
|
314
|
+
#
|
|
315
|
+
# Searches Temu for `keyword` and returns the matching product urls (~40 per page). This is
|
|
316
|
+
# discovery only - fetch each url's data (name, price, rating, reviews, images) through
|
|
317
|
+
# `POST /goods/temu-product`. Use `sort` to pick Temu's own result order (relevance,
|
|
318
|
+
# best_selling, recent, price_asc, price_desc). Response time is typically ~2s.
|
|
319
|
+
#
|
|
320
|
+
# @param keyword [String] Search phrase, e.g. 'kids toys' or 'phone case'.
|
|
321
|
+
# @param limit [Integer] Max product urls to return (one search page carries ~40). API
|
|
322
|
+
# default: 40.
|
|
323
|
+
# @param sort [String] Result order, using Temu's own sort: `relevance` (default),
|
|
324
|
+
# `best_selling` (top sales), `recent` (most recent), `price_asc` (price low to high) or
|
|
325
|
+
# `price_desc` (price high to low). API default: relevance.
|
|
326
|
+
# Calls POST /goods/temu-search.
|
|
327
|
+
def temu_search(keyword, limit: nil, sort: nil)
|
|
328
|
+
post_json("/goods/temu-search", {
|
|
329
|
+
keyword: keyword,
|
|
330
|
+
limit: limit,
|
|
331
|
+
sort: sort,
|
|
332
|
+
})
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# Scrape a Temu product page.
|
|
336
|
+
#
|
|
337
|
+
# Fetches a single Temu product and returns it as JSON: `name`, `description`, `brand`,
|
|
338
|
+
# `sku`, `price`, `priceCurrency`, `availability`, `itemCondition`, `rating`, `reviewCount`,
|
|
339
|
+
# `variantGroupId`, `images`, `reviews` (per-review rating and date; Temu masks the review
|
|
340
|
+
# text and author name in this document) and `video` (a `{contentUrl, thumbnailUrl,
|
|
341
|
+
# uploadDate}` object when the product ships one, else null). Temu serves product data only
|
|
342
|
+
# to search-engine crawlers, so this reads the ld+json SEO document rather than a browser
|
|
343
|
+
# render. The url must be a product page (path ends in `-g-<id>.html`).
|
|
344
|
+
#
|
|
345
|
+
# @param url [String] Full Temu product url, e.g.
|
|
346
|
+
# 'https://www.temu.com/xyz-g-601099512345678.html'. Must be a product page (its path ends
|
|
347
|
+
# in `-g-<id>.html`); search/category urls have no product data and are rejected.
|
|
348
|
+
# Calls POST /goods/temu-product.
|
|
349
|
+
def temu_product(url)
|
|
350
|
+
post_json("/goods/temu-product", {
|
|
351
|
+
url: url,
|
|
352
|
+
})
|
|
353
|
+
end
|
|
354
|
+
|
|
355
|
+
# Scrape AliExpress search results.
|
|
356
|
+
#
|
|
357
|
+
# Searches AliExpress for `keyword` and returns the products as JSON: `title`, product
|
|
358
|
+
# `url`, `price` (numeric) + `priceFormatted` + `currency`/`currencySymbol`,
|
|
359
|
+
# `originalPrice`, `rating`, `orders` (sold count) and `images`. Sort with `sort` (relevance
|
|
360
|
+
# / orders / price_asc / price_desc / newest) and collect up to `max_results` across pages.
|
|
361
|
+
# AliExpress localises price by region, so the currency follows the proxy country; pass
|
|
362
|
+
# `proxy_country` to pin it.
|
|
363
|
+
#
|
|
364
|
+
# @param keyword [String] What to search, e.g. 'wireless earbuds' or 'phone case'.
|
|
365
|
+
# @param sort [String] Ordering: relevance, orders, price_asc, price_desc, newest. API
|
|
366
|
+
# default: relevance.
|
|
367
|
+
# @param max_results [Integer] How many products to collect across pages (~60 per page, max
|
|
368
|
+
# 1200). API default: 60.
|
|
369
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
370
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
371
|
+
# 'DE'); also pins AliExpress price currency. Leave blank to let the API pick.
|
|
372
|
+
# Calls POST /goods/aliexpress-search.
|
|
373
|
+
def aliexpress_search(keyword, sort: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
374
|
+
post_json("/goods/aliexpress-search", {
|
|
375
|
+
keyword: keyword,
|
|
376
|
+
sort: sort,
|
|
377
|
+
max_results: max_results,
|
|
378
|
+
page: page,
|
|
379
|
+
proxy_country: proxy_country,
|
|
380
|
+
})
|
|
381
|
+
end
|
|
382
|
+
|
|
383
|
+
# Scrape Alibaba.com product search results.
|
|
384
|
+
#
|
|
385
|
+
# Searches Alibaba.com for `q` and returns the products as JSON: `productId`, `title`,
|
|
386
|
+
# `url`, `image`/`images`, `priceText` with `priceMin`/`priceMax` and `currency` (USD),
|
|
387
|
+
# `priceTiers` (price per quantity range), promotion price, `moq` + `moqUnit` (minimum
|
|
388
|
+
# order), `sold`, store `rating` + `reviewsCount`, `leadTimeDays`, `category`,
|
|
389
|
+
# `certifications`, `sponsored`, and the `supplier` (name, store and profile URL, country,
|
|
390
|
+
# `yearsOnAlibaba`, `verified`, `verifiedPro`, `goldSupplier`, `tradeAssurance`,
|
|
391
|
+
# `responseRate`). Sort by relevance or sales, filter with `verified`, `verified_pro`,
|
|
392
|
+
# `trade_assurance` and `supplier_country`, and collect up to `max_results` across pages (48
|
|
393
|
+
# per page, each page billed as one request, see `pagesFetched`). Each product also has a
|
|
394
|
+
# `thumbnail` (300x300 rendition of `image`, which is the full-size original). Paging:
|
|
395
|
+
# `nextPage` is the next Alibaba page not yet returned (pass it as `page` to continue) and
|
|
396
|
+
# `hasMore` says whether there is one; when `max_results` ends inside a page, the rest of
|
|
397
|
+
# that page is not returned (`droppedFromLastPage`), so use multiples of 48 to get every
|
|
398
|
+
# product.
|
|
399
|
+
#
|
|
400
|
+
# @param q [String] What to search, e.g. 'bluetooth speaker' or 'yoga mat'. Any language.
|
|
401
|
+
# @param page [Integer] 1-based results page to start from (1-100). Use `nextPage` of a
|
|
402
|
+
# previous response to continue. API default: 1.
|
|
403
|
+
# @param max_results [Integer] Products to collect across pages (1-480); Alibaba serves 48
|
|
404
|
+
# per page and each page fetched is billed as one request. API default: 48.
|
|
405
|
+
# @param sort [String] Order: relevance, sales (sales = most sold in the last 180 days
|
|
406
|
+
# first). API default: relevance.
|
|
407
|
+
# @param verified [Boolean] Only Verified suppliers (third-party inspected). API default:
|
|
408
|
+
# false.
|
|
409
|
+
# @param verified_pro [Boolean] Only Verified PRO suppliers. API default: false.
|
|
410
|
+
# @param trade_assurance [Boolean] Only products covered by Trade Assurance. API default:
|
|
411
|
+
# false.
|
|
412
|
+
# @param supplier_country [String] Only suppliers located in this country (ISO-2), e.g. CN,
|
|
413
|
+
# US, IN.
|
|
414
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
415
|
+
# 'DE'). Leave blank to let the API pick.
|
|
416
|
+
# Calls POST /marketplace/alibaba-search.
|
|
417
|
+
def alibaba_search(q, page: nil, max_results: nil, sort: nil, verified: nil, verified_pro: nil, trade_assurance: nil, supplier_country: nil, proxy_country: nil)
|
|
418
|
+
post_json("/marketplace/alibaba-search", {
|
|
419
|
+
q: q,
|
|
420
|
+
page: page,
|
|
421
|
+
max_results: max_results,
|
|
422
|
+
sort: sort,
|
|
423
|
+
verified: verified,
|
|
424
|
+
verified_pro: verified_pro,
|
|
425
|
+
trade_assurance: trade_assurance,
|
|
426
|
+
supplier_country: supplier_country,
|
|
427
|
+
proxy_country: proxy_country,
|
|
428
|
+
})
|
|
429
|
+
end
|
|
430
|
+
|
|
431
|
+
# Scrape one Alibaba.com product's full detail.
|
|
432
|
+
#
|
|
433
|
+
# Fetches one Alibaba.com product (by `product_id` or `url`) and returns it as JSON:
|
|
434
|
+
# `title`, `categories`, `images`, `videos`, price summary (`priceText`,
|
|
435
|
+
# `priceMin`/`priceMax` in the local `currency` of the proxy country, which varies by
|
|
436
|
+
# country, and `priceMinUsd`/`priceMaxUsd` in USD, the same for every country), `priceTiers`
|
|
437
|
+
# (price per quantity range, local `price` and `priceUsd`), `moq` + `unit`, SKU `variants`
|
|
438
|
+
# (`skuPricing` is `sku` when SKUs carry their own prices, `tiers` when every SKU is priced
|
|
439
|
+
# by `priceTiers` and the SKU prices are null), `attributes`, `leadTimes` (days per quantity
|
|
440
|
+
# range), `packaging`, `shippingMethods`, `shipFrom`, `tradeAssurance`, and the `supplier`
|
|
441
|
+
# (name, store/profile URL, country, location, years on Alibaba, business type,
|
|
442
|
+
# `goldSupplier`, `verified` + `verifiedBy`, rating, reviews, response time, on-time
|
|
443
|
+
# delivery rate). Companion to `/marketplace/alibaba-search`.
|
|
444
|
+
#
|
|
445
|
+
# @param product_id [String] Numeric Alibaba product id, e.g. '1601919386560' (the number
|
|
446
|
+
# before .html).
|
|
447
|
+
# @param url [String] Alibaba product URL, e.g. 'https://www.alibaba.com/product-
|
|
448
|
+
# detail/..._1601919386560.html'. Either url or product_id is required.
|
|
449
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
450
|
+
# 'DE'). Alibaba prices in that country's currency; USD figures are always included. Leave
|
|
451
|
+
# blank to let the API pick.
|
|
452
|
+
# Calls POST /marketplace/alibaba-product.
|
|
453
|
+
def alibaba_product(product_id = nil, url: nil, proxy_country: nil)
|
|
454
|
+
post_json("/marketplace/alibaba-product", {
|
|
455
|
+
product_id: product_id,
|
|
456
|
+
url: url,
|
|
457
|
+
proxy_country: proxy_country,
|
|
458
|
+
})
|
|
459
|
+
end
|
|
460
|
+
|
|
461
|
+
# Scrape 1688.com wholesale search results.
|
|
462
|
+
#
|
|
463
|
+
# Searches 1688.com (China's B2B wholesale marketplace) for `q` and returns the offers as
|
|
464
|
+
# JSON: `offerId`, `title`, `url`, `image`/`images`, `price` in yuan (`currency` CNY),
|
|
465
|
+
# `priceTiers` (price per quantity range with `minQuantity`/`maxQuantity`), `soldText` +
|
|
466
|
+
# `sold` (lower bound of units sold), `repurchaseRatePercent`, `serviceTags`,
|
|
467
|
+
# `promotionTags`, `tags`, `sponsored`, and the `supplier` (`name`, `loginId`, `memberId`,
|
|
468
|
+
# `shopUrl`, `yearsOnPlatform`, `province`, `city`, `factoryInspected`, `businessInspected`,
|
|
469
|
+
# `superFactory`, `tradeScores`). Sort by relevance, sales or price, filter with
|
|
470
|
+
# `price_min`/`price_max` (yuan) and `province`, and collect up to `max_results` across
|
|
471
|
+
# pages (60 per page, each page billed as one request, see `pagesFetched`). Paging:
|
|
472
|
+
# `nextPage` is the next 1688 page not yet returned (pass it as `page` to continue) and
|
|
473
|
+
# `hasMore` says whether there is one; when `max_results` ends inside a page, the rest of
|
|
474
|
+
# that page is not returned (`droppedFromLastPage`), so use multiples of 60 to get every
|
|
475
|
+
# offer. Text is returned in Chinese as 1688 publishes it. Companion to
|
|
476
|
+
# `/marketplace/1688-product`.
|
|
477
|
+
#
|
|
478
|
+
# @param q [String] What to search, in Chinese or English, e.g. '手机壳', '连衣裙', 'bluetooth
|
|
479
|
+
# earphones'. Chinese keywords match the most offers.
|
|
480
|
+
# @param page [Integer] 1-based results page to start from (1-34). Use `nextPage` of a
|
|
481
|
+
# previous response to continue. API default: 1.
|
|
482
|
+
# @param max_results [Integer] Offers to collect across pages (1-300); 1688 serves 60 per
|
|
483
|
+
# page and each page fetched is billed as one request. API default: 60.
|
|
484
|
+
# @param sort [String] Order: relevance, sales, price_asc, price_desc (sales = most sold in
|
|
485
|
+
# the last 30 days first; price in yuan). API default: relevance.
|
|
486
|
+
# @param price_min [Float] Only offers priced at or above this many yuan (CNY).
|
|
487
|
+
# @param price_max [Float] Only offers priced at or below this many yuan (CNY).
|
|
488
|
+
# @param province [String] Only suppliers in this Chinese province, by its Chinese name,
|
|
489
|
+
# e.g. 广东 (Guangdong), 浙江 (Zhejiang), 江苏 (Jiangsu).
|
|
490
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'DE',
|
|
491
|
+
# 'SG'). Leave blank to let the API pick.
|
|
492
|
+
# Calls POST /marketplace/1688-search.
|
|
493
|
+
def alibaba1688_search(q, page: nil, max_results: nil, sort: nil, price_min: nil, price_max: nil, province: nil, proxy_country: nil)
|
|
494
|
+
post_json("/marketplace/1688-search", {
|
|
495
|
+
q: q,
|
|
496
|
+
page: page,
|
|
497
|
+
max_results: max_results,
|
|
498
|
+
sort: sort,
|
|
499
|
+
price_min: price_min,
|
|
500
|
+
price_max: price_max,
|
|
501
|
+
province: province,
|
|
502
|
+
proxy_country: proxy_country,
|
|
503
|
+
})
|
|
504
|
+
end
|
|
505
|
+
|
|
506
|
+
# Scrape one 1688.com offer's full detail.
|
|
507
|
+
#
|
|
508
|
+
# Fetches one 1688.com offer (by `offer_id` or `url`) and returns it as JSON: `title`,
|
|
509
|
+
# `status`, `brand`, `category`, prices in yuan (`currency` CNY: `priceText`,
|
|
510
|
+
# `priceMin`/`priceMax`), `pricing` (`tiers` when every SKU is priced by the quantity ladder
|
|
511
|
+
# `priceTiers`, `sku` when each SKU carries its own `price`), `originalPriceTiers` (before a
|
|
512
|
+
# promotion), `unit`, `moq` (minimum order), `available` (stock), `sold` + `soldText` +
|
|
513
|
+
# `soldPeriod`, `variantOptions` (option names and values with images) and `variants` (every
|
|
514
|
+
# SKU with `skuId`, `options`, `price`, `available`), `attributes`, `images`, `video`,
|
|
515
|
+
# `descriptionUrl`, `shipping` (`from`, `dispatch` promise, `unitWeightKg`, `freeShipping`),
|
|
516
|
+
# `services` (guarantees), `labels`, `rating` (`goodRatePercent`, `score`, `reviewCount`,
|
|
517
|
+
# `reviewsWithPhotos`, review `tags`), `favorites` and the `supplier` (`name`, `loginId`,
|
|
518
|
+
# `memberId`, `userId`, `shopUrl`, `serviceScore`, `repeatPurchaseRate3m`, `factory`,
|
|
519
|
+
# `tpMember`, `location`). Text is returned in Chinese as 1688 publishes it. Companion to
|
|
520
|
+
# `/marketplace/1688-search`.
|
|
521
|
+
#
|
|
522
|
+
# @param offer_id [String] Numeric 1688 offer id, e.g. '735353578254' (the number in
|
|
523
|
+
# detail.1688.com/offer/<id>.html).
|
|
524
|
+
# @param url [String] 1688 offer link, e.g.
|
|
525
|
+
# 'https://detail.1688.com/offer/735353578254.html' (mobile m.1688.com and offerId= links
|
|
526
|
+
# work too). Either offer_id or url is required.
|
|
527
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'DE',
|
|
528
|
+
# 'SG'). Leave blank to let the API pick.
|
|
529
|
+
# Calls POST /marketplace/1688-product.
|
|
530
|
+
def alibaba1688_product(offer_id = nil, url: nil, proxy_country: nil)
|
|
531
|
+
post_json("/marketplace/1688-product", {
|
|
532
|
+
offer_id: offer_id,
|
|
533
|
+
url: url,
|
|
534
|
+
proxy_country: proxy_country,
|
|
535
|
+
})
|
|
536
|
+
end
|
|
537
|
+
|
|
538
|
+
# Search TikTok Shop products by keyword or category.
|
|
539
|
+
#
|
|
540
|
+
# Lists TikTok Shop (US) products for a keyword (`q`) or a category (`category`, URL or id).
|
|
541
|
+
# Each product card carries `productId`, `url`, `title`, `image`, `price` (current,
|
|
542
|
+
# original, discountPercent, currency, fromPrice), `rating`, `reviewCount`, `soldCount`,
|
|
543
|
+
# `brand`, `seller` (id, name, logo), promotion `labels` and, when TikTok binds one, the
|
|
544
|
+
# promoting `video` (id, url, author, play and like counts, `durationSeconds`). A keyword
|
|
545
|
+
# search also returns `relatedSearches` and `relatedCategories`; a category returns its
|
|
546
|
+
# `subcategories` and `breadcrumbs`. Up to 30 products (one page, one billed request). A
|
|
547
|
+
# keyword or category TikTok has no listing for returns 404.
|
|
548
|
+
#
|
|
549
|
+
# @param q [String] Search keyword, e.g. 'led face mask' (or a TikTok Shop keyword URL like
|
|
550
|
+
# 'https://www.tiktok.com/shop/product/led-face-mask'). Either this or `category`.
|
|
551
|
+
# @param category [String] List a category instead of a keyword: a category URL
|
|
552
|
+
# ('https://www.tiktok.com/shop/c/beauty-personal-care/601450') or its numeric id
|
|
553
|
+
# ('601450').
|
|
554
|
+
# @param max_results [Integer] Products to return (1-30). One page is read: a keyword lists
|
|
555
|
+
# up to 30 products, a category about 15. API default: 30.
|
|
556
|
+
# Calls POST /social/tiktok-shop-search.
|
|
557
|
+
def tiktok_shop_search(q = nil, category: nil, max_results: nil)
|
|
558
|
+
post_json("/social/tiktok-shop-search", {
|
|
559
|
+
q: q,
|
|
560
|
+
category: category,
|
|
561
|
+
max_results: max_results,
|
|
562
|
+
})
|
|
563
|
+
end
|
|
564
|
+
|
|
565
|
+
# Scrape one TikTok Shop product.
|
|
566
|
+
#
|
|
567
|
+
# Fetches a public TikTok Shop (US) product page by `url` or `product_id` and returns it as
|
|
568
|
+
# JSON: `productId`, `url`, `title`, `description` (plus `descriptionImages`), `status`,
|
|
569
|
+
# `available`, `brand`, `price` (current, original, discountPercent, currency, display
|
|
570
|
+
# strings, min/max for a price range), `soldCount`, `rating`, `reviewCount`,
|
|
571
|
+
# `ratingDistribution`, `categories`, `images`, `options` and `variants` (per-SKU price,
|
|
572
|
+
# stock and availability), `specifications`, `shipping` (fee, free, delivery days and
|
|
573
|
+
# estimate), promotion `labels`, `seller` (id, name, store URL, rating, followers, sold
|
|
574
|
+
# count, product / video / review counts, service scores, business name and address) and the
|
|
575
|
+
# top `reviews` the page shows (rating, text, date, variant, images). One billed request. No
|
|
576
|
+
# login or cookies; a product that does not exist or is not sold in the US returns 404.
|
|
577
|
+
#
|
|
578
|
+
# @param url [String] A TikTok Shop product URL ('https://www.tiktok.com/shop/pdp/goli-
|
|
579
|
+
# ashwagandha/1729527313880355335', 'https://shop.tiktok.com/us/pdp/1729527313880355335', a
|
|
580
|
+
# /view/product/ link) or a bare product id. Either this or `product_id`.
|
|
581
|
+
# @param product_id [String] The numeric product id (e.g. '1729527313880355335').
|
|
582
|
+
# Calls POST /social/tiktok-shop-product.
|
|
583
|
+
def tiktok_shop_product(url = nil, product_id: nil)
|
|
584
|
+
post_json("/social/tiktok-shop-product", {
|
|
585
|
+
url: url,
|
|
586
|
+
product_id: product_id,
|
|
587
|
+
})
|
|
588
|
+
end
|
|
589
|
+
|
|
590
|
+
# Scrape a TikTok Shop store and its products.
|
|
591
|
+
#
|
|
592
|
+
# Fetches a TikTok Shop (US) store by URL or seller id and returns its profile as `store`
|
|
593
|
+
# (id, name, url, rating, followers, soldCount, productCount, reviewCount, videoCount, logo,
|
|
594
|
+
# label such as 'OFFICIAL SHOP', service `scores` like positive feedback and on-time
|
|
595
|
+
# delivery, description) plus the `products` on its storefront page (up to 20, same card
|
|
596
|
+
# shape as /social/tiktok-shop-search) and `hasMore`. The store page does not carry the
|
|
597
|
+
# seller's business name and address, so `businessName` / `businessAddress` are null here; a
|
|
598
|
+
# product's `seller` (/social/tiktok-shop-product) has them. A store with no products listed
|
|
599
|
+
# returns its profile and an empty `products`. One billed request. Most stores answer in a
|
|
600
|
+
# few seconds; a store TikTok shows only past its verification puzzle (common for smaller
|
|
601
|
+
# shops) is read through that puzzle and can take up to about 50 s. A seller id TikTok has
|
|
602
|
+
# no US store for returns 404.
|
|
603
|
+
#
|
|
604
|
+
# @param store [String] A TikTok Shop store URL ('https://www.tiktok.com/shop/store/goli-
|
|
605
|
+
# nutrition/7495794203056835079') or the seller id ('7495794203056835079').
|
|
606
|
+
# @param max_results [Integer] Storefront products to return (0-20); 0 = store profile only.
|
|
607
|
+
# One page is read. API default: 20.
|
|
608
|
+
# Calls POST /social/tiktok-shop-store.
|
|
609
|
+
def tiktok_shop_store(store, max_results: nil)
|
|
610
|
+
post_json("/social/tiktok-shop-store", {
|
|
611
|
+
store: store,
|
|
612
|
+
max_results: max_results,
|
|
613
|
+
})
|
|
614
|
+
end
|
|
615
|
+
|
|
616
|
+
# Scrape Vinted product search.
|
|
617
|
+
#
|
|
618
|
+
# Searches Vinted for `query` and returns the listings as JSON: `title`, `price` and
|
|
619
|
+
# `currency`, `brand`, `size`, `condition`, `image` and an item `url`. Pick the marketplace
|
|
620
|
+
# with `domain`, sort with `order`, filter with `price_from`/`price_to`, and collect up to
|
|
621
|
+
# `max_results` across pages.
|
|
622
|
+
#
|
|
623
|
+
# @param query [String] What to search, e.g. 'nike air max' or 'levis 501'.
|
|
624
|
+
# @param domain [String] Marketplace TLD: co.uk, com, de, fr, it, es, nl, pl. API default:
|
|
625
|
+
# co.uk.
|
|
626
|
+
# @param order [String] Ordering: relevance, newest_first, price_low_to_high,
|
|
627
|
+
# price_high_to_low. API default: relevance.
|
|
628
|
+
# @param price_from [Integer] Lowest price to include.
|
|
629
|
+
# @param price_to [Integer] Highest price to include.
|
|
630
|
+
# @param max_results [Integer] How many listings to collect across pages. API default: 96.
|
|
631
|
+
# @param page [Integer] 1-based page to start from (~96 per page). API default: 1.
|
|
632
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
633
|
+
# 'DE'). Defaults to an EU country.
|
|
634
|
+
# Calls POST /marketplace/vinted-search.
|
|
635
|
+
def vinted_search(query, domain: nil, order: nil, price_from: nil, price_to: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
636
|
+
post_json("/marketplace/vinted-search", {
|
|
637
|
+
query: query,
|
|
638
|
+
domain: domain,
|
|
639
|
+
order: order,
|
|
640
|
+
price_from: price_from,
|
|
641
|
+
price_to: price_to,
|
|
642
|
+
max_results: max_results,
|
|
643
|
+
page: page,
|
|
644
|
+
proxy_country: proxy_country,
|
|
645
|
+
})
|
|
646
|
+
end
|
|
647
|
+
|
|
648
|
+
# Scrape Leboncoin search.
|
|
649
|
+
#
|
|
650
|
+
# Searches Leboncoin (France) for `query` and returns the ads as JSON: `id`, `title`,
|
|
651
|
+
# `price` and `currency`, `location` (city + postcode), `category`, publication `date`,
|
|
652
|
+
# `image`, listing `url`, `sellerType` (pro/private) and the ad's `attributes`. Narrow with
|
|
653
|
+
# `category`, `location` and `min_price`/`max_price`, order with `sort`, and collect up to
|
|
654
|
+
# `max_results` across pages. Paid 'a la une' placements (`sponsored: true`) are left out
|
|
655
|
+
# unless `include_sponsored=true`.
|
|
656
|
+
#
|
|
657
|
+
# @param query [String] What to search, e.g. 'velo electrique' or 'iphone 13'.
|
|
658
|
+
# @param category [String] Category: vehicles, cars, motorcycles, vans, car_parts,
|
|
659
|
+
# real_estate, real_estate_sale, rentals, flatshare, holiday_rentals, commercial_property,
|
|
660
|
+
# electronics, computers, photo_audio_video, phones, consoles, video_games,
|
|
661
|
+
# computer_accessories, home_garden, furniture, appliances, diy, decoration, garden,
|
|
662
|
+
# fashion, clothing, shoes, bags_accessories, watches_jewelry, baby_equipment, leisure,
|
|
663
|
+
# books, sports, bikes, toys, music_instruments, collectibles, antiques, animals, jobs,
|
|
664
|
+
# job_offers, or a numeric Leboncoin category id.
|
|
665
|
+
# @param location [String] Where to search: a city ('Bordeaux'), a city with postcode
|
|
666
|
+
# ('Toulouse 31000'), a department code ('75') or a region ('Ile-de-France').
|
|
667
|
+
# @param min_price [Integer] Lowest price to include (EUR).
|
|
668
|
+
# @param max_price [Integer] Highest price to include (EUR).
|
|
669
|
+
# @param sort [String] Ordering: relevance, newest, price_low, price_high. API default:
|
|
670
|
+
# relevance.
|
|
671
|
+
# @param max_results [Integer] How many listings to collect across pages. API default: 35.
|
|
672
|
+
# @param page [Integer] 1-based page to start from (35 per page). API default: 1.
|
|
673
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
674
|
+
# 'DE'). Optional.
|
|
675
|
+
# @param include_sponsored [Boolean] Also return Leboncoin's paid 'a la une' placements
|
|
676
|
+
# (`sponsored: true`). Off by default: they ignore the query, category and sort. API
|
|
677
|
+
# default: false.
|
|
678
|
+
# Calls POST /marketplace/leboncoin-search.
|
|
679
|
+
def leboncoin_search(query, category: nil, location: nil, min_price: nil, max_price: nil, sort: nil, max_results: nil, page: nil, proxy_country: nil, include_sponsored: nil)
|
|
680
|
+
post_json("/marketplace/leboncoin-search", {
|
|
681
|
+
query: query,
|
|
682
|
+
category: category,
|
|
683
|
+
location: location,
|
|
684
|
+
min_price: min_price,
|
|
685
|
+
max_price: max_price,
|
|
686
|
+
sort: sort,
|
|
687
|
+
max_results: max_results,
|
|
688
|
+
page: page,
|
|
689
|
+
proxy_country: proxy_country,
|
|
690
|
+
include_sponsored: include_sponsored,
|
|
691
|
+
})
|
|
692
|
+
end
|
|
693
|
+
|
|
694
|
+
# Scrape Woolworths (Australia) groceries.
|
|
695
|
+
#
|
|
696
|
+
# Woolworths Australia (woolworths.com.au) products as JSON. `mode=search&query=`,
|
|
697
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (stockcode), `name`,
|
|
698
|
+
# `brand`, `size`, `url`, `barcode`, `price`, `wasPrice`, `onSale`, `savings`, `promotion`
|
|
699
|
+
# (`type` half_price / special / multibuy / offer, `text`, multibuy `quantity` and `price`),
|
|
700
|
+
# `unitPrice` (`price`, `per`, `text`), `soldBy`, `inStock`, `available`, `purchaseLimit`,
|
|
701
|
+
# `isNew`, `sponsored`, `image`, `categories`, `healthStarRating` and `rating`, plus
|
|
702
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=product&product=` returns `product`
|
|
703
|
+
# with those fields and `description`, `ingredients`, `allergens` (`contains`, `mayContain`,
|
|
704
|
+
# `freeFrom`), `dietary`, `nutrition` (per serving and per 100 g / mL), `servingSize`,
|
|
705
|
+
# `servingsPerPack`, `storage`, `countryOfOrigin`, `department`, `aisle`, `images` and
|
|
706
|
+
# `variants`. Prices are Woolworths' online prices in AUD. Listings collect up to
|
|
707
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
708
|
+
#
|
|
709
|
+
# @param query [String] What to search for (mode=search), e.g. 'milk', 'tim tam', 'chicken
|
|
710
|
+
# breast'.
|
|
711
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc,
|
|
712
|
+
# unit_price_desc, name. API default: relevance.
|
|
713
|
+
# @param specials_only [Boolean] mode=search: only products on special. API default: false.
|
|
714
|
+
# @param page [Integer] 1-based listing page to start from (36 products per page). API
|
|
715
|
+
# default: 1.
|
|
716
|
+
# @param max_results [Integer] Products to collect across pages, 1-360. Each page fetched is
|
|
717
|
+
# billed one unit. API default: 36.
|
|
718
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
719
|
+
# 'US'). Leave blank: Woolworths answers every country, so the load spreads over all exits.
|
|
720
|
+
# Calls POST /groceries/woolworths.
|
|
721
|
+
def woolworths_search(query, sort: nil, specials_only: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
722
|
+
post_json("/groceries/woolworths", {
|
|
723
|
+
mode: "search",
|
|
724
|
+
query: query,
|
|
725
|
+
sort: sort,
|
|
726
|
+
specials_only: specials_only,
|
|
727
|
+
page: page,
|
|
728
|
+
max_results: max_results,
|
|
729
|
+
proxy_country: proxy_country,
|
|
730
|
+
})
|
|
731
|
+
end
|
|
732
|
+
|
|
733
|
+
# Scrape Woolworths (Australia) groceries.
|
|
734
|
+
#
|
|
735
|
+
# Woolworths Australia (woolworths.com.au) products as JSON. `mode=search&query=`,
|
|
736
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (stockcode), `name`,
|
|
737
|
+
# `brand`, `size`, `url`, `barcode`, `price`, `wasPrice`, `onSale`, `savings`, `promotion`
|
|
738
|
+
# (`type` half_price / special / multibuy / offer, `text`, multibuy `quantity` and `price`),
|
|
739
|
+
# `unitPrice` (`price`, `per`, `text`), `soldBy`, `inStock`, `available`, `purchaseLimit`,
|
|
740
|
+
# `isNew`, `sponsored`, `image`, `categories`, `healthStarRating` and `rating`, plus
|
|
741
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=product&product=` returns `product`
|
|
742
|
+
# with those fields and `description`, `ingredients`, `allergens` (`contains`, `mayContain`,
|
|
743
|
+
# `freeFrom`), `dietary`, `nutrition` (per serving and per 100 g / mL), `servingSize`,
|
|
744
|
+
# `servingsPerPack`, `storage`, `countryOfOrigin`, `department`, `aisle`, `images` and
|
|
745
|
+
# `variants`. Prices are Woolworths' online prices in AUD. Listings collect up to
|
|
746
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
747
|
+
#
|
|
748
|
+
# @param category [String] mode=category: a browse link
|
|
749
|
+
# (https://www.woolworths.com.au/shop/browse/dairy-eggs-fridge/milk), its path ('dairy-eggs-
|
|
750
|
+
# fridge/milk') or a category id ('1_6E4F4E4'). mode=specials: the specials page - 'half-
|
|
751
|
+
# price' (default), 'lower-shelf-price', 'everyday-low-price', 'seasonal-price', 'buy-more-
|
|
752
|
+
# save-more', 'online-only-specials' or a specials link.
|
|
753
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc,
|
|
754
|
+
# unit_price_desc, name. API default: relevance.
|
|
755
|
+
# @param page [Integer] 1-based listing page to start from (36 products per page). API
|
|
756
|
+
# default: 1.
|
|
757
|
+
# @param max_results [Integer] Products to collect across pages, 1-360. Each page fetched is
|
|
758
|
+
# billed one unit. API default: 36.
|
|
759
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
760
|
+
# 'US'). Leave blank: Woolworths answers every country, so the load spreads over all exits.
|
|
761
|
+
# Calls POST /groceries/woolworths.
|
|
762
|
+
def woolworths_category(category, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
763
|
+
post_json("/groceries/woolworths", {
|
|
764
|
+
mode: "category",
|
|
765
|
+
category: category,
|
|
766
|
+
sort: sort,
|
|
767
|
+
page: page,
|
|
768
|
+
max_results: max_results,
|
|
769
|
+
proxy_country: proxy_country,
|
|
770
|
+
})
|
|
771
|
+
end
|
|
772
|
+
|
|
773
|
+
# Scrape Woolworths (Australia) groceries.
|
|
774
|
+
#
|
|
775
|
+
# Woolworths Australia (woolworths.com.au) products as JSON. `mode=search&query=`,
|
|
776
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (stockcode), `name`,
|
|
777
|
+
# `brand`, `size`, `url`, `barcode`, `price`, `wasPrice`, `onSale`, `savings`, `promotion`
|
|
778
|
+
# (`type` half_price / special / multibuy / offer, `text`, multibuy `quantity` and `price`),
|
|
779
|
+
# `unitPrice` (`price`, `per`, `text`), `soldBy`, `inStock`, `available`, `purchaseLimit`,
|
|
780
|
+
# `isNew`, `sponsored`, `image`, `categories`, `healthStarRating` and `rating`, plus
|
|
781
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=product&product=` returns `product`
|
|
782
|
+
# with those fields and `description`, `ingredients`, `allergens` (`contains`, `mayContain`,
|
|
783
|
+
# `freeFrom`), `dietary`, `nutrition` (per serving and per 100 g / mL), `servingSize`,
|
|
784
|
+
# `servingsPerPack`, `storage`, `countryOfOrigin`, `department`, `aisle`, `images` and
|
|
785
|
+
# `variants`. Prices are Woolworths' online prices in AUD. Listings collect up to
|
|
786
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
787
|
+
#
|
|
788
|
+
# @param category [String] mode=category: a browse link
|
|
789
|
+
# (https://www.woolworths.com.au/shop/browse/dairy-eggs-fridge/milk), its path ('dairy-eggs-
|
|
790
|
+
# fridge/milk') or a category id ('1_6E4F4E4'). mode=specials: the specials page - 'half-
|
|
791
|
+
# price' (default), 'lower-shelf-price', 'everyday-low-price', 'seasonal-price', 'buy-more-
|
|
792
|
+
# save-more', 'online-only-specials' or a specials link.
|
|
793
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc,
|
|
794
|
+
# unit_price_desc, name. API default: relevance.
|
|
795
|
+
# @param page [Integer] 1-based listing page to start from (36 products per page). API
|
|
796
|
+
# default: 1.
|
|
797
|
+
# @param max_results [Integer] Products to collect across pages, 1-360. Each page fetched is
|
|
798
|
+
# billed one unit. API default: 36.
|
|
799
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
800
|
+
# 'US'). Leave blank: Woolworths answers every country, so the load spreads over all exits.
|
|
801
|
+
# Calls POST /groceries/woolworths.
|
|
802
|
+
def woolworths_specials(category: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
803
|
+
post_json("/groceries/woolworths", {
|
|
804
|
+
mode: "specials",
|
|
805
|
+
category: category,
|
|
806
|
+
sort: sort,
|
|
807
|
+
page: page,
|
|
808
|
+
max_results: max_results,
|
|
809
|
+
proxy_country: proxy_country,
|
|
810
|
+
})
|
|
811
|
+
end
|
|
812
|
+
|
|
813
|
+
# Scrape Woolworths (Australia) groceries.
|
|
814
|
+
#
|
|
815
|
+
# Woolworths Australia (woolworths.com.au) products as JSON. `mode=search&query=`,
|
|
816
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (stockcode), `name`,
|
|
817
|
+
# `brand`, `size`, `url`, `barcode`, `price`, `wasPrice`, `onSale`, `savings`, `promotion`
|
|
818
|
+
# (`type` half_price / special / multibuy / offer, `text`, multibuy `quantity` and `price`),
|
|
819
|
+
# `unitPrice` (`price`, `per`, `text`), `soldBy`, `inStock`, `available`, `purchaseLimit`,
|
|
820
|
+
# `isNew`, `sponsored`, `image`, `categories`, `healthStarRating` and `rating`, plus
|
|
821
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=product&product=` returns `product`
|
|
822
|
+
# with those fields and `description`, `ingredients`, `allergens` (`contains`, `mayContain`,
|
|
823
|
+
# `freeFrom`), `dietary`, `nutrition` (per serving and per 100 g / mL), `servingSize`,
|
|
824
|
+
# `servingsPerPack`, `storage`, `countryOfOrigin`, `department`, `aisle`, `images` and
|
|
825
|
+
# `variants`. Prices are Woolworths' online prices in AUD. Listings collect up to
|
|
826
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
827
|
+
#
|
|
828
|
+
# @param product [String] mode=product: a stockcode ('888137') or a product link
|
|
829
|
+
# (https://www.woolworths.com.au/shop/productdetails/888137/woolworths-full-cream-milk).
|
|
830
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
831
|
+
# 'US'). Leave blank: Woolworths answers every country, so the load spreads over all exits.
|
|
832
|
+
# Calls POST /groceries/woolworths.
|
|
833
|
+
def woolworths_product(product, proxy_country: nil)
|
|
834
|
+
post_json("/groceries/woolworths", {
|
|
835
|
+
mode: "product",
|
|
836
|
+
product: product,
|
|
837
|
+
proxy_country: proxy_country,
|
|
838
|
+
})
|
|
839
|
+
end
|
|
840
|
+
|
|
841
|
+
# Scrape Coles (Australia) groceries.
|
|
842
|
+
#
|
|
843
|
+
# Coles Australia (coles.com.au) products as JSON. `mode=search&query=`,
|
|
844
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id`, `name`, `brand`,
|
|
845
|
+
# `size`, `url`, `price`, `wasPrice`, `onSale`, `savings`, `promotion` (`type` half_price /
|
|
846
|
+
# special / multibuy / offer, `text`, multibuy `quantity` and `price`), `unitPrice`
|
|
847
|
+
# (`price`, `per`, `text`), `inStock`, `available`, `availableQuantity`, `purchaseLimit`,
|
|
848
|
+
# `promotionalLimit`, `onlineOnly`, `sponsored`, `image`, `categories` and
|
|
849
|
+
# `minimumShelfLife`, plus `totalResults`, `totalPages` and `nextPage`. `special` limits a
|
|
850
|
+
# search or the specials listing to half-price, multibuy or online-only deals.
|
|
851
|
+
# `mode=product&product=` returns `product` with those fields and `barcode`, `description`,
|
|
852
|
+
# `ingredients`, `allergens`, `dietary`, `storage`, `nutrition` (per serving and per 100 g /
|
|
853
|
+
# mL), `servingSize`, `servingsPerPack`, `countryOfOrigin`, `images` and `variants`. Prices
|
|
854
|
+
# are Coles' online prices in AUD. Listings collect up to `max_results` from `page` on,
|
|
855
|
+
# billed one unit per page.
|
|
856
|
+
#
|
|
857
|
+
# @param query [String] What to search for (mode=search), e.g. 'milk', 'tim tam', 'chicken
|
|
858
|
+
# breast'.
|
|
859
|
+
# @param special [String] Products on special only: all, half-price, multibuy, online-only.
|
|
860
|
+
# mode=specials defaults to 'all'; with mode=search it filters the results.
|
|
861
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc. API
|
|
862
|
+
# default: relevance.
|
|
863
|
+
# @param page [Integer] 1-based listing page to start from (48 products per page). API
|
|
864
|
+
# default: 1.
|
|
865
|
+
# @param max_results [Integer] Products to collect across pages, 1-480. Each page fetched is
|
|
866
|
+
# billed one unit. API default: 48.
|
|
867
|
+
# @param proxy_country [String] Proxy country the request is sent from: AU, NZ, US or CA
|
|
868
|
+
# (Coles serves only these). Leave blank to spread over all of them.
|
|
869
|
+
# Calls POST /groceries/coles.
|
|
870
|
+
def coles_search(query, special: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
871
|
+
post_json("/groceries/coles", {
|
|
872
|
+
mode: "search",
|
|
873
|
+
query: query,
|
|
874
|
+
special: special,
|
|
875
|
+
sort: sort,
|
|
876
|
+
page: page,
|
|
877
|
+
max_results: max_results,
|
|
878
|
+
proxy_country: proxy_country,
|
|
879
|
+
})
|
|
880
|
+
end
|
|
881
|
+
|
|
882
|
+
# Scrape Coles (Australia) groceries.
|
|
883
|
+
#
|
|
884
|
+
# Coles Australia (coles.com.au) products as JSON. `mode=search&query=`,
|
|
885
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id`, `name`, `brand`,
|
|
886
|
+
# `size`, `url`, `price`, `wasPrice`, `onSale`, `savings`, `promotion` (`type` half_price /
|
|
887
|
+
# special / multibuy / offer, `text`, multibuy `quantity` and `price`), `unitPrice`
|
|
888
|
+
# (`price`, `per`, `text`), `inStock`, `available`, `availableQuantity`, `purchaseLimit`,
|
|
889
|
+
# `promotionalLimit`, `onlineOnly`, `sponsored`, `image`, `categories` and
|
|
890
|
+
# `minimumShelfLife`, plus `totalResults`, `totalPages` and `nextPage`. `special` limits a
|
|
891
|
+
# search or the specials listing to half-price, multibuy or online-only deals.
|
|
892
|
+
# `mode=product&product=` returns `product` with those fields and `barcode`, `description`,
|
|
893
|
+
# `ingredients`, `allergens`, `dietary`, `storage`, `nutrition` (per serving and per 100 g /
|
|
894
|
+
# mL), `servingSize`, `servingsPerPack`, `countryOfOrigin`, `images` and `variants`. Prices
|
|
895
|
+
# are Coles' online prices in AUD. Listings collect up to `max_results` from `page` on,
|
|
896
|
+
# billed one unit per page.
|
|
897
|
+
#
|
|
898
|
+
# @param category [String] mode=category: a browse link
|
|
899
|
+
# (https://www.coles.com.au/browse/dairy-eggs-fridge/milk) or its path ('dairy-eggs-
|
|
900
|
+
# fridge/milk').
|
|
901
|
+
# @param special [String] Products on special only: all, half-price, multibuy, online-only.
|
|
902
|
+
# mode=specials defaults to 'all'; with mode=search it filters the results.
|
|
903
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc. API
|
|
904
|
+
# default: relevance.
|
|
905
|
+
# @param page [Integer] 1-based listing page to start from (48 products per page). API
|
|
906
|
+
# default: 1.
|
|
907
|
+
# @param max_results [Integer] Products to collect across pages, 1-480. Each page fetched is
|
|
908
|
+
# billed one unit. API default: 48.
|
|
909
|
+
# @param proxy_country [String] Proxy country the request is sent from: AU, NZ, US or CA
|
|
910
|
+
# (Coles serves only these). Leave blank to spread over all of them.
|
|
911
|
+
# Calls POST /groceries/coles.
|
|
912
|
+
def coles_category(category, special: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
913
|
+
post_json("/groceries/coles", {
|
|
914
|
+
mode: "category",
|
|
915
|
+
category: category,
|
|
916
|
+
special: special,
|
|
917
|
+
sort: sort,
|
|
918
|
+
page: page,
|
|
919
|
+
max_results: max_results,
|
|
920
|
+
proxy_country: proxy_country,
|
|
921
|
+
})
|
|
922
|
+
end
|
|
923
|
+
|
|
924
|
+
# Scrape Coles (Australia) groceries.
|
|
925
|
+
#
|
|
926
|
+
# Coles Australia (coles.com.au) products as JSON. `mode=search&query=`,
|
|
927
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id`, `name`, `brand`,
|
|
928
|
+
# `size`, `url`, `price`, `wasPrice`, `onSale`, `savings`, `promotion` (`type` half_price /
|
|
929
|
+
# special / multibuy / offer, `text`, multibuy `quantity` and `price`), `unitPrice`
|
|
930
|
+
# (`price`, `per`, `text`), `inStock`, `available`, `availableQuantity`, `purchaseLimit`,
|
|
931
|
+
# `promotionalLimit`, `onlineOnly`, `sponsored`, `image`, `categories` and
|
|
932
|
+
# `minimumShelfLife`, plus `totalResults`, `totalPages` and `nextPage`. `special` limits a
|
|
933
|
+
# search or the specials listing to half-price, multibuy or online-only deals.
|
|
934
|
+
# `mode=product&product=` returns `product` with those fields and `barcode`, `description`,
|
|
935
|
+
# `ingredients`, `allergens`, `dietary`, `storage`, `nutrition` (per serving and per 100 g /
|
|
936
|
+
# mL), `servingSize`, `servingsPerPack`, `countryOfOrigin`, `images` and `variants`. Prices
|
|
937
|
+
# are Coles' online prices in AUD. Listings collect up to `max_results` from `page` on,
|
|
938
|
+
# billed one unit per page.
|
|
939
|
+
#
|
|
940
|
+
# @param special [String] Products on special only: all, half-price, multibuy, online-only.
|
|
941
|
+
# mode=specials defaults to 'all'; with mode=search it filters the results.
|
|
942
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, unit_price_asc. API
|
|
943
|
+
# default: relevance.
|
|
944
|
+
# @param page [Integer] 1-based listing page to start from (48 products per page). API
|
|
945
|
+
# default: 1.
|
|
946
|
+
# @param max_results [Integer] Products to collect across pages, 1-480. Each page fetched is
|
|
947
|
+
# billed one unit. API default: 48.
|
|
948
|
+
# @param proxy_country [String] Proxy country the request is sent from: AU, NZ, US or CA
|
|
949
|
+
# (Coles serves only these). Leave blank to spread over all of them.
|
|
950
|
+
# Calls POST /groceries/coles.
|
|
951
|
+
def coles_specials(special: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
952
|
+
post_json("/groceries/coles", {
|
|
953
|
+
mode: "specials",
|
|
954
|
+
special: special,
|
|
955
|
+
sort: sort,
|
|
956
|
+
page: page,
|
|
957
|
+
max_results: max_results,
|
|
958
|
+
proxy_country: proxy_country,
|
|
959
|
+
})
|
|
960
|
+
end
|
|
961
|
+
|
|
962
|
+
# Scrape Coles (Australia) groceries.
|
|
963
|
+
#
|
|
964
|
+
# Coles Australia (coles.com.au) products as JSON. `mode=search&query=`,
|
|
965
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id`, `name`, `brand`,
|
|
966
|
+
# `size`, `url`, `price`, `wasPrice`, `onSale`, `savings`, `promotion` (`type` half_price /
|
|
967
|
+
# special / multibuy / offer, `text`, multibuy `quantity` and `price`), `unitPrice`
|
|
968
|
+
# (`price`, `per`, `text`), `inStock`, `available`, `availableQuantity`, `purchaseLimit`,
|
|
969
|
+
# `promotionalLimit`, `onlineOnly`, `sponsored`, `image`, `categories` and
|
|
970
|
+
# `minimumShelfLife`, plus `totalResults`, `totalPages` and `nextPage`. `special` limits a
|
|
971
|
+
# search or the specials listing to half-price, multibuy or online-only deals.
|
|
972
|
+
# `mode=product&product=` returns `product` with those fields and `barcode`, `description`,
|
|
973
|
+
# `ingredients`, `allergens`, `dietary`, `storage`, `nutrition` (per serving and per 100 g /
|
|
974
|
+
# mL), `servingSize`, `servingsPerPack`, `countryOfOrigin`, `images` and `variants`. Prices
|
|
975
|
+
# are Coles' online prices in AUD. Listings collect up to `max_results` from `page` on,
|
|
976
|
+
# billed one unit per page.
|
|
977
|
+
#
|
|
978
|
+
# @param product [String] mode=product: a Coles product id ('8150288') or a product link
|
|
979
|
+
# (https://www.coles.com.au/product/coles-full-cream-milk-3l-8150288).
|
|
980
|
+
# @param proxy_country [String] Proxy country the request is sent from: AU, NZ, US or CA
|
|
981
|
+
# (Coles serves only these). Leave blank to spread over all of them.
|
|
982
|
+
# Calls POST /groceries/coles.
|
|
983
|
+
def coles_product(product, proxy_country: nil)
|
|
984
|
+
post_json("/groceries/coles", {
|
|
985
|
+
mode: "product",
|
|
986
|
+
product: product,
|
|
987
|
+
proxy_country: proxy_country,
|
|
988
|
+
})
|
|
989
|
+
end
|
|
990
|
+
|
|
991
|
+
# Scrape ALDI Australia groceries.
|
|
992
|
+
#
|
|
993
|
+
# ALDI Australia (aldi.com.au) products as JSON. `mode=search&query=`,
|
|
994
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (SKU), `name`,
|
|
995
|
+
# `brand`, `size`, `url`, `price`, `priceText`, `wasPrice`, `onSale`, `unitPrice` (`price`,
|
|
996
|
+
# `per`, `text`), `bottleDeposit`, `specialBuy` (on-sale date and label), `badges`,
|
|
997
|
+
# `available`, `soldOnline`, `ageRestricted`, `image`, `categories` and `categoryKeys`, plus
|
|
998
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=specials` lists one Special Buys drop
|
|
999
|
+
# (the current one, or the date / link in `category`; `theme` narrows it) and returns
|
|
1000
|
+
# `promotions`, every drop ALDI has announced with its themes. `mode=product&product=`
|
|
1001
|
+
# returns `product` with those fields and `description`, `ingredients`, `allergens`,
|
|
1002
|
+
# `claims`, `storage`, `usage`, `countryOfOrigin` and `images`. Prices are ALDI's in-store
|
|
1003
|
+
# prices in AUD (ALDI Australia does not sell groceries online). Listings collect up to
|
|
1004
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
1005
|
+
#
|
|
1006
|
+
# @param query [String] What to search for (mode=search), e.g. 'milk', 'chocolate',
|
|
1007
|
+
# 'nappies'.
|
|
1008
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, name_asc, name_desc.
|
|
1009
|
+
# API default: relevance.
|
|
1010
|
+
# @param page [Integer] 1-based listing page to start from (60 products per page). API
|
|
1011
|
+
# default: 1.
|
|
1012
|
+
# @param max_results [Integer] Products to collect across pages, 1-600. Each page fetched is
|
|
1013
|
+
# billed one unit. API default: 60.
|
|
1014
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
1015
|
+
# 'US'). Leave blank: ALDI answers every country, so the load spreads over all exits.
|
|
1016
|
+
# Calls POST /groceries/aldi-au.
|
|
1017
|
+
def aldi_au_search(query, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
1018
|
+
post_json("/groceries/aldi-au", {
|
|
1019
|
+
mode: "search",
|
|
1020
|
+
query: query,
|
|
1021
|
+
sort: sort,
|
|
1022
|
+
page: page,
|
|
1023
|
+
max_results: max_results,
|
|
1024
|
+
proxy_country: proxy_country,
|
|
1025
|
+
})
|
|
1026
|
+
end
|
|
1027
|
+
|
|
1028
|
+
# Scrape ALDI Australia groceries.
|
|
1029
|
+
#
|
|
1030
|
+
# ALDI Australia (aldi.com.au) products as JSON. `mode=search&query=`,
|
|
1031
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (SKU), `name`,
|
|
1032
|
+
# `brand`, `size`, `url`, `price`, `priceText`, `wasPrice`, `onSale`, `unitPrice` (`price`,
|
|
1033
|
+
# `per`, `text`), `bottleDeposit`, `specialBuy` (on-sale date and label), `badges`,
|
|
1034
|
+
# `available`, `soldOnline`, `ageRestricted`, `image`, `categories` and `categoryKeys`, plus
|
|
1035
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=specials` lists one Special Buys drop
|
|
1036
|
+
# (the current one, or the date / link in `category`; `theme` narrows it) and returns
|
|
1037
|
+
# `promotions`, every drop ALDI has announced with its themes. `mode=product&product=`
|
|
1038
|
+
# returns `product` with those fields and `description`, `ingredients`, `allergens`,
|
|
1039
|
+
# `claims`, `storage`, `usage`, `countryOfOrigin` and `images`. Prices are ALDI's in-store
|
|
1040
|
+
# prices in AUD (ALDI Australia does not sell groceries online). Listings collect up to
|
|
1041
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
1042
|
+
#
|
|
1043
|
+
# @param category [String] mode=category: a category link
|
|
1044
|
+
# (https://www.aldi.com.au/products/dairy-eggs-fridge/milk/k/1111111160) or its key
|
|
1045
|
+
# ('1111111160'; '1588161425841179' = Lower Prices). mode=specials: the Special Buys drop -
|
|
1046
|
+
# its date ('2026-10-14') or link (https://www.aldi.com.au/special-buys/2026-10-14); blank =
|
|
1047
|
+
# the current drop.
|
|
1048
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, name_asc, name_desc.
|
|
1049
|
+
# API default: relevance.
|
|
1050
|
+
# @param page [Integer] 1-based listing page to start from (60 products per page). API
|
|
1051
|
+
# default: 1.
|
|
1052
|
+
# @param max_results [Integer] Products to collect across pages, 1-600. Each page fetched is
|
|
1053
|
+
# billed one unit. API default: 60.
|
|
1054
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
1055
|
+
# 'US'). Leave blank: ALDI answers every country, so the load spreads over all exits.
|
|
1056
|
+
# Calls POST /groceries/aldi-au.
|
|
1057
|
+
def aldi_au_category(category, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
1058
|
+
post_json("/groceries/aldi-au", {
|
|
1059
|
+
mode: "category",
|
|
1060
|
+
category: category,
|
|
1061
|
+
sort: sort,
|
|
1062
|
+
page: page,
|
|
1063
|
+
max_results: max_results,
|
|
1064
|
+
proxy_country: proxy_country,
|
|
1065
|
+
})
|
|
1066
|
+
end
|
|
1067
|
+
|
|
1068
|
+
# Scrape ALDI Australia groceries.
|
|
1069
|
+
#
|
|
1070
|
+
# ALDI Australia (aldi.com.au) products as JSON. `mode=search&query=`,
|
|
1071
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (SKU), `name`,
|
|
1072
|
+
# `brand`, `size`, `url`, `price`, `priceText`, `wasPrice`, `onSale`, `unitPrice` (`price`,
|
|
1073
|
+
# `per`, `text`), `bottleDeposit`, `specialBuy` (on-sale date and label), `badges`,
|
|
1074
|
+
# `available`, `soldOnline`, `ageRestricted`, `image`, `categories` and `categoryKeys`, plus
|
|
1075
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=specials` lists one Special Buys drop
|
|
1076
|
+
# (the current one, or the date / link in `category`; `theme` narrows it) and returns
|
|
1077
|
+
# `promotions`, every drop ALDI has announced with its themes. `mode=product&product=`
|
|
1078
|
+
# returns `product` with those fields and `description`, `ingredients`, `allergens`,
|
|
1079
|
+
# `claims`, `storage`, `usage`, `countryOfOrigin` and `images`. Prices are ALDI's in-store
|
|
1080
|
+
# prices in AUD (ALDI Australia does not sell groceries online). Listings collect up to
|
|
1081
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
1082
|
+
#
|
|
1083
|
+
# @param category [String] mode=category: a category link
|
|
1084
|
+
# (https://www.aldi.com.au/products/dairy-eggs-fridge/milk/k/1111111160) or its key
|
|
1085
|
+
# ('1111111160'; '1588161425841179' = Lower Prices). mode=specials: the Special Buys drop -
|
|
1086
|
+
# its date ('2026-10-14') or link (https://www.aldi.com.au/special-buys/2026-10-14); blank =
|
|
1087
|
+
# the current drop.
|
|
1088
|
+
# @param theme [String] mode=specials: one theme of the drop, e.g. 'Gardening' (see
|
|
1089
|
+
# `promotions[].themes`).
|
|
1090
|
+
# @param sort [String] Listing order: relevance, price_asc, price_desc, name_asc, name_desc.
|
|
1091
|
+
# API default: relevance.
|
|
1092
|
+
# @param page [Integer] 1-based listing page to start from (60 products per page). API
|
|
1093
|
+
# default: 1.
|
|
1094
|
+
# @param max_results [Integer] Products to collect across pages, 1-600. Each page fetched is
|
|
1095
|
+
# billed one unit. API default: 60.
|
|
1096
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
1097
|
+
# 'US'). Leave blank: ALDI answers every country, so the load spreads over all exits.
|
|
1098
|
+
# Calls POST /groceries/aldi-au.
|
|
1099
|
+
def aldi_au_specials(category: nil, theme: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
1100
|
+
post_json("/groceries/aldi-au", {
|
|
1101
|
+
mode: "specials",
|
|
1102
|
+
category: category,
|
|
1103
|
+
theme: theme,
|
|
1104
|
+
sort: sort,
|
|
1105
|
+
page: page,
|
|
1106
|
+
max_results: max_results,
|
|
1107
|
+
proxy_country: proxy_country,
|
|
1108
|
+
})
|
|
1109
|
+
end
|
|
1110
|
+
|
|
1111
|
+
# Scrape ALDI Australia groceries.
|
|
1112
|
+
#
|
|
1113
|
+
# ALDI Australia (aldi.com.au) products as JSON. `mode=search&query=`,
|
|
1114
|
+
# `mode=category&category=` and `mode=specials` return `products`: `id` (SKU), `name`,
|
|
1115
|
+
# `brand`, `size`, `url`, `price`, `priceText`, `wasPrice`, `onSale`, `unitPrice` (`price`,
|
|
1116
|
+
# `per`, `text`), `bottleDeposit`, `specialBuy` (on-sale date and label), `badges`,
|
|
1117
|
+
# `available`, `soldOnline`, `ageRestricted`, `image`, `categories` and `categoryKeys`, plus
|
|
1118
|
+
# `totalResults`, `totalPages` and `nextPage`. `mode=specials` lists one Special Buys drop
|
|
1119
|
+
# (the current one, or the date / link in `category`; `theme` narrows it) and returns
|
|
1120
|
+
# `promotions`, every drop ALDI has announced with its themes. `mode=product&product=`
|
|
1121
|
+
# returns `product` with those fields and `description`, `ingredients`, `allergens`,
|
|
1122
|
+
# `claims`, `storage`, `usage`, `countryOfOrigin` and `images`. Prices are ALDI's in-store
|
|
1123
|
+
# prices in AUD (ALDI Australia does not sell groceries online). Listings collect up to
|
|
1124
|
+
# `max_results` from `page` on, billed one unit per page.
|
|
1125
|
+
#
|
|
1126
|
+
# @param product [String] mode=product: an ALDI SKU ('000000000000398691') or a product link
|
|
1127
|
+
# (https://www.aldi.com.au/product/farmdale-light-milk-2l-000000000000398691).
|
|
1128
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
|
|
1129
|
+
# 'US'). Leave blank: ALDI answers every country, so the load spreads over all exits.
|
|
1130
|
+
# Calls POST /groceries/aldi-au.
|
|
1131
|
+
def aldi_au_product(product, proxy_country: nil)
|
|
1132
|
+
post_json("/groceries/aldi-au", {
|
|
1133
|
+
mode: "product",
|
|
1134
|
+
product: product,
|
|
1135
|
+
proxy_country: proxy_country,
|
|
1136
|
+
})
|
|
1137
|
+
end
|
|
1138
|
+
|
|
1139
|
+
# Scrape Booking.com hotels.
|
|
1140
|
+
#
|
|
1141
|
+
# Searches Booking.com for `location` and returns the properties as JSON: `name`, `price`
|
|
1142
|
+
# (with `checkin`/`checkout`), `reviewScore`, `address`, coordinates, `image` and a hotel
|
|
1143
|
+
# `url`. Set `adults`, `children`, `rooms` and `currency`, and collect up to `max_results`
|
|
1144
|
+
# across pages. Prices only appear when check-in/check-out dates are given.
|
|
1145
|
+
#
|
|
1146
|
+
# @param location [String] City or place to search, e.g. 'Austin' or 'Paris'.
|
|
1147
|
+
# @param checkin [String] Check-in YYYY-MM-DD (needed for prices).
|
|
1148
|
+
# @param checkout [String] Check-out YYYY-MM-DD.
|
|
1149
|
+
# @param adults [Integer] Number of adults.
|
|
1150
|
+
# @param children [Integer] Number of children.
|
|
1151
|
+
# @param rooms [Integer] Number of rooms.
|
|
1152
|
+
# @param currency [String] Price currency: USD, EUR, GBP, AUD, CAD, CHF, JPY, CNY, INR, BRL,
|
|
1153
|
+
# MXN, PLN, SEK, NOK, DKK, AED, SGD, ZAR, TRY, RUB. API default: USD.
|
|
1154
|
+
# @param max_results [Integer] How many properties to collect across pages. API default: 25.
|
|
1155
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1156
|
+
# 'DE').
|
|
1157
|
+
# Calls POST /travel/booking.
|
|
1158
|
+
def booking_hotels(location, checkin: nil, checkout: nil, adults: nil, children: nil, rooms: nil, currency: nil, max_results: nil, proxy_country: nil)
|
|
1159
|
+
post_json("/travel/booking", {
|
|
1160
|
+
location: location,
|
|
1161
|
+
checkin: checkin,
|
|
1162
|
+
checkout: checkout,
|
|
1163
|
+
adults: adults,
|
|
1164
|
+
children: children,
|
|
1165
|
+
rooms: rooms,
|
|
1166
|
+
currency: currency,
|
|
1167
|
+
max_results: max_results,
|
|
1168
|
+
proxy_country: proxy_country,
|
|
1169
|
+
})
|
|
1170
|
+
end
|
|
1171
|
+
|
|
1172
|
+
# Scrape Booking.com hotel reviews.
|
|
1173
|
+
#
|
|
1174
|
+
# Given a Booking.com `hotel` (page URL or `<cc>/<slug>`), returns its newest guest
|
|
1175
|
+
# `reviews`: `id`, `reviewDate`, `rating`, `ratingText`, `title`, `ratingBand`,
|
|
1176
|
+
# `positiveText`, `negativeText`, `reviewer` (`name`, `country`, `countryCode`), `tripType`,
|
|
1177
|
+
# `travelerType`, `nights`, `stayMonth`, `helpfulVotes`, `submittedFromMobile`, `language`
|
|
1178
|
+
# and the raw `tags`. Booking serves the newest 15 reviews written in each language: pick
|
|
1179
|
+
# the languages with `languages` (or `all`), cap the total with `max_results` and drop older
|
|
1180
|
+
# reviews with `since`. Billed one unit per language page fetched (`pagesFetched`).
|
|
1181
|
+
#
|
|
1182
|
+
# @param hotel [String] Booking.com hotel page URL, e.g.
|
|
1183
|
+
# https://www.booking.com/hotel/fr/le-meurice.html (a locale suffix such as .en-gb.html and
|
|
1184
|
+
# query parameters are fine), or `<cc>/<slug>` such as fr/le-meurice.
|
|
1185
|
+
# @param languages [String] Review languages to read, comma-separated, or `all`: en, de, fr,
|
|
1186
|
+
# es, it, nl, pt, pl, ru, ja, zh, ko, sv, da, no, fi, cs, tr. Default en. Booking serves the
|
|
1187
|
+
# newest 15 reviews written in each language; each language is one page and one billed unit,
|
|
1188
|
+
# fetched in the order given until max_results is reached.
|
|
1189
|
+
# @param max_results [Integer] How many reviews to return across languages (1-270, at most
|
|
1190
|
+
# 15 per language). API default: 15.
|
|
1191
|
+
# @param since [String] Only reviews written on or after this date (YYYY-MM-DD).
|
|
1192
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1193
|
+
# 'DE').
|
|
1194
|
+
# Calls POST /travel/booking-reviews.
|
|
1195
|
+
def booking_reviews(hotel, languages: nil, max_results: nil, since: nil, proxy_country: nil)
|
|
1196
|
+
post_json("/travel/booking-reviews", {
|
|
1197
|
+
hotel: hotel,
|
|
1198
|
+
languages: languages,
|
|
1199
|
+
max_results: max_results,
|
|
1200
|
+
since: since,
|
|
1201
|
+
proxy_country: proxy_country,
|
|
1202
|
+
})
|
|
1203
|
+
end
|
|
1204
|
+
|
|
1205
|
+
# Scrape Airbnb stay listings.
|
|
1206
|
+
#
|
|
1207
|
+
# Searches Airbnb for `location` and returns the stay listings as JSON: `name`, `roomType`,
|
|
1208
|
+
# `price` (USD by default) with any original price, `rating` and `reviewCount`, coordinates,
|
|
1209
|
+
# `badges` (e.g. Guest favourite), `image` and a clean room `url`. Filter with
|
|
1210
|
+
# `checkin`/`checkout`, `adults`, `min_price`/`max_price` and `room_type`, and collect up to
|
|
1211
|
+
# `max_results` across pages.
|
|
1212
|
+
#
|
|
1213
|
+
# @param location [String] City to search, e.g. 'Austin, TX' or 'Paris, France'.
|
|
1214
|
+
# @param checkin [String] Check-in date YYYY-MM-DD (with checkout).
|
|
1215
|
+
# @param checkout [String] Check-out date YYYY-MM-DD (with checkin).
|
|
1216
|
+
# @param adults [Integer] Number of adult guests.
|
|
1217
|
+
# @param room_type [String] Room type filter: entire_home, private_room, hotel_room,
|
|
1218
|
+
# shared_room. Omit for all.
|
|
1219
|
+
# @param min_price [Integer] Minimum nightly price.
|
|
1220
|
+
# @param max_price [Integer] Maximum nightly price.
|
|
1221
|
+
# @param max_results [Integer] How many listings to collect across pages. API default: 48.
|
|
1222
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1223
|
+
# 'DE'). Defaults to US so prices are in USD.
|
|
1224
|
+
# Calls POST /travel/airbnb.
|
|
1225
|
+
def airbnb_search(location, checkin: nil, checkout: nil, adults: nil, room_type: nil, min_price: nil, max_price: nil, max_results: nil, proxy_country: nil)
|
|
1226
|
+
post_json("/travel/airbnb", {
|
|
1227
|
+
location: location,
|
|
1228
|
+
checkin: checkin,
|
|
1229
|
+
checkout: checkout,
|
|
1230
|
+
adults: adults,
|
|
1231
|
+
room_type: room_type,
|
|
1232
|
+
min_price: min_price,
|
|
1233
|
+
max_price: max_price,
|
|
1234
|
+
max_results: max_results,
|
|
1235
|
+
proxy_country: proxy_country,
|
|
1236
|
+
})
|
|
1237
|
+
end
|
|
1238
|
+
|
|
1239
|
+
# Scrape an Airbnb listing's full details.
|
|
1240
|
+
#
|
|
1241
|
+
# Fetches one Airbnb stay (by listing id or room URL) and returns it as JSON: `title`,
|
|
1242
|
+
# `description`, `roomType`, `propertyType`, `location` (city, country, `lat`/`lng`), `host`
|
|
1243
|
+
# (name, `isSuperhost`, `responseRate`, `yearsHosting`, photo), `rating` (overall plus
|
|
1244
|
+
# accuracy, cleanliness, check-in, communication, location, value), `reviewsCount`,
|
|
1245
|
+
# `guests`, `bedrooms`, `beds`, `bathrooms`, grouped `amenities`, `photos` with captions,
|
|
1246
|
+
# `highlights`, `houseRules`, `checkIn`/`checkOut` times and `isGuestFavorite`. Add
|
|
1247
|
+
# `check_in`/`check_out` for `available`, the `cancellationPolicy` and `price` (total,
|
|
1248
|
+
# nightly, breakdown) in `currency`; add `include_reviews` for up to `max_reviews` guest
|
|
1249
|
+
# `reviews`. Companion to the Airbnb search plugin (`/travel/airbnb`).
|
|
1250
|
+
#
|
|
1251
|
+
# @param room [String] Airbnb listing id (e.g. 23860926) or a room URL on any Airbnb domain,
|
|
1252
|
+
# e.g. 'https://www.airbnb.co.uk/rooms/23860926'.
|
|
1253
|
+
# @param check_in [String] Check-in date YYYY-MM-DD. With check_out, adds price,
|
|
1254
|
+
# availability and the cancellation policy.
|
|
1255
|
+
# @param check_out [String] Check-out date YYYY-MM-DD (with check_in).
|
|
1256
|
+
# @param adults [Integer] Adult guests (affects the price). API default: 1.
|
|
1257
|
+
# @param children [Integer] Children aged 2-12. API default: 0.
|
|
1258
|
+
# @param infants [Integer] Infants under 2. API default: 0.
|
|
1259
|
+
# @param pets [Integer] Pets. API default: 0.
|
|
1260
|
+
# @param currency [String] Price currency, ISO code such as USD, EUR, GBP. API default: USD.
|
|
1261
|
+
# @param locale [String] Language for Airbnb's texts, e.g. en, de, fr, es, pt-BR. Host-
|
|
1262
|
+
# written text is returned translated when Airbnb has a translation. API default: en.
|
|
1263
|
+
# @param include_reviews [Boolean] Also return guest reviews. API default: false.
|
|
1264
|
+
# @param max_reviews [Integer] How many reviews to return with include_reviews (1-100). API
|
|
1265
|
+
# default: 20.
|
|
1266
|
+
# @param reviews_sort [String] Review order: most_relevant, most_recent, highest_rated,
|
|
1267
|
+
# lowest_rated. API default: most_relevant.
|
|
1268
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1269
|
+
# 'DE'). Defaults to US.
|
|
1270
|
+
# Calls POST /travel/airbnb-room.
|
|
1271
|
+
def airbnb_room(room, check_in: nil, check_out: nil, adults: nil, children: nil, infants: nil, pets: nil, currency: nil, locale: nil, include_reviews: nil, max_reviews: nil, reviews_sort: nil, proxy_country: nil)
|
|
1272
|
+
post_json("/travel/airbnb-room", {
|
|
1273
|
+
room: room,
|
|
1274
|
+
check_in: check_in,
|
|
1275
|
+
check_out: check_out,
|
|
1276
|
+
adults: adults,
|
|
1277
|
+
children: children,
|
|
1278
|
+
infants: infants,
|
|
1279
|
+
pets: pets,
|
|
1280
|
+
currency: currency,
|
|
1281
|
+
locale: locale,
|
|
1282
|
+
include_reviews: include_reviews,
|
|
1283
|
+
max_reviews: max_reviews,
|
|
1284
|
+
reviews_sort: reviews_sort,
|
|
1285
|
+
proxy_country: proxy_country,
|
|
1286
|
+
})
|
|
1287
|
+
end
|
|
1288
|
+
|
|
1289
|
+
# Scrape all reviews of an Airbnb listing.
|
|
1290
|
+
#
|
|
1291
|
+
# Given an Airbnb `room` (listing id or room URL), returns its guest `reviews`: `id`,
|
|
1292
|
+
# `rating`, `text` (as written) with its `language`, `translatedText` (Airbnb's translation
|
|
1293
|
+
# into `locale`), `date` (ISO), `localizedDate`, `stayDetail`, `photos`, `reviewer` (`id`,
|
|
1294
|
+
# `name`, `photo`, `location` or `tenure`, `profileUrl`, `profileId`, `isSuperhost`,
|
|
1295
|
+
# `deleted`) and the host's `response`. `reviewer.id` is Airbnb's account id (the one in its
|
|
1296
|
+
# photo URLs; airbnb.com/users/show/<id> asks anonymous visitors to log in), while
|
|
1297
|
+
# `profileUrl` is the public profile link Airbnb's own review cards open, keyed by a
|
|
1298
|
+
# separate public id given as `profileId`. `deleted` is true when the guest has since closed
|
|
1299
|
+
# their Airbnb account; Airbnb still sends the name, photo and location or tenure from the
|
|
1300
|
+
# review, and they are returned as sent. Reads every review, not just the few on the room
|
|
1301
|
+
# page: page with `page` (or `offset`) and `max_reviews` (up to 500 per call), order with
|
|
1302
|
+
# `sort` (most_relevant, most_recent, highest_rated, lowest_rated) and drop older reviews
|
|
1303
|
+
# with `since`. Also returns the `listing` summary (title, overall and category `rating`,
|
|
1304
|
+
# `ratingDistribution`, `ratingCounts`, `reviewsCount`, `reviewTags`, host) and
|
|
1305
|
+
# `totalResults`, `hasMore`, `nextPage`, `nextOffset`. `ratingDistribution` is Airbnb's own
|
|
1306
|
+
# list of each star's share of the ratings as a 0-1 fraction (`{"stars": 5, "percentage":
|
|
1307
|
+
# 0.957}` = 95.7%), not a count like the other review plugins return; `ratingCounts` gives
|
|
1308
|
+
# the number of ratings per star (`{"5": 692, ...}`) worked out from those shares and
|
|
1309
|
+
# `reviewsCount`, and is null when the shares do not fit it (seen on hotel listings, whose
|
|
1310
|
+
# distribution covers more ratings than `reviewsCount`). Billed one unit per 50-review page
|
|
1311
|
+
# read (`pagesFetched`). Companion to the Airbnb search (`/travel/airbnb`) and listing
|
|
1312
|
+
# (`/travel/airbnb-room`) plugins.
|
|
1313
|
+
#
|
|
1314
|
+
# @param room [String] Airbnb listing id (e.g. 23860926) or a room URL on any Airbnb domain,
|
|
1315
|
+
# e.g. 'https://www.airbnb.co.uk/rooms/23860926'.
|
|
1316
|
+
# @param max_reviews [Integer] How many reviews to return (1-500). Airbnb serves 50 reviews
|
|
1317
|
+
# per request; each request is one billed unit. API default: 50.
|
|
1318
|
+
# @param page [Integer] 1-based page of `max_reviews`-sized pages (1-1000). Use the
|
|
1319
|
+
# `nextPage` of a previous response to continue. API default: 1.
|
|
1320
|
+
# @param offset [Integer] Start at this review position instead of `page` (0-20000). Use the
|
|
1321
|
+
# `nextOffset` of a previous response to resume exactly.
|
|
1322
|
+
# @param sort [String] Review order: most_relevant, most_recent, highest_rated, lowest_rated
|
|
1323
|
+
# (the default is Airbnb's own order). API default: most_relevant.
|
|
1324
|
+
# @param locale [String] Language to translate reviews and host responses into, e.g. en, de,
|
|
1325
|
+
# fr, es, pt-BR. The original text and its language are always returned too. API default:
|
|
1326
|
+
# en.
|
|
1327
|
+
# @param since [String] Only reviews written on or after this date (YYYY-MM-DD). With
|
|
1328
|
+
# sort=most_recent it also stops paging once older reviews start.
|
|
1329
|
+
# @param include_listing [Boolean] Also return the listing summary (title, overall and
|
|
1330
|
+
# category ratings, rating distribution, host). Set false to skip that request. API default:
|
|
1331
|
+
# true.
|
|
1332
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1333
|
+
# 'DE').
|
|
1334
|
+
# Calls POST /travel/airbnb-reviews.
|
|
1335
|
+
def airbnb_reviews(room, max_reviews: nil, page: nil, offset: nil, sort: nil, locale: nil, since: nil, include_listing: nil, proxy_country: nil)
|
|
1336
|
+
post_json("/travel/airbnb-reviews", {
|
|
1337
|
+
room: room,
|
|
1338
|
+
max_reviews: max_reviews,
|
|
1339
|
+
page: page,
|
|
1340
|
+
offset: offset,
|
|
1341
|
+
sort: sort,
|
|
1342
|
+
locale: locale,
|
|
1343
|
+
since: since,
|
|
1344
|
+
include_listing: include_listing,
|
|
1345
|
+
proxy_country: proxy_country,
|
|
1346
|
+
})
|
|
1347
|
+
end
|
|
1348
|
+
|
|
1349
|
+
# Scrape a TripAdvisor list (restaurants / hotels / attractions).
|
|
1350
|
+
#
|
|
1351
|
+
# Given a TripAdvisor list `url` (Restaurants, Hotels or Attractions for a city/area),
|
|
1352
|
+
# returns the places as JSON: `name`, `url`, `image`, `rating`, `reviewCount`, `priceRange`,
|
|
1353
|
+
# `cuisine`, `phone`, `address`, `country` and `coordinates`. Paginates through
|
|
1354
|
+
# TripAdvisor's `-oaN-` pages up to `max_results`. Data is read from TripAdvisor's
|
|
1355
|
+
# schema.org ld+json, which is more stable than its HTML.
|
|
1356
|
+
#
|
|
1357
|
+
# @param url [String] TripAdvisor list URL (Restaurants / Hotels / Attractions), e.g.
|
|
1358
|
+
# https://www.tripadvisor.com/Restaurants-g60763-New_York_City_New_York.html
|
|
1359
|
+
# @param max_results [Integer] How many places to collect across pages (~30 per page, max
|
|
1360
|
+
# 1020). TripAdvisor renders slowly, so large values take a while. API default: 30.
|
|
1361
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
1362
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1363
|
+
# 'DE'). Leave blank to let the API pick.
|
|
1364
|
+
# Calls POST /travel/tripadvisor.
|
|
1365
|
+
def tripadvisor_list(url, max_results: nil, page: nil, proxy_country: nil)
|
|
1366
|
+
post_json("/travel/tripadvisor", {
|
|
1367
|
+
url: url,
|
|
1368
|
+
max_results: max_results,
|
|
1369
|
+
page: page,
|
|
1370
|
+
proxy_country: proxy_country,
|
|
1371
|
+
})
|
|
1372
|
+
end
|
|
1373
|
+
|
|
1374
|
+
# Scrape TripAdvisor reviews (hotels / restaurants / attractions).
|
|
1375
|
+
#
|
|
1376
|
+
# Given a TripAdvisor place review `url` (Hotel_Review, Restaurant_Review, Attraction_Review
|
|
1377
|
+
# or AttractionProductReview, any TripAdvisor site) or a `location_id`, returns the place
|
|
1378
|
+
# summary (`name`, `type`, `rating`, `reviewsCount`, `ratingHistogram`, `subratings`,
|
|
1379
|
+
# `address`, `coordinates`, `phone`, `website`) and its `reviews`: `title`, `text`,
|
|
1380
|
+
# `rating`, `publishedDate`, `travelDate`, `tripType`, `language`, `helpfulVotes`, `user`,
|
|
1381
|
+
# `subratings`, `photos` and `ownerResponse`. Newest first; pages through TripAdvisor's
|
|
1382
|
+
# review pages up to `max_reviews`. Pick the review language with `language` and cut off old
|
|
1383
|
+
# reviews with `since`. Billed one unit per review page fetched (`pagesFetched`).
|
|
1384
|
+
#
|
|
1385
|
+
# @param url [String] TripAdvisor place review page on any TripAdvisor site: Hotel_Review,
|
|
1386
|
+
# Restaurant_Review, Attraction_Review or AttractionProductReview, e.g.
|
|
1387
|
+
# https://www.tripadvisor.com/Hotel_Review-g60763-d93589-Reviews-The_Michelangelo_New_York-
|
|
1388
|
+
# New_York_City_New_York.html. Pass this or `location_id`.
|
|
1389
|
+
# @param location_id [Integer] TripAdvisor location id (the digits after -d in a place URL,
|
|
1390
|
+
# e.g. 93589). Pass this or `url`.
|
|
1391
|
+
# @param max_reviews [Integer] How many reviews to collect across pages (1-200). TripAdvisor
|
|
1392
|
+
# shows 10 reviews per page (15 for restaurants); each page is one billed unit. API default:
|
|
1393
|
+
# 20.
|
|
1394
|
+
# @param page [Integer] 1-based review page to start from (10 reviews per page, 15 for
|
|
1395
|
+
# restaurants). Use the `nextPage` of a previous response to continue. API default: 1.
|
|
1396
|
+
# @param language [String] Review language, which TripAdvisor ties to its site: en, de, fr,
|
|
1397
|
+
# it, es, pt, nl, ja, ko, ru, sv, tr, da. Default: the language of the URL's site
|
|
1398
|
+
# (tripadvisor.com = en), or - when that site lists no reviews for the place - the place's
|
|
1399
|
+
# most-reviewed of these languages. Reviews written in other languages appear machine-
|
|
1400
|
+
# translated, flagged by `isMachineTranslated` and `originalLanguage`.
|
|
1401
|
+
# @param since [String] Only reviews published on or after this date (YYYY-MM-DD). Stops
|
|
1402
|
+
# paging once older reviews start, so it also saves pages.
|
|
1403
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1404
|
+
# 'DE'). Leave blank: pinning a country makes the call much slower.
|
|
1405
|
+
# Calls POST /travel/tripadvisor-reviews.
|
|
1406
|
+
def tripadvisor_reviews(url = nil, location_id: nil, max_reviews: nil, page: nil, language: nil, since: nil, proxy_country: nil)
|
|
1407
|
+
post_json("/travel/tripadvisor-reviews", {
|
|
1408
|
+
url: url,
|
|
1409
|
+
location_id: location_id,
|
|
1410
|
+
max_reviews: max_reviews,
|
|
1411
|
+
page: page,
|
|
1412
|
+
language: language,
|
|
1413
|
+
since: since,
|
|
1414
|
+
proxy_country: proxy_country,
|
|
1415
|
+
})
|
|
1416
|
+
end
|
|
1417
|
+
|
|
1418
|
+
# Scrape Google Shopping product listings.
|
|
1419
|
+
#
|
|
1420
|
+
# Runs the Google Shopping vertical (`udm=28`) for `q` and returns the product cards as
|
|
1421
|
+
# JSON: `title`, `price`, `currency`, `originalPrice`, `discount`, `merchant`, `rating` and
|
|
1422
|
+
# `reviewCount`. Set `proxy_country`/`gl` to target a market (prices/merchants are region-
|
|
1423
|
+
# specific) and `min_price`/`max_price` to filter. One render returns the first page of
|
|
1424
|
+
# products (v1). Note: Google Shopping exposes no per-product URL, so `url` is omitted and
|
|
1425
|
+
# `image` is best-effort.
|
|
1426
|
+
#
|
|
1427
|
+
# @param q [String] Product search term, e.g. 'wireless headphones'.
|
|
1428
|
+
# @param gl [String] Market of search, ISO-2, e.g. 'us', 'de'. Sets prices/merchants. API
|
|
1429
|
+
# default: us.
|
|
1430
|
+
# @param min_price [Integer] Minimum price filter (whole units, gl currency).
|
|
1431
|
+
# @param max_price [Integer] Maximum price filter (whole units, gl currency).
|
|
1432
|
+
# @param max_results [Integer] How many products to return (v1: first page). API default:
|
|
1433
|
+
# 60.
|
|
1434
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1435
|
+
# 'DE'). Prices/merchants are market-specific.
|
|
1436
|
+
# Calls POST /goods/google-shopping.
|
|
1437
|
+
def google_shopping(q, gl: nil, min_price: nil, max_price: nil, max_results: nil, proxy_country: nil)
|
|
1438
|
+
post_json("/goods/google-shopping", {
|
|
1439
|
+
q: q,
|
|
1440
|
+
gl: gl,
|
|
1441
|
+
min_price: min_price,
|
|
1442
|
+
max_price: max_price,
|
|
1443
|
+
max_results: max_results,
|
|
1444
|
+
proxy_country: proxy_country,
|
|
1445
|
+
})
|
|
1446
|
+
end
|
|
1447
|
+
|
|
1448
|
+
# Scrape Google Jobs listings.
|
|
1449
|
+
#
|
|
1450
|
+
# Runs the Google Jobs vertical (`udm=8`) for `q` and returns the job cards as JSON:
|
|
1451
|
+
# `title`, `company`, `location`, `via` (source), `salary`, `employmentType`, `remote`,
|
|
1452
|
+
# `postedAge`, `qualifications`, `benefits`, `logo`, a stable `jobId` and a `shareUrl`.
|
|
1453
|
+
# Append a `location` to narrow the search, and set `proxy_country`/`gl`/`hl` to target a
|
|
1454
|
+
# market. One render returns ~10 jobs (v1).
|
|
1455
|
+
#
|
|
1456
|
+
# @param q [String] Job search term, e.g. 'software engineer' or 'nurse'.
|
|
1457
|
+
# @param location [String] Optional location, appended to the query, e.g. 'Austin, TX'.
|
|
1458
|
+
# @param hl [String] Google UI language, e.g. 'en', 'de'. API default: en.
|
|
1459
|
+
# @param gl [String] Google country of search, ISO-2, e.g. 'us'. API default: us.
|
|
1460
|
+
# @param max_results [Integer] How many jobs to return. One render yields ~10 (v1). API
|
|
1461
|
+
# default: 10.
|
|
1462
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1463
|
+
# 'DE'). Jobs are location-sensitive; pick the market you want.
|
|
1464
|
+
# Calls POST /jobs/google-jobs.
|
|
1465
|
+
def google_jobs(q, location: nil, hl: nil, gl: nil, max_results: nil, proxy_country: nil)
|
|
1466
|
+
post_json("/jobs/google-jobs", {
|
|
1467
|
+
q: q,
|
|
1468
|
+
location: location,
|
|
1469
|
+
hl: hl,
|
|
1470
|
+
gl: gl,
|
|
1471
|
+
max_results: max_results,
|
|
1472
|
+
proxy_country: proxy_country,
|
|
1473
|
+
})
|
|
1474
|
+
end
|
|
1475
|
+
|
|
1476
|
+
# Scrape Google Trends (interest over time, by region, related queries).
|
|
1477
|
+
#
|
|
1478
|
+
# Compares 1-5 `keywords` on Google Trends and returns the data behind the explore page as
|
|
1479
|
+
# JSON: `interestOverTime` (one point per time bucket with a 0-100 value per keyword, ISO
|
|
1480
|
+
# `date`, `timestamp`, `isPartial`), `averages`, `interestByRegion` (`geoCode`, `geoName`,
|
|
1481
|
+
# values per keyword), and per keyword `relatedQueries` and `relatedTopics` (opt-in via
|
|
1482
|
+
# `sections`), each with `top` and `rising` lists. Filter with `geo` (worldwide, country,
|
|
1483
|
+
# region or US metro), `timeframe` (presets such as 'today 12-m' or a custom date range),
|
|
1484
|
+
# `category` and `property` (web, images, news, youtube, shopping). Use `sections` to fetch
|
|
1485
|
+
# only what you need. Billed as one request per keyword compared.
|
|
1486
|
+
#
|
|
1487
|
+
# @param keywords [String, Array<String>] 1-5 search terms to compare, comma-separated, e.g.
|
|
1488
|
+
# 'coffee,tea'. A Knowledge Graph topic id such as '/m/0dl567' also works.
|
|
1489
|
+
# @param geo [String] Where: empty = worldwide, an ISO-2 country ('US'), a region ('US-CA')
|
|
1490
|
+
# or a US metro ('US-CA-807').
|
|
1491
|
+
# @param timeframe [String] Time range: now 1-H, now 4-H, now 1-d, now 7-d, today 1-m, today
|
|
1492
|
+
# 3-m, today 12-m, today 5-y, all, or a custom 'YYYY-MM-DD YYYY-MM-DD' (hourly for the last
|
|
1493
|
+
# 7 days: 'YYYY-MM-DDTHH YYYY-MM-DDTHH'). API default: today 12-m.
|
|
1494
|
+
# @param category [Integer] Google Trends category id (0 = all categories, e.g. 71 = Food &
|
|
1495
|
+
# Drink, 5 = Computers & Electronics). API default: 0.
|
|
1496
|
+
# @param property [String] Which Google search to measure: web, images, news, youtube,
|
|
1497
|
+
# shopping. API default: web.
|
|
1498
|
+
# @param hl [String] Language of labels (region names, topic titles), e.g. 'en-US', 'de',
|
|
1499
|
+
# 'fr'. API default: en-US.
|
|
1500
|
+
# @param tz [Integer] Timezone offset in minutes that day/hour buckets align to (Trends'
|
|
1501
|
+
# convention: 300 = UTC-5, -60 = UTC+1). Default 0 = UTC. API default: 0.
|
|
1502
|
+
# @param region_resolution [String] Level of the interest-by-region map: country, region,
|
|
1503
|
+
# city, dma. Default = Trends' own choice (countries worldwide, regions in a country).
|
|
1504
|
+
# @param include_low_volume [Boolean] Include regions with low search volume in the region
|
|
1505
|
+
# map. API default: false.
|
|
1506
|
+
# @param sections [String] What to fetch: a comma list of interest_over_time,
|
|
1507
|
+
# interest_by_region, related_queries, related_topics, or 'all'. Fewer sections = faster.
|
|
1508
|
+
# related_topics is opt-in (Google often returns it empty). API default:
|
|
1509
|
+
# interest_over_time,interest_by_region,related_queries.
|
|
1510
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1511
|
+
# 'DE'). Optional - by default a country outside the US and Europe is used; results follow
|
|
1512
|
+
# `geo`, not the proxy country.
|
|
1513
|
+
# Calls POST /trends/google-trends.
|
|
1514
|
+
def google_trends(keywords, geo: nil, timeframe: nil, category: nil, property: nil, hl: nil, tz: nil, region_resolution: nil, include_low_volume: nil, sections: nil, proxy_country: nil)
|
|
1515
|
+
post_json("/trends/google-trends", {
|
|
1516
|
+
keywords: plugin_join(keywords, ","),
|
|
1517
|
+
geo: geo,
|
|
1518
|
+
timeframe: timeframe,
|
|
1519
|
+
category: category,
|
|
1520
|
+
property: property,
|
|
1521
|
+
hl: hl,
|
|
1522
|
+
tz: tz,
|
|
1523
|
+
region_resolution: region_resolution,
|
|
1524
|
+
include_low_volume: include_low_volume,
|
|
1525
|
+
sections: sections,
|
|
1526
|
+
proxy_country: proxy_country,
|
|
1527
|
+
})
|
|
1528
|
+
end
|
|
1529
|
+
|
|
1530
|
+
# Scrape Google Maps place details.
|
|
1531
|
+
#
|
|
1532
|
+
# Returns the full data for one Google Maps place as JSON: `name`, `address` (+
|
|
1533
|
+
# `addressLines`), `phone`, `website`, `rating`, `categories`, `latitude`/`longitude`,
|
|
1534
|
+
# `plusCode`, `hours` + `openNow`, `description`, `photo`, `cid`, `featureId`,
|
|
1535
|
+
# `googleMapsUrl` and `relatedPlaces`. Identify the place with `place_id` (ChIJ...), a Maps
|
|
1536
|
+
# `url`, or a raw `feature_id` (0xHEX:0xHEX). Location-neutral - the feature id resolves the
|
|
1537
|
+
# place directly; `proxy_country` defaults to a US proxy to clear Google's consent wall.
|
|
1538
|
+
#
|
|
1539
|
+
# @param url [String] A Google Maps place URL (share link or /maps/place/... URL). The
|
|
1540
|
+
# feature id or place_id is read out of it.
|
|
1541
|
+
# @param place_id [String] A Google place_id, e.g. 'ChIJN1t_tDeuEmsRUsoyG83frY4'.
|
|
1542
|
+
# @param feature_id [String] A Maps feature id '0xHEX:0xHEX', e.g.
|
|
1543
|
+
# '0x47e66e2964e34e2d:0x8ddca9ee380ef7e0'. Fastest - no resolve step.
|
|
1544
|
+
# @param hl [String] Google UI language, e.g. 'en', 'de', 'fr'. API default: en.
|
|
1545
|
+
# @param gl [String] Google country, ISO-2 lowercase, e.g. 'us'. API default: us.
|
|
1546
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1547
|
+
# 'DE'). Defaults to US - an EU country hits Google's consent wall. API default: us.
|
|
1548
|
+
# @param raw [Boolean] Include the decoded place core array under `raw`. API default: false.
|
|
1549
|
+
# Calls POST /maps/place-details.
|
|
1550
|
+
def maps_place_details(url: nil, place_id: nil, feature_id: nil, hl: nil, gl: nil, proxy_country: nil, raw: nil)
|
|
1551
|
+
post_json("/maps/place-details", {
|
|
1552
|
+
url: url,
|
|
1553
|
+
place_id: place_id,
|
|
1554
|
+
feature_id: feature_id,
|
|
1555
|
+
hl: hl,
|
|
1556
|
+
gl: gl,
|
|
1557
|
+
proxy_country: proxy_country,
|
|
1558
|
+
raw: raw,
|
|
1559
|
+
})
|
|
1560
|
+
end
|
|
1561
|
+
|
|
1562
|
+
# Search Bing.
|
|
1563
|
+
#
|
|
1564
|
+
# Searches Bing for `keyword` in a real browser and returns the results page as JSON:
|
|
1565
|
+
# `topAds`, `bottomAds`, `organic` (each with `position`, `title`, `url`, `displayedUrl`,
|
|
1566
|
+
# `domain`, `description`), `relatedSearches` and `totalResults`. Same parameters and
|
|
1567
|
+
# response shape as `/serpApi`. `proxy_country` sets both the market and the location the
|
|
1568
|
+
# search is sent from, which decides the ads.
|
|
1569
|
+
#
|
|
1570
|
+
# @param keyword [String] Search terms.
|
|
1571
|
+
# @param pages_to_check [Integer] How many result pages to collect (1-10, ~10 results each).
|
|
1572
|
+
# API default: 1.
|
|
1573
|
+
# @param proxy_country [String] Country to search from (ISO-2). Sets the result market and
|
|
1574
|
+
# the location the search is sent from, which decides the ads. Defaults to US.
|
|
1575
|
+
# @param language [String] Interface language (ISO-639-1, e.g. en, de). Defaults to en.
|
|
1576
|
+
# Calls POST /search/bing.
|
|
1577
|
+
def bing_search(keyword, pages_to_check: nil, proxy_country: nil, language: nil)
|
|
1578
|
+
post_json("/search/bing", {
|
|
1579
|
+
keyword: keyword,
|
|
1580
|
+
pages_to_check: pages_to_check,
|
|
1581
|
+
proxy_country: proxy_country,
|
|
1582
|
+
language: language,
|
|
1583
|
+
})
|
|
1584
|
+
end
|
|
1585
|
+
|
|
1586
|
+
# Search Bing Images.
|
|
1587
|
+
#
|
|
1588
|
+
# Runs Bing Images for `q` in a real browser and returns the image results as JSON, in the
|
|
1589
|
+
# same shape as `/images/google-search`. Every result carries `imageUrl` (the origin image,
|
|
1590
|
+
# not a Bing thumbnail), `sourceDomain`, `sourcePageUrl`, `title`, `sourceName`,
|
|
1591
|
+
# `thumbnailUrl`, `width`, `height` and `bingImagesUrl` (Bing's image viewer link). `pages`
|
|
1592
|
+
# fetches up to 10 result pages of ~35 each; every page fetched is billed as one request.
|
|
1593
|
+
#
|
|
1594
|
+
# @param q [String] Image search term, e.g. 'eiffel tower'.
|
|
1595
|
+
# @param pages [Integer] How many Bing Images result pages to fetch (1-10, ~35 results
|
|
1596
|
+
# each). Each page fetched is billed as one request. API default: 1.
|
|
1597
|
+
# @param max_results [Integer] Optional cap on the number of results returned (default: all
|
|
1598
|
+
# results of the fetched pages).
|
|
1599
|
+
# @param proxy_country [String] Country to search from (ISO-2); sets the Bing market.
|
|
1600
|
+
# Defaults to US.
|
|
1601
|
+
# @param language [String] Interface language (ISO-639-1, e.g. en, de). Defaults to en.
|
|
1602
|
+
# Calls POST /images/bing-search.
|
|
1603
|
+
def bing_images(q, pages: nil, max_results: nil, proxy_country: nil, language: nil)
|
|
1604
|
+
post_json("/images/bing-search", {
|
|
1605
|
+
q: q,
|
|
1606
|
+
pages: pages,
|
|
1607
|
+
max_results: max_results,
|
|
1608
|
+
proxy_country: proxy_country,
|
|
1609
|
+
language: language,
|
|
1610
|
+
})
|
|
1611
|
+
end
|
|
1612
|
+
|
|
1613
|
+
# Search DuckDuckGo.
|
|
1614
|
+
#
|
|
1615
|
+
# Searches DuckDuckGo for `keyword` and returns the results as JSON: `topAds`, `bottomAds`,
|
|
1616
|
+
# `organic` (each with `position`, `title`, `url`, `displayedUrl`, `domain`, `description`)
|
|
1617
|
+
# and `relatedSearches`. Same parameters and response shape as `/serpApi`. The first page is
|
|
1618
|
+
# rendered in a real browser and carries the ads; further pages add organic results.
|
|
1619
|
+
#
|
|
1620
|
+
# @param keyword [String] Search terms.
|
|
1621
|
+
# @param pages_to_check [Integer] How many result pages to collect (1-10, ~10 results each).
|
|
1622
|
+
# API default: 1.
|
|
1623
|
+
# @param proxy_country [String] Country to search from (ISO-2). Sets the result market and
|
|
1624
|
+
# the location the search is sent from, which decides the ads. Defaults to US.
|
|
1625
|
+
# @param language [String] Interface language (ISO-639-1, e.g. en, de). Defaults to en.
|
|
1626
|
+
# Calls POST /search/duckduckgo.
|
|
1627
|
+
def duckduckgo_search(keyword, pages_to_check: nil, proxy_country: nil, language: nil)
|
|
1628
|
+
post_json("/search/duckduckgo", {
|
|
1629
|
+
keyword: keyword,
|
|
1630
|
+
pages_to_check: pages_to_check,
|
|
1631
|
+
proxy_country: proxy_country,
|
|
1632
|
+
language: language,
|
|
1633
|
+
})
|
|
1634
|
+
end
|
|
1635
|
+
|
|
1636
|
+
# Search Yahoo.
|
|
1637
|
+
#
|
|
1638
|
+
# Searches Yahoo for `keyword` in a real browser and returns the results page as JSON:
|
|
1639
|
+
# `topAds`, `bottomAds`, `organic` (each with `position`, `title`, `url`, `displayedUrl`,
|
|
1640
|
+
# `domain`, `description`) and `relatedSearches`. Same parameters and response shape as
|
|
1641
|
+
# `/serpApi`. `proxy_country` picks Yahoo's regional site and the location the search is
|
|
1642
|
+
# sent from, which decides the ads; `language`, when set, restricts results to that
|
|
1643
|
+
# language.
|
|
1644
|
+
#
|
|
1645
|
+
# @param keyword [String] Search terms.
|
|
1646
|
+
# @param pages_to_check [Integer] How many result pages to collect (1-10, ~10 results each).
|
|
1647
|
+
# API default: 1.
|
|
1648
|
+
# @param proxy_country [String] Country to search from (ISO-2). Sets the result market and
|
|
1649
|
+
# the location the search is sent from, which decides the ads. Defaults to US.
|
|
1650
|
+
# @param language [String] Interface language (ISO-639-1, e.g. en, de). Defaults to en.
|
|
1651
|
+
# Calls POST /search/yahoo.
|
|
1652
|
+
def yahoo_search(keyword, pages_to_check: nil, proxy_country: nil, language: nil)
|
|
1653
|
+
post_json("/search/yahoo", {
|
|
1654
|
+
keyword: keyword,
|
|
1655
|
+
pages_to_check: pages_to_check,
|
|
1656
|
+
proxy_country: proxy_country,
|
|
1657
|
+
language: language,
|
|
1658
|
+
})
|
|
1659
|
+
end
|
|
1660
|
+
|
|
1661
|
+
# Search Yahoo Images.
|
|
1662
|
+
#
|
|
1663
|
+
# Runs Yahoo Images for `q` in a real browser and returns the image results as JSON, in the
|
|
1664
|
+
# same shape as `/images/google-search`. Every result carries `imageUrl` (the origin image,
|
|
1665
|
+
# not a thumbnail), `sourceDomain`, `sourcePageUrl`, `title`, `sourceName`, `thumbnailUrl`,
|
|
1666
|
+
# `width`, `height` and `yahooImagesUrl` (Yahoo's image viewer link). `pages` fetches up to
|
|
1667
|
+
# 10 result pages of ~60 each; every page fetched is billed as one request.
|
|
1668
|
+
#
|
|
1669
|
+
# @param q [String] Image search term, e.g. 'eiffel tower'.
|
|
1670
|
+
# @param pages [Integer] How many Yahoo Images result pages to fetch (1-10, ~60 results
|
|
1671
|
+
# each). Each page fetched is billed as one request. API default: 1.
|
|
1672
|
+
# @param max_results [Integer] Optional cap on the number of results returned (default: all
|
|
1673
|
+
# results of the fetched pages).
|
|
1674
|
+
# @param proxy_country [String] Country to search from (ISO-2); picks Yahoo's regional site.
|
|
1675
|
+
# Defaults to US.
|
|
1676
|
+
# Calls POST /images/yahoo-search.
|
|
1677
|
+
def yahoo_images(q, pages: nil, max_results: nil, proxy_country: nil)
|
|
1678
|
+
post_json("/images/yahoo-search", {
|
|
1679
|
+
q: q,
|
|
1680
|
+
pages: pages,
|
|
1681
|
+
max_results: max_results,
|
|
1682
|
+
proxy_country: proxy_country,
|
|
1683
|
+
})
|
|
1684
|
+
end
|
|
1685
|
+
|
|
1686
|
+
# Search Brave.
|
|
1687
|
+
#
|
|
1688
|
+
# Searches Brave Search for `keyword` in a real browser and returns the results as JSON:
|
|
1689
|
+
# `topAds`, `bottomAds`, `organic` (each with `position`, `title`, `url`, `displayedUrl`,
|
|
1690
|
+
# `domain`, `description`) and `relatedSearches`. Same parameters and response shape as
|
|
1691
|
+
# `/serpApi`; ~20 organic results per page. Brave picks the result region from the
|
|
1692
|
+
# searcher's location, so `proxy_country` sets it through the proxy's location.
|
|
1693
|
+
#
|
|
1694
|
+
# @param keyword [String] Search terms.
|
|
1695
|
+
# @param pages_to_check [Integer] How many result pages to collect (1-10, ~10 results each).
|
|
1696
|
+
# API default: 1.
|
|
1697
|
+
# @param proxy_country [String] Country to search from (ISO-2). Sets the result market and
|
|
1698
|
+
# the location the search is sent from, which decides the ads. Defaults to US.
|
|
1699
|
+
# @param language [String] Interface language (ISO-639-1, e.g. en, de). Defaults to en.
|
|
1700
|
+
# Calls POST /search/brave.
|
|
1701
|
+
def brave_search(keyword, pages_to_check: nil, proxy_country: nil, language: nil)
|
|
1702
|
+
post_json("/search/brave", {
|
|
1703
|
+
keyword: keyword,
|
|
1704
|
+
pages_to_check: pages_to_check,
|
|
1705
|
+
proxy_country: proxy_country,
|
|
1706
|
+
language: language,
|
|
1707
|
+
})
|
|
1708
|
+
end
|
|
1709
|
+
|
|
1710
|
+
# Search Brave Images.
|
|
1711
|
+
#
|
|
1712
|
+
# Runs Brave Images for `q` in a real browser and returns the image results as JSON, in the
|
|
1713
|
+
# same shape as `/images/google-search`. Every result carries `imageUrl` (the origin image,
|
|
1714
|
+
# not a thumbnail), `sourceDomain`, `sourcePageUrl`, `title`, `sourceName`, `thumbnailUrl`,
|
|
1715
|
+
# `width` and `height`. Brave returns its whole result set (~200 images) on one page, billed
|
|
1716
|
+
# as one request; `max_results` caps it.
|
|
1717
|
+
#
|
|
1718
|
+
# @param q [String] Image search term, e.g. 'eiffel tower'.
|
|
1719
|
+
# @param max_results [Integer] Optional cap on the number of results returned (default: all,
|
|
1720
|
+
# ~200).
|
|
1721
|
+
# @param proxy_country [String] Country to search from (ISO-2); Brave takes the region from
|
|
1722
|
+
# it. Defaults to US.
|
|
1723
|
+
# Calls POST /images/brave-search.
|
|
1724
|
+
def brave_images(q, max_results: nil, proxy_country: nil)
|
|
1725
|
+
post_json("/images/brave-search", {
|
|
1726
|
+
q: q,
|
|
1727
|
+
max_results: max_results,
|
|
1728
|
+
proxy_country: proxy_country,
|
|
1729
|
+
})
|
|
1730
|
+
end
|
|
1731
|
+
|
|
1732
|
+
# Scrape Zillow real-estate listings.
|
|
1733
|
+
#
|
|
1734
|
+
# Searches Zillow for `location` (a city/state or ZIP) and returns the listings as JSON:
|
|
1735
|
+
# `price`, `beds`, `baths`, `area`, `address`, `homeType`, `zestimate`, `rentZestimate`,
|
|
1736
|
+
# `daysOnZillow`, coordinates, broker and a clean `url`. Filter with `status` (for_sale /
|
|
1737
|
+
# for_rent / sold), `min_price`/`max_price`, `beds`, `baths` and `home_type`, sort with
|
|
1738
|
+
# `sort`, and collect up to `max_results` across pages. Rental results can include apartment
|
|
1739
|
+
# buildings (`isBuilding: true`): they have no `zpid` and carry `lotId`, `buildingName`, a
|
|
1740
|
+
# rent range (`price` to `maxPrice`), `units` and `availableUnits` instead.
|
|
1741
|
+
#
|
|
1742
|
+
# @param location [String] City/state or ZIP to search, e.g. 'Austin, TX' or '78704'.
|
|
1743
|
+
# @param status [String] Listing status: for_sale, for_rent, sold. API default: for_sale.
|
|
1744
|
+
# @param max_results [Integer] How many listings to collect across pages (~41 per page). API
|
|
1745
|
+
# default: 41.
|
|
1746
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
1747
|
+
# @param sort [String] Ordering: newest, price_asc, price_desc, relevant, beds, sqft, lot.
|
|
1748
|
+
# API default: newest.
|
|
1749
|
+
# @param home_type [String] Property type filter: houses, condos, townhomes, multi_family,
|
|
1750
|
+
# apartments, manufactured, lots_land. Omit for all.
|
|
1751
|
+
# @param min_price [Integer] Minimum price (or monthly rent).
|
|
1752
|
+
# @param max_price [Integer] Maximum price (or monthly rent).
|
|
1753
|
+
# @param beds [Integer] Minimum bedrooms.
|
|
1754
|
+
# @param baths [Integer] Minimum bathrooms.
|
|
1755
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1756
|
+
# 'DE'). Defaults to US.
|
|
1757
|
+
# Calls POST /realestate/zillow.
|
|
1758
|
+
def zillow_search(location, status: nil, max_results: nil, page: nil, sort: nil, home_type: nil, min_price: nil, max_price: nil, beds: nil, baths: nil, proxy_country: nil)
|
|
1759
|
+
post_json("/realestate/zillow", {
|
|
1760
|
+
location: location,
|
|
1761
|
+
status: status,
|
|
1762
|
+
max_results: max_results,
|
|
1763
|
+
page: page,
|
|
1764
|
+
sort: sort,
|
|
1765
|
+
home_type: home_type,
|
|
1766
|
+
min_price: min_price,
|
|
1767
|
+
max_price: max_price,
|
|
1768
|
+
beds: beds,
|
|
1769
|
+
baths: baths,
|
|
1770
|
+
proxy_country: proxy_country,
|
|
1771
|
+
})
|
|
1772
|
+
end
|
|
1773
|
+
|
|
1774
|
+
# Scrape a single Zillow property's full detail.
|
|
1775
|
+
#
|
|
1776
|
+
# Fetches one Zillow property (by `url` or `zpid`) and returns the full record as JSON:
|
|
1777
|
+
# `price`, `zestimate`, `rentZestimate`, `beds`, `baths`, `area`, `lotSize`, `yearBuilt`,
|
|
1778
|
+
# `homeType`, `status`, `description`, `priceHistory`, `taxHistory`, `schools`, `facts`,
|
|
1779
|
+
# `parking`, listing `brokerName`/`listingAgent`, `mlsid`, `monthlyHoaFee`, `pricePerSqft`,
|
|
1780
|
+
# `daysOnZillow`, `taxAssessedValue`, coordinates, the `photos` gallery and
|
|
1781
|
+
# `virtualTourUrl`. Every field is always present and is `null` when Zillow does not show it
|
|
1782
|
+
# for that listing (for-sale and sold pages carry fewer sections than rentals). Companion to
|
|
1783
|
+
# the Zillow search plugin (`/realestate/zillow`).
|
|
1784
|
+
#
|
|
1785
|
+
# @param url [String] Zillow property URL, e.g.
|
|
1786
|
+
# 'https://www.zillow.com/homedetails/.../29432984_zpid/'. Either url or zpid is required.
|
|
1787
|
+
# @param zpid [String] Zillow property id (zpid), e.g. '29432984'. Alternative to url.
|
|
1788
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1789
|
+
# 'DE'). Leave blank to let the API pick.
|
|
1790
|
+
# Calls POST /realestate/zillow-detail.
|
|
1791
|
+
def zillow_detail(url: nil, zpid: nil, proxy_country: nil)
|
|
1792
|
+
post_json("/realestate/zillow-detail", {
|
|
1793
|
+
url: url,
|
|
1794
|
+
zpid: zpid,
|
|
1795
|
+
proxy_country: proxy_country,
|
|
1796
|
+
})
|
|
1797
|
+
end
|
|
1798
|
+
|
|
1799
|
+
# Scrape Redfin real-estate listings.
|
|
1800
|
+
#
|
|
1801
|
+
# Searches Redfin for `location` (a city/state or ZIP) and returns the listings as JSON:
|
|
1802
|
+
# `price`, `beds`, `baths`, `sqFt`, `lotSize`, address, coordinates, `propertyType`,
|
|
1803
|
+
# `mlsStatus`, `mlsId`, `daysOnMarket`, `pricePerSqFt` and a clean `url`. Filter with
|
|
1804
|
+
# `status` (for_sale / sold), `min_price`/`max_price`, `beds`, `baths` and `property_type`,
|
|
1805
|
+
# sort with `sort`, and collect up to `max_results` across pages. Use `mode=detail` with a
|
|
1806
|
+
# listing `url` for one home in full (agents, public records, tax, price history, Redfin
|
|
1807
|
+
# Estimate).
|
|
1808
|
+
#
|
|
1809
|
+
# @param location [String] City/state or ZIP to search, e.g. 'Austin, TX' or '78704'. Search
|
|
1810
|
+
# mode.
|
|
1811
|
+
# @param status [String] Listing status: for_sale, sold. API default: for_sale.
|
|
1812
|
+
# @param max_results [Integer] How many listings to collect across pages (~350 per page).
|
|
1813
|
+
# API default: 350.
|
|
1814
|
+
# @param sort [String] Ordering: recommended, newest, price_asc, price_desc, sqft,
|
|
1815
|
+
# price_per_sqft. API default: recommended.
|
|
1816
|
+
# @param property_type [String] Property type filter: houses, condos, townhomes,
|
|
1817
|
+
# multi_family, land, manufactured, other. Omit for all.
|
|
1818
|
+
# @param min_price [Integer] Minimum price.
|
|
1819
|
+
# @param max_price [Integer] Maximum price.
|
|
1820
|
+
# @param beds [Integer] Minimum bedrooms.
|
|
1821
|
+
# @param baths [Integer] Minimum bathrooms.
|
|
1822
|
+
# @param sold_within_days [Integer] For status=sold: look back this many days (default 90).
|
|
1823
|
+
# API default: 90.
|
|
1824
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1825
|
+
# 'DE'). Defaults to US.
|
|
1826
|
+
# Calls POST /realestate/redfin.
|
|
1827
|
+
def redfin_search(location, status: nil, max_results: nil, sort: nil, property_type: nil, min_price: nil, max_price: nil, beds: nil, baths: nil, sold_within_days: nil, proxy_country: nil)
|
|
1828
|
+
post_json("/realestate/redfin", {
|
|
1829
|
+
mode: "search",
|
|
1830
|
+
location: location,
|
|
1831
|
+
status: status,
|
|
1832
|
+
max_results: max_results,
|
|
1833
|
+
sort: sort,
|
|
1834
|
+
property_type: property_type,
|
|
1835
|
+
min_price: min_price,
|
|
1836
|
+
max_price: max_price,
|
|
1837
|
+
beds: beds,
|
|
1838
|
+
baths: baths,
|
|
1839
|
+
sold_within_days: sold_within_days,
|
|
1840
|
+
proxy_country: proxy_country,
|
|
1841
|
+
})
|
|
1842
|
+
end
|
|
1843
|
+
|
|
1844
|
+
# Scrape Redfin real-estate listings.
|
|
1845
|
+
#
|
|
1846
|
+
# Searches Redfin for `location` (a city/state or ZIP) and returns the listings as JSON:
|
|
1847
|
+
# `price`, `beds`, `baths`, `sqFt`, `lotSize`, address, coordinates, `propertyType`,
|
|
1848
|
+
# `mlsStatus`, `mlsId`, `daysOnMarket`, `pricePerSqFt` and a clean `url`. Filter with
|
|
1849
|
+
# `status` (for_sale / sold), `min_price`/`max_price`, `beds`, `baths` and `property_type`,
|
|
1850
|
+
# sort with `sort`, and collect up to `max_results` across pages. Use `mode=detail` with a
|
|
1851
|
+
# listing `url` for one home in full (agents, public records, tax, price history, Redfin
|
|
1852
|
+
# Estimate).
|
|
1853
|
+
#
|
|
1854
|
+
# @param url [String] Redfin URL. Required for detail mode (a listing URL); optional in
|
|
1855
|
+
# search mode (a city/zip page URL).
|
|
1856
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1857
|
+
# 'DE'). Defaults to US.
|
|
1858
|
+
# Calls POST /realestate/redfin.
|
|
1859
|
+
def redfin_detail(url, proxy_country: nil)
|
|
1860
|
+
post_json("/realestate/redfin", {
|
|
1861
|
+
mode: "detail",
|
|
1862
|
+
url: url,
|
|
1863
|
+
proxy_country: proxy_country,
|
|
1864
|
+
})
|
|
1865
|
+
end
|
|
1866
|
+
|
|
1867
|
+
# Scrape Rightmove property listings.
|
|
1868
|
+
#
|
|
1869
|
+
# Searches Rightmove (the UK's largest property portal) for `location` and returns the
|
|
1870
|
+
# listings as JSON: `address`, `price` (`amount` + `currency` + `display` + `frequency`),
|
|
1871
|
+
# `transactionType`, `bedrooms`, `bathrooms`, `propertyType`, `summary`, `coordinates`,
|
|
1872
|
+
# `images`, `agent` (name / branch / phone) and listing dates. Filter with `channel` (buy /
|
|
1873
|
+
# rent), `min_price`/`max_price`, `min_bedrooms`/`max_bedrooms` and `property_type`, sort
|
|
1874
|
+
# with `sort`, and collect up to `max_results` across pages.
|
|
1875
|
+
#
|
|
1876
|
+
# @param location [String] UK place name, e.g. 'London', 'Manchester', 'Milton Keynes'.
|
|
1877
|
+
# @param channel [String] buy (for sale) or rent (to rent). API default: buy.
|
|
1878
|
+
# @param min_price [Integer] Minimum price (GBP).
|
|
1879
|
+
# @param max_price [Integer] Maximum price (GBP).
|
|
1880
|
+
# @param min_bedrooms [Integer] Minimum bedrooms.
|
|
1881
|
+
# @param max_bedrooms [Integer] Maximum bedrooms.
|
|
1882
|
+
# @param property_type [String] Rightmove property types, comma-separated, e.g.
|
|
1883
|
+
# 'flat,detached'.
|
|
1884
|
+
# @param sort [String] Ordering: relevance, newest, price_asc, price_desc, oldest. API
|
|
1885
|
+
# default: relevance.
|
|
1886
|
+
# @param max_results [Integer] How many listings to collect across pages (~24 per page, max
|
|
1887
|
+
# 1008). API default: 48.
|
|
1888
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
1889
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
1890
|
+
# 'DE'). Leave blank to let the API pick.
|
|
1891
|
+
# Calls POST /realestate/rightmove.
|
|
1892
|
+
def rightmove_search(location, channel: nil, min_price: nil, max_price: nil, min_bedrooms: nil, max_bedrooms: nil, property_type: nil, sort: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
1893
|
+
post_json("/realestate/rightmove", {
|
|
1894
|
+
location: location,
|
|
1895
|
+
channel: channel,
|
|
1896
|
+
min_price: min_price,
|
|
1897
|
+
max_price: max_price,
|
|
1898
|
+
min_bedrooms: min_bedrooms,
|
|
1899
|
+
max_bedrooms: max_bedrooms,
|
|
1900
|
+
property_type: property_type,
|
|
1901
|
+
sort: sort,
|
|
1902
|
+
max_results: max_results,
|
|
1903
|
+
page: page,
|
|
1904
|
+
proxy_country: proxy_country,
|
|
1905
|
+
})
|
|
1906
|
+
end
|
|
1907
|
+
|
|
1908
|
+
# Scrape idealista property listings (Spain, Italy, Portugal).
|
|
1909
|
+
#
|
|
1910
|
+
# Searches idealista (idealista.com, idealista.it, idealista.pt) for `location` - or takes
|
|
1911
|
+
# any idealista search `url` - and returns the listings as JSON: `id`, `url`, `title`,
|
|
1912
|
+
# `price` + `currency` (+ `pricePeriod` for rentals), `originalPrice` and `priceDropPercent`
|
|
1913
|
+
# after a price cut, `pricePerM2`, `area`, `rooms`, `floor`, `lift`, `exterior`, `parking`,
|
|
1914
|
+
# `description`, `tags`, `images`, `imageCount`, `multimedia` (plan, video, 3D tour),
|
|
1915
|
+
# `agency` and `professional`. Filter by `operation` (sale / rent), `property_type`, price
|
|
1916
|
+
# and size ranges, `rooms`, `bathrooms` and `features`, order with `sort`, and collect up to
|
|
1917
|
+
# `max_results` across pages (30 per page; idealista shows at most 60 pages per search). A
|
|
1918
|
+
# search with no match answers 200 with an empty `listings`. Billed one unit per page
|
|
1919
|
+
# fetched (`pagesFetched`).
|
|
1920
|
+
#
|
|
1921
|
+
# @param location [String] Location as idealista writes it in its URLs: a municipality
|
|
1922
|
+
# ('madrid-madrid', 'barcelona-barcelona', 'milano-milano', 'lisboa'), a district
|
|
1923
|
+
# ('madrid/centro', 'lisboa/arroios') or a province ('malaga-provincia'). Required unless
|
|
1924
|
+
# url is given.
|
|
1925
|
+
# @param url [String] Any idealista search URL (idealista.com, idealista.it or idealista.pt)
|
|
1926
|
+
# with its filters, e.g. https://www.idealista.com/alquiler-viviendas/barcelona-
|
|
1927
|
+
# barcelona/con-precio-hasta_2000,de-dos-dormitorios/. Overrides location and the filter
|
|
1928
|
+
# parameters.
|
|
1929
|
+
# @param country [String] es, it, pt (idealista.com / .it / .pt). API default: es.
|
|
1930
|
+
# @param operation [String] sale or rent. API default: sale.
|
|
1931
|
+
# @param property_type [String] One of: homes, new_homes, offices, premises, garages, lands,
|
|
1932
|
+
# storage, buildings, rooms. API default: homes.
|
|
1933
|
+
# @param min_price [Integer] Minimum price in EUR (monthly rent with operation=rent).
|
|
1934
|
+
# @param max_price [Integer] Maximum price in EUR.
|
|
1935
|
+
# @param min_size [Integer] Minimum area in m².
|
|
1936
|
+
# @param max_size [Integer] Maximum area in m².
|
|
1937
|
+
# @param rooms [String] Comma-separated room counts: bedrooms in Spain (0 = studio ... 4 =
|
|
1938
|
+
# four or more) and Portugal (T0 ... T4+), total rooms in Italy (1 ... 5 = five or more).
|
|
1939
|
+
# E.g. '2,3'.
|
|
1940
|
+
# @param bathrooms [String] Comma-separated 1, 2, 3 (3 = three or more).
|
|
1941
|
+
# @param features [String] Comma-separated: lift, terrace, balcony, garage, pool, garden,
|
|
1942
|
+
# air_conditioning, storage_room, fitted_wardrobes, accessible, luxury, new_construction,
|
|
1943
|
+
# good_condition, to_renovate.
|
|
1944
|
+
# @param sort [String] One of: relevance, price_asc, price_desc, newest, oldest, size_asc,
|
|
1945
|
+
# size_desc, price_per_m2_asc, price_per_m2_desc, biggest_drop. API default: relevance.
|
|
1946
|
+
# @param page [Integer] 1-based results page to start from (30 listings per page, at most
|
|
1947
|
+
# 60). Use the `nextPage` of a previous response to continue.
|
|
1948
|
+
# @param max_results [Integer] How many listings to collect across pages (1-300); each page
|
|
1949
|
+
# fetched is one billed unit. API default: 30.
|
|
1950
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2). Leave blank
|
|
1951
|
+
# for the exit chosen for the site.
|
|
1952
|
+
# Calls POST /realestate/idealista.
|
|
1953
|
+
def idealista_search(location, url: nil, country: nil, operation: nil, property_type: nil, min_price: nil, max_price: nil, min_size: nil, max_size: nil, rooms: nil, bathrooms: nil, features: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
1954
|
+
post_json("/realestate/idealista", {
|
|
1955
|
+
location: location,
|
|
1956
|
+
url: url,
|
|
1957
|
+
country: country,
|
|
1958
|
+
operation: operation,
|
|
1959
|
+
property_type: property_type,
|
|
1960
|
+
min_price: min_price,
|
|
1961
|
+
max_price: max_price,
|
|
1962
|
+
min_size: min_size,
|
|
1963
|
+
max_size: max_size,
|
|
1964
|
+
rooms: rooms,
|
|
1965
|
+
bathrooms: bathrooms,
|
|
1966
|
+
features: features,
|
|
1967
|
+
sort: sort,
|
|
1968
|
+
page: page,
|
|
1969
|
+
max_results: max_results,
|
|
1970
|
+
proxy_country: proxy_country,
|
|
1971
|
+
})
|
|
1972
|
+
end
|
|
1973
|
+
|
|
1974
|
+
# Scrape one idealista listing (Spain, Italy, Portugal).
|
|
1975
|
+
#
|
|
1976
|
+
# Given an idealista `listing` id (with `country`) or URL, returns the listing as JSON:
|
|
1977
|
+
# `title`, `locationLine`, `operation`, `propertyType`, `price` + `currency` (+
|
|
1978
|
+
# `pricePeriod`), `originalPrice`/`priceDropPercent`, `pricePerM2`, `priceDetails` (the
|
|
1979
|
+
# price breakdown idealista shows, e.g. community fees or deposit), `area`, `usableArea`,
|
|
1980
|
+
# `rooms`, `bathrooms`, `floor`, `lift`, `builtYear`, `energyRating`, `features` (every
|
|
1981
|
+
# feature group idealista lists), `description`, `address`, `coordinates`
|
|
1982
|
+
# (`locationApproximate` when the advertiser hid the exact spot), `images`, `floorPlans`,
|
|
1983
|
+
# `multimediaCounts`, `advertiser` (name, type, reference, agency page) and `updated`. A
|
|
1984
|
+
# listing its advertiser removed answers 404. Billed one unit.
|
|
1985
|
+
#
|
|
1986
|
+
# @param listing [String] idealista listing id ('111278716') or URL, e.g.
|
|
1987
|
+
# https://www.idealista.com/inmueble/111278716/, https://www.idealista.it/immobile/37068709/
|
|
1988
|
+
# or https://www.idealista.pt/imovel/34868387/.
|
|
1989
|
+
# @param country [String] Country of a bare listing id: es, it, pt. Ignored for a URL. API
|
|
1990
|
+
# default: es.
|
|
1991
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2). Leave blank
|
|
1992
|
+
# for the exit chosen for the site.
|
|
1993
|
+
# Calls POST /realestate/idealista-detail.
|
|
1994
|
+
def idealista_detail(listing, country: nil, proxy_country: nil)
|
|
1995
|
+
post_json("/realestate/idealista-detail", {
|
|
1996
|
+
listing: listing,
|
|
1997
|
+
country: country,
|
|
1998
|
+
proxy_country: proxy_country,
|
|
1999
|
+
})
|
|
2000
|
+
end
|
|
2001
|
+
|
|
2002
|
+
# Scrape ImmobilienScout24 property listings.
|
|
2003
|
+
#
|
|
2004
|
+
# Searches ImmobilienScout24 (Germany's largest property portal) for `location` and returns
|
|
2005
|
+
# the listings as JSON: `title`, `price` + `currency` + `priceType` (`rent_cold` /
|
|
2006
|
+
# `rent_warm` / `purchase`), `warmRent`, `livingArea`, `rooms`, `plotArea`, `address`,
|
|
2007
|
+
# `district`, `coordinates`, `image`/`images`, `realtor` and a listing `url`. Filter with
|
|
2008
|
+
# `channel` (rent / buy), `property_type` (apartment / house), `min_price`/`max_price`,
|
|
2009
|
+
# `min_rooms`/`max_rooms` and `min_area`/`max_area`, and collect up to `max_results` across
|
|
2010
|
+
# pages.
|
|
2011
|
+
#
|
|
2012
|
+
# @param location [String] German city ('Berlin', 'Muenchen', 'Köln', 'Frankfurt am Main'),
|
|
2013
|
+
# a state ('Bayern'), or an ImmoScout24 region path such as 'berlin/berlin/mitte' or
|
|
2014
|
+
# 'bayern/muenchen-kreis/unterschleissheim'.
|
|
2015
|
+
# @param channel [String] rent or buy. API default: rent.
|
|
2016
|
+
# @param property_type [String] apartment or house. API default: apartment.
|
|
2017
|
+
# @param min_price [Float] Minimum price in EUR (cold rent per month, or purchase price).
|
|
2018
|
+
# @param max_price [Float] Maximum price in EUR (cold rent per month, or purchase price).
|
|
2019
|
+
# @param min_rooms [Float] Minimum number of rooms (half rooms allowed, e.g. 2.5).
|
|
2020
|
+
# @param max_rooms [Float] Maximum number of rooms.
|
|
2021
|
+
# @param min_area [Float] Minimum living area in m².
|
|
2022
|
+
# @param max_area [Float] Maximum living area in m².
|
|
2023
|
+
# @param max_results [Integer] How many listings to collect across pages (20 per page, max
|
|
2024
|
+
# 500). API default: 20.
|
|
2025
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
2026
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2027
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2028
|
+
# Calls POST /realestate/immobilienscout24.
|
|
2029
|
+
def immobilienscout24_search(location, channel: nil, property_type: nil, min_price: nil, max_price: nil, min_rooms: nil, max_rooms: nil, min_area: nil, max_area: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
2030
|
+
post_json("/realestate/immobilienscout24", {
|
|
2031
|
+
location: location,
|
|
2032
|
+
channel: channel,
|
|
2033
|
+
property_type: property_type,
|
|
2034
|
+
min_price: min_price,
|
|
2035
|
+
max_price: max_price,
|
|
2036
|
+
min_rooms: min_rooms,
|
|
2037
|
+
max_rooms: max_rooms,
|
|
2038
|
+
min_area: min_area,
|
|
2039
|
+
max_area: max_area,
|
|
2040
|
+
max_results: max_results,
|
|
2041
|
+
page: page,
|
|
2042
|
+
proxy_country: proxy_country,
|
|
2043
|
+
})
|
|
2044
|
+
end
|
|
2045
|
+
|
|
2046
|
+
# Scrape mobile.de car listings.
|
|
2047
|
+
#
|
|
2048
|
+
# Searches mobile.de (Germany's largest car marketplace) and returns the result-list cards
|
|
2049
|
+
# as JSON: `title`, `makeModel`, `price` (numeric EUR) plus `priceFormatted`,
|
|
2050
|
+
# `firstRegistration`, `mileage`, `power` (`kw` + `ps`), `fuel`, and the detail `id` +
|
|
2051
|
+
# `url`. Pass a friendly `make` name (or a raw `make_id`), and filter with
|
|
2052
|
+
# `price_min`/`price_max`, `year_min`, `mileage_max` and `fuel`; collect up to `max_results`
|
|
2053
|
+
# across pages. `model` is accepted as free text but is not used to filter (mobile.de needs
|
|
2054
|
+
# a numeric model id we do not resolve).
|
|
2055
|
+
#
|
|
2056
|
+
# @param make [String] Make name, e.g. 'bmw', 'volkswagen', 'audi'. Required unless
|
|
2057
|
+
# `make_id` is given.
|
|
2058
|
+
# @param make_id [Integer] Raw mobile.de numeric make id, overrides `make` (e.g. 3500 =
|
|
2059
|
+
# BMW).
|
|
2060
|
+
# @param model [String] Model name (free text). Model-id resolution is not available, so it
|
|
2061
|
+
# is used only as a note; results are not filtered by model.
|
|
2062
|
+
# @param price_min [Integer] Minimum price in EUR.
|
|
2063
|
+
# @param price_max [Integer] Maximum price in EUR.
|
|
2064
|
+
# @param year_min [Integer] Earliest first-registration year (YYYY).
|
|
2065
|
+
# @param mileage_max [Integer] Maximum mileage in km.
|
|
2066
|
+
# @param fuel [String] Fuel: petrol, diesel, electric, hybrid (or a raw mobile.de code).
|
|
2067
|
+
# @param max_results [Integer] How many listings to collect across pages (~20 per page). API
|
|
2068
|
+
# default: 20.
|
|
2069
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
2070
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2071
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2072
|
+
# Calls POST /vehicles/mobile-de.
|
|
2073
|
+
def mobile_de(make = nil, make_id: nil, model: nil, price_min: nil, price_max: nil, year_min: nil, mileage_max: nil, fuel: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
2074
|
+
post_json("/vehicles/mobile-de", {
|
|
2075
|
+
make: make,
|
|
2076
|
+
make_id: make_id,
|
|
2077
|
+
model: model,
|
|
2078
|
+
price_min: price_min,
|
|
2079
|
+
price_max: price_max,
|
|
2080
|
+
year_min: year_min,
|
|
2081
|
+
mileage_max: mileage_max,
|
|
2082
|
+
fuel: fuel,
|
|
2083
|
+
max_results: max_results,
|
|
2084
|
+
page: page,
|
|
2085
|
+
proxy_country: proxy_country,
|
|
2086
|
+
})
|
|
2087
|
+
end
|
|
2088
|
+
|
|
2089
|
+
# Scrape AutoScout24 car listings.
|
|
2090
|
+
#
|
|
2091
|
+
# Searches AutoScout24 (Europe's largest car marketplace) and returns the listings as JSON:
|
|
2092
|
+
# `title`, `make`, `model`, `condition` (used / new), `price` (`raw` + `formatted` +
|
|
2093
|
+
# `label`), `mileageKm`, `firstRegistration`, `fuel`, `transmission`, `powerKw`/`powerHp`,
|
|
2094
|
+
# `engineCc`, `location` (country / zip / city / street), `seller` (type, name, phones) and
|
|
2095
|
+
# `images`. Filter with `fuel`, `transmission`, `price_from`/`price_to`,
|
|
2096
|
+
# `year_from`/`year_to`, `mileage_to`, `power_from`/`power_to` (`power_unit`) and `country`,
|
|
2097
|
+
# sort with `sort`, and collect up to `max_results` across pages.
|
|
2098
|
+
#
|
|
2099
|
+
# @param make [String] Make slug, e.g. 'bmw', 'volkswagen', 'audi'.
|
|
2100
|
+
# @param model [String] Model slug, e.g. 'x5', 'golf'. Requires `make`.
|
|
2101
|
+
# @param fuel [String] Fuel: gasoline, diesel, electric, hybrid, lpg, cng (or a raw
|
|
2102
|
+
# AutoScout24 code).
|
|
2103
|
+
# @param transmission [String] Transmission: automatic, manual, semi-automatic.
|
|
2104
|
+
# @param price_from [Integer] Minimum price in EUR.
|
|
2105
|
+
# @param price_to [Integer] Maximum price in EUR.
|
|
2106
|
+
# @param year_from [Integer] Earliest first-registration year.
|
|
2107
|
+
# @param year_to [Integer] Latest first-registration year.
|
|
2108
|
+
# @param mileage_to [Integer] Maximum mileage in km.
|
|
2109
|
+
# @param power_from [Integer] Minimum power (unit set by `power_unit`).
|
|
2110
|
+
# @param power_to [Integer] Maximum power (unit set by `power_unit`).
|
|
2111
|
+
# @param power_unit [String] Power unit for the power filters: 'kw' or 'hp'. API default:
|
|
2112
|
+
# kw.
|
|
2113
|
+
# @param country [String] Seller country (ISO-2), e.g. 'DE', 'IT', 'NL'.
|
|
2114
|
+
# @param sort [String] Ordering: relevance, price_asc, price_desc, newest, mileage_asc,
|
|
2115
|
+
# year_desc, power_desc. API default: relevance.
|
|
2116
|
+
# @param max_results [Integer] How many listings to collect across pages (~20 per page, max
|
|
2117
|
+
# 400). API default: 40.
|
|
2118
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
2119
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2120
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2121
|
+
# Calls POST /vehicles/autoscout24.
|
|
2122
|
+
def autoscout24_search(make: nil, model: nil, fuel: nil, transmission: nil, price_from: nil, price_to: nil, year_from: nil, year_to: nil, mileage_to: nil, power_from: nil, power_to: nil, power_unit: nil, country: nil, sort: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
2123
|
+
post_json("/vehicles/autoscout24", {
|
|
2124
|
+
make: make,
|
|
2125
|
+
model: model,
|
|
2126
|
+
fuel: fuel,
|
|
2127
|
+
transmission: transmission,
|
|
2128
|
+
price_from: price_from,
|
|
2129
|
+
price_to: price_to,
|
|
2130
|
+
year_from: year_from,
|
|
2131
|
+
year_to: year_to,
|
|
2132
|
+
mileage_to: mileage_to,
|
|
2133
|
+
power_from: power_from,
|
|
2134
|
+
power_to: power_to,
|
|
2135
|
+
power_unit: power_unit,
|
|
2136
|
+
country: country,
|
|
2137
|
+
sort: sort,
|
|
2138
|
+
max_results: max_results,
|
|
2139
|
+
page: page,
|
|
2140
|
+
proxy_country: proxy_country,
|
|
2141
|
+
})
|
|
2142
|
+
end
|
|
2143
|
+
|
|
2144
|
+
# Scrape Indeed job listings.
|
|
2145
|
+
#
|
|
2146
|
+
# Searches Indeed for `query` in `location` and returns the job listings as JSON: `title`,
|
|
2147
|
+
# `company`, `location`, `salary` (with parsed `min`/`max`), `jobTypes`, `companyRating`, a
|
|
2148
|
+
# `snippet`, `datePosted` (YYYY-MM-DD), `sponsored` (Indeed's paid placement) and a clean
|
|
2149
|
+
# job `url`. Filter with `country`, `job_type`, `days` (posting age), `remote`, sort with
|
|
2150
|
+
# `sort` (`date` returns the collected jobs newest first, sponsored ones included), and
|
|
2151
|
+
# collect up to `max_results` across pages. Indeed sits behind Cloudflare + PerimeterX -
|
|
2152
|
+
# handled for you.
|
|
2153
|
+
#
|
|
2154
|
+
# @param query [String] What to search, e.g. 'software engineer' or 'nurse'.
|
|
2155
|
+
# @param country [String] Indeed country site: us, uk, ca, au, in, ie, de, fr, nl, es, it,
|
|
2156
|
+
# at, ch, be, pl, se, sg, za, nz, mx. API default: us.
|
|
2157
|
+
# @param location [String] Where, e.g. 'Austin, TX' or 'London'.
|
|
2158
|
+
# @param radius [Integer] Search radius in miles/km around the location.
|
|
2159
|
+
# @param job_type [String] Job type: fulltime, parttime, contract, internship, temporary.
|
|
2160
|
+
# @param days [Integer] Max posting age in days: 1, 3, 7, 14.
|
|
2161
|
+
# @param remote [Boolean] Only remote jobs. API default: false.
|
|
2162
|
+
# @param sort [String] Ordering: relevance, date. API default: relevance.
|
|
2163
|
+
# @param max_results [Integer] How many jobs to collect across pages. API default: 30.
|
|
2164
|
+
# @param start [Integer] 0-based result offset to start from. API default: 0.
|
|
2165
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2166
|
+
# 'DE'). Defaults to the country's site.
|
|
2167
|
+
# Calls POST /jobs/indeed.
|
|
2168
|
+
def indeed_jobs(query, country: nil, location: nil, radius: nil, job_type: nil, days: nil, remote: nil, sort: nil, max_results: nil, start: nil, proxy_country: nil)
|
|
2169
|
+
post_json("/jobs/indeed", {
|
|
2170
|
+
query: query,
|
|
2171
|
+
country: country,
|
|
2172
|
+
location: location,
|
|
2173
|
+
radius: radius,
|
|
2174
|
+
job_type: job_type,
|
|
2175
|
+
days: days,
|
|
2176
|
+
remote: remote,
|
|
2177
|
+
sort: sort,
|
|
2178
|
+
max_results: max_results,
|
|
2179
|
+
start: start,
|
|
2180
|
+
proxy_country: proxy_country,
|
|
2181
|
+
})
|
|
2182
|
+
end
|
|
2183
|
+
|
|
2184
|
+
# Scrape Glassdoor job listings.
|
|
2185
|
+
#
|
|
2186
|
+
# Searches Glassdoor for `keyword` and returns the job listings as JSON: `title`, `jobId`,
|
|
2187
|
+
# `url`, `company` (+ `companyId`, `companyRating`, `companyLogo`), `location`, `ageInDays`,
|
|
2188
|
+
# `salary` (p10/median/p90, payPeriod, currency, source), `snippet`, `skills`, `easyApply`
|
|
2189
|
+
# and `sponsored`. Filter with `location`, `from_age`, `job_type`, `min_salary`,
|
|
2190
|
+
# `experience` and `remote`, sort with `sort` (recency / relevance), and collect up to
|
|
2191
|
+
# `max_results` across pages.
|
|
2192
|
+
#
|
|
2193
|
+
# @param keyword [String] Job title / keywords to search, e.g. 'software engineer'.
|
|
2194
|
+
# @param location [String] Location name to filter by, e.g. 'New York, NY'. Resolved to
|
|
2195
|
+
# Glassdoor's location id automatically; pass loc_id to skip resolution. A name Glassdoor
|
|
2196
|
+
# does not know is a 400.
|
|
2197
|
+
# @param loc_id [Integer] Glassdoor numeric location id (overrides `location` resolution).
|
|
2198
|
+
# @param loc_type [String] Location type for loc_id: C=city, S=state, N=nation, M=metro.
|
|
2199
|
+
# @param sort [String] Ordering: recency, relevance. API default: recency.
|
|
2200
|
+
# @param job_type [String] Employment type: fulltime, parttime, contract, internship,
|
|
2201
|
+
# temporary, apprenticeship. Omit for any.
|
|
2202
|
+
# @param experience [String] Experience level: internship, entrylevel, midseniorlevel,
|
|
2203
|
+
# director, executive. Omit for any.
|
|
2204
|
+
# @param from_age [Integer] Posted within N days (1, 3, 7, 14 or 30).
|
|
2205
|
+
# @param min_salary [Integer] Minimum annual salary.
|
|
2206
|
+
# @param remote [Boolean] Remote jobs only. API default: false.
|
|
2207
|
+
# @param radius [Integer] Search radius in miles around the location.
|
|
2208
|
+
# @param max_results [Integer] How many jobs to collect across pages (~30 per page). API
|
|
2209
|
+
# default: 30.
|
|
2210
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
2211
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2212
|
+
# 'DE'). Defaults to US.
|
|
2213
|
+
# Calls POST /jobs/glassdoor.
|
|
2214
|
+
def glassdoor_jobs(keyword, location: nil, loc_id: nil, loc_type: nil, sort: nil, job_type: nil, experience: nil, from_age: nil, min_salary: nil, remote: nil, radius: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
2215
|
+
post_json("/jobs/glassdoor", {
|
|
2216
|
+
keyword: keyword,
|
|
2217
|
+
location: location,
|
|
2218
|
+
loc_id: loc_id,
|
|
2219
|
+
loc_type: loc_type,
|
|
2220
|
+
sort: sort,
|
|
2221
|
+
job_type: job_type,
|
|
2222
|
+
experience: experience,
|
|
2223
|
+
from_age: from_age,
|
|
2224
|
+
min_salary: min_salary,
|
|
2225
|
+
remote: remote,
|
|
2226
|
+
radius: radius,
|
|
2227
|
+
max_results: max_results,
|
|
2228
|
+
page: page,
|
|
2229
|
+
proxy_country: proxy_country,
|
|
2230
|
+
})
|
|
2231
|
+
end
|
|
2232
|
+
|
|
2233
|
+
# Scrape Glassdoor company reviews.
|
|
2234
|
+
#
|
|
2235
|
+
# Returns a company's Glassdoor employee reviews as JSON. Each review has `id`, `url`,
|
|
2236
|
+
# `title`, `rating` (1-5), `subratings` (work/life balance, culture and values, diversity
|
|
2237
|
+
# and inclusion, career opportunities, compensation and benefits, senior management),
|
|
2238
|
+
# `recommendToFriend`, `ceoApproval`, `businessOutlook`, `pros`, `cons`, `advice`,
|
|
2239
|
+
# `jobTitle`, `employmentStatus`, `isCurrentEmployee`, `yearsOfEmployment`, `location`,
|
|
2240
|
+
# `date`, `helpfulCount`, `notHelpfulCount`, `featured`, `employerResponse` and `view` (the
|
|
2241
|
+
# sort / filter view it came from). The response also carries `summary` (overall rating,
|
|
2242
|
+
# review counts, recommend / CEO approval / business outlook percentages, the CEO, sub-
|
|
2243
|
+
# rating averages, star distribution, reviews in the last 7 and 30 days), `companyMeta`
|
|
2244
|
+
# (name, logo, website, industry, sector, headquarters, size) and `totalReviews`. The counts
|
|
2245
|
+
# are Glassdoor's own and measure different sets: `totalReviews` is the number of reviews
|
|
2246
|
+
# matching every filter of the call (`rating`, `job_title`, `keyword`, `topic`,
|
|
2247
|
+
# `employment_status`, `language`, `current_only`; the default view is English reviews by
|
|
2248
|
+
# full-time and part-time employees), the set the views read from. `summary.allReviewsCount`
|
|
2249
|
+
# is every review Glassdoor holds for the employer, rated or not, in any language and
|
|
2250
|
+
# employment status; no filter changes it. `summary.reviewCount` is the number of ratings
|
|
2251
|
+
# behind `summary.overallRating`, and `summary.ratedReviewsCount` the number of reviews that
|
|
2252
|
+
# carry a star rating; Glassdoor counts the two separately, so they can differ by a few.
|
|
2253
|
+
# Both follow `rating` but not the language or employment filters, so they are usually above
|
|
2254
|
+
# `totalReviews`. `summary.ratingDistribution` holds the star counts Glassdoor sends for its
|
|
2255
|
+
# rating bars (`overall`, each sub-rating, and `recommendToFriend`); for employers with many
|
|
2256
|
+
# reviews they cover only part of them (Google: 18,643 overall against a `reviewCount` of
|
|
2257
|
+
# 52,845), so read them as proportions, not totals, and with `rating` set only that star is
|
|
2258
|
+
# counted. `summary.reviewsLast7Days` and `reviewsLast30Days` count reviews matching the
|
|
2259
|
+
# filters. Give `company` as a Glassdoor employer id, a Glassdoor company URL or a company
|
|
2260
|
+
# name. Only US proxies are used (`proxy_country` accepts US or blank): Glassdoor redirects
|
|
2261
|
+
# other countries to their local site, where these pages do not exist. Glassdoor shows
|
|
2262
|
+
# anonymous visitors only the first page (3 reviews) of each sort / filter view and
|
|
2263
|
+
# redirects deeper pages to sign-in, so there is no page parameter: use `sort`, `rating` and
|
|
2264
|
+
# the filters to reach other reviews, or raise `max_results` to read several views in one
|
|
2265
|
+
# call. `hasMore` is always false and `nextPage` null; `publicLimitReached` is true when
|
|
2266
|
+
# fewer reviews came back than asked because the views anonymous visitors may read ran out.
|
|
2267
|
+
# Every field is always present and is `null` when Glassdoor does not show it. Billed per
|
|
2268
|
+
# review view fetched (`pagesFetched`).
|
|
2269
|
+
#
|
|
2270
|
+
# @param company [String] The company: a Glassdoor employer id (e.g. '9079'), any Glassdoor
|
|
2271
|
+
# company URL (e.g. 'https://www.glassdoor.com/Reviews/Google-Reviews-E9079.htm'), or a
|
|
2272
|
+
# company name (e.g. 'Bank of America', matched with Glassdoor's company search).
|
|
2273
|
+
# @param sort [String] Review order: recent, helpful, highest, lowest. API default: recent.
|
|
2274
|
+
# @param rating [Integer] Only reviews with this overall star rating (1-5).
|
|
2275
|
+
# @param job_title [String] Only reviews from job titles matching this text, e.g. 'software
|
|
2276
|
+
# engineer' (Glassdoor's own job-title search).
|
|
2277
|
+
# @param keyword [String] Only reviews mentioning this keyword, e.g. 'remote'.
|
|
2278
|
+
# @param topic [String] Only reviews about one topic: compensation, culture,
|
|
2279
|
+
# career_development, management, work_life_balance, diversity_and_inclusion.
|
|
2280
|
+
# @param employment_status [String] Comma list of full_time, part_time, contract, intern,
|
|
2281
|
+
# freelance. Defaults to Glassdoor's own view: full_time,part_time.
|
|
2282
|
+
# @param language [String] Comma list of review languages (ISO 639-3): eng, fra, deu, nld,
|
|
2283
|
+
# por, spa, ita. Defaults to eng.
|
|
2284
|
+
# @param current_only [Boolean] Only reviews by current employees. API default: false.
|
|
2285
|
+
# @param max_results [Integer] How many reviews to return. Glassdoor shows anonymous
|
|
2286
|
+
# visitors 3 reviews per view; above that the call also reads the other sort orders and each
|
|
2287
|
+
# star rating, and merges the distinct reviews. API default: 3.
|
|
2288
|
+
# @param proxy_country [String] Proxy country the request is sent from. Only US (or blank,
|
|
2289
|
+
# the default) is accepted: Glassdoor redirects visitors from other countries to their local
|
|
2290
|
+
# site, where the English reviews pages do not exist.
|
|
2291
|
+
# Calls POST /reviews/glassdoor.
|
|
2292
|
+
def glassdoor_reviews(company, sort: nil, rating: nil, job_title: nil, keyword: nil, topic: nil, employment_status: nil, language: nil, current_only: nil, max_results: nil, proxy_country: nil)
|
|
2293
|
+
post_json("/reviews/glassdoor", {
|
|
2294
|
+
company: company,
|
|
2295
|
+
sort: sort,
|
|
2296
|
+
rating: rating,
|
|
2297
|
+
job_title: job_title,
|
|
2298
|
+
keyword: keyword,
|
|
2299
|
+
topic: topic,
|
|
2300
|
+
employment_status: employment_status,
|
|
2301
|
+
language: language,
|
|
2302
|
+
current_only: current_only,
|
|
2303
|
+
max_results: max_results,
|
|
2304
|
+
proxy_country: proxy_country,
|
|
2305
|
+
})
|
|
2306
|
+
end
|
|
2307
|
+
|
|
2308
|
+
# Scrape Glassdoor company salaries.
|
|
2309
|
+
#
|
|
2310
|
+
# Returns a company's Glassdoor salary estimates by job title as JSON. Each row has
|
|
2311
|
+
# `jobTitle`, `jobTitleId`, `basePay` and `totalPay` percentiles (`P10`, `P25`, `P50`,
|
|
2312
|
+
# `P75`, `P90`), `currency`, `salaryCount` (salaries behind the estimate) and `mostRecent`
|
|
2313
|
+
# (latest report). The response also carries `companyMeta` (name, rating, logo, website,
|
|
2314
|
+
# industry, sector, headquarters, size, total salary count), `jobTitleCount` and `numPages`.
|
|
2315
|
+
# Give `company` as a Glassdoor employer id, a Glassdoor company URL or a company name;
|
|
2316
|
+
# narrow with `job_title` (Glassdoor's own job-title search) and collect up to `max_results`
|
|
2317
|
+
# rows across pages, most-reported job titles first. Every field is always present and is
|
|
2318
|
+
# `null` when Glassdoor does not show it. Billed per salary page fetched (10 rows per page).
|
|
2319
|
+
#
|
|
2320
|
+
# @param company [String] The company: a Glassdoor employer id (e.g. '9079'), any Glassdoor
|
|
2321
|
+
# company URL (e.g. 'https://www.glassdoor.com/Overview/Working-at-Google-
|
|
2322
|
+
# EI_IE9079.11,17.htm'), or a company name (e.g. 'Bank of America', matched with Glassdoor's
|
|
2323
|
+
# company search).
|
|
2324
|
+
# @param job_title [String] Narrow to job titles matching this text, using Glassdoor's own
|
|
2325
|
+
# job-title search, e.g. 'software engineer'.
|
|
2326
|
+
# @param max_results [Integer] How many job-title rows to return (10 per Glassdoor page,
|
|
2327
|
+
# most-reported titles first). API default: 20.
|
|
2328
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2329
|
+
# 'DE'). Defaults to US, so pay is reported for the US market in USD.
|
|
2330
|
+
# Calls POST /salaries/glassdoor.
|
|
2331
|
+
def glassdoor_salaries(company, job_title: nil, max_results: nil, proxy_country: nil)
|
|
2332
|
+
post_json("/salaries/glassdoor", {
|
|
2333
|
+
company: company,
|
|
2334
|
+
job_title: job_title,
|
|
2335
|
+
max_results: max_results,
|
|
2336
|
+
proxy_country: proxy_country,
|
|
2337
|
+
})
|
|
2338
|
+
end
|
|
2339
|
+
|
|
2340
|
+
# Scrape Naukri job listings.
|
|
2341
|
+
#
|
|
2342
|
+
# Searches Naukri (India's largest job board) for `keyword` in `location` and returns the
|
|
2343
|
+
# job listings as JSON: `title`, `company`, `location` (with `workMode` hybrid / remote /
|
|
2344
|
+
# on-site), `experience` (with parsed `experienceMinYears`/`experienceMaxYears`), `salary`
|
|
2345
|
+
# (with parsed `min`/`max`), `skills`, `companyRating`, `reviewsCount`, a `description`
|
|
2346
|
+
# snippet, `postedDate` and a clean job `url`. Filter with `location`, `experience` (years),
|
|
2347
|
+
# `days` (posting age) and `remote`, sort with `sort`, and collect up to `max_results`
|
|
2348
|
+
# across pages.
|
|
2349
|
+
#
|
|
2350
|
+
# @param keyword [String] What to search, e.g. 'python developer' or 'data analyst'.
|
|
2351
|
+
# @param location [String] City to search in, e.g. 'Bangalore' or 'Mumbai'.
|
|
2352
|
+
# @param experience [Integer] Candidate experience in years (0-30).
|
|
2353
|
+
# @param days [Integer] Max posting age in days: 1, 3, 7, 15, 30.
|
|
2354
|
+
# @param remote [Boolean] Only work-from-home / remote jobs. API default: false.
|
|
2355
|
+
# @param sort [String] Ordering: relevance, date. API default: relevance.
|
|
2356
|
+
# @param max_results [Integer] How many jobs to collect across pages (~20 per page). API
|
|
2357
|
+
# default: 40.
|
|
2358
|
+
# @param page [Integer] 1-based results page to start from. API default: 1.
|
|
2359
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2360
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2361
|
+
# Calls POST /jobs/naukri.
|
|
2362
|
+
def naukri_jobs(keyword, location: nil, experience: nil, days: nil, remote: nil, sort: nil, max_results: nil, page: nil, proxy_country: nil)
|
|
2363
|
+
post_json("/jobs/naukri", {
|
|
2364
|
+
keyword: keyword,
|
|
2365
|
+
location: location,
|
|
2366
|
+
experience: experience,
|
|
2367
|
+
days: days,
|
|
2368
|
+
remote: remote,
|
|
2369
|
+
sort: sort,
|
|
2370
|
+
max_results: max_results,
|
|
2371
|
+
page: page,
|
|
2372
|
+
proxy_country: proxy_country,
|
|
2373
|
+
})
|
|
2374
|
+
end
|
|
2375
|
+
|
|
2376
|
+
# Scrape Wellfound startup jobs.
|
|
2377
|
+
#
|
|
2378
|
+
# Searches Wellfound (formerly AngelList Talent) startup jobs by `role`, `location` or
|
|
2379
|
+
# remote `role` (or a Wellfound search `url`) and returns `jobs`: `title`, `url`, the full
|
|
2380
|
+
# `description`, `compensation` with parsed `salary` (`min`, `max`, `currency`) and `equity`
|
|
2381
|
+
# (`min`, `max` in %), `jobType` (`full-time`, `part-time`, `contract`, `internship`),
|
|
2382
|
+
# `locations`, `remote`, `remotePolicy` (`ONSITE`, `REMOTE`, `ONSITE_OR_REMOTE`, `HYBRID`),
|
|
2383
|
+
# `acceptedRemoteLocations`, `yearsExperienceMin`/`Max`, `postedAt` and the hiring `company`
|
|
2384
|
+
# (`name`, `slug`, `url`, `tagline`, `companySize`, `stage`, `badges`). A listing page holds
|
|
2385
|
+
# 20 companies with up to 3 matching jobs each; by default the call returns one page, with
|
|
2386
|
+
# `max_results` it pages through up to that many jobs. Totals: `totalJobs`,
|
|
2387
|
+
# `totalCompanies`, `pageCount`, `nextPage`. `mode=job&job=<url>` returns one job in full:
|
|
2388
|
+
# description (text and HTML), salary with period, the same `jobType` and `remotePolicy`
|
|
2389
|
+
# values (the page's wording in `remotePolicyText`, plus `wfhFlexible`), skills,
|
|
2390
|
+
# `visaSponsorship`, `relocation`, `hiresRemotelyIn`, preferred timezones, hiring contact,
|
|
2391
|
+
# posting date, and the company's size, markets, badges, funding and founders. Billed one
|
|
2392
|
+
# unit per page fetched (`pagesFetched`).
|
|
2393
|
+
#
|
|
2394
|
+
# @param role [String] Role to search, as text or Wellfound slug: 'software engineer',
|
|
2395
|
+
# 'product-manager', 'data-scientist'.
|
|
2396
|
+
# @param location [String] Location, as text or Wellfound slug: 'new york', 'san-francisco',
|
|
2397
|
+
# 'london', 'berlin'.
|
|
2398
|
+
# @param remote [Boolean] Remote jobs only (needs role, no location). API default: false.
|
|
2399
|
+
# @param url [String] A Wellfound search URL instead of role / location / remote:
|
|
2400
|
+
# https://wellfound.com/role/l/software-engineer/san-francisco, /role/r/<role>,
|
|
2401
|
+
# /location/<location>.
|
|
2402
|
+
# @param page [Integer] 1-based listing page to start from (1-500); use `nextPage` of a
|
|
2403
|
+
# previous response to continue. API default: 1.
|
|
2404
|
+
# @param max_results [Integer] How many jobs to collect across pages (1-500). Default: the
|
|
2405
|
+
# whole first page (20 companies with up to 3 jobs each, usually 30-60 jobs). Each page
|
|
2406
|
+
# fetched is one billed unit.
|
|
2407
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2408
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2409
|
+
# Calls POST /jobs/wellfound.
|
|
2410
|
+
def wellfound_jobs(role = nil, location: nil, remote: nil, url: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
2411
|
+
post_json("/jobs/wellfound", {
|
|
2412
|
+
mode: "search",
|
|
2413
|
+
role: role,
|
|
2414
|
+
location: location,
|
|
2415
|
+
remote: remote,
|
|
2416
|
+
url: url,
|
|
2417
|
+
page: page,
|
|
2418
|
+
max_results: max_results,
|
|
2419
|
+
proxy_country: proxy_country,
|
|
2420
|
+
})
|
|
2421
|
+
end
|
|
2422
|
+
|
|
2423
|
+
# Scrape Wellfound startup jobs.
|
|
2424
|
+
#
|
|
2425
|
+
# Searches Wellfound (formerly AngelList Talent) startup jobs by `role`, `location` or
|
|
2426
|
+
# remote `role` (or a Wellfound search `url`) and returns `jobs`: `title`, `url`, the full
|
|
2427
|
+
# `description`, `compensation` with parsed `salary` (`min`, `max`, `currency`) and `equity`
|
|
2428
|
+
# (`min`, `max` in %), `jobType` (`full-time`, `part-time`, `contract`, `internship`),
|
|
2429
|
+
# `locations`, `remote`, `remotePolicy` (`ONSITE`, `REMOTE`, `ONSITE_OR_REMOTE`, `HYBRID`),
|
|
2430
|
+
# `acceptedRemoteLocations`, `yearsExperienceMin`/`Max`, `postedAt` and the hiring `company`
|
|
2431
|
+
# (`name`, `slug`, `url`, `tagline`, `companySize`, `stage`, `badges`). A listing page holds
|
|
2432
|
+
# 20 companies with up to 3 matching jobs each; by default the call returns one page, with
|
|
2433
|
+
# `max_results` it pages through up to that many jobs. Totals: `totalJobs`,
|
|
2434
|
+
# `totalCompanies`, `pageCount`, `nextPage`. `mode=job&job=<url>` returns one job in full:
|
|
2435
|
+
# description (text and HTML), salary with period, the same `jobType` and `remotePolicy`
|
|
2436
|
+
# values (the page's wording in `remotePolicyText`, plus `wfhFlexible`), skills,
|
|
2437
|
+
# `visaSponsorship`, `relocation`, `hiresRemotelyIn`, preferred timezones, hiring contact,
|
|
2438
|
+
# posting date, and the company's size, markets, badges, funding and founders. Billed one
|
|
2439
|
+
# unit per page fetched (`pagesFetched`).
|
|
2440
|
+
#
|
|
2441
|
+
# @param job [String] mode=job: the job URL (https://wellfound.com/jobs/4787364-software-
|
|
2442
|
+
# engineer) or '<id>-<slug>', as returned in `jobs[].url` by mode=search.
|
|
2443
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2444
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2445
|
+
# Calls POST /jobs/wellfound.
|
|
2446
|
+
def wellfound_job(job, proxy_country: nil)
|
|
2447
|
+
post_json("/jobs/wellfound", {
|
|
2448
|
+
mode: "job",
|
|
2449
|
+
job: job,
|
|
2450
|
+
proxy_country: proxy_country,
|
|
2451
|
+
})
|
|
2452
|
+
end
|
|
2453
|
+
|
|
2454
|
+
# Scrape Wellfound startup company profiles.
|
|
2455
|
+
#
|
|
2456
|
+
# Given a Wellfound `company` (slug like `stripe` or a wellfound.com/company/<slug> URL),
|
|
2457
|
+
# returns its profile: `name`, `tagline`, `description`, `website` and social links,
|
|
2458
|
+
# `companySize`, `stage`, `badges`, `markets`, `locations`, `companyTypes`, `hiring`,
|
|
2459
|
+
# `openJobs` with `openJobsByRole`, `totalRaised`, `fundingRounds` (`type`, `date`,
|
|
2460
|
+
# `valuation`), `founders` (name, profile URL, role, bio, location), `teamMembersCount`,
|
|
2461
|
+
# `perks`, `media`, the first open `jobs` (with `jobsTotal`) and `similarCompanies`. Company
|
|
2462
|
+
# profiles sit behind a Cloudflare challenge, so a call can take up to about a minute.
|
|
2463
|
+
# `mode=search` lists Wellfound's startup directory by `location` or `industry` (10
|
|
2464
|
+
# companies per page, paged up to `max_results`) with website, description, markets,
|
|
2465
|
+
# locations and open-job counts. Billed one unit per page fetched (`pagesFetched`).
|
|
2466
|
+
#
|
|
2467
|
+
# @param company [String] mode=profile: the company slug ('stripe') or URL
|
|
2468
|
+
# (https://wellfound.com/company/stripe).
|
|
2469
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2470
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2471
|
+
# Calls POST /companies/wellfound.
|
|
2472
|
+
def wellfound_company(company, proxy_country: nil)
|
|
2473
|
+
post_json("/companies/wellfound", {
|
|
2474
|
+
mode: "profile",
|
|
2475
|
+
company: company,
|
|
2476
|
+
proxy_country: proxy_country,
|
|
2477
|
+
})
|
|
2478
|
+
end
|
|
2479
|
+
|
|
2480
|
+
# Scrape Wellfound startup company profiles.
|
|
2481
|
+
#
|
|
2482
|
+
# Given a Wellfound `company` (slug like `stripe` or a wellfound.com/company/<slug> URL),
|
|
2483
|
+
# returns its profile: `name`, `tagline`, `description`, `website` and social links,
|
|
2484
|
+
# `companySize`, `stage`, `badges`, `markets`, `locations`, `companyTypes`, `hiring`,
|
|
2485
|
+
# `openJobs` with `openJobsByRole`, `totalRaised`, `fundingRounds` (`type`, `date`,
|
|
2486
|
+
# `valuation`), `founders` (name, profile URL, role, bio, location), `teamMembersCount`,
|
|
2487
|
+
# `perks`, `media`, the first open `jobs` (with `jobsTotal`) and `similarCompanies`. Company
|
|
2488
|
+
# profiles sit behind a Cloudflare challenge, so a call can take up to about a minute.
|
|
2489
|
+
# `mode=search` lists Wellfound's startup directory by `location` or `industry` (10
|
|
2490
|
+
# companies per page, paged up to `max_results`) with website, description, markets,
|
|
2491
|
+
# locations and open-job counts. Billed one unit per page fetched (`pagesFetched`).
|
|
2492
|
+
#
|
|
2493
|
+
# @param location [String] mode=search: location as text or slug ('new york', 'london').
|
|
2494
|
+
# @param industry [String] mode=search: Wellfound industry slug ('saas', 'fintech-2',
|
|
2495
|
+
# 'artificial-intelligence', 'health-care', 'e-commerce'; see wellfound.com/browse/tech-
|
|
2496
|
+
# industries). Common names such as 'fintech', 'healthcare', 'crypto' or 'edtech' are mapped
|
|
2497
|
+
# to their slug.
|
|
2498
|
+
# @param url [String] mode=search: a Wellfound directory URL
|
|
2499
|
+
# (https://wellfound.com/startups/location/new-york or /startups/industry/<industry>)
|
|
2500
|
+
# instead of location / industry.
|
|
2501
|
+
# @param page [Integer] mode=search: 1-based page to start from (10 companies per page). API
|
|
2502
|
+
# default: 1.
|
|
2503
|
+
# @param max_results [Integer] mode=search: how many companies to collect (1-100); each page
|
|
2504
|
+
# fetched is one billed unit. API default: 10.
|
|
2505
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2506
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2507
|
+
# Calls POST /companies/wellfound.
|
|
2508
|
+
def wellfound_company_search(location: nil, industry: nil, url: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
2509
|
+
post_json("/companies/wellfound", {
|
|
2510
|
+
mode: "search",
|
|
2511
|
+
location: location,
|
|
2512
|
+
industry: industry,
|
|
2513
|
+
url: url,
|
|
2514
|
+
page: page,
|
|
2515
|
+
max_results: max_results,
|
|
2516
|
+
proxy_country: proxy_country,
|
|
2517
|
+
})
|
|
2518
|
+
end
|
|
2519
|
+
|
|
2520
|
+
# Scrape company career sites (Greenhouse, Lever, Ashby, Workday and more).
|
|
2521
|
+
#
|
|
2522
|
+
# Every open job on a company's career site hosted on Greenhouse, Lever, Ashby, Workday,
|
|
2523
|
+
# SmartRecruiters, Workable or Recruitee, from its career page `url` (or `provider` +
|
|
2524
|
+
# `company`). Each job (`jobs`) has `id`, `title`, `company`, `provider`, `url`, `applyUrl`,
|
|
2525
|
+
# `location` and `locations`, `country`, `remote`, `workplaceType` (remote / hybrid /
|
|
2526
|
+
# onsite), `department`, `team`, `employmentType`, `postedAt`, `updatedAt`, `requisitionId`,
|
|
2527
|
+
# `salary` and `salaryRanges` (min, max, currency, interval) where the ATS publishes pay,
|
|
2528
|
+
# `customFields` (the ATS's extra fields) and, with `include_description=true`,
|
|
2529
|
+
# `descriptionText` and `descriptionHtml`. A field the ATS does not expose is null. Filter
|
|
2530
|
+
# with `keyword` (title words), `location`, `department` and `remote`; `limit` caps the jobs
|
|
2531
|
+
# returned. The answer also carries the detected `provider`, `company`, `companyName`,
|
|
2532
|
+
# `boardUrl`, `totalJobs` on the board and `matchedJobs`. A link to one posting returns just
|
|
2533
|
+
# that posting with its description. Billed one unit per upstream request that returned data
|
|
2534
|
+
# (`pagesFetched`): one for a whole Greenhouse, Ashby, Workable or Recruitee board, one per
|
|
2535
|
+
# Lever (100 jobs), Workday (20 jobs) or SmartRecruiters (100 jobs) listing page, plus one
|
|
2536
|
+
# per Workday / SmartRecruiters posting whose description is read.
|
|
2537
|
+
#
|
|
2538
|
+
# @param url [String] The company's career page on its ATS, e.g. 'https://job-
|
|
2539
|
+
# boards.greenhouse.io/stripe', 'https://jobs.lever.co/palantir',
|
|
2540
|
+
# 'https://jobs.ashbyhq.com/ramp',
|
|
2541
|
+
# 'https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite',
|
|
2542
|
+
# 'https://jobs.smartrecruiters.com/Ubisoft2', 'https://apply.workable.com/huggingface',
|
|
2543
|
+
# 'https://bunq.recruitee.com'. A link to one posting returns just that posting.
|
|
2544
|
+
# @param provider [String] Instead of `url`: the ATS (greenhouse, lever, ashby, workday,
|
|
2545
|
+
# smartrecruiters, workable, recruitee), with `company`.
|
|
2546
|
+
# @param company [String] Instead of `url`: the company's board name on that ATS, as in its
|
|
2547
|
+
# career URL: Greenhouse token ('stripe'), Lever / Ashby / Workable / Recruitee slug,
|
|
2548
|
+
# SmartRecruiters company id ('Ubisoft2'), Workday '<tenant>.wd<N>/<site>'
|
|
2549
|
+
# ('nvidia.wd5/NVIDIAExternalCareerSite').
|
|
2550
|
+
# @param keyword [String] Only jobs whose title contains every word, e.g. 'software
|
|
2551
|
+
# engineer'.
|
|
2552
|
+
# @param location [String] Only jobs in this place (city, region or country as the company
|
|
2553
|
+
# writes it, e.g. 'London', 'Germany', 'Remote'); case-insensitive.
|
|
2554
|
+
# @param department [String] Only jobs in this department or team, e.g. 'Engineering'.
|
|
2555
|
+
# @param remote [Boolean] Only remote jobs. API default: false.
|
|
2556
|
+
# @param limit [Integer] How many jobs to return (1-2000). API default: 100.
|
|
2557
|
+
# @param include_description [Boolean] Add each job's full description (`descriptionText`,
|
|
2558
|
+
# `descriptionHtml`). On Workday and SmartRecruiters this reads every posting separately
|
|
2559
|
+
# (one billed request each, at most 300 per call). API default: false.
|
|
2560
|
+
# @param proxy_country [String] Proxy country the requests are sent from (ISO-2, e.g. 'US',
|
|
2561
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2562
|
+
# Calls POST /jobs/career-site.
|
|
2563
|
+
def career_site_jobs(url = nil, provider: nil, company: nil, keyword: nil, location: nil, department: nil, remote: nil, limit: nil, include_description: nil, proxy_country: nil)
|
|
2564
|
+
post_json("/jobs/career-site", {
|
|
2565
|
+
url: url,
|
|
2566
|
+
provider: provider,
|
|
2567
|
+
company: company,
|
|
2568
|
+
keyword: keyword,
|
|
2569
|
+
location: location,
|
|
2570
|
+
department: department,
|
|
2571
|
+
remote: remote,
|
|
2572
|
+
limit: limit,
|
|
2573
|
+
include_description: include_description,
|
|
2574
|
+
proxy_country: proxy_country,
|
|
2575
|
+
})
|
|
2576
|
+
end
|
|
2577
|
+
|
|
2578
|
+
# Scrape Trustpilot company reviews.
|
|
2579
|
+
#
|
|
2580
|
+
# Returns a company's Trustpilot profile and reviews as JSON. `company` is the reviewed
|
|
2581
|
+
# domain (e.g. `www.nike.com`). The response has a `company` summary (`trustScore`,
|
|
2582
|
+
# `totalReviews`, `ratingDistribution` 1-5, `categories`, `contact`) and a `reviews` array -
|
|
2583
|
+
# each with `rating`, `title`, `text`, `verified`, `author` (name, country, review count),
|
|
2584
|
+
# `reply` (the company's response) and dates. Page with `page`/`pages` (20 reviews per
|
|
2585
|
+
# page), and filter with `stars`, `sort` (recency/relevancy), `language`, `date`, `search`,
|
|
2586
|
+
# `verified` or `replies`.
|
|
2587
|
+
#
|
|
2588
|
+
# @param company [String] The reviewed company's domain, e.g. 'www.nike.com' or 'nike.com'.
|
|
2589
|
+
# A full site URL or a pasted Trustpilot review URL is also accepted.
|
|
2590
|
+
# @param page [Integer] 1-based page to start from (20 reviews/page). API default: 1.
|
|
2591
|
+
# @param pages [Integer] How many consecutive pages to fetch from `page` onward (20 reviews
|
|
2592
|
+
# each). Stops early at the last page. API default: 1.
|
|
2593
|
+
# @param stars [Array<Integer>] Only reviews with these star ratings (1, 2, 3, 4, 5).
|
|
2594
|
+
# Repeatable, e.g. stars=4&stars=5.
|
|
2595
|
+
# @param sort [String] Ordering: recency, relevancy. Default 'recency' (newest first). API
|
|
2596
|
+
# default: recency.
|
|
2597
|
+
# @param language [String] Review language ISO-2 code (e.g. 'en', 'de') or 'all'. Default
|
|
2598
|
+
# 'all'. API default: all.
|
|
2599
|
+
# @param date [String] Only reviews from a recent window: last30days, last3months,
|
|
2600
|
+
# last6months, last12months. Omit for all time.
|
|
2601
|
+
# @param search [String] Keyword to search within the company's reviews.
|
|
2602
|
+
# @param verified [Boolean] Only verified reviews (Trustpilot's verified badge). API
|
|
2603
|
+
# default: false.
|
|
2604
|
+
# @param replies [Boolean] Only reviews the company has replied to. API default: false.
|
|
2605
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2606
|
+
# 'DE'). Defaults to a country chosen for the site.
|
|
2607
|
+
# Calls POST /reviews/trustpilot.
|
|
2608
|
+
def trustpilot_reviews(company, page: nil, pages: nil, stars: nil, sort: nil, language: nil, date: nil, search: nil, verified: nil, replies: nil, proxy_country: nil)
|
|
2609
|
+
post_json("/reviews/trustpilot", {
|
|
2610
|
+
company: company,
|
|
2611
|
+
page: page,
|
|
2612
|
+
pages: pages,
|
|
2613
|
+
stars: stars,
|
|
2614
|
+
sort: sort,
|
|
2615
|
+
language: language,
|
|
2616
|
+
date: date,
|
|
2617
|
+
search: search,
|
|
2618
|
+
verified: verified,
|
|
2619
|
+
replies: replies,
|
|
2620
|
+
proxy_country: proxy_country,
|
|
2621
|
+
})
|
|
2622
|
+
end
|
|
2623
|
+
|
|
2624
|
+
# Scrape G2 software reviews.
|
|
2625
|
+
#
|
|
2626
|
+
# Given a G2 `product` (slug like `slack`, or a g2.com product URL), returns the product
|
|
2627
|
+
# summary (`name`, `rating`, `reviewCount`, `starDistribution`, `categories`, `aiSummary`,
|
|
2628
|
+
# `pros`, `cons`, `segments`) and its `reviews`: `title`, `rating`, `date`, `reviewer` (job
|
|
2629
|
+
# title, industry, company size), `likeBest`, `dislike`, `problemsSolved`, the full
|
|
2630
|
+
# `answers`, badges (`validatedReviewer`, `currentUser`, `incentivized`, `source`) and
|
|
2631
|
+
# `vendorResponse`. Pages through G2's review pages up to `max_results` (10 per page, at
|
|
2632
|
+
# most 10 pages per view); pick the view with `sort`, `stars` and `search`. `details=true`
|
|
2633
|
+
# adds the vendor, description, website and languages. `mode=search&query=` searches G2's
|
|
2634
|
+
# product catalog. Billed one unit per page fetched (`pagesFetched`).
|
|
2635
|
+
#
|
|
2636
|
+
# @param product [String] G2 product slug, e.g. 'slack' or 'hubspot-sales-hub', or a g2.com
|
|
2637
|
+
# product URL such as https://www.g2.com/products/slack/reviews. Required unless
|
|
2638
|
+
# mode=search.
|
|
2639
|
+
# @param page [Integer] 1-based page to start from: 10 reviews per page, at most 10 (G2's
|
|
2640
|
+
# limit per sort/filter view); with mode=search 20 products per page, at most 20. Use the
|
|
2641
|
+
# `nextPage` of a previous response to continue. API default: 1.
|
|
2642
|
+
# @param max_results [Integer] How many reviews to collect across pages (1-100, default 10).
|
|
2643
|
+
# With mode=search, how many products (1-100, default 20) - search answers whole pages of
|
|
2644
|
+
# 20, so it can return a few more. Each page fetched is one billed unit.
|
|
2645
|
+
# @param sort [String] Review order: g2_default, most_recent, most_helpful, highest_rated,
|
|
2646
|
+
# lowest_rated. Default: G2's own ranking. API default: g2_default.
|
|
2647
|
+
# @param stars [Integer] Only reviews with this star rating (1-5).
|
|
2648
|
+
# @param search [String] Only reviews whose answers mention this keyword.
|
|
2649
|
+
# @param details [Boolean] Also fetch the product page for its vendor, description, website
|
|
2650
|
+
# and languages (one more billed unit). API default: false.
|
|
2651
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2652
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2653
|
+
# Calls POST /reviews/g2.
|
|
2654
|
+
def g2_reviews(product, page: nil, max_results: nil, sort: nil, stars: nil, search: nil, details: nil, proxy_country: nil)
|
|
2655
|
+
post_json("/reviews/g2", {
|
|
2656
|
+
mode: "reviews",
|
|
2657
|
+
product: product,
|
|
2658
|
+
page: page,
|
|
2659
|
+
max_results: max_results,
|
|
2660
|
+
sort: sort,
|
|
2661
|
+
stars: stars,
|
|
2662
|
+
search: search,
|
|
2663
|
+
details: details,
|
|
2664
|
+
proxy_country: proxy_country,
|
|
2665
|
+
})
|
|
2666
|
+
end
|
|
2667
|
+
|
|
2668
|
+
# Scrape G2 software reviews.
|
|
2669
|
+
#
|
|
2670
|
+
# Given a G2 `product` (slug like `slack`, or a g2.com product URL), returns the product
|
|
2671
|
+
# summary (`name`, `rating`, `reviewCount`, `starDistribution`, `categories`, `aiSummary`,
|
|
2672
|
+
# `pros`, `cons`, `segments`) and its `reviews`: `title`, `rating`, `date`, `reviewer` (job
|
|
2673
|
+
# title, industry, company size), `likeBest`, `dislike`, `problemsSolved`, the full
|
|
2674
|
+
# `answers`, badges (`validatedReviewer`, `currentUser`, `incentivized`, `source`) and
|
|
2675
|
+
# `vendorResponse`. Pages through G2's review pages up to `max_results` (10 per page, at
|
|
2676
|
+
# most 10 pages per view); pick the view with `sort`, `stars` and `search`. `details=true`
|
|
2677
|
+
# adds the vendor, description, website and languages. `mode=search&query=` searches G2's
|
|
2678
|
+
# product catalog. Billed one unit per page fetched (`pagesFetched`).
|
|
2679
|
+
#
|
|
2680
|
+
# @param query [String] Keyword to search G2 products for (mode=search), e.g. 'crm'.
|
|
2681
|
+
# @param page [Integer] 1-based page to start from: 10 reviews per page, at most 10 (G2's
|
|
2682
|
+
# limit per sort/filter view); with mode=search 20 products per page, at most 20. Use the
|
|
2683
|
+
# `nextPage` of a previous response to continue. API default: 1.
|
|
2684
|
+
# @param max_results [Integer] How many reviews to collect across pages (1-100, default 10).
|
|
2685
|
+
# With mode=search, how many products (1-100, default 20) - search answers whole pages of
|
|
2686
|
+
# 20, so it can return a few more. Each page fetched is one billed unit.
|
|
2687
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2688
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2689
|
+
# Calls POST /reviews/g2.
|
|
2690
|
+
def g2_search(query, page: nil, max_results: nil, proxy_country: nil)
|
|
2691
|
+
post_json("/reviews/g2", {
|
|
2692
|
+
mode: "search",
|
|
2693
|
+
query: query,
|
|
2694
|
+
page: page,
|
|
2695
|
+
max_results: max_results,
|
|
2696
|
+
proxy_country: proxy_country,
|
|
2697
|
+
})
|
|
2698
|
+
end
|
|
2699
|
+
|
|
2700
|
+
# Scrape Capterra software reviews.
|
|
2701
|
+
#
|
|
2702
|
+
# Given a Capterra `product` (numeric id like `135003`, `id/slug` like `135003/Slack`, or a
|
|
2703
|
+
# capterra.com/p/<id>/<slug>/ URL), returns the product summary (`name`, `vendor`, `rating`,
|
|
2704
|
+
# `reviewCount`, sub-`ratings` `easeOfUse`, `customerSupport` (Capterra's "Customer Service"
|
|
2705
|
+
# rating), `features` and `valueForMoney`, `categories`, `description`, `pricing`) and its
|
|
2706
|
+
# `reviews`: `title`, `rating`, the same sub-`ratings` with `likelihoodToRecommend` (0-10),
|
|
2707
|
+
# `date`, `reviewer` (job title, industry, company size, time used, validations),
|
|
2708
|
+
# `comments`, `pros`, `cons`, `reasonsForChoosing`, `reasonsForSwitching` with
|
|
2709
|
+
# `alternativesConsidered` / `switchedFrom`, `advice`, the incentive badges (`incentive`,
|
|
2710
|
+
# `incentivized`, `vendorReferred`, `reviewSource`), `source` (Capterra, GetApp or Software
|
|
2711
|
+
# Advice) and `vendorResponse`. Pages through Capterra's review pages up to `max_results`
|
|
2712
|
+
# (25 per page, at most 100 pages, in Capterra's "Most Helpful" order). A product with only
|
|
2713
|
+
# a few reviews has no review pages; its reviews come from its profile in one page. A
|
|
2714
|
+
# product without reviews answers 200 with an empty `reviews` list. `mode=search&query=`
|
|
2715
|
+
# searches Capterra's product catalog (one page of up to 20 products, plus matching
|
|
2716
|
+
# categories). A bare product id costs one extra fetch the first time a page past the first
|
|
2717
|
+
# is asked for (Capterra redirects it to the canonical slug); `id/slug` (echoed in `query`)
|
|
2718
|
+
# avoids it. Billed one unit per page fetched (`pagesFetched`).
|
|
2719
|
+
#
|
|
2720
|
+
# @param product [String] Capterra product: its numeric id ('135003'), 'id/slug'
|
|
2721
|
+
# ('135003/Slack') or a capterra.com product URL such as
|
|
2722
|
+
# https://www.capterra.com/p/135003/Slack/reviews/. Required unless mode=search.
|
|
2723
|
+
# @param slug [String] Optional product slug ('Slack') to go with a bare id. Capterra finds
|
|
2724
|
+
# the product by id alone; the canonical slug is returned.
|
|
2725
|
+
# @param page [Integer] 1-based review page to start from: 25 reviews per page, at most 100
|
|
2726
|
+
# (Capterra's limit per product). Use the `nextPage` of a previous response to continue. API
|
|
2727
|
+
# default: 1.
|
|
2728
|
+
# @param max_results [Integer] How many reviews to collect across pages (1-500, default 25);
|
|
2729
|
+
# each page fetched is one billed unit. With mode=search, how many products (1-20, default
|
|
2730
|
+
# 20; one page).
|
|
2731
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2732
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2733
|
+
# Calls POST /reviews/capterra.
|
|
2734
|
+
def capterra_reviews(product, slug: nil, page: nil, max_results: nil, proxy_country: nil)
|
|
2735
|
+
post_json("/reviews/capterra", {
|
|
2736
|
+
mode: "reviews",
|
|
2737
|
+
product: product,
|
|
2738
|
+
slug: slug,
|
|
2739
|
+
page: page,
|
|
2740
|
+
max_results: max_results,
|
|
2741
|
+
proxy_country: proxy_country,
|
|
2742
|
+
})
|
|
2743
|
+
end
|
|
2744
|
+
|
|
2745
|
+
# Scrape Capterra software reviews.
|
|
2746
|
+
#
|
|
2747
|
+
# Given a Capterra `product` (numeric id like `135003`, `id/slug` like `135003/Slack`, or a
|
|
2748
|
+
# capterra.com/p/<id>/<slug>/ URL), returns the product summary (`name`, `vendor`, `rating`,
|
|
2749
|
+
# `reviewCount`, sub-`ratings` `easeOfUse`, `customerSupport` (Capterra's "Customer Service"
|
|
2750
|
+
# rating), `features` and `valueForMoney`, `categories`, `description`, `pricing`) and its
|
|
2751
|
+
# `reviews`: `title`, `rating`, the same sub-`ratings` with `likelihoodToRecommend` (0-10),
|
|
2752
|
+
# `date`, `reviewer` (job title, industry, company size, time used, validations),
|
|
2753
|
+
# `comments`, `pros`, `cons`, `reasonsForChoosing`, `reasonsForSwitching` with
|
|
2754
|
+
# `alternativesConsidered` / `switchedFrom`, `advice`, the incentive badges (`incentive`,
|
|
2755
|
+
# `incentivized`, `vendorReferred`, `reviewSource`), `source` (Capterra, GetApp or Software
|
|
2756
|
+
# Advice) and `vendorResponse`. Pages through Capterra's review pages up to `max_results`
|
|
2757
|
+
# (25 per page, at most 100 pages, in Capterra's "Most Helpful" order). A product with only
|
|
2758
|
+
# a few reviews has no review pages; its reviews come from its profile in one page. A
|
|
2759
|
+
# product without reviews answers 200 with an empty `reviews` list. `mode=search&query=`
|
|
2760
|
+
# searches Capterra's product catalog (one page of up to 20 products, plus matching
|
|
2761
|
+
# categories). A bare product id costs one extra fetch the first time a page past the first
|
|
2762
|
+
# is asked for (Capterra redirects it to the canonical slug); `id/slug` (echoed in `query`)
|
|
2763
|
+
# avoids it. Billed one unit per page fetched (`pagesFetched`).
|
|
2764
|
+
#
|
|
2765
|
+
# @param query [String] Product name or keyword to search for (mode=search), e.g. 'crm'.
|
|
2766
|
+
# @param max_results [Integer] How many reviews to collect across pages (1-500, default 25);
|
|
2767
|
+
# each page fetched is one billed unit. With mode=search, how many products (1-20, default
|
|
2768
|
+
# 20; one page).
|
|
2769
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2770
|
+
# 'DE'). Leave blank for the country chosen for the site.
|
|
2771
|
+
# Calls POST /reviews/capterra.
|
|
2772
|
+
def capterra_search(query, max_results: nil, proxy_country: nil)
|
|
2773
|
+
post_json("/reviews/capterra", {
|
|
2774
|
+
mode: "search",
|
|
2775
|
+
query: query,
|
|
2776
|
+
max_results: max_results,
|
|
2777
|
+
proxy_country: proxy_country,
|
|
2778
|
+
})
|
|
2779
|
+
end
|
|
2780
|
+
|
|
2781
|
+
# Scrape Product Hunt launches, products, search and topics.
|
|
2782
|
+
#
|
|
2783
|
+
# Turns Product Hunt into JSON. `mode=leaderboard` returns a day's ranked launches
|
|
2784
|
+
# (`date`=YYYY-MM-DD, or today) with votes, comments, topics and rank; `mode=search` returns
|
|
2785
|
+
# products matching `query`; `mode=topic` returns products under a `topic` slug;
|
|
2786
|
+
# `mode=detail` returns one product by `slug` with description, website, socials, review
|
|
2787
|
+
# rating, followers and categories. Set `max_results` for the list modes.
|
|
2788
|
+
#
|
|
2789
|
+
# @param mode [String] What to fetch: leaderboard, search, topic, detail. API default:
|
|
2790
|
+
# leaderboard.
|
|
2791
|
+
# @param query [String] Search terms (mode=search), e.g. 'ai chatbot'.
|
|
2792
|
+
# @param date [String] Leaderboard day as YYYY-MM-DD (mode=leaderboard). Omit for today.
|
|
2793
|
+
# Past days are finalised and ranked; today's board is live and not ranked yet.
|
|
2794
|
+
# @param topic [String] Topic slug (mode=topic), e.g. 'artificial-intelligence'.
|
|
2795
|
+
# @param slug [String] Product slug (mode=detail), e.g. 'notion'.
|
|
2796
|
+
# @param max_results [Integer] Max products to return for list modes
|
|
2797
|
+
# (leaderboard/search/topic). API default: 30.
|
|
2798
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2799
|
+
# 'DE'). Leave blank - Product Hunt is global and a forced country only hurts.
|
|
2800
|
+
# Calls POST /products/producthunt.
|
|
2801
|
+
def product_hunt(mode: nil, query: nil, date: nil, topic: nil, slug: nil, max_results: nil, proxy_country: nil)
|
|
2802
|
+
post_json("/products/producthunt", {
|
|
2803
|
+
mode: mode,
|
|
2804
|
+
query: query,
|
|
2805
|
+
date: date,
|
|
2806
|
+
topic: topic,
|
|
2807
|
+
slug: slug,
|
|
2808
|
+
max_results: max_results,
|
|
2809
|
+
proxy_country: proxy_country,
|
|
2810
|
+
})
|
|
2811
|
+
end
|
|
2812
|
+
|
|
2813
|
+
# Scrape a Crunchbase company profile, or search companies by name.
|
|
2814
|
+
#
|
|
2815
|
+
# With `organization` (a Crunchbase slug like 'stripe' or a crunchbase.com/organization/...
|
|
2816
|
+
# URL) returns the organization as JSON: `name`, `slug`, `uuid`, `url`, `legalName`,
|
|
2817
|
+
# `aliases`, `shortDescription`, `description`, `logo`, `website`, `headquarters` (city,
|
|
2818
|
+
# region, country, continent), `operatingStatus`, `companyType`, `ipoStatus`, `stockSymbol`
|
|
2819
|
+
# + `stockExchange`, `employeeRange`, `industries`, `industryGroups`, `hubTags`, CB `rank`
|
|
2820
|
+
# with `rankDelta90d` (Crunchbase's 90-day trend score for the rank, from -10 to 10:
|
|
2821
|
+
# positive means the company moved up the ranking, toward #1), `growthScore` and `heatScore`
|
|
2822
|
+
# (0-100) with `growthScoreDelta90d` and `heatScoreDelta90d` (their change in points over 90
|
|
2823
|
+
# days), `founders`, `lastFundingType`, counts (`numFundingRounds`, `numInvestors`,
|
|
2824
|
+
# `numAcquisitions`, `numInvestments`, `numEmployeeProfiles` ...), `acquiredBy`, `owner`,
|
|
2825
|
+
# the latest `fundingRounds` (type, investors, lead investors, each with `name`, `slug`,
|
|
2826
|
+
# `url`, `type` and `image` like every other company or person reference), `investors`,
|
|
2827
|
+
# `acquisitions`, `investments`, `exits`, `subOrganizations`, `keyPeople` (title, start
|
|
2828
|
+
# date), `boardMembers`, `similarCompanies`, `products`, `contactEmail`, `phone`,
|
|
2829
|
+
# `socialLinks`, `webTraffic` (`monthlyVisits`, and `monthlyVisitsChange`, the month-over-
|
|
2830
|
+
# month change as a fraction: 0.069 = +6.9%, -0.2 = -20%), `technology`, `predictions`,
|
|
2831
|
+
# `events` (product launches, partnerships, awards, layoffs, key employee changes, legal
|
|
2832
|
+
# proceedings, each with its press source), `offices` and recent `news`. Every one of the
|
|
2833
|
+
# record's 73 keys is always present. Data Crunchbase shows only to paying users (founded /
|
|
2834
|
+
# closed / IPO dates, funding totals, round dates and amounts, acquisition prices, exits,
|
|
2835
|
+
# valuation) is null, and `paywalledFields` names the ones Crunchbase holds for this
|
|
2836
|
+
# organization. With `query` instead, searches organizations by name and returns up to
|
|
2837
|
+
# `max_results` (max 25) matches with `name`, `slug`, `url`, `shortDescription` and `logo`,
|
|
2838
|
+
# plus the total `count` of matches. No Crunchbase login required.
|
|
2839
|
+
#
|
|
2840
|
+
# @param organization [String] Crunchbase organization slug (the part after /organization/
|
|
2841
|
+
# in its profile URL, e.g. 'stripe' or 'tesla-motors') or the full profile URL. Either
|
|
2842
|
+
# organization or query is required.
|
|
2843
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2844
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2845
|
+
# Calls POST /companies/crunchbase.
|
|
2846
|
+
def crunchbase_company(organization, proxy_country: nil)
|
|
2847
|
+
post_json("/companies/crunchbase", {
|
|
2848
|
+
organization: organization,
|
|
2849
|
+
proxy_country: proxy_country,
|
|
2850
|
+
})
|
|
2851
|
+
end
|
|
2852
|
+
|
|
2853
|
+
# Scrape a Crunchbase company profile, or search companies by name.
|
|
2854
|
+
#
|
|
2855
|
+
# With `organization` (a Crunchbase slug like 'stripe' or a crunchbase.com/organization/...
|
|
2856
|
+
# URL) returns the organization as JSON: `name`, `slug`, `uuid`, `url`, `legalName`,
|
|
2857
|
+
# `aliases`, `shortDescription`, `description`, `logo`, `website`, `headquarters` (city,
|
|
2858
|
+
# region, country, continent), `operatingStatus`, `companyType`, `ipoStatus`, `stockSymbol`
|
|
2859
|
+
# + `stockExchange`, `employeeRange`, `industries`, `industryGroups`, `hubTags`, CB `rank`
|
|
2860
|
+
# with `rankDelta90d` (Crunchbase's 90-day trend score for the rank, from -10 to 10:
|
|
2861
|
+
# positive means the company moved up the ranking, toward #1), `growthScore` and `heatScore`
|
|
2862
|
+
# (0-100) with `growthScoreDelta90d` and `heatScoreDelta90d` (their change in points over 90
|
|
2863
|
+
# days), `founders`, `lastFundingType`, counts (`numFundingRounds`, `numInvestors`,
|
|
2864
|
+
# `numAcquisitions`, `numInvestments`, `numEmployeeProfiles` ...), `acquiredBy`, `owner`,
|
|
2865
|
+
# the latest `fundingRounds` (type, investors, lead investors, each with `name`, `slug`,
|
|
2866
|
+
# `url`, `type` and `image` like every other company or person reference), `investors`,
|
|
2867
|
+
# `acquisitions`, `investments`, `exits`, `subOrganizations`, `keyPeople` (title, start
|
|
2868
|
+
# date), `boardMembers`, `similarCompanies`, `products`, `contactEmail`, `phone`,
|
|
2869
|
+
# `socialLinks`, `webTraffic` (`monthlyVisits`, and `monthlyVisitsChange`, the month-over-
|
|
2870
|
+
# month change as a fraction: 0.069 = +6.9%, -0.2 = -20%), `technology`, `predictions`,
|
|
2871
|
+
# `events` (product launches, partnerships, awards, layoffs, key employee changes, legal
|
|
2872
|
+
# proceedings, each with its press source), `offices` and recent `news`. Every one of the
|
|
2873
|
+
# record's 73 keys is always present. Data Crunchbase shows only to paying users (founded /
|
|
2874
|
+
# closed / IPO dates, funding totals, round dates and amounts, acquisition prices, exits,
|
|
2875
|
+
# valuation) is null, and `paywalledFields` names the ones Crunchbase holds for this
|
|
2876
|
+
# organization. With `query` instead, searches organizations by name and returns up to
|
|
2877
|
+
# `max_results` (max 25) matches with `name`, `slug`, `url`, `shortDescription` and `logo`,
|
|
2878
|
+
# plus the total `count` of matches. No Crunchbase login required.
|
|
2879
|
+
#
|
|
2880
|
+
# @param query [String] Company name to search for, e.g. 'Revolut'. Returns the matching
|
|
2881
|
+
# organizations with their slugs instead of one full profile.
|
|
2882
|
+
# @param max_results [Integer] Search matches to return with query (1-25). API default: 10.
|
|
2883
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2884
|
+
# 'DE'). Leave blank to let the API pick.
|
|
2885
|
+
# Calls POST /companies/crunchbase.
|
|
2886
|
+
def crunchbase_search(query, max_results: nil, proxy_country: nil)
|
|
2887
|
+
post_json("/companies/crunchbase", {
|
|
2888
|
+
query: query,
|
|
2889
|
+
max_results: max_results,
|
|
2890
|
+
proxy_country: proxy_country,
|
|
2891
|
+
})
|
|
2892
|
+
end
|
|
2893
|
+
|
|
2894
|
+
# Scrape Yelp businesses (search + detail).
|
|
2895
|
+
#
|
|
2896
|
+
# Two modes. **Search** (`mode=search`): give `find_desc` (a keyword or category) and
|
|
2897
|
+
# `find_loc` (a city/area) and get the businesses as JSON - `name`, `rating`, `reviewCount`,
|
|
2898
|
+
# `priceRange`, `categories`, `address`, `neighborhoods`, `url`, `businessId`, `image` and
|
|
2899
|
+
# `isAd`. Filter with `sort`, `price` (1-4) and `open_now`, and collect up to `max_results`
|
|
2900
|
+
# across pages. **Detail** (`mode=detail`): give a business `url` and get a rich object -
|
|
2901
|
+
# phone, full address, website, hours, attributes, categories, specialties, review
|
|
2902
|
+
# highlights, photos and popular dishes.
|
|
2903
|
+
#
|
|
2904
|
+
# @param find_desc [String] Search keyword or category, e.g. 'coffee' or 'plumbers'.
|
|
2905
|
+
# @param find_loc [String] Location to search, e.g. 'San Francisco, CA' or '94109'.
|
|
2906
|
+
# @param sort [String] Search ordering: recommended, rating, review_count, distance. API
|
|
2907
|
+
# default: recommended.
|
|
2908
|
+
# @param price [String] Price filter 1-4 ($ to $$$$); combine like '1,2'.
|
|
2909
|
+
# @param open_now [Boolean] Search: only businesses open now. API default: false.
|
|
2910
|
+
# @param max_results [Integer] Search: how many businesses to collect across pages (10 per
|
|
2911
|
+
# page). API default: 30.
|
|
2912
|
+
# @param start [Integer] Search: result offset to start from. API default: 0.
|
|
2913
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2914
|
+
# 'DE'). Defaults to US.
|
|
2915
|
+
# Calls POST /local/yelp.
|
|
2916
|
+
def yelp_search(find_desc, find_loc, sort: nil, price: nil, open_now: nil, max_results: nil, start: nil, proxy_country: nil)
|
|
2917
|
+
post_json("/local/yelp", {
|
|
2918
|
+
mode: "search",
|
|
2919
|
+
find_desc: find_desc,
|
|
2920
|
+
find_loc: find_loc,
|
|
2921
|
+
sort: sort,
|
|
2922
|
+
price: price,
|
|
2923
|
+
open_now: open_now,
|
|
2924
|
+
max_results: max_results,
|
|
2925
|
+
start: start,
|
|
2926
|
+
proxy_country: proxy_country,
|
|
2927
|
+
})
|
|
2928
|
+
end
|
|
2929
|
+
|
|
2930
|
+
# Scrape Yelp businesses (search + detail).
|
|
2931
|
+
#
|
|
2932
|
+
# Two modes. **Search** (`mode=search`): give `find_desc` (a keyword or category) and
|
|
2933
|
+
# `find_loc` (a city/area) and get the businesses as JSON - `name`, `rating`, `reviewCount`,
|
|
2934
|
+
# `priceRange`, `categories`, `address`, `neighborhoods`, `url`, `businessId`, `image` and
|
|
2935
|
+
# `isAd`. Filter with `sort`, `price` (1-4) and `open_now`, and collect up to `max_results`
|
|
2936
|
+
# across pages. **Detail** (`mode=detail`): give a business `url` and get a rich object -
|
|
2937
|
+
# phone, full address, website, hours, attributes, categories, specialties, review
|
|
2938
|
+
# highlights, photos and popular dishes.
|
|
2939
|
+
#
|
|
2940
|
+
# @param url [String] Detail mode: a Yelp business URL or /biz alias, e.g.
|
|
2941
|
+
# 'https://www.yelp.com/biz/gary-danko-san-francisco'.
|
|
2942
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2943
|
+
# 'DE'). Defaults to US.
|
|
2944
|
+
# Calls POST /local/yelp.
|
|
2945
|
+
def yelp_business(url, proxy_country: nil)
|
|
2946
|
+
post_json("/local/yelp", {
|
|
2947
|
+
mode: "detail",
|
|
2948
|
+
url: url,
|
|
2949
|
+
proxy_country: proxy_country,
|
|
2950
|
+
})
|
|
2951
|
+
end
|
|
2952
|
+
|
|
2953
|
+
# Scrape Yelp business reviews.
|
|
2954
|
+
#
|
|
2955
|
+
# Given a Yelp `business` (page URL, alias or business id), returns the `business` summary
|
|
2956
|
+
# (`businessId`, `alias`, `name`, `url`, `rating`, `reviewCount`, `ratingHistogram`,
|
|
2957
|
+
# `reviewsCountByLanguage`, `categories`, `city`, `region`) and its `reviews`: `id`, `url`,
|
|
2958
|
+
# `rating`, `text`, `language`, `publishedDate`, `date`, `user` (`name`, `location`,
|
|
2959
|
+
# `reviewCount`, `friendCount`, `photoCount`, `isElite`, `eliteYear`), `photos`, `videos`,
|
|
2960
|
+
# `reactions` (`helpful`, `thanks`, `loveThis`, `ohNo`), `ownerResponse`, `checkIns` and the
|
|
2961
|
+
# author's `previousReviews`. Sort like yelp.com (`relevance`, `newest`, `oldest`,
|
|
2962
|
+
# `highest`, `lowest`, `elites`), filter by `language`, star `rating`, a search `query` or
|
|
2963
|
+
# `since` a date, and page with `page` / `start` up to `max_reviews` per call. To continue,
|
|
2964
|
+
# call again with `start` = `nextStart` while `hasMore` is true: `nextStart` is the exact
|
|
2965
|
+
# offset where this call stopped. `nextPage` is only set when that offset falls on a page
|
|
2966
|
+
# boundary (a multiple of 10) and is null otherwise, because a page number would repeat
|
|
2967
|
+
# reviews already returned. Counts: `business.reviewCount` is the review count Yelp shows
|
|
2968
|
+
# for the business; `business.ratingHistogram` and `business.reviewsCountByLanguage` count
|
|
2969
|
+
# the reviews in every language, and Yelp updates them separately from `reviewCount`, so
|
|
2970
|
+
# their sums can differ from it by a few; `totalResults` is how many reviews match this
|
|
2971
|
+
# call's `language`, `rating` and `query` (the total that `start` pages through). A business
|
|
2972
|
+
# with no reviews returns an empty list; an unknown alias or URL is a 404. Billed one unit
|
|
2973
|
+
# per 10 reviews returned (`pagesFetched`).
|
|
2974
|
+
#
|
|
2975
|
+
# @param business [String] The business: its Yelp page URL (https://www.yelp.com/biz/gary-
|
|
2976
|
+
# danko-san-francisco), its alias (gary-danko-san-francisco) or its business id
|
|
2977
|
+
# (WavvLdfdP6g8aZTtbBQHTw, the `businessId` of /local/yelp results). An id is answered
|
|
2978
|
+
# fastest; a URL or alias costs one business-page fetch first.
|
|
2979
|
+
# @param max_reviews [Integer] How many reviews to return (1-500). API default: 20.
|
|
2980
|
+
# @param page [Integer] 1-based Yelp review page to start from (10 reviews per page, like
|
|
2981
|
+
# yelp.com's own pages; 1-1000). To continue a previous response, pass its `nextStart` as
|
|
2982
|
+
# `start`. API default: 1.
|
|
2983
|
+
# @param start [Integer] 0-based review offset to start from instead of `page`. Pass
|
|
2984
|
+
# `nextStart` from a previous response to continue exactly where it stopped.
|
|
2985
|
+
# @param sort [String] Review order, as yelp.com's "Sort by" menu: relevance, newest,
|
|
2986
|
+
# oldest, highest, lowest, elites. `relevance` is Yelp's default order. API default:
|
|
2987
|
+
# relevance.
|
|
2988
|
+
# @param language [String] Reviews written in this language (two-letter code: en, fr, de,
|
|
2989
|
+
# es, ...). Yelp lists one language at a time. Default: the language most of the business's
|
|
2990
|
+
# reviews are in; `business.reviewsCountByLanguage` lists the others.
|
|
2991
|
+
# @param rating [String] Only reviews with these star ratings, comma-separated (e.g. `1,2`).
|
|
2992
|
+
# @param query [String] Only reviews that mention this phrase (yelp.com's review search).
|
|
2993
|
+
# @param since [String] Only reviews written on or after this date (YYYY-MM-DD). With
|
|
2994
|
+
# `sort=newest` paging stops at the first older review.
|
|
2995
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
2996
|
+
# 'DE'). Leave blank: Yelp serves reviews the same to every country.
|
|
2997
|
+
# Calls POST /local/yelp-reviews.
|
|
2998
|
+
def yelp_reviews(business, max_reviews: nil, page: nil, start: nil, sort: nil, language: nil, rating: nil, query: nil, since: nil, proxy_country: nil)
|
|
2999
|
+
post_json("/local/yelp-reviews", {
|
|
3000
|
+
business: business,
|
|
3001
|
+
max_reviews: max_reviews,
|
|
3002
|
+
page: page,
|
|
3003
|
+
start: start,
|
|
3004
|
+
sort: sort,
|
|
3005
|
+
language: language,
|
|
3006
|
+
rating: rating,
|
|
3007
|
+
query: query,
|
|
3008
|
+
since: since,
|
|
3009
|
+
proxy_country: proxy_country,
|
|
3010
|
+
})
|
|
3011
|
+
end
|
|
3012
|
+
|
|
3013
|
+
# Scrape Naver Map places.
|
|
3014
|
+
#
|
|
3015
|
+
# Naver Map (Naver Place), Korea's main local search. `mode=search&query=` returns up to 50
|
|
3016
|
+
# places (`results`): `id`, `name`, `category`, `businessType`, `address` / `roadAddress` /
|
|
3017
|
+
# `jibunAddress`, `phone`, `latitude` / `longitude`, `rating`, `visitorReviewCount`,
|
|
3018
|
+
# `blogReviewCount`, opening `status`, `microReview`, `popularMenus`, `options`, images,
|
|
3019
|
+
# booking and Naver Pay flags, `hotelMinPrice` (stays), `representativePrice` (salons) and
|
|
3020
|
+
# the place's Naver Map `url`, plus `totalResults`. `longitude` / `latitude` move the search
|
|
3021
|
+
# centre (each place then carries its `distance`). `mode=details&place=<id or Naver Map
|
|
3022
|
+
# link>` returns the `place` - addresses, phone, coordinates, `businessHours` per day,
|
|
3023
|
+
# `description`, `directions`, `conveniences`, `facilities`, `keywords`, `menus` with
|
|
3024
|
+
# prices, `photos`, `homepage` and `links`, `bookingUrl`, `subwayStations` and `reviewStats`
|
|
3025
|
+
# (average rating, star distribution, keyword votes, themes) - with its first ten visitor
|
|
3026
|
+
# `reviews` (`body`, `rating`, `created` / `createdDate`, `visitedAt`, `verifiedBy`,
|
|
3027
|
+
# `keywords`, `media`, `reactions`, `author` stats, `ownerReply`) and its first
|
|
3028
|
+
# `blogReviews` (blog and cafe posts). `sort=recent` returns the newest visitor reviews.
|
|
3029
|
+
# Korean text is returned as is. One page per call, billed one unit.
|
|
3030
|
+
#
|
|
3031
|
+
# @param query [String] What to search for (mode=search), in Korean or English: a category,
|
|
3032
|
+
# a place name or both, e.g. '강남 맛집', '홍대 카페', '부산 호텔', 'Gangnam cafe'.
|
|
3033
|
+
# @param longitude [Float] Search centre longitude (mode=search, with `latitude`), e.g.
|
|
3034
|
+
# 127.0276. Naver ranks places around the centre, and each result then carries its
|
|
3035
|
+
# `distance` from it.
|
|
3036
|
+
# @param latitude [Float] Search centre latitude (mode=search, with `longitude`), e.g.
|
|
3037
|
+
# 37.4979.
|
|
3038
|
+
# @param max_results [Integer] How many places to return (mode=search, 1-50). Naver answers
|
|
3039
|
+
# one page of up to 50 places per search; move the search centre to reach other places. API
|
|
3040
|
+
# default: 50.
|
|
3041
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'KR',
|
|
3042
|
+
# 'US'). Leave blank for the country chosen for the site.
|
|
3043
|
+
# Calls POST /maps/naver-place.
|
|
3044
|
+
def naver_place_search(query, longitude: nil, latitude: nil, max_results: nil, proxy_country: nil)
|
|
3045
|
+
post_json("/maps/naver-place", {
|
|
3046
|
+
mode: "search",
|
|
3047
|
+
query: query,
|
|
3048
|
+
longitude: longitude,
|
|
3049
|
+
latitude: latitude,
|
|
3050
|
+
max_results: max_results,
|
|
3051
|
+
proxy_country: proxy_country,
|
|
3052
|
+
})
|
|
3053
|
+
end
|
|
3054
|
+
|
|
3055
|
+
# Scrape Naver Map places.
|
|
3056
|
+
#
|
|
3057
|
+
# Naver Map (Naver Place), Korea's main local search. `mode=search&query=` returns up to 50
|
|
3058
|
+
# places (`results`): `id`, `name`, `category`, `businessType`, `address` / `roadAddress` /
|
|
3059
|
+
# `jibunAddress`, `phone`, `latitude` / `longitude`, `rating`, `visitorReviewCount`,
|
|
3060
|
+
# `blogReviewCount`, opening `status`, `microReview`, `popularMenus`, `options`, images,
|
|
3061
|
+
# booking and Naver Pay flags, `hotelMinPrice` (stays), `representativePrice` (salons) and
|
|
3062
|
+
# the place's Naver Map `url`, plus `totalResults`. `longitude` / `latitude` move the search
|
|
3063
|
+
# centre (each place then carries its `distance`). `mode=details&place=<id or Naver Map
|
|
3064
|
+
# link>` returns the `place` - addresses, phone, coordinates, `businessHours` per day,
|
|
3065
|
+
# `description`, `directions`, `conveniences`, `facilities`, `keywords`, `menus` with
|
|
3066
|
+
# prices, `photos`, `homepage` and `links`, `bookingUrl`, `subwayStations` and `reviewStats`
|
|
3067
|
+
# (average rating, star distribution, keyword votes, themes) - with its first ten visitor
|
|
3068
|
+
# `reviews` (`body`, `rating`, `created` / `createdDate`, `visitedAt`, `verifiedBy`,
|
|
3069
|
+
# `keywords`, `media`, `reactions`, `author` stats, `ownerReply`) and its first
|
|
3070
|
+
# `blogReviews` (blog and cafe posts). `sort=recent` returns the newest visitor reviews.
|
|
3071
|
+
# Korean text is returned as is. One page per call, billed one unit.
|
|
3072
|
+
#
|
|
3073
|
+
# @param place [String] The place (mode=details): its Naver place id ('2097258400'), a Naver
|
|
3074
|
+
# Map link (https://map.naver.com/p/entry/place/2097258400) or a Naver Place link
|
|
3075
|
+
# (https://m.place.naver.com/restaurant/2097258400/home).
|
|
3076
|
+
# @param sort [String] Visitor review order (mode=details): 'relevance' (Naver's default) or
|
|
3077
|
+
# 'recent' (newest first). API default: relevance.
|
|
3078
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'KR',
|
|
3079
|
+
# 'US'). Leave blank for the country chosen for the site.
|
|
3080
|
+
# Calls POST /maps/naver-place.
|
|
3081
|
+
def naver_place_details(place, sort: nil, proxy_country: nil)
|
|
3082
|
+
post_json("/maps/naver-place", {
|
|
3083
|
+
mode: "details",
|
|
3084
|
+
place: place,
|
|
3085
|
+
sort: sort,
|
|
3086
|
+
proxy_country: proxy_country,
|
|
3087
|
+
})
|
|
3088
|
+
end
|
|
3089
|
+
|
|
3090
|
+
# Scrape Google Play app details and search.
|
|
3091
|
+
#
|
|
3092
|
+
# Scrapes the Google Play Store. `mode=app` with `app_id` (e.g. `com.whatsapp`) returns one
|
|
3093
|
+
# app's full detail as JSON: title, developer and contact, category,
|
|
3094
|
+
# `score`/`ratings`/`reviews` and the star `histogram`, `installs`, price / in-app
|
|
3095
|
+
# purchases, content rating, description, screenshots, icon, `version`, `updated`, what's
|
|
3096
|
+
# new and similar apps. `mode=search` with `q` returns matching apps (package id, title,
|
|
3097
|
+
# developer, score, installs, icon, price). `hl`/`gl` set language and store country.
|
|
3098
|
+
#
|
|
3099
|
+
# @param app_id [String] Play package id for mode=app, e.g. 'com.whatsapp'.
|
|
3100
|
+
# @param hl [String] Language code (ISO), e.g. 'en'. API default: en.
|
|
3101
|
+
# @param gl [String] Store country (ISO-2), e.g. 'us'. API default: us.
|
|
3102
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3103
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3104
|
+
# Calls POST /apps/google-play.
|
|
3105
|
+
def google_play_app(app_id, hl: nil, gl: nil, proxy_country: nil)
|
|
3106
|
+
post_json("/apps/google-play", {
|
|
3107
|
+
mode: "app",
|
|
3108
|
+
app_id: app_id,
|
|
3109
|
+
hl: hl,
|
|
3110
|
+
gl: gl,
|
|
3111
|
+
proxy_country: proxy_country,
|
|
3112
|
+
})
|
|
3113
|
+
end
|
|
3114
|
+
|
|
3115
|
+
# Scrape Google Play app details and search.
|
|
3116
|
+
#
|
|
3117
|
+
# Scrapes the Google Play Store. `mode=app` with `app_id` (e.g. `com.whatsapp`) returns one
|
|
3118
|
+
# app's full detail as JSON: title, developer and contact, category,
|
|
3119
|
+
# `score`/`ratings`/`reviews` and the star `histogram`, `installs`, price / in-app
|
|
3120
|
+
# purchases, content rating, description, screenshots, icon, `version`, `updated`, what's
|
|
3121
|
+
# new and similar apps. `mode=search` with `q` returns matching apps (package id, title,
|
|
3122
|
+
# developer, score, installs, icon, price). `hl`/`gl` set language and store country.
|
|
3123
|
+
#
|
|
3124
|
+
# @param q [String] Search term for mode=search, e.g. 'vpn'.
|
|
3125
|
+
# @param max_results [Integer] How many apps to return for mode=search (one page). API
|
|
3126
|
+
# default: 30.
|
|
3127
|
+
# @param hl [String] Language code (ISO), e.g. 'en'. API default: en.
|
|
3128
|
+
# @param gl [String] Store country (ISO-2), e.g. 'us'. API default: us.
|
|
3129
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3130
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3131
|
+
# Calls POST /apps/google-play.
|
|
3132
|
+
def google_play_search(q, max_results: nil, hl: nil, gl: nil, proxy_country: nil)
|
|
3133
|
+
post_json("/apps/google-play", {
|
|
3134
|
+
mode: "search",
|
|
3135
|
+
q: q,
|
|
3136
|
+
max_results: max_results,
|
|
3137
|
+
hl: hl,
|
|
3138
|
+
gl: gl,
|
|
3139
|
+
proxy_country: proxy_country,
|
|
3140
|
+
})
|
|
3141
|
+
end
|
|
3142
|
+
|
|
3143
|
+
# Scrape Apple App Store app details, search and reviews.
|
|
3144
|
+
#
|
|
3145
|
+
# Scrapes the Apple App Store. `mode=app` with `app_id` (numeric id, `id310633997`, an
|
|
3146
|
+
# apps.apple.com URL or a bundle id) returns one app's full detail: title, `subtitle`,
|
|
3147
|
+
# developer (id, store page, website), price / currency / `free`, `rating`, `ratingsCount`
|
|
3148
|
+
# and the star `histogram`, `contentRating`, genres, `releaseDate`, `updated`, `version`,
|
|
3149
|
+
# `releaseNotes` and `versionHistory`, `size` in bytes, `minimumOsVersion`, `languages`,
|
|
3150
|
+
# description, screenshots, icon, `inAppPurchases`, `privacy` labels, `chartRank` and
|
|
3151
|
+
# `similarApps`. `mode=search` with `q` returns matching apps (up to 200). `mode=reviews`
|
|
3152
|
+
# returns customer reviews (title, text, rating, author, date, app version, helpful votes),
|
|
3153
|
+
# newest or most helpful first; Apple serves 50 per page and 500 per country. `country`
|
|
3154
|
+
# picks the storefront. mode=reviews is billed one request per review page fetched
|
|
3155
|
+
# (`pagesFetched`).
|
|
3156
|
+
#
|
|
3157
|
+
# @param app_id [String] App for mode=app / mode=reviews: the numeric App Store id
|
|
3158
|
+
# ('310633997'), 'id310633997', an apps.apple.com app URL, or a bundle id
|
|
3159
|
+
# ('net.whatsapp.WhatsApp').
|
|
3160
|
+
# @param country [String] App Store country (ISO-2), e.g. 'us', 'gb', 'de'. Prices, ratings
|
|
3161
|
+
# and reviews are per country. Default: the country in an app URL, else 'us'.
|
|
3162
|
+
# @param lang [String] Listing language for mode=app / mode=search, e.g. 'fr' or 'fr-CA'
|
|
3163
|
+
# (for countries whose store has several languages). Default: the store's own.
|
|
3164
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3165
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3166
|
+
# Calls POST /apps/app-store.
|
|
3167
|
+
def app_store_app(app_id, country: nil, lang: nil, proxy_country: nil)
|
|
3168
|
+
post_json("/apps/app-store", {
|
|
3169
|
+
mode: "app",
|
|
3170
|
+
app_id: app_id,
|
|
3171
|
+
country: country,
|
|
3172
|
+
lang: lang,
|
|
3173
|
+
proxy_country: proxy_country,
|
|
3174
|
+
})
|
|
3175
|
+
end
|
|
3176
|
+
|
|
3177
|
+
# Scrape Apple App Store app details, search and reviews.
|
|
3178
|
+
#
|
|
3179
|
+
# Scrapes the Apple App Store. `mode=app` with `app_id` (numeric id, `id310633997`, an
|
|
3180
|
+
# apps.apple.com URL or a bundle id) returns one app's full detail: title, `subtitle`,
|
|
3181
|
+
# developer (id, store page, website), price / currency / `free`, `rating`, `ratingsCount`
|
|
3182
|
+
# and the star `histogram`, `contentRating`, genres, `releaseDate`, `updated`, `version`,
|
|
3183
|
+
# `releaseNotes` and `versionHistory`, `size` in bytes, `minimumOsVersion`, `languages`,
|
|
3184
|
+
# description, screenshots, icon, `inAppPurchases`, `privacy` labels, `chartRank` and
|
|
3185
|
+
# `similarApps`. `mode=search` with `q` returns matching apps (up to 200). `mode=reviews`
|
|
3186
|
+
# returns customer reviews (title, text, rating, author, date, app version, helpful votes),
|
|
3187
|
+
# newest or most helpful first; Apple serves 50 per page and 500 per country. `country`
|
|
3188
|
+
# picks the storefront. mode=reviews is billed one request per review page fetched
|
|
3189
|
+
# (`pagesFetched`).
|
|
3190
|
+
#
|
|
3191
|
+
# @param q [String] Search term for mode=search, e.g. 'vpn'.
|
|
3192
|
+
# @param country [String] App Store country (ISO-2), e.g. 'us', 'gb', 'de'. Prices, ratings
|
|
3193
|
+
# and reviews are per country. Default: the country in an app URL, else 'us'.
|
|
3194
|
+
# @param lang [String] Listing language for mode=app / mode=search, e.g. 'fr' or 'fr-CA'
|
|
3195
|
+
# (for countries whose store has several languages). Default: the store's own.
|
|
3196
|
+
# @param max_results [Integer] mode=search: apps to return (1-200). mode=reviews: reviews to
|
|
3197
|
+
# collect (1-500); Apple serves 50 per page and at most 10 pages per country, each page
|
|
3198
|
+
# billed as one request. API default: 50.
|
|
3199
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3200
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3201
|
+
# Calls POST /apps/app-store.
|
|
3202
|
+
def app_store_search(q, country: nil, lang: nil, max_results: nil, proxy_country: nil)
|
|
3203
|
+
post_json("/apps/app-store", {
|
|
3204
|
+
mode: "search",
|
|
3205
|
+
q: q,
|
|
3206
|
+
country: country,
|
|
3207
|
+
lang: lang,
|
|
3208
|
+
max_results: max_results,
|
|
3209
|
+
proxy_country: proxy_country,
|
|
3210
|
+
})
|
|
3211
|
+
end
|
|
3212
|
+
|
|
3213
|
+
# Scrape Apple App Store app details, search and reviews.
|
|
3214
|
+
#
|
|
3215
|
+
# Scrapes the Apple App Store. `mode=app` with `app_id` (numeric id, `id310633997`, an
|
|
3216
|
+
# apps.apple.com URL or a bundle id) returns one app's full detail: title, `subtitle`,
|
|
3217
|
+
# developer (id, store page, website), price / currency / `free`, `rating`, `ratingsCount`
|
|
3218
|
+
# and the star `histogram`, `contentRating`, genres, `releaseDate`, `updated`, `version`,
|
|
3219
|
+
# `releaseNotes` and `versionHistory`, `size` in bytes, `minimumOsVersion`, `languages`,
|
|
3220
|
+
# description, screenshots, icon, `inAppPurchases`, `privacy` labels, `chartRank` and
|
|
3221
|
+
# `similarApps`. `mode=search` with `q` returns matching apps (up to 200). `mode=reviews`
|
|
3222
|
+
# returns customer reviews (title, text, rating, author, date, app version, helpful votes),
|
|
3223
|
+
# newest or most helpful first; Apple serves 50 per page and 500 per country. `country`
|
|
3224
|
+
# picks the storefront. mode=reviews is billed one request per review page fetched
|
|
3225
|
+
# (`pagesFetched`).
|
|
3226
|
+
#
|
|
3227
|
+
# @param app_id [String] App for mode=app / mode=reviews: the numeric App Store id
|
|
3228
|
+
# ('310633997'), 'id310633997', an apps.apple.com app URL, or a bundle id
|
|
3229
|
+
# ('net.whatsapp.WhatsApp').
|
|
3230
|
+
# @param country [String] App Store country (ISO-2), e.g. 'us', 'gb', 'de'. Prices, ratings
|
|
3231
|
+
# and reviews are per country. Default: the country in an app URL, else 'us'.
|
|
3232
|
+
# @param max_results [Integer] mode=search: apps to return (1-200). mode=reviews: reviews to
|
|
3233
|
+
# collect (1-500); Apple serves 50 per page and at most 10 pages per country, each page
|
|
3234
|
+
# billed as one request. API default: 50.
|
|
3235
|
+
# @param sort [String] Review order for mode=reviews: mostrecent, mosthelpful. API default:
|
|
3236
|
+
# mostrecent.
|
|
3237
|
+
# @param page [Integer] mode=reviews: review page to start from (1-10). Use `nextPage` of a
|
|
3238
|
+
# previous response to continue. When the last page was only partly used, `nextPage` points
|
|
3239
|
+
# at it again, so skip reviews whose `id` you already have. API default: 1.
|
|
3240
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3241
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3242
|
+
# Calls POST /apps/app-store.
|
|
3243
|
+
def app_store_reviews(app_id, country: nil, max_results: nil, sort: nil, page: nil, proxy_country: nil)
|
|
3244
|
+
post_json("/apps/app-store", {
|
|
3245
|
+
mode: "reviews",
|
|
3246
|
+
app_id: app_id,
|
|
3247
|
+
country: country,
|
|
3248
|
+
max_results: max_results,
|
|
3249
|
+
sort: sort,
|
|
3250
|
+
page: page,
|
|
3251
|
+
proxy_country: proxy_country,
|
|
3252
|
+
})
|
|
3253
|
+
end
|
|
3254
|
+
|
|
3255
|
+
# Scrape a YouTube channel's videos.
|
|
3256
|
+
#
|
|
3257
|
+
# Returns a YouTube channel's recent videos as JSON: `videoId`, `title`, `views`,
|
|
3258
|
+
# `published`, `thumbnail` and a watch `url`, plus a `channel` summary (name, id). Pass a
|
|
3259
|
+
# handle (`@mkbhd`), a channel id (`UC...`) or a channel URL. Returns the channel's first
|
|
3260
|
+
# page of videos (about 30).
|
|
3261
|
+
#
|
|
3262
|
+
# @param channel [String] Channel handle (@mkbhd), channel id (UC...) or a channel URL.
|
|
3263
|
+
# @param tab [String] Which tab: videos, shorts, streams. API default: videos.
|
|
3264
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3265
|
+
# 'DE').
|
|
3266
|
+
# @param limit [Integer] Max videos to return (1-200). Honoured on the 'videos' tab; other
|
|
3267
|
+
# tabs return a single scraped page. API default: 30.
|
|
3268
|
+
# Calls POST /content/youtube-channel.
|
|
3269
|
+
def youtube_channel(channel, tab: nil, proxy_country: nil, limit: nil)
|
|
3270
|
+
post_json("/content/youtube-channel", {
|
|
3271
|
+
channel: channel,
|
|
3272
|
+
tab: tab,
|
|
3273
|
+
proxy_country: proxy_country,
|
|
3274
|
+
limit: limit,
|
|
3275
|
+
})
|
|
3276
|
+
end
|
|
3277
|
+
|
|
3278
|
+
# Search YouTube.
|
|
3279
|
+
#
|
|
3280
|
+
# Searches YouTube for `query` and returns the results as JSON. For each video: `videoId`,
|
|
3281
|
+
# `title`, `url`, `channel` (name/id/url), `publishedText`, `viewCount` (+ text),
|
|
3282
|
+
# `lengthSeconds` (+ text), `thumbnails`, `descriptionSnippet`, `badges`, `verified` and
|
|
3283
|
+
# `isLive`. Filter with `type` (video / channel / playlist / all) and `sort` (relevance /
|
|
3284
|
+
# date / views / rating), and collect up to `max_results` across pages.
|
|
3285
|
+
#
|
|
3286
|
+
# @param query [String] Search terms.
|
|
3287
|
+
# @param max_results [Integer] How many results to collect across pages (~20 per page). API
|
|
3288
|
+
# default: 20.
|
|
3289
|
+
# @param type [String] Result type: video, channel, playlist, movie, all. API default:
|
|
3290
|
+
# video.
|
|
3291
|
+
# @param sort [String] Ordering: relevance, rating, date, views. API default: relevance.
|
|
3292
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3293
|
+
# 'DE'). Defaults to US.
|
|
3294
|
+
# Calls POST /content/youtube-search.
|
|
3295
|
+
def youtube_search(query, max_results: nil, type: nil, sort: nil, proxy_country: nil)
|
|
3296
|
+
post_json("/content/youtube-search", {
|
|
3297
|
+
query: query,
|
|
3298
|
+
max_results: max_results,
|
|
3299
|
+
type: type,
|
|
3300
|
+
sort: sort,
|
|
3301
|
+
proxy_country: proxy_country,
|
|
3302
|
+
})
|
|
3303
|
+
end
|
|
3304
|
+
|
|
3305
|
+
# Scrape a YouTube video's comments.
|
|
3306
|
+
#
|
|
3307
|
+
# Returns a YouTube video's comments as JSON. For each comment: `commentId`, `author`,
|
|
3308
|
+
# `authorChannelId`, `authorThumbnail`, `text`, `likeCount`, `publishedTimeText`,
|
|
3309
|
+
# `replyCount`, and the `isPinned`, `isHearted`, `isAuthorVerified` and `isUploader` flags,
|
|
3310
|
+
# plus the video `title` and total `commentCount`. Sort by `top` or `newest` and collect up
|
|
3311
|
+
# to `max_results` across pages.
|
|
3312
|
+
#
|
|
3313
|
+
# @param video [String] Video id (dQw4w9WgXcQ) or any YouTube video URL.
|
|
3314
|
+
# @param sort [String] Comment order: top, newest. API default: top.
|
|
3315
|
+
# @param max_results [Integer] How many comments to collect across pages (~20 per page). API
|
|
3316
|
+
# default: 20.
|
|
3317
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3318
|
+
# 'DE') for the watch page. Defaults to US.
|
|
3319
|
+
# Calls POST /content/youtube-comments.
|
|
3320
|
+
def youtube_comments(video, sort: nil, max_results: nil, proxy_country: nil)
|
|
3321
|
+
post_json("/content/youtube-comments", {
|
|
3322
|
+
video: video,
|
|
3323
|
+
sort: sort,
|
|
3324
|
+
max_results: max_results,
|
|
3325
|
+
proxy_country: proxy_country,
|
|
3326
|
+
})
|
|
3327
|
+
end
|
|
3328
|
+
|
|
3329
|
+
# Get a YouTube video transcript.
|
|
3330
|
+
#
|
|
3331
|
+
# Returns a YouTube video's captions as JSON: timestamped `segments` (`start`, `dur`,
|
|
3332
|
+
# `text`) and a joined `text`, plus `availableLanguages`, the picked `language`,
|
|
3333
|
+
# `isAutoGenerated`, and the video's `title`, `author` and `lengthSeconds`. Pass a video id
|
|
3334
|
+
# or any YouTube video URL; choose a caption language with `lang`. Ideal for LLM/RAG
|
|
3335
|
+
# pipelines, subtitles and content analysis.
|
|
3336
|
+
#
|
|
3337
|
+
# @param video [String] Video id (dQw4w9WgXcQ) or any YouTube video URL.
|
|
3338
|
+
# @param lang [String] Preferred caption language code, e.g. 'en'. Omit for the
|
|
3339
|
+
# default/first track. A prefix like 'en' matches 'en-US'.
|
|
3340
|
+
# @param format [String] Output shape: segments, text. 'segments' returns timestamps; 'text'
|
|
3341
|
+
# returns only the joined plaintext. API default: segments.
|
|
3342
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3343
|
+
# 'DE') for the watch page. Defaults to US.
|
|
3344
|
+
# Calls POST /content/youtube-transcript.
|
|
3345
|
+
def youtube_transcript(video, lang: nil, format: nil, proxy_country: nil)
|
|
3346
|
+
post_json("/content/youtube-transcript", {
|
|
3347
|
+
video: video,
|
|
3348
|
+
lang: lang,
|
|
3349
|
+
format: format,
|
|
3350
|
+
proxy_country: proxy_country,
|
|
3351
|
+
})
|
|
3352
|
+
end
|
|
3353
|
+
|
|
3354
|
+
# Scrape Spotify artists, albums, tracks, playlists and search.
|
|
3355
|
+
#
|
|
3356
|
+
# Returns Spotify catalog data as JSON, including the play counts and audience numbers the
|
|
3357
|
+
# official Web API does not expose. `type=artist`: `artist` (`followers`,
|
|
3358
|
+
# `monthlyListeners`, `worldRank`, `topCities`, `biography`, `externalLinks`, images,
|
|
3359
|
+
# `verified`), `topTracks` with `playcount` (a top track's `album` carries `name` and
|
|
3360
|
+
# `releaseDate` when Spotify lists that album among the artist's releases), `discography`
|
|
3361
|
+
# groups (popular releases, albums, singles, compilations with totals), `relatedArtists`,
|
|
3362
|
+
# `appearsOn`, `featuredOn` / `discoveredOn` playlists; `discography=true` adds the full
|
|
3363
|
+
# `releases` list, paged. `type=album`: `album` (type, release date, label, copyright,
|
|
3364
|
+
# artists, cover) and `tracks` with `playcount`, paged. `type=track`: `track` with
|
|
3365
|
+
# `playcount`, duration, explicit flag, album and every credited artist. `type=playlist`:
|
|
3366
|
+
# `playlist` (name, description, `followers`, owner, cover, `totalTracks`) and `tracks` with
|
|
3367
|
+
# `playcount`, `addedAt`, `bpm` and `key`, paged. `type=search&query=`: `results` of
|
|
3368
|
+
# `search_type` tracks, artists, albums or playlists, paged; a search with no match answers
|
|
3369
|
+
# 200 with empty `results`. `playcount` comes only on artist, album, track and playlist
|
|
3370
|
+
# lookups: Spotify's search does not return it, so search track results have no `playcount`
|
|
3371
|
+
# key (look a track up with `type=track` for its count). `id` takes an open.spotify.com URL,
|
|
3372
|
+
# a Spotify URI or a bare id. Lists page with `offset` / `max_results` and answer `hasMore`
|
|
3373
|
+
# / `nextOffset`; each page fetched is one billed unit (`pagesFetched`).
|
|
3374
|
+
#
|
|
3375
|
+
# @param query [String] Search text for type=search, e.g. 'daft punk'.
|
|
3376
|
+
# @param search_type [String] What a search returns: tracks, artists, albums, playlists. API
|
|
3377
|
+
# default: tracks.
|
|
3378
|
+
# @param offset [Integer] 0-based position to start from in the list being paged: album
|
|
3379
|
+
# tracks, playlist tracks, search results, or the full discography with discography=true.
|
|
3380
|
+
# Use the `nextOffset` of a previous response to continue. Spotify serves search results up
|
|
3381
|
+
# to position 1000. API default: 0.
|
|
3382
|
+
# @param max_results [Integer] How many list items to collect. Album tracks: 1-1000 (default
|
|
3383
|
+
# 100, 100 per page). Playlist tracks: 1-2000 (default 100, 100 per page). Search: 1-1000
|
|
3384
|
+
# (default 20, 50 per page). Artist releases with discography=true: 1-1000 (default 50, 50
|
|
3385
|
+
# per page). Each page fetched is one billed unit.
|
|
3386
|
+
# @param proxy_country [String] Country the request is sent from (ISO-2, e.g. 'DE', 'GB').
|
|
3387
|
+
# Spotify's market follows it, which can change `playable` and search order; play counts are
|
|
3388
|
+
# the same everywhere. Leave blank for any.
|
|
3389
|
+
# Calls POST /music/spotify.
|
|
3390
|
+
def spotify_search(query, search_type: nil, offset: nil, max_results: nil, proxy_country: nil)
|
|
3391
|
+
post_json("/music/spotify", {
|
|
3392
|
+
type: "search",
|
|
3393
|
+
query: query,
|
|
3394
|
+
search_type: search_type,
|
|
3395
|
+
offset: offset,
|
|
3396
|
+
max_results: max_results,
|
|
3397
|
+
proxy_country: proxy_country,
|
|
3398
|
+
})
|
|
3399
|
+
end
|
|
3400
|
+
|
|
3401
|
+
# Scrape Spotify artists, albums, tracks, playlists and search.
|
|
3402
|
+
#
|
|
3403
|
+
# Returns Spotify catalog data as JSON, including the play counts and audience numbers the
|
|
3404
|
+
# official Web API does not expose. `type=artist`: `artist` (`followers`,
|
|
3405
|
+
# `monthlyListeners`, `worldRank`, `topCities`, `biography`, `externalLinks`, images,
|
|
3406
|
+
# `verified`), `topTracks` with `playcount` (a top track's `album` carries `name` and
|
|
3407
|
+
# `releaseDate` when Spotify lists that album among the artist's releases), `discography`
|
|
3408
|
+
# groups (popular releases, albums, singles, compilations with totals), `relatedArtists`,
|
|
3409
|
+
# `appearsOn`, `featuredOn` / `discoveredOn` playlists; `discography=true` adds the full
|
|
3410
|
+
# `releases` list, paged. `type=album`: `album` (type, release date, label, copyright,
|
|
3411
|
+
# artists, cover) and `tracks` with `playcount`, paged. `type=track`: `track` with
|
|
3412
|
+
# `playcount`, duration, explicit flag, album and every credited artist. `type=playlist`:
|
|
3413
|
+
# `playlist` (name, description, `followers`, owner, cover, `totalTracks`) and `tracks` with
|
|
3414
|
+
# `playcount`, `addedAt`, `bpm` and `key`, paged. `type=search&query=`: `results` of
|
|
3415
|
+
# `search_type` tracks, artists, albums or playlists, paged; a search with no match answers
|
|
3416
|
+
# 200 with empty `results`. `playcount` comes only on artist, album, track and playlist
|
|
3417
|
+
# lookups: Spotify's search does not return it, so search track results have no `playcount`
|
|
3418
|
+
# key (look a track up with `type=track` for its count). `id` takes an open.spotify.com URL,
|
|
3419
|
+
# a Spotify URI or a bare id. Lists page with `offset` / `max_results` and answer `hasMore`
|
|
3420
|
+
# / `nextOffset`; each page fetched is one billed unit (`pagesFetched`).
|
|
3421
|
+
#
|
|
3422
|
+
# @param id [String] The artist, album, track or playlist: an open.spotify.com URL
|
|
3423
|
+
# (https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02), a Spotify URI
|
|
3424
|
+
# (spotify:artist:06HL4z0CvFAxyc27GXpf02) or a bare 22-character id (then set `type`).
|
|
3425
|
+
# @param discography [Boolean] type=artist only: also return the artist's full release list
|
|
3426
|
+
# (albums, singles, compilations, newest first) in `releases`, paged by offset /
|
|
3427
|
+
# max_results. API default: false.
|
|
3428
|
+
# @param proxy_country [String] Country the request is sent from (ISO-2, e.g. 'DE', 'GB').
|
|
3429
|
+
# Spotify's market follows it, which can change `playable` and search order; play counts are
|
|
3430
|
+
# the same everywhere. Leave blank for any.
|
|
3431
|
+
# Calls POST /music/spotify.
|
|
3432
|
+
def spotify_artist(id, discography: nil, proxy_country: nil)
|
|
3433
|
+
post_json("/music/spotify", {
|
|
3434
|
+
type: "artist",
|
|
3435
|
+
id: id,
|
|
3436
|
+
discography: discography,
|
|
3437
|
+
proxy_country: proxy_country,
|
|
3438
|
+
})
|
|
3439
|
+
end
|
|
3440
|
+
|
|
3441
|
+
# Scrape Spotify artists, albums, tracks, playlists and search.
|
|
3442
|
+
#
|
|
3443
|
+
# Returns Spotify catalog data as JSON, including the play counts and audience numbers the
|
|
3444
|
+
# official Web API does not expose. `type=artist`: `artist` (`followers`,
|
|
3445
|
+
# `monthlyListeners`, `worldRank`, `topCities`, `biography`, `externalLinks`, images,
|
|
3446
|
+
# `verified`), `topTracks` with `playcount` (a top track's `album` carries `name` and
|
|
3447
|
+
# `releaseDate` when Spotify lists that album among the artist's releases), `discography`
|
|
3448
|
+
# groups (popular releases, albums, singles, compilations with totals), `relatedArtists`,
|
|
3449
|
+
# `appearsOn`, `featuredOn` / `discoveredOn` playlists; `discography=true` adds the full
|
|
3450
|
+
# `releases` list, paged. `type=album`: `album` (type, release date, label, copyright,
|
|
3451
|
+
# artists, cover) and `tracks` with `playcount`, paged. `type=track`: `track` with
|
|
3452
|
+
# `playcount`, duration, explicit flag, album and every credited artist. `type=playlist`:
|
|
3453
|
+
# `playlist` (name, description, `followers`, owner, cover, `totalTracks`) and `tracks` with
|
|
3454
|
+
# `playcount`, `addedAt`, `bpm` and `key`, paged. `type=search&query=`: `results` of
|
|
3455
|
+
# `search_type` tracks, artists, albums or playlists, paged; a search with no match answers
|
|
3456
|
+
# 200 with empty `results`. `playcount` comes only on artist, album, track and playlist
|
|
3457
|
+
# lookups: Spotify's search does not return it, so search track results have no `playcount`
|
|
3458
|
+
# key (look a track up with `type=track` for its count). `id` takes an open.spotify.com URL,
|
|
3459
|
+
# a Spotify URI or a bare id. Lists page with `offset` / `max_results` and answer `hasMore`
|
|
3460
|
+
# / `nextOffset`; each page fetched is one billed unit (`pagesFetched`).
|
|
3461
|
+
#
|
|
3462
|
+
# @param id [String] The artist, album, track or playlist: an open.spotify.com URL
|
|
3463
|
+
# (https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02), a Spotify URI
|
|
3464
|
+
# (spotify:artist:06HL4z0CvFAxyc27GXpf02) or a bare 22-character id (then set `type`).
|
|
3465
|
+
# @param offset [Integer] 0-based position to start from in the list being paged: album
|
|
3466
|
+
# tracks, playlist tracks, search results, or the full discography with discography=true.
|
|
3467
|
+
# Use the `nextOffset` of a previous response to continue. Spotify serves search results up
|
|
3468
|
+
# to position 1000. API default: 0.
|
|
3469
|
+
# @param max_results [Integer] How many list items to collect. Album tracks: 1-1000 (default
|
|
3470
|
+
# 100, 100 per page). Playlist tracks: 1-2000 (default 100, 100 per page). Search: 1-1000
|
|
3471
|
+
# (default 20, 50 per page). Artist releases with discography=true: 1-1000 (default 50, 50
|
|
3472
|
+
# per page). Each page fetched is one billed unit.
|
|
3473
|
+
# @param proxy_country [String] Country the request is sent from (ISO-2, e.g. 'DE', 'GB').
|
|
3474
|
+
# Spotify's market follows it, which can change `playable` and search order; play counts are
|
|
3475
|
+
# the same everywhere. Leave blank for any.
|
|
3476
|
+
# Calls POST /music/spotify.
|
|
3477
|
+
def spotify_album(id, offset: nil, max_results: nil, proxy_country: nil)
|
|
3478
|
+
post_json("/music/spotify", {
|
|
3479
|
+
type: "album",
|
|
3480
|
+
id: id,
|
|
3481
|
+
offset: offset,
|
|
3482
|
+
max_results: max_results,
|
|
3483
|
+
proxy_country: proxy_country,
|
|
3484
|
+
})
|
|
3485
|
+
end
|
|
3486
|
+
|
|
3487
|
+
# Scrape Spotify artists, albums, tracks, playlists and search.
|
|
3488
|
+
#
|
|
3489
|
+
# Returns Spotify catalog data as JSON, including the play counts and audience numbers the
|
|
3490
|
+
# official Web API does not expose. `type=artist`: `artist` (`followers`,
|
|
3491
|
+
# `monthlyListeners`, `worldRank`, `topCities`, `biography`, `externalLinks`, images,
|
|
3492
|
+
# `verified`), `topTracks` with `playcount` (a top track's `album` carries `name` and
|
|
3493
|
+
# `releaseDate` when Spotify lists that album among the artist's releases), `discography`
|
|
3494
|
+
# groups (popular releases, albums, singles, compilations with totals), `relatedArtists`,
|
|
3495
|
+
# `appearsOn`, `featuredOn` / `discoveredOn` playlists; `discography=true` adds the full
|
|
3496
|
+
# `releases` list, paged. `type=album`: `album` (type, release date, label, copyright,
|
|
3497
|
+
# artists, cover) and `tracks` with `playcount`, paged. `type=track`: `track` with
|
|
3498
|
+
# `playcount`, duration, explicit flag, album and every credited artist. `type=playlist`:
|
|
3499
|
+
# `playlist` (name, description, `followers`, owner, cover, `totalTracks`) and `tracks` with
|
|
3500
|
+
# `playcount`, `addedAt`, `bpm` and `key`, paged. `type=search&query=`: `results` of
|
|
3501
|
+
# `search_type` tracks, artists, albums or playlists, paged; a search with no match answers
|
|
3502
|
+
# 200 with empty `results`. `playcount` comes only on artist, album, track and playlist
|
|
3503
|
+
# lookups: Spotify's search does not return it, so search track results have no `playcount`
|
|
3504
|
+
# key (look a track up with `type=track` for its count). `id` takes an open.spotify.com URL,
|
|
3505
|
+
# a Spotify URI or a bare id. Lists page with `offset` / `max_results` and answer `hasMore`
|
|
3506
|
+
# / `nextOffset`; each page fetched is one billed unit (`pagesFetched`).
|
|
3507
|
+
#
|
|
3508
|
+
# @param id [String] The artist, album, track or playlist: an open.spotify.com URL
|
|
3509
|
+
# (https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02), a Spotify URI
|
|
3510
|
+
# (spotify:artist:06HL4z0CvFAxyc27GXpf02) or a bare 22-character id (then set `type`).
|
|
3511
|
+
# @param proxy_country [String] Country the request is sent from (ISO-2, e.g. 'DE', 'GB').
|
|
3512
|
+
# Spotify's market follows it, which can change `playable` and search order; play counts are
|
|
3513
|
+
# the same everywhere. Leave blank for any.
|
|
3514
|
+
# Calls POST /music/spotify.
|
|
3515
|
+
def spotify_track(id, proxy_country: nil)
|
|
3516
|
+
post_json("/music/spotify", {
|
|
3517
|
+
type: "track",
|
|
3518
|
+
id: id,
|
|
3519
|
+
proxy_country: proxy_country,
|
|
3520
|
+
})
|
|
3521
|
+
end
|
|
3522
|
+
|
|
3523
|
+
# Scrape Spotify artists, albums, tracks, playlists and search.
|
|
3524
|
+
#
|
|
3525
|
+
# Returns Spotify catalog data as JSON, including the play counts and audience numbers the
|
|
3526
|
+
# official Web API does not expose. `type=artist`: `artist` (`followers`,
|
|
3527
|
+
# `monthlyListeners`, `worldRank`, `topCities`, `biography`, `externalLinks`, images,
|
|
3528
|
+
# `verified`), `topTracks` with `playcount` (a top track's `album` carries `name` and
|
|
3529
|
+
# `releaseDate` when Spotify lists that album among the artist's releases), `discography`
|
|
3530
|
+
# groups (popular releases, albums, singles, compilations with totals), `relatedArtists`,
|
|
3531
|
+
# `appearsOn`, `featuredOn` / `discoveredOn` playlists; `discography=true` adds the full
|
|
3532
|
+
# `releases` list, paged. `type=album`: `album` (type, release date, label, copyright,
|
|
3533
|
+
# artists, cover) and `tracks` with `playcount`, paged. `type=track`: `track` with
|
|
3534
|
+
# `playcount`, duration, explicit flag, album and every credited artist. `type=playlist`:
|
|
3535
|
+
# `playlist` (name, description, `followers`, owner, cover, `totalTracks`) and `tracks` with
|
|
3536
|
+
# `playcount`, `addedAt`, `bpm` and `key`, paged. `type=search&query=`: `results` of
|
|
3537
|
+
# `search_type` tracks, artists, albums or playlists, paged; a search with no match answers
|
|
3538
|
+
# 200 with empty `results`. `playcount` comes only on artist, album, track and playlist
|
|
3539
|
+
# lookups: Spotify's search does not return it, so search track results have no `playcount`
|
|
3540
|
+
# key (look a track up with `type=track` for its count). `id` takes an open.spotify.com URL,
|
|
3541
|
+
# a Spotify URI or a bare id. Lists page with `offset` / `max_results` and answer `hasMore`
|
|
3542
|
+
# / `nextOffset`; each page fetched is one billed unit (`pagesFetched`).
|
|
3543
|
+
#
|
|
3544
|
+
# @param id [String] The artist, album, track or playlist: an open.spotify.com URL
|
|
3545
|
+
# (https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02), a Spotify URI
|
|
3546
|
+
# (spotify:artist:06HL4z0CvFAxyc27GXpf02) or a bare 22-character id (then set `type`).
|
|
3547
|
+
# @param offset [Integer] 0-based position to start from in the list being paged: album
|
|
3548
|
+
# tracks, playlist tracks, search results, or the full discography with discography=true.
|
|
3549
|
+
# Use the `nextOffset` of a previous response to continue. Spotify serves search results up
|
|
3550
|
+
# to position 1000. API default: 0.
|
|
3551
|
+
# @param max_results [Integer] How many list items to collect. Album tracks: 1-1000 (default
|
|
3552
|
+
# 100, 100 per page). Playlist tracks: 1-2000 (default 100, 100 per page). Search: 1-1000
|
|
3553
|
+
# (default 20, 50 per page). Artist releases with discography=true: 1-1000 (default 50, 50
|
|
3554
|
+
# per page). Each page fetched is one billed unit.
|
|
3555
|
+
# @param proxy_country [String] Country the request is sent from (ISO-2, e.g. 'DE', 'GB').
|
|
3556
|
+
# Spotify's market follows it, which can change `playable` and search order; play counts are
|
|
3557
|
+
# the same everywhere. Leave blank for any.
|
|
3558
|
+
# Calls POST /music/spotify.
|
|
3559
|
+
def spotify_playlist(id, offset: nil, max_results: nil, proxy_country: nil)
|
|
3560
|
+
post_json("/music/spotify", {
|
|
3561
|
+
type: "playlist",
|
|
3562
|
+
id: id,
|
|
3563
|
+
offset: offset,
|
|
3564
|
+
max_results: max_results,
|
|
3565
|
+
proxy_country: proxy_country,
|
|
3566
|
+
})
|
|
3567
|
+
end
|
|
3568
|
+
|
|
3569
|
+
# Scrape a public Telegram channel: info and posts.
|
|
3570
|
+
#
|
|
3571
|
+
# Reads a public Telegram channel's web preview. Returns `channel` (`username`, `title`,
|
|
3572
|
+
# `description`, `avatar`, `verified`, `subscribers` and the other `counters`) and `posts`,
|
|
3573
|
+
# newest first: `id`, `url`, `date` (ISO-8601 UTC), `text` (plain), `textHtml`, `views`,
|
|
3574
|
+
# `author` (signature), `edited`, `forwardedFrom`, `replyTo`, `media` (photo / video /
|
|
3575
|
+
# animation / round_video / voice / audio / document / sticker, with CDN `url`, `thumbnail`,
|
|
3576
|
+
# `duration`), `poll`, `linkPreview`, `reactions` (a custom-emoji reaction has its
|
|
3577
|
+
# `customEmojiId`, plus the standard `emoji` and `image` Telegram falls back to when the
|
|
3578
|
+
# page shows that emoji anywhere), `buttons`, `links`, `hashtags` (the tags Telegram links;
|
|
3579
|
+
# a bare '#7' is text) and `mentions`. One page (about 20 posts) by default; `max_posts` (up
|
|
3580
|
+
# to 500) pages back through history; `before`, `after` and `since` bound the range, `q`
|
|
3581
|
+
# searches inside the channel and `post_id` returns one post. `nextBefore` continues a
|
|
3582
|
+
# capped answer. Only channels have a public preview: users, bots, groups and private
|
|
3583
|
+
# channels answer 404. Billed one request per page fetched (`pagesFetched`).
|
|
3584
|
+
#
|
|
3585
|
+
# @param channel [String] Public channel: username ('durov'), '@durov', or a t.me link
|
|
3586
|
+
# ('https://t.me/durov', 'https://t.me/s/durov'). A post link ('https://t.me/durov/400')
|
|
3587
|
+
# returns that one post.
|
|
3588
|
+
# @param max_posts [Integer] Posts to collect, newest first (1-500), paging back through the
|
|
3589
|
+
# channel's history; each page fetched is billed as one request. Omit it for one page,
|
|
3590
|
+
# billed once: the newest posts (a page has 20 message slots and an album fills one per
|
|
3591
|
+
# photo, so it can hold fewer posts).
|
|
3592
|
+
# @param before [Integer] Only posts with an id below this one. Pass `nextBefore` of a
|
|
3593
|
+
# previous response to continue where it stopped.
|
|
3594
|
+
# @param after [Integer] Stop at this post id: only posts with a higher id are returned.
|
|
3595
|
+
# @param since [String] Stop at this date: only posts published on or after it (YYYY-MM-DD
|
|
3596
|
+
# or an ISO-8601 datetime, UTC when no offset is given).
|
|
3597
|
+
# @param q [String] Search inside the channel: only posts matching this text.
|
|
3598
|
+
# @param post_id [Integer] Return only this post (a post id of the channel).
|
|
3599
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3600
|
+
# 'DE'). Leave blank to let the API pick.
|
|
3601
|
+
# Calls POST /social/telegram-channel.
|
|
3602
|
+
def telegram_channel(channel, max_posts: nil, before: nil, after: nil, since: nil, q: nil, post_id: nil, proxy_country: nil)
|
|
3603
|
+
post_json("/social/telegram-channel", {
|
|
3604
|
+
channel: channel,
|
|
3605
|
+
max_posts: max_posts,
|
|
3606
|
+
before: before,
|
|
3607
|
+
after: after,
|
|
3608
|
+
since: since,
|
|
3609
|
+
q: q,
|
|
3610
|
+
post_id: post_id,
|
|
3611
|
+
proxy_country: proxy_country,
|
|
3612
|
+
})
|
|
3613
|
+
end
|
|
3614
|
+
|
|
3615
|
+
# Scrape Quora questions, answers and profiles.
|
|
3616
|
+
#
|
|
3617
|
+
# Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
|
|
3618
|
+
# `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
|
|
3619
|
+
# `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
|
|
3620
|
+
# merged into this one, `related` with `include_related=true`), the answered `question`,
|
|
3621
|
+
# `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
|
|
3622
|
+
# `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
|
|
3623
|
+
# `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
|
|
3624
|
+
# are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
|
|
3625
|
+
# within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
|
|
3626
|
+
# every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
|
|
3627
|
+
# `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
|
|
3628
|
+
# `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
|
|
3629
|
+
# follower / following / answer / question / post counts, `contentViews`, `credentials`,
|
|
3630
|
+
# `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
|
|
3631
|
+
# only what Quora shows an anonymous visitor - Quora's search is not available that way, and
|
|
3632
|
+
# Quora lists only a profile's newest few dozen answers to logged-out visitors
|
|
3633
|
+
# (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
|
|
3634
|
+
# ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
|
|
3635
|
+
# (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
|
|
3636
|
+
#
|
|
3637
|
+
# @param question [String] Quora question URL (https://www.quora.com/What-is-the-best-way-
|
|
3638
|
+
# to-learn-programming), an answer URL on it, or the question slug (What-is-the-best-way-to-
|
|
3639
|
+
# learn-programming). Language editions such as es.quora.com are accepted as URLs. Required
|
|
3640
|
+
# with mode=question.
|
|
3641
|
+
# @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
|
|
3642
|
+
# 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
|
|
3643
|
+
# first few; each page is one billed unit.
|
|
3644
|
+
# @param include_related [Boolean] mode=question: also return the answers Quora shows from
|
|
3645
|
+
# related questions (its 'All related' view, `relation`='related'). By default only answers
|
|
3646
|
+
# to this question and to questions merged into it. API default: false.
|
|
3647
|
+
# @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
|
|
3648
|
+
# each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
|
|
3649
|
+
# can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
|
|
3650
|
+
# ask for a larger `max_answers` in one call (up to 300).
|
|
3651
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3652
|
+
# 'DE'). Leave blank for the countries chosen for the site.
|
|
3653
|
+
# Calls POST /social/quora.
|
|
3654
|
+
def quora_question(question, max_answers: nil, include_related: nil, cursor: nil, proxy_country: nil)
|
|
3655
|
+
post_json("/social/quora", {
|
|
3656
|
+
mode: "question",
|
|
3657
|
+
question: question,
|
|
3658
|
+
max_answers: max_answers,
|
|
3659
|
+
include_related: include_related,
|
|
3660
|
+
cursor: cursor,
|
|
3661
|
+
proxy_country: proxy_country,
|
|
3662
|
+
})
|
|
3663
|
+
end
|
|
3664
|
+
|
|
3665
|
+
# Scrape Quora questions, answers and profiles.
|
|
3666
|
+
#
|
|
3667
|
+
# Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
|
|
3668
|
+
# `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
|
|
3669
|
+
# `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
|
|
3670
|
+
# merged into this one, `related` with `include_related=true`), the answered `question`,
|
|
3671
|
+
# `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
|
|
3672
|
+
# `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
|
|
3673
|
+
# `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
|
|
3674
|
+
# are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
|
|
3675
|
+
# within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
|
|
3676
|
+
# every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
|
|
3677
|
+
# `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
|
|
3678
|
+
# `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
|
|
3679
|
+
# follower / following / answer / question / post counts, `contentViews`, `credentials`,
|
|
3680
|
+
# `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
|
|
3681
|
+
# only what Quora shows an anonymous visitor - Quora's search is not available that way, and
|
|
3682
|
+
# Quora lists only a profile's newest few dozen answers to logged-out visitors
|
|
3683
|
+
# (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
|
|
3684
|
+
# ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
|
|
3685
|
+
# (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
|
|
3686
|
+
#
|
|
3687
|
+
# @param profile [String] Quora profile URL (https://www.quora.com/profile/Adam-DAngelo) or
|
|
3688
|
+
# profile slug (Adam-DAngelo). Required with mode=profile.
|
|
3689
|
+
# @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
|
|
3690
|
+
# 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
|
|
3691
|
+
# first few; each page is one billed unit.
|
|
3692
|
+
# @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
|
|
3693
|
+
# each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
|
|
3694
|
+
# can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
|
|
3695
|
+
# ask for a larger `max_answers` in one call (up to 300).
|
|
3696
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3697
|
+
# 'DE'). Leave blank for the countries chosen for the site.
|
|
3698
|
+
# Calls POST /social/quora.
|
|
3699
|
+
def quora_profile(profile, max_answers: nil, cursor: nil, proxy_country: nil)
|
|
3700
|
+
post_json("/social/quora", {
|
|
3701
|
+
mode: "profile",
|
|
3702
|
+
profile: profile,
|
|
3703
|
+
max_answers: max_answers,
|
|
3704
|
+
cursor: cursor,
|
|
3705
|
+
proxy_country: proxy_country,
|
|
3706
|
+
})
|
|
3707
|
+
end
|
|
3708
|
+
|
|
3709
|
+
# Get a website's Similarweb traffic overview.
|
|
3710
|
+
#
|
|
3711
|
+
# Returns Similarweb's public traffic overview for `domain` as JSON: `globalRank`,
|
|
3712
|
+
# `countryRank` (with `country`), `categoryRank` (with `category`) and their month-over-
|
|
3713
|
+
# month changes; `totalVisits` for the latest month with `visitsChangePercent`;
|
|
3714
|
+
# `bounceRate`, `pagesPerVisit`, `avgVisitDuration`/`avgVisitDurationSeconds`;
|
|
3715
|
+
# `visitsHistory` (the last 3 months - earlier months are derived from Similarweb's own
|
|
3716
|
+
# month-over-month change) and `rankHistory`; `trafficSources` (channel shares -
|
|
3717
|
+
# Similarweb's free page reveals only some, the rest are null); `topCountries`;
|
|
3718
|
+
# `demographics` (gender, age groups); `competitors` (similar sites with affinity); `search`
|
|
3719
|
+
# (organic/paid split and top keywords); `socialNetworks`, `referrals`, `outgoingLinks`,
|
|
3720
|
+
# `displayAds`; `technologies`; `audienceInterests`; and the `company` profile.
|
|
3721
|
+
# `snapshotDate` is the month the data describes. A domain Similarweb has no page for
|
|
3722
|
+
# returns `found: false` with the same keys set to null/empty; a known domain too small to
|
|
3723
|
+
# estimate returns `hasTrafficData: false`.
|
|
3724
|
+
#
|
|
3725
|
+
# @param domain [String] Website to look up: a bare domain ('github.com'), a URL
|
|
3726
|
+
# ('https://www.github.com/about'), or a Similarweb page URL. A leading 'www.' is dropped;
|
|
3727
|
+
# other subdomains ('docs.python.org') are kept.
|
|
3728
|
+
# @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
|
|
3729
|
+
# 'DE'). Optional.
|
|
3730
|
+
# Calls POST /seo/similarweb.
|
|
3731
|
+
def similarweb_overview(domain, proxy_country: nil)
|
|
3732
|
+
post_json("/seo/similarweb", {
|
|
3733
|
+
domain: domain,
|
|
3734
|
+
proxy_country: proxy_country,
|
|
3735
|
+
})
|
|
3736
|
+
end
|
|
3737
|
+
|
|
3738
|
+
# Run Ahrefs' free SEO tools (DR, backlinks, traffic, keywords, KD, SERP).
|
|
3739
|
+
#
|
|
3740
|
+
# Ahrefs' free SEO tools as JSON, without an Ahrefs account.
|
|
3741
|
+
# `tool=authority&target=<domain>`: `domainRating`, `backlinks`, `referringDomains`,
|
|
3742
|
+
# `dofollowBacklinksPercent`, `dofollowReferringDomainsPercent`. `tool=backlinks`: the same
|
|
3743
|
+
# plus `topBacklinks` (`urlFrom`, `urlTo`, `title`, `anchor`, `textBefore`/`textAfter`,
|
|
3744
|
+
# `domainRating`). `tool=traffic`: `organicTraffic`, `trafficValueUsd`, `trafficHistory`,
|
|
3745
|
+
# `topPages`, `topCountries`, `topKeywords` (`country` narrows it to one market).
|
|
3746
|
+
# `tool=broken-links`: `inbound` broken backlinks and `outbound` broken links on the site's
|
|
3747
|
+
# pages. `tool=keyword-difficulty&keyword=<phrase>`: `difficulty` (0-100),
|
|
3748
|
+
# `referringDomainsToRank` and the SERP `results`. `tool=keywords`: `ideas` and
|
|
3749
|
+
# `questionIdeas` with `difficulty` and `volume` buckets, plus totals (Google). `tool=serp`:
|
|
3750
|
+
# the top Google `results` (organic with `metrics` - DR, UR, backlinks, referring domains,
|
|
3751
|
+
# traffic, value, keywords - plus AI overview, questions, shopping, local and news blocks).
|
|
3752
|
+
# Domain tools take `mode` (subdomains, domain, prefix, exact). One call, one unit.
|
|
3753
|
+
#
|
|
3754
|
+
# @param tool [String] Which Ahrefs free tool to run: 'authority' (Domain Rating),
|
|
3755
|
+
# 'backlinks', 'traffic', 'broken-links' (these take `target`), 'keyword-difficulty',
|
|
3756
|
+
# 'keywords' (keyword ideas), 'serp' (these take `keyword`).
|
|
3757
|
+
# @param target [String] Domain or URL for the domain tools, e.g. 'github.com' or
|
|
3758
|
+
# 'https://github.com/features'.
|
|
3759
|
+
# @param keyword [String] Search phrase for the keyword tools, e.g. 'running shoes'.
|
|
3760
|
+
# @param country [String] Two-letter market code. Keyword tools: the search market (default
|
|
3761
|
+
# 'us'). traffic: limits the estimate to that country (default: worldwide). Not used by the
|
|
3762
|
+
# other domain tools.
|
|
3763
|
+
# @param mode [String] Target scope for the domain tools: 'subdomains' (the domain with all
|
|
3764
|
+
# its subdomains), 'domain' (without subdomains), 'prefix' (everything under the URL) or
|
|
3765
|
+
# 'exact' (that URL only). API default: subdomains.
|
|
3766
|
+
# @param protocol [String] traffic only: 'both', 'http' or 'https'. API default: both.
|
|
3767
|
+
# @param questions [Boolean] keywords only: also return question ideas. API default: true.
|
|
3768
|
+
# Calls POST /seo/ahrefs.
|
|
3769
|
+
def ahrefs(tool, target: nil, keyword: nil, country: nil, mode: nil, protocol: nil, questions: nil)
|
|
3770
|
+
post_json("/seo/ahrefs", {
|
|
3771
|
+
tool: tool,
|
|
3772
|
+
target: target,
|
|
3773
|
+
keyword: keyword,
|
|
3774
|
+
country: country,
|
|
3775
|
+
mode: mode,
|
|
3776
|
+
protocol: protocol,
|
|
3777
|
+
questions: questions,
|
|
3778
|
+
})
|
|
3779
|
+
end
|
|
3780
|
+
|
|
3781
|
+
# Get a website's Semrush overview (Authority Score, traffic, backlinks).
|
|
3782
|
+
#
|
|
3783
|
+
# Semrush's public website data for `domain`, without a Semrush account. `report=overview`
|
|
3784
|
+
# (default): `authorityScore`, `globalRank`, `countryRank`, `categoryRank`, `visits` with
|
|
3785
|
+
# `visitsChangePercent` and `visitsHistory` (desktop/mobile), `pagesPerVisit`,
|
|
3786
|
+
# `timeOnSiteSeconds`, `bounceRate`, `searchTraffic` (organic and paid with history),
|
|
3787
|
+
# `backlinks` (backlinks, referring domains, Google penalty risk, history), `topKeywords`
|
|
3788
|
+
# (position, search volume, CPC, traffic share, intent), `topCountries`, `competitors`,
|
|
3789
|
+
# `trafficSources` (incoming and outgoing) and `aiTraffic` (visits from AI assistants).
|
|
3790
|
+
# `report=competitors`: the domain's competitors, each with Authority Score, ranks, visits
|
|
3791
|
+
# and engagement. Semrush publishes these pages for about 200k of the most visited sites;
|
|
3792
|
+
# any other domain returns `found: false`. One call, one unit.
|
|
3793
|
+
#
|
|
3794
|
+
# @param domain [String] Website to look up: a domain ('github.com'), a URL
|
|
3795
|
+
# ('https://www.github.com/about') or a Semrush website page URL. A leading 'www.' is
|
|
3796
|
+
# dropped; other subdomains are kept.
|
|
3797
|
+
# @param report [String] 'overview' (traffic, rank, search traffic, backlinks, keywords,
|
|
3798
|
+
# countries, competitors, sources) or 'competitors' (the competitor list with each one's
|
|
3799
|
+
# ranks, visits and engagement). API default: overview.
|
|
3800
|
+
# Calls POST /seo/semrush.
|
|
3801
|
+
def semrush_overview(domain, report: nil)
|
|
3802
|
+
post_json("/seo/semrush", {
|
|
3803
|
+
domain: domain,
|
|
3804
|
+
report: report,
|
|
3805
|
+
})
|
|
3806
|
+
end
|
|
3807
|
+
|
|
3808
|
+
private
|
|
3809
|
+
|
|
3810
|
+
def plugin_join(value, sep)
|
|
3811
|
+
value.is_a?(Array) ? value.join(sep) : value
|
|
3812
|
+
end
|
|
3813
|
+
end
|
|
3814
|
+
end
|