scrapeunblocker 0.9.0 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f20529c692d1de01f3f02acca1accaf677cd5df2ec9d075bfb949b4b230272d2
-  data.tar.gz: b0eff5160bbd56e6e45700609ab17966d061d74537909c7a1a06df9242cddb7a
+  metadata.gz: 14be9ee13840a5574c9924bf39c16b15ecf05c7e4cde02091840034013474c90
+  data.tar.gz: e915ed083458bad3fcc0bdbf1be03347052582a97f8e0955daa86ede3b535eff
 SHA512:
-  metadata.gz: 2421c59b2bc7c32d769d01546b4f796b7f71237edf500af4ee21b56f0a1393532b6361cc4207264c6072c5510bd31dc840986adda94dfbc18e7d2fdbf03a1df8
-  data.tar.gz: 98532dd9a066ef4e52f1123a958e36f1f40df7f603a3055b78784337bedfb256a237343d011e086ba8d0feb94cd17100e2501d5317055b35bae15fc57bea27ea
+  metadata.gz: 5d9ccb478cbf809a1c747b578946a08e0fb30a0e8d968d5eda0cd8389f7c572825e37772c1f0ec590b8ba618d47748262de8a65c773e8fa0ade262b0cb75d471
+  data.tar.gz: c43112450a0ffebabf538bb3a0cb50d2a74be3c0de0712f8e138a34d226b8eb7710f6e9b78560280412d2001e7860439ef3c1d1975559f67519df0bb6d838262

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 0.10.1 (2026-10-13)
+
+- `tiktok_shop_search` takes its keyword as optional, so a category-only search works (`category` option).
+- `aldi_au_specials` takes `category`: a Special Buys drop date such as `2026-10-14`.
+- `capterra_search` no longer offers `page` and `app_store_reviews` no longer offers `lang`; the API ignored both for those calls.
+- Method docs say "proxy country" where they said "exit-IP country", and drop internal routing notes.
+
+## 0.10.0 (2026-10-13)
+
+- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Telegram, Quora, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
+- Each method takes the plugin's main input positionally and the rest as keyword options; options left `nil` are not sent, so the API's defaults apply. Methods for one mode of a multi-mode endpoint set the mode themselves (`woolworths_product("888137")`, `spotify_artist(...)`).
+- New `plugin(path, **params)` calls any plugin endpoint by path. Array values are sent as a repeated query parameter.
+
+No breaking changes.
+
 ## 0.9.0 (2026-10-07)
 
 - New `ScrapeUnblocker::BudgetExceededError` (a subclass of `PaymentRequiredError`): raised when the API answers 402 with `User set budget exceeded`, meaning this billing period's spend reached the monthly budget limit (EUR, excluding VAT) set in the [profile](https://app.scrapeunblocker.com/dashboard/profile). It clears at the start of the next billing period, or within about a minute after the limit is raised or removed. Like the other 402s it is never retried and not billed. Before, this body surfaced as a plain `PaymentRequiredError`.

data/README.md

@@ -114,6 +114,48 @@ p result.data         # the fields
 fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")
 ```
 
+## Plugins
+
+Ready-made scrapers for specific sites that return structured JSON. Each
+method takes the plugin's main input positionally and everything else as
+keyword options; options you leave out are not sent, so the API's own
+defaults apply. Every parameter is documented on the
+[plugin pages](https://docs.scrapeunblocker.com/plugins/overview?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk).
+
+```ruby
+products = su.target_search("coffee maker", sort: "price_low", max_results: 48)
+milk = su.woolworths_product("888137")                     # one product, with nutrition
+trends = su.google_trends(%w[coffee tea], geo: "US")       # a list is sent comma-separated
+reviews = su.trustpilot_reviews("www.nike.com", stars: [4, 5])
+jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")
+```
+
+<!-- plugin-methods:start -->
+| Category | Methods |
+| --- | --- |
+| Travel | `skyscanner.flights`, `skyscanner.hotels`, `skyscanner.carhire`, `booking_hotels`, `booking_reviews`, `airbnb_search`, `airbnb_room`, `airbnb_reviews`, `tripadvisor_list`, `tripadvisor_reviews` |
+| Google | `google_local`, `google_images`, `maps_place_details`, `google_trends`, `google_shopping`, `google_jobs` |
+| Search engines | `bing_search`, `bing_images`, `duckduckgo_search`, `yahoo_search`, `yahoo_images`, `brave_search`, `brave_images` |
+| E-commerce | `amazon_product`, `amazon_search`, `amazon_bestsellers`, `amazon_reviews`, `walmart_search`, `target_search`, `bestbuy_search`, `homedepot_search`, `etsy_search`, `temu_search`, `temu_product`, `shein_search`, `zalando_search`, `aliexpress_search`, `alibaba_search`, `alibaba_product`, `alibaba1688_search`, `alibaba1688_product`, `tiktok_shop_search`, `tiktok_shop_product`, `tiktok_shop_store`, `oopbuy_search` |
+| Classifieds & Resale | `ebay_search`, `vinted_search`, `leboncoin_search` |
+| Groceries | `woolworths_search`, `woolworths_category`, `woolworths_specials`, `woolworths_product`, `coles_search`, `coles_category`, `coles_specials`, `coles_product`, `aldi_au_search`, `aldi_au_category`, `aldi_au_specials`, `aldi_au_product` |
+| Real estate | `zillow_search`, `zillow_detail`, `redfin_search`, `redfin_detail`, `rightmove_search`, `idealista_search`, `idealista_detail`, `immobilienscout24_search` |
+| Vehicles | `mobile_de`, `autoscout24_search` |
+| Jobs | `indeed_jobs`, `glassdoor_jobs`, `glassdoor_reviews`, `glassdoor_salaries`, `naukri_jobs`, `wellfound_jobs`, `wellfound_job`, `wellfound_company`, `wellfound_company_search`, `career_site_jobs` |
+| Reviews & Companies | `trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search` |
+| Local & Maps | `yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details` |
+| Apps | `google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews` |
+| Video, Music & Communities | `youtube_channel`, `youtube_search`, `youtube_comments`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`, `telegram_channel`, `quora_question`, `quora_profile` |
+| SEO & Ads | `similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library` |
+<!-- plugin-methods:end -->
+
+Any plugin endpoint can also be called by path, with parameters named as the
+API names them - handy for an endpoint added after your gem version:
+
+```ruby
+data = su.plugin("/marketplace/target-search", query: "tv", max_results: 24)
+```
+
 ## Google search (SERP)
 
 ```ruby
@@ -184,19 +226,6 @@ end
 
 `marketplace` is any of the 19 regional eBay hosts (`ebay.com` default). `condition` is one of `"new"`, `"open_box"`, `"refurbished"`, `"used"` or `"for_parts"`; `sort` is one of `"best_match"` (default), `"newly_listed"`, `"ending_soon"`, `"price_asc"` or `"price_desc"`; `page_size` is 60, 120 or 240. `exactMatches` is `false` when eBay found nothing for the keyword and answered with its own loosely-related suggestions instead, so check it before using the listings.
 
-### TikTok
-
-```ruby
-profile = su.tiktok_profile("nasa", max_videos: 5)      # exact stats + newest videos
-video = su.tiktok_video("https://www.tiktok.com/@nasa/video/7665075736742530317", include_transcript: true)
-tag = su.tiktok_hashtag("nasa", max_videos: 10)
-results = su.tiktok_search("space telescope", max_results: 25)   # TikTok's own ranking
-comments = su.tiktok_comments("https://www.tiktok.com/@nasa/video/7665075736742530317", max_comments: 40)
-puts profile["stats"]["followers"], video["stats"]["plays"], tag["stats"]["views"], comments["totalComments"]
-```
-
-Profiles and hashtags list up to 10 videos in a couple of seconds from TikTok's server-rendered widget; ask for more (up to 200) and the real grid is scrolled in a browser session.
-
 ## Cookies and the serving proxy
 
 ```ruby
@@ -229,25 +258,6 @@ hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", chec
 cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")
 ```
 
-## Southwest plugin
-
-Southwest Airlines flight quotes as JSON. `origin` and `dest` are IATA airport codes; omit `return_date` for a one-way search.
-
-```ruby
-flights = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", return_date: "2026-10-27"
-)
-
-# One-way, priced in Rapid Rewards points
-oneway = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", adults: 2, fare_type: "points"
-)
-```
-
-`adults` is 1-8 (default 1), `fare_type` is `"dollars"` (default) or `"points"`, `proxy_country` defaults to `"US"` and `max_attempts` is 1-5 (default 3). The call returns the raw booking / shopping JSON.
-
 ## Error handling
 
 Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`.

data/lib/scrapeunblocker/client.rb

@@ -6,6 +6,7 @@ require "json"
 
 require_relative "errors"
 require_relative "parsed_page"
+require_relative "plugins"
 require_relative "skyscanner"
 require_relative "southwest"
 require_relative "version"
@@ -16,6 +17,8 @@ module ScrapeUnblocker
   #   su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
   #   html = su.get_page_source("https://example.com")
   class Client
+    include Plugins
+
     DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
     API_KEY_HEADER = "x-scrapeunblocker-key"
     RETRYABLE = [429, 502, 503, 504].freeze
@@ -148,7 +151,7 @@ module ScrapeUnblocker
     # originalPrice, priceCny, monthSold, image and url. +channel+ is one of
     # "1688" (default), "taobao" or "official"; +sort+ is one of "default",
     # "price_asc", "price_desc" or "best_selling". +page_size+ max is 60.
-    # Brand keywords return HTTP 422.
+    # A brand keyword returns 200 with keywordRejected = true and no products.
     def oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil)
       post_json("/goods/oopbuy-search",
                 keyword: keyword, channel: channel, page: page,
@@ -189,7 +192,7 @@ module ScrapeUnblocker
     # availability, rating, review count, seller, feature bullets, categories
     # and images. Prices come back in the marketplace's own currency:
     # +proxy_country+ defaults to the marketplace's home country
-    # (amazon.com -> US), pinning the exit over the ISP pool. Pass either
+    # (amazon.com -> US). Pass either
     # +asin+ (with +marketplace+) or a full product +url+.
     def amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil)
       post_json("/marketplace/amazon-product",
@@ -302,8 +305,11 @@ module ScrapeUnblocker
       params.each_with_object([]) do |(key, value), acc|
         next if value.nil?
 
-        value = value ? "true" : "false" if value == true || value == false
-        acc << "#{key}=#{URI.encode_www_form_component(value.to_s)}"
+        # A list is a repeated parameter (stars=4&stars=5), as the API reads it.
+        Array(value).each do |item|
+          item = item ? "true" : "false" if item == true || item == false
+          acc << "#{key}=#{URI.encode_www_form_component(item.to_s)}"
+        end
       end.join("&")
     end