scrapeunblocker 0.9.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f20529c692d1de01f3f02acca1accaf677cd5df2ec9d075bfb949b4b230272d2
-  data.tar.gz: b0eff5160bbd56e6e45700609ab17966d061d74537909c7a1a06df9242cddb7a
+  metadata.gz: 3ffe8b5253c990d2b8f3754b32da58cd5a74150513c5592606dcc70c24d98e6c
+  data.tar.gz: a086e3553aeb1fd48bd57884a2315adfbe92b55763277298db932365b42c44bf
 SHA512:
-  metadata.gz: 2421c59b2bc7c32d769d01546b4f796b7f71237edf500af4ee21b56f0a1393532b6361cc4207264c6072c5510bd31dc840986adda94dfbc18e7d2fdbf03a1df8
-  data.tar.gz: 98532dd9a066ef4e52f1123a958e36f1f40df7f603a3055b78784337bedfb256a237343d011e086ba8d0feb94cd17100e2501d5317055b35bae15fc57bea27ea
+  metadata.gz: 2c0efc8818178442559f1d4ee6ab1e1fe16938e5f5f3c4371eae0506a6ec5d498df37345e726d26293807be62783051bf5824510cb23b3099fa6eb44cdb54b44
+  data.tar.gz: 15d718a1db74279f32f17bfbfda2034987f411e61514c15c122567115edcabb3849dcbeae4ed1fbe9e94e1c913ac6997eea1c638ba2c645d93ef618ea234562e

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.10.0 (2026-10-13)
+
+- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Telegram, Quora, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
+- Each method takes the plugin's main input positionally and the rest as keyword options; options left `nil` are not sent, so the API's defaults apply. Methods for one mode of a multi-mode endpoint set the mode themselves (`woolworths_product("888137")`, `spotify_artist(...)`).
+- New `plugin(path, **params)` calls any plugin endpoint by path. Array values are sent as a repeated query parameter.
+
+No breaking changes.
+
 ## 0.9.0 (2026-10-07)
 
 - New `ScrapeUnblocker::BudgetExceededError` (a subclass of `PaymentRequiredError`): raised when the API answers 402 with `User set budget exceeded`, meaning this billing period's spend reached the monthly budget limit (EUR, excluding VAT) set in the [profile](https://app.scrapeunblocker.com/dashboard/profile). It clears at the start of the next billing period, or within about a minute after the limit is raised or removed. Like the other 402s it is never retried and not billed. Before, this body surfaced as a plain `PaymentRequiredError`.

data/README.md

@@ -114,6 +114,48 @@ p result.data         # the fields
 fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")
 ```
 
+## Plugins
+
+Ready-made scrapers for specific sites that return structured JSON. Each
+method takes the plugin's main input positionally and everything else as
+keyword options; options you leave out are not sent, so the API's own
+defaults apply. Every parameter is documented on the
+[plugin pages](https://docs.scrapeunblocker.com/plugins/overview?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk).
+
+```ruby
+products = su.target_search("coffee maker", sort: "price_low", max_results: 48)
+milk = su.woolworths_product("888137")                     # one product, with nutrition
+trends = su.google_trends(%w[coffee tea], geo: "US")       # a list is sent comma-separated
+reviews = su.trustpilot_reviews("www.nike.com", stars: [4, 5])
+jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")
+```
+
+<!-- plugin-methods:start -->
+| Category | Methods |
+| --- | --- |
+| Travel | `skyscanner.flights`, `skyscanner.hotels`, `skyscanner.carhire`, `booking_hotels`, `booking_reviews`, `airbnb_search`, `airbnb_room`, `airbnb_reviews`, `tripadvisor_list`, `tripadvisor_reviews` |
+| Google | `google_local`, `google_images`, `maps_place_details`, `google_trends`, `google_shopping`, `google_jobs` |
+| Search engines | `bing_search`, `bing_images`, `duckduckgo_search`, `yahoo_search`, `yahoo_images`, `brave_search`, `brave_images` |
+| E-commerce | `amazon_product`, `amazon_search`, `amazon_bestsellers`, `amazon_reviews`, `walmart_search`, `target_search`, `bestbuy_search`, `homedepot_search`, `etsy_search`, `temu_search`, `temu_product`, `shein_search`, `zalando_search`, `aliexpress_search`, `alibaba_search`, `alibaba_product`, `alibaba1688_search`, `alibaba1688_product`, `tiktok_shop_search`, `tiktok_shop_product`, `tiktok_shop_store`, `oopbuy_search` |
+| Classifieds & Resale | `ebay_search`, `vinted_search`, `leboncoin_search` |
+| Groceries | `woolworths_search`, `woolworths_category`, `woolworths_specials`, `woolworths_product`, `coles_search`, `coles_category`, `coles_specials`, `coles_product`, `aldi_au_search`, `aldi_au_category`, `aldi_au_specials`, `aldi_au_product` |
+| Real estate | `zillow_search`, `zillow_detail`, `redfin_search`, `redfin_detail`, `rightmove_search`, `idealista_search`, `idealista_detail`, `immobilienscout24_search` |
+| Vehicles | `mobile_de`, `autoscout24_search` |
+| Jobs | `indeed_jobs`, `glassdoor_jobs`, `glassdoor_reviews`, `glassdoor_salaries`, `naukri_jobs`, `wellfound_jobs`, `wellfound_job`, `wellfound_company`, `wellfound_company_search`, `career_site_jobs` |
+| Reviews & Companies | `trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search` |
+| Local & Maps | `yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details` |
+| Apps | `google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews` |
+| Video, Music & Communities | `youtube_channel`, `youtube_search`, `youtube_comments`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`, `telegram_channel`, `quora_question`, `quora_profile` |
+| SEO & Ads | `similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library` |
+<!-- plugin-methods:end -->
+
+Any plugin endpoint can also be called by path, with parameters named as the
+API names them - handy for an endpoint added after your gem version:
+
+```ruby
+data = su.plugin("/marketplace/target-search", query: "tv", max_results: 24)
+```
+
 ## Google search (SERP)
 
 ```ruby
@@ -184,19 +226,6 @@ end
 
 `marketplace` is any of the 19 regional eBay hosts (`ebay.com` default). `condition` is one of `"new"`, `"open_box"`, `"refurbished"`, `"used"` or `"for_parts"`; `sort` is one of `"best_match"` (default), `"newly_listed"`, `"ending_soon"`, `"price_asc"` or `"price_desc"`; `page_size` is 60, 120 or 240. `exactMatches` is `false` when eBay found nothing for the keyword and answered with its own loosely-related suggestions instead, so check it before using the listings.
 
-### TikTok
-
-```ruby
-profile = su.tiktok_profile("nasa", max_videos: 5)      # exact stats + newest videos
-video = su.tiktok_video("https://www.tiktok.com/@nasa/video/7665075736742530317", include_transcript: true)
-tag = su.tiktok_hashtag("nasa", max_videos: 10)
-results = su.tiktok_search("space telescope", max_results: 25)   # TikTok's own ranking
-comments = su.tiktok_comments("https://www.tiktok.com/@nasa/video/7665075736742530317", max_comments: 40)
-puts profile["stats"]["followers"], video["stats"]["plays"], tag["stats"]["views"], comments["totalComments"]
-```
-
-Profiles and hashtags list up to 10 videos in a couple of seconds from TikTok's server-rendered widget; ask for more (up to 200) and the real grid is scrolled in a browser session.
-
 ## Cookies and the serving proxy
 
 ```ruby
@@ -229,25 +258,6 @@ hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", chec
 cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")
 ```
 
-## Southwest plugin
-
-Southwest Airlines flight quotes as JSON. `origin` and `dest` are IATA airport codes; omit `return_date` for a one-way search.
-
-```ruby
-flights = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", return_date: "2026-10-27"
-)
-
-# One-way, priced in Rapid Rewards points
-oneway = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", adults: 2, fare_type: "points"
-)
-```
-
-`adults` is 1-8 (default 1), `fare_type` is `"dollars"` (default) or `"points"`, `proxy_country` defaults to `"US"` and `max_attempts` is 1-5 (default 3). The call returns the raw booking / shopping JSON.
-
 ## Error handling
 
 Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`.

data/lib/scrapeunblocker/client.rb

@@ -6,6 +6,7 @@ require "json"
 
 require_relative "errors"
 require_relative "parsed_page"
+require_relative "plugins"
 require_relative "skyscanner"
 require_relative "southwest"
 require_relative "version"
@@ -16,6 +17,8 @@ module ScrapeUnblocker
   #   su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
   #   html = su.get_page_source("https://example.com")
   class Client
+    include Plugins
+
     DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
     API_KEY_HEADER = "x-scrapeunblocker-key"
     RETRYABLE = [429, 502, 503, 504].freeze
@@ -302,8 +305,11 @@ module ScrapeUnblocker
       params.each_with_object([]) do |(key, value), acc|
         next if value.nil?
 
-        value = value ? "true" : "false" if value == true || value == false
-        acc << "#{key}=#{URI.encode_www_form_component(value.to_s)}"
+        # A list is a repeated parameter (stars=4&stars=5), as the API reads it.
+        Array(value).each do |item|
+          item = item ? "true" : "false" if item == true || item == false
+          acc << "#{key}=#{URI.encode_www_form_component(item.to_s)}"
+        end
       end.join("&")
     end