scrapeunblocker 0.8.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e8096ca55e9f93c01077d947e60e36ec71c595e7ee92d1405a413a0746c303f0
-  data.tar.gz: 28c0667c78459edf026b26effd0209ea0635409eb078f152b8e4963be2312dd3
+  metadata.gz: 3ffe8b5253c990d2b8f3754b32da58cd5a74150513c5592606dcc70c24d98e6c
+  data.tar.gz: a086e3553aeb1fd48bd57884a2315adfbe92b55763277298db932365b42c44bf
 SHA512:
-  metadata.gz: f1b51dee641104d7d77fd600c5cb2299f30d6dcdf3e155686befc2d3024c981fb0c904d2aa8d5d1f360797a9cdbd331356ec2c99f1a3832907704198689306c6
-  data.tar.gz: 8274485fa1644d6dfb1512bc33e93fc034f3c01a0248e450869315f3d0669244761ef146db9e11cac0ccfb3fe6361148d5b3ef64a4c706d967d825ef23fba9e6
+  metadata.gz: 2c0efc8818178442559f1d4ee6ab1e1fe16938e5f5f3c4371eae0506a6ec5d498df37345e726d26293807be62783051bf5824510cb23b3099fa6eb44cdb54b44
+  data.tar.gz: 15d718a1db74279f32f17bfbfda2034987f411e61514c15c122567115edcabb3849dcbeae4ed1fbe9e94e1c913ac6997eea1c638ba2c645d93ef618ea234562e

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.10.0 (2026-10-13)
+
+- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Telegram, Quora, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
+- Each method takes the plugin's main input positionally and the rest as keyword options; options left `nil` are not sent, so the API's defaults apply. Methods for one mode of a multi-mode endpoint set the mode themselves (`woolworths_product("888137")`, `spotify_artist(...)`).
+- New `plugin(path, **params)` calls any plugin endpoint by path. Array values are sent as a repeated query parameter.
+
+No breaking changes.
+
+## 0.9.0 (2026-10-07)
+
+- New `ScrapeUnblocker::BudgetExceededError` (a subclass of `PaymentRequiredError`): raised when the API answers 402 with `User set budget exceeded`, meaning this billing period's spend reached the monthly budget limit (EUR, excluding VAT) set in the [profile](https://app.scrapeunblocker.com/dashboard/profile). It clears at the start of the next billing period, or within about a minute after the limit is raised or removed. Like the other 402s it is never retried and not billed. Before, this body surfaced as a plain `PaymentRequiredError`.
+
+No breaking changes: `rescue ScrapeUnblocker::PaymentRequiredError` still catches it.
+
 ## 0.8.0 (2026-10-06)
 
 - `get_parsed` on a page that rendered but held no structured data now returns a normal `ParsedPage` instead of raising `NoDataExtractedError`. The API answers this case with a billed 200: `data` is empty, the new `data_extracted?` is `false`, the new `detail` explains it and the new `html` carries the rendered page, so no second call is needed. On a successful parse `data_extracted?` is `true` and `html` / `detail` are `nil`.

data/README.md

@@ -108,13 +108,54 @@ result["elements"].each { |el| p el }
 ```ruby
 result = su.get_parsed("https://www.walmart.com/ip/12345")
 puts result.page_type # e.g. "product"
-puts result.source    # how it was extracted
 p result.data         # the fields
 
 # If a parse ever comes back wrong, force a fresh set of rules:
 fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")
 ```
 
+## Plugins
+
+Ready-made scrapers for specific sites that return structured JSON. Each
+method takes the plugin's main input positionally and everything else as
+keyword options; options you leave out are not sent, so the API's own
+defaults apply. Every parameter is documented on the
+[plugin pages](https://docs.scrapeunblocker.com/plugins/overview?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk).
+
+```ruby
+products = su.target_search("coffee maker", sort: "price_low", max_results: 48)
+milk = su.woolworths_product("888137")                     # one product, with nutrition
+trends = su.google_trends(%w[coffee tea], geo: "US")       # a list is sent comma-separated
+reviews = su.trustpilot_reviews("www.nike.com", stars: [4, 5])
+jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")
+```
+
+<!-- plugin-methods:start -->
+| Category | Methods |
+| --- | --- |
+| Travel | `skyscanner.flights`, `skyscanner.hotels`, `skyscanner.carhire`, `booking_hotels`, `booking_reviews`, `airbnb_search`, `airbnb_room`, `airbnb_reviews`, `tripadvisor_list`, `tripadvisor_reviews` |
+| Google | `google_local`, `google_images`, `maps_place_details`, `google_trends`, `google_shopping`, `google_jobs` |
+| Search engines | `bing_search`, `bing_images`, `duckduckgo_search`, `yahoo_search`, `yahoo_images`, `brave_search`, `brave_images` |
+| E-commerce | `amazon_product`, `amazon_search`, `amazon_bestsellers`, `amazon_reviews`, `walmart_search`, `target_search`, `bestbuy_search`, `homedepot_search`, `etsy_search`, `temu_search`, `temu_product`, `shein_search`, `zalando_search`, `aliexpress_search`, `alibaba_search`, `alibaba_product`, `alibaba1688_search`, `alibaba1688_product`, `tiktok_shop_search`, `tiktok_shop_product`, `tiktok_shop_store`, `oopbuy_search` |
+| Classifieds & Resale | `ebay_search`, `vinted_search`, `leboncoin_search` |
+| Groceries | `woolworths_search`, `woolworths_category`, `woolworths_specials`, `woolworths_product`, `coles_search`, `coles_category`, `coles_specials`, `coles_product`, `aldi_au_search`, `aldi_au_category`, `aldi_au_specials`, `aldi_au_product` |
+| Real estate | `zillow_search`, `zillow_detail`, `redfin_search`, `redfin_detail`, `rightmove_search`, `idealista_search`, `idealista_detail`, `immobilienscout24_search` |
+| Vehicles | `mobile_de`, `autoscout24_search` |
+| Jobs | `indeed_jobs`, `glassdoor_jobs`, `glassdoor_reviews`, `glassdoor_salaries`, `naukri_jobs`, `wellfound_jobs`, `wellfound_job`, `wellfound_company`, `wellfound_company_search`, `career_site_jobs` |
+| Reviews & Companies | `trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search` |
+| Local & Maps | `yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details` |
+| Apps | `google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews` |
+| Video, Music & Communities | `youtube_channel`, `youtube_search`, `youtube_comments`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`, `telegram_channel`, `quora_question`, `quora_profile` |
+| SEO & Ads | `similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library` |
+<!-- plugin-methods:end -->
+
+Any plugin endpoint can also be called by path, with parameters named as the
+API names them - handy for an endpoint added after your gem version:
+
+```ruby
+data = su.plugin("/marketplace/target-search", query: "tv", max_results: 24)
+```
+
 ## Google search (SERP)
 
 ```ruby
@@ -185,19 +226,6 @@ end
 
 `marketplace` is any of the 19 regional eBay hosts (`ebay.com` default). `condition` is one of `"new"`, `"open_box"`, `"refurbished"`, `"used"` or `"for_parts"`; `sort` is one of `"best_match"` (default), `"newly_listed"`, `"ending_soon"`, `"price_asc"` or `"price_desc"`; `page_size` is 60, 120 or 240. `exactMatches` is `false` when eBay found nothing for the keyword and answered with its own loosely-related suggestions instead, so check it before using the listings.
 
-### TikTok
-
-```ruby
-profile = su.tiktok_profile("nasa", max_videos: 5)      # exact stats + newest videos
-video = su.tiktok_video("https://www.tiktok.com/@nasa/video/7665075736742530317", include_transcript: true)
-tag = su.tiktok_hashtag("nasa", max_videos: 10)
-results = su.tiktok_search("space telescope", max_results: 25)   # TikTok's own ranking
-comments = su.tiktok_comments("https://www.tiktok.com/@nasa/video/7665075736742530317", max_comments: 40)
-puts profile["stats"]["followers"], video["stats"]["plays"], tag["stats"]["views"], comments["totalComments"]
-```
-
-Profiles and hashtags list up to 10 videos in a couple of seconds from TikTok's server-rendered widget; ask for more (up to 200) and the real grid is scrolled in a browser session.
-
 ## Cookies and the serving proxy
 
 ```ruby
@@ -230,25 +258,6 @@ hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", chec
 cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")
 ```
 
-## Southwest plugin
-
-Southwest Airlines flight quotes as JSON. `origin` and `dest` are IATA airport codes; omit `return_date` for a one-way search.
-
-```ruby
-flights = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", return_date: "2026-10-27"
-)
-
-# One-way, priced in Rapid Rewards points
-oneway = su.southwest.flights(
-  origin: "DAL", dest: "HOU",
-  depart_date: "2026-10-20", adults: 2, fare_type: "points"
-)
-```
-
-`adults` is 1-8 (default 1), `fare_type` is `"dollars"` (default) or `"points"`, `proxy_country` defaults to `"US"` and `max_attempts` is 1-5 (default 3). The call returns the raw booking / shopping JSON.
-
 ## Error handling
 
 Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`.
@@ -259,7 +268,7 @@ begin
 rescue ScrapeUnblocker::BlockedError
   # 403: the target blocked every bypass path (not billed)
 rescue ScrapeUnblocker::PaymentRequiredError
-  # 402: quota, credit limit, or a failed payment - fix billing
+  # 402: quota, credit limit, your budget limit, or a failed payment - fix billing
 rescue ScrapeUnblocker::RateLimitError
   # 429: slow down
 rescue ScrapeUnblocker::UpstreamOutageError
@@ -272,9 +281,10 @@ end
 | `InvalidRequestError` | 400 | Bad URL, unsupported scheme, or the API key header was not sent |
 | `AuthenticationError` | 401 | Key not recognised - typo, stray whitespace, or a rotated key |
 | `NoSubscriptionError` | 401 | Key is fine, but the account has no active plan |
-| `PaymentRequiredError` | 402 | Billing block - base class for the three below |
+| `PaymentRequiredError` | 402 | Billing block - base class for the four below |
 | `QuotaExceededError` | 402 | The plan's requests for this period are used up |
 | `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
+| `BudgetExceededError` | 402 | This billing period's spend reached the monthly budget limit you set |
 | `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
 | `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
@@ -323,7 +333,7 @@ Transient failures (429, 502, 503, 504 and network errors) are retried automatic
 
 ### Billing errors (402)
 
-The three billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
+The four billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
 
 ```ruby
 begin
@@ -332,12 +342,16 @@ rescue ScrapeUnblocker::QuotaExceededError
   # plan quota (plus any overage allowance) is used up for this period
 rescue ScrapeUnblocker::CreditLimitExceededError
   # unpaid balance passed the account credit limit
+rescue ScrapeUnblocker::BudgetExceededError
+  # this period's spend reached the monthly budget limit set in your profile
 rescue ScrapeUnblocker::PaymentFailedError
   # card declined three times - update the payment method
 end
 ```
 
-When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota. All three lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
+When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota, which outranks your own budget limit. All four lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
+
+`BudgetExceededError` (body `User set budget exceeded`) means this billing period's spend reached the monthly budget limit you set in your [profile](https://app.scrapeunblocker.com/dashboard/profile?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk) (EUR, excluding VAT). The key works again at the start of the next billing period, or within about a minute after you raise or remove the limit.
 
 Full details for every status code: [docs.scrapeunblocker.com/errors](https://docs.scrapeunblocker.com/errors).
 

data/lib/scrapeunblocker/client.rb

@@ -6,6 +6,7 @@ require "json"
 
 require_relative "errors"
 require_relative "parsed_page"
+require_relative "plugins"
 require_relative "skyscanner"
 require_relative "southwest"
 require_relative "version"
@@ -16,6 +17,8 @@ module ScrapeUnblocker
   #   su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
   #   html = su.get_page_source("https://example.com")
   class Client
+    include Plugins
+
     DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
     API_KEY_HEADER = "x-scrapeunblocker-key"
     RETRYABLE = [429, 502, 503, 504].freeze
@@ -302,8 +305,11 @@ module ScrapeUnblocker
       params.each_with_object([]) do |(key, value), acc|
         next if value.nil?
 
-        value = value ? "true" : "false" if value == true || value == false
-        acc << "#{key}=#{URI.encode_www_form_component(value.to_s)}"
+        # A list is a repeated parameter (stars=4&stars=5), as the API reads it.
+        Array(value).each do |item|
+          item = item ? "true" : "false" if item == true || item == false
+          acc << "#{key}=#{URI.encode_www_form_component(item.to_s)}"
+        end
       end.join("&")
     end
 

data/lib/scrapeunblocker/errors.rb

@@ -36,15 +36,16 @@ module ScrapeUnblocker
   # The account has a billing problem (HTTP 402).
   #
   # Credentials are fine - the request was stopped for a billing reason. There
-  # are three, each raised as a dedicated subclass: QuotaExceededError,
-  # CreditLimitExceededError and PaymentFailedError. Rescue this base class to
-  # handle all three.
+  # are four, each raised as a dedicated subclass: QuotaExceededError,
+  # CreditLimitExceededError, BudgetExceededError and PaymentFailedError.
+  # Rescue this base class to handle all four.
   #
   # When more than one applies, the most serious wins: failed payment outranks
-  # credit limit, which outranks quota. All three lift by themselves once the
-  # billing state changes - access returns within roughly a minute, with no key
-  # change needed. Like a 401, a 402 is refused before anything is scraped, so
-  # it is never billed. Retrying is pointless; fix the billing state first.
+  # credit limit, which outranks quota, which outranks your own budget limit.
+  # All four lift by themselves once the billing state changes - access returns
+  # within roughly a minute, with no key change needed. Like a 401, a 402 is
+  # refused before anything is scraped, so it is never billed. Retrying is
+  # pointless; fix the billing state first.
   class PaymentRequiredError < APIError; end
 
   # Every request the plan allows this period has been used (HTTP 402).
@@ -64,6 +65,17 @@ module ScrapeUnblocker
   # usually clears itself within about a minute.
   class CreditLimitExceededError < PaymentRequiredError; end
 
+  # This billing period's spend reached the monthly budget limit you set
+  # (HTTP 402).
+  #
+  # The limit is set in your profile (EUR, excluding VAT), and spend is counted
+  # the way the invoice is: the plan's fixed monthly fee, if any, plus the
+  # requests billed on top of it. Requests paid from coupon credit do not count.
+  # The key works again at the start of the next billing period, or within
+  # about a minute after you raise or remove the limit at
+  # https://app.scrapeunblocker.com/dashboard/profile.
+  class BudgetExceededError < PaymentRequiredError; end
+
   # A card payment has been declined three times (HTTP 402).
   #
   # Those attempts are the payment provider's automatic retries spread over
@@ -167,7 +179,7 @@ module ScrapeUnblocker
   BASE_MESSAGES = {
     400 => "Invalid request (bad URL, unsupported scheme, or missing API key header)",
     401 => "Authentication failed - key not recognised, or account has no active plan",
-    402 => "Billing block - quota exceeded, credit limit exceeded, or a failed payment",
+    402 => "Billing block - quota exceeded, credit limit exceeded, budget limit reached, or a failed payment",
     403 => "Target blocked by bot protection on every bypass path",
     404 => "Requested element not found on the page",
     408 => "Browser run timed out before the page was ready",
@@ -187,12 +199,13 @@ module ScrapeUnblocker
   end
   private_class_method :auth_error_class
 
-  # The three billing blocks share a status code and differ only in their
+  # The four billing blocks share a status code and differ only in their
   # plain-text body. An unrecognised body falls back to PaymentRequiredError.
   def self.billing_error_class(body)
     text = (body || "").downcase
     return QuotaExceededError if text.include?("quota exceeded")
     return CreditLimitExceededError if text.include?("credit limit exceeded")
+    return BudgetExceededError if text.include?("user set budget exceeded")
     return PaymentFailedError if text.include?("payment failed")
 
     PaymentRequiredError