scrapeunblocker 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3ffe8b5253c990d2b8f3754b32da58cd5a74150513c5592606dcc70c24d98e6c
-  data.tar.gz: a086e3553aeb1fd48bd57884a2315adfbe92b55763277298db932365b42c44bf
+  metadata.gz: 506dfab9e7e8c63f8b2808a9f7cdf8b113a509fe94b1841c2e8970a33e2bc451
+  data.tar.gz: b1bc717e7851b88651cf6960cabfb8d9298589d0494040be702a1c9ce5d953e4
 SHA512:
-  metadata.gz: 2c0efc8818178442559f1d4ee6ab1e1fe16938e5f5f3c4371eae0506a6ec5d498df37345e726d26293807be62783051bf5824510cb23b3099fa6eb44cdb54b44
-  data.tar.gz: 15d718a1db74279f32f17bfbfda2034987f411e61514c15c122567115edcabb3849dcbeae4ed1fbe9e94e1c913ac6997eea1c638ba2c645d93ef618ea234562e
+  metadata.gz: 5f1901a2940ae350e8603fe8cb4776ecdb433173340010ec8c320ef063c6f4d906ebd135ca36359a3d708b2722826f1db77d56d785724ff7c5a7bbddef01db4b
+  data.tar.gz: 0dce31734e4c853bae21c43581880ff84251990ba1ea8a8023002b4705d669206141b10ec8dea45db4fe825a384334a3d7cbe5b07d3feeba0bb5ef50c7b49f94

data/CHANGELOG.md

@@ -1,8 +1,19 @@
 # Changelog
 
+## 0.11.0 (2026-10-13)
+
+- Removed four plugin methods added in 0.10.0: `youtube_comments`, `telegram_channel`, `quora_question` and `quora_profile`. Those plugins are no longer offered through the SDKs; `plugin(path, params)` still reaches any endpoint your account can use.
+
+## 0.10.1 (2026-10-13)
+
+- `tiktok_shop_search` takes its keyword as optional, so a category-only search works (`category` option).
+- `aldi_au_specials` takes `category`: a Special Buys drop date such as `2026-10-14`.
+- `capterra_search` no longer offers `page` and `app_store_reviews` no longer offers `lang`; the API ignored both for those calls.
+- Method docs say "proxy country" where they said "exit-IP country", and drop internal routing notes.
+
 ## 0.10.0 (2026-10-13)
 
-- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Telegram, Quora, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
+- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
 - Each method takes the plugin's main input positionally and the rest as keyword options; options left `nil` are not sent, so the API's defaults apply. Methods for one mode of a multi-mode endpoint set the mode themselves (`woolworths_product("888137")`, `spotify_artist(...)`).
 - New `plugin(path, **params)` calls any plugin endpoint by path. Array values are sent as a repeated query parameter.
 

data/README.md

@@ -145,7 +145,7 @@ jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")
 | Reviews & Companies | `trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search` |
 | Local & Maps | `yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details` |
 | Apps | `google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews` |
-| Video, Music & Communities | `youtube_channel`, `youtube_search`, `youtube_comments`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`, `telegram_channel`, `quora_question`, `quora_profile` |
+| Video & Music | `youtube_channel`, `youtube_search`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist` |
 | SEO & Ads | `similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library` |
 <!-- plugin-methods:end -->
 

data/lib/scrapeunblocker/client.rb

@@ -151,7 +151,7 @@ module ScrapeUnblocker
     # originalPrice, priceCny, monthSold, image and url. +channel+ is one of
     # "1688" (default), "taobao" or "official"; +sort+ is one of "default",
     # "price_asc", "price_desc" or "best_selling". +page_size+ max is 60.
-    # Brand keywords return HTTP 422.
+    # A brand keyword returns 200 with keywordRejected = true and no products.
     def oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil)
       post_json("/goods/oopbuy-search",
                 keyword: keyword, channel: channel, page: page,
@@ -192,7 +192,7 @@ module ScrapeUnblocker
     # availability, rating, review count, seller, feature bullets, categories
     # and images. Prices come back in the marketplace's own currency:
     # +proxy_country+ defaults to the marketplace's home country
-    # (amazon.com -> US), pinning the exit over the ISP pool. Pass either
+    # (amazon.com -> US). Pass either
     # +asin+ (with +marketplace+) or a full product +url+.
     def amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil)
       post_json("/marketplace/amazon-product",

data/lib/scrapeunblocker/plugins.rb

@@ -554,7 +554,7 @@ module ScrapeUnblocker
     # @param max_results [Integer] Products to return (1-30). One page is read: a keyword lists
     # up to 30 products, a category about 15. API default: 30.
     # Calls POST /social/tiktok-shop-search.
-    def tiktok_shop_search(q, category: nil, max_results: nil)
+    def tiktok_shop_search(q = nil, category: nil, max_results: nil)
       post_json("/social/tiktok-shop-search", {
         q: q,
         category: category,
@@ -1080,6 +1080,11 @@ module ScrapeUnblocker
     # prices in AUD (ALDI Australia does not sell groceries online). Listings collect up to
     # `max_results` from `page` on, billed one unit per page.
     #
+    # @param category [String] mode=category: a category link
+    # (https://www.aldi.com.au/products/dairy-eggs-fridge/milk/k/1111111160) or its key
+    # ('1111111160'; '1588161425841179' = Lower Prices). mode=specials: the Special Buys drop -
+    # its date ('2026-10-14') or link (https://www.aldi.com.au/special-buys/2026-10-14); blank =
+    # the current drop.
     # @param theme [String] mode=specials: one theme of the drop, e.g. 'Gardening' (see
     # `promotions[].themes`).
     # @param sort [String] Listing order: relevance, price_asc, price_desc, name_asc, name_desc.
@@ -1091,9 +1096,10 @@ module ScrapeUnblocker
     # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'AU',
     # 'US'). Leave blank: ALDI answers every country, so the load spreads over all exits.
     # Calls POST /groceries/aldi-au.
-    def aldi_au_specials(theme: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
+    def aldi_au_specials(category: nil, theme: nil, sort: nil, page: nil, max_results: nil, proxy_country: nil)
       post_json("/groceries/aldi-au", {
         mode: "specials",
+        category: category,
         theme: theme,
         sort: sort,
         page: page,
@@ -2757,20 +2763,16 @@ module ScrapeUnblocker
     # avoids it. Billed one unit per page fetched (`pagesFetched`).
     #
     # @param query [String] Product name or keyword to search for (mode=search), e.g. 'crm'.
-    # @param page [Integer] 1-based review page to start from: 25 reviews per page, at most 100
-    # (Capterra's limit per product). Use the `nextPage` of a previous response to continue. API
-    # default: 1.
     # @param max_results [Integer] How many reviews to collect across pages (1-500, default 25);
     # each page fetched is one billed unit. With mode=search, how many products (1-20, default
     # 20; one page).
     # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
     # 'DE'). Leave blank for the country chosen for the site.
     # Calls POST /reviews/capterra.
-    def capterra_search(query, page: nil, max_results: nil, proxy_country: nil)
+    def capterra_search(query, max_results: nil, proxy_country: nil)
       post_json("/reviews/capterra", {
         mode: "search",
         query: query,
-        page: page,
         max_results: max_results,
         proxy_country: proxy_country,
       })
@@ -3227,8 +3229,6 @@ module ScrapeUnblocker
     # ('net.whatsapp.WhatsApp').
     # @param country [String] App Store country (ISO-2), e.g. 'us', 'gb', 'de'. Prices, ratings
     # and reviews are per country. Default: the country in an app URL, else 'us'.
-    # @param lang [String] Listing language for mode=app / mode=search, e.g. 'fr' or 'fr-CA'
-    # (for countries whose store has several languages). Default: the store's own.
     # @param max_results [Integer] mode=search: apps to return (1-200). mode=reviews: reviews to
     # collect (1-500); Apple serves 50 per page and at most 10 pages per country, each page
     # billed as one request. API default: 50.
@@ -3240,12 +3240,11 @@ module ScrapeUnblocker
     # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
     # 'DE'). Leave blank to let the API pick.
     # Calls POST /apps/app-store.
-    def app_store_reviews(app_id, country: nil, lang: nil, max_results: nil, sort: nil, page: nil, proxy_country: nil)
+    def app_store_reviews(app_id, country: nil, max_results: nil, sort: nil, page: nil, proxy_country: nil)
       post_json("/apps/app-store", {
         mode: "reviews",
         app_id: app_id,
         country: country,
-        lang: lang,
         max_results: max_results,
         sort: sort,
         page: page,
@@ -3303,30 +3302,6 @@ module ScrapeUnblocker
       })
     end
 
-    # Scrape a YouTube video's comments.
-    #
-    # Returns a YouTube video's comments as JSON. For each comment: `commentId`, `author`,
-    # `authorChannelId`, `authorThumbnail`, `text`, `likeCount`, `publishedTimeText`,
-    # `replyCount`, and the `isPinned`, `isHearted`, `isAuthorVerified` and `isUploader` flags,
-    # plus the video `title` and total `commentCount`. Sort by `top` or `newest` and collect up
-    # to `max_results` across pages.
-    #
-    # @param video [String] Video id (dQw4w9WgXcQ) or any YouTube video URL.
-    # @param sort [String] Comment order: top, newest. API default: top.
-    # @param max_results [Integer] How many comments to collect across pages (~20 per page). API
-    # default: 20.
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE') for the watch page. Defaults to US.
-    # Calls POST /content/youtube-comments.
-    def youtube_comments(video, sort: nil, max_results: nil, proxy_country: nil)
-      post_json("/content/youtube-comments", {
-        video: video,
-        sort: sort,
-        max_results: max_results,
-        proxy_country: proxy_country,
-      })
-    end
-
     # Get a YouTube video transcript.
     #
     # Returns a YouTube video's captions as JSON: timestamped `segments` (`start`, `dur`,
@@ -3567,146 +3542,6 @@ module ScrapeUnblocker
       })
     end
 
-    # Scrape a public Telegram channel: info and posts.
-    #
-    # Reads a public Telegram channel's web preview. Returns `channel` (`username`, `title`,
-    # `description`, `avatar`, `verified`, `subscribers` and the other `counters`) and `posts`,
-    # newest first: `id`, `url`, `date` (ISO-8601 UTC), `text` (plain), `textHtml`, `views`,
-    # `author` (signature), `edited`, `forwardedFrom`, `replyTo`, `media` (photo / video /
-    # animation / round_video / voice / audio / document / sticker, with CDN `url`, `thumbnail`,
-    # `duration`), `poll`, `linkPreview`, `reactions` (a custom-emoji reaction has its
-    # `customEmojiId`, plus the standard `emoji` and `image` Telegram falls back to when the
-    # page shows that emoji anywhere), `buttons`, `links`, `hashtags` (the tags Telegram links;
-    # a bare '#7' is text) and `mentions`. One page (about 20 posts) by default; `max_posts` (up
-    # to 500) pages back through history; `before`, `after` and `since` bound the range, `q`
-    # searches inside the channel and `post_id` returns one post. `nextBefore` continues a
-    # capped answer. Only channels have a public preview: users, bots, groups and private
-    # channels answer 404. Billed one request per page fetched (`pagesFetched`).
-    #
-    # @param channel [String] Public channel: username ('durov'), '@durov', or a t.me link
-    # ('https://t.me/durov', 'https://t.me/s/durov'). A post link ('https://t.me/durov/400')
-    # returns that one post.
-    # @param max_posts [Integer] Posts to collect, newest first (1-500), paging back through the
-    # channel's history; each page fetched is billed as one request. Omit it for one page,
-    # billed once: the newest posts (a page has 20 message slots and an album fills one per
-    # photo, so it can hold fewer posts).
-    # @param before [Integer] Only posts with an id below this one. Pass `nextBefore` of a
-    # previous response to continue where it stopped.
-    # @param after [Integer] Stop at this post id: only posts with a higher id are returned.
-    # @param since [String] Stop at this date: only posts published on or after it (YYYY-MM-DD
-    # or an ISO-8601 datetime, UTC when no offset is given).
-    # @param q [String] Search inside the channel: only posts matching this text.
-    # @param post_id [Integer] Return only this post (a post id of the channel).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank to let the API pick.
-    # Calls POST /social/telegram-channel.
-    def telegram_channel(channel, max_posts: nil, before: nil, after: nil, since: nil, q: nil, post_id: nil, proxy_country: nil)
-      post_json("/social/telegram-channel", {
-        channel: channel,
-        max_posts: max_posts,
-        before: before,
-        after: after,
-        since: since,
-        q: q,
-        post_id: post_id,
-        proxy_country: proxy_country,
-      })
-    end
-
-    # Scrape Quora questions, answers and profiles.
-    #
-    # Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
-    # `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
-    # `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
-    # merged into this one, `related` with `include_related=true`), the answered `question`,
-    # `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
-    # `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
-    # `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
-    # are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
-    # within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
-    # every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
-    # `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
-    # `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
-    # follower / following / answer / question / post counts, `contentViews`, `credentials`,
-    # `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
-    # only what Quora shows an anonymous visitor - Quora's search is not available that way, and
-    # Quora lists only a profile's newest few dozen answers to logged-out visitors
-    # (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
-    # ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
-    # (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
-    #
-    # @param question [String] Quora question URL (https://www.quora.com/What-is-the-best-way-
-    # to-learn-programming), an answer URL on it, or the question slug (What-is-the-best-way-to-
-    # learn-programming). Language editions such as es.quora.com are accepted as URLs. Required
-    # with mode=question.
-    # @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
-    # 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
-    # first few; each page is one billed unit.
-    # @param include_related [Boolean] mode=question: also return the answers Quora shows from
-    # related questions (its 'All related' view, `relation`='related'). By default only answers
-    # to this question and to questions merged into it. API default: false.
-    # @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
-    # each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
-    # can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
-    # ask for a larger `max_answers` in one call (up to 300).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank for the countries chosen for the site.
-    # Calls POST /social/quora.
-    def quora_question(question, max_answers: nil, include_related: nil, cursor: nil, proxy_country: nil)
-      post_json("/social/quora", {
-        mode: "question",
-        question: question,
-        max_answers: max_answers,
-        include_related: include_related,
-        cursor: cursor,
-        proxy_country: proxy_country,
-      })
-    end
-
-    # Scrape Quora questions, answers and profiles.
-    #
-    # Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
-    # `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
-    # `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
-    # merged into this one, `related` with `include_related=true`), the answered `question`,
-    # `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
-    # `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
-    # `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
-    # are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
-    # within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
-    # every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
-    # `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
-    # `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
-    # follower / following / answer / question / post counts, `contentViews`, `credentials`,
-    # `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
-    # only what Quora shows an anonymous visitor - Quora's search is not available that way, and
-    # Quora lists only a profile's newest few dozen answers to logged-out visitors
-    # (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
-    # ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
-    # (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
-    #
-    # @param profile [String] Quora profile URL (https://www.quora.com/profile/Adam-DAngelo) or
-    # profile slug (Adam-DAngelo). Required with mode=profile.
-    # @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
-    # 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
-    # first few; each page is one billed unit.
-    # @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
-    # each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
-    # can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
-    # ask for a larger `max_answers` in one call (up to 300).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank for the countries chosen for the site.
-    # Calls POST /social/quora.
-    def quora_profile(profile, max_answers: nil, cursor: nil, proxy_country: nil)
-      post_json("/social/quora", {
-        mode: "profile",
-        profile: profile,
-        max_answers: max_answers,
-        cursor: cursor,
-        proxy_country: proxy_country,
-      })
-    end
-
     # Get a website's Similarweb traffic overview.
     #
     # Returns Similarweb's public traffic overview for `domain` as JSON: `globalRank`,

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.10.0"
+  VERSION = "0.11.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.11.0
 platform: ruby
 authors:
 - ScrapeUnblocker