scrapeunblocker 0.10.1 → 0.11.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 14be9ee13840a5574c9924bf39c16b15ecf05c7e4cde02091840034013474c90
-  data.tar.gz: e915ed083458bad3fcc0bdbf1be03347052582a97f8e0955daa86ede3b535eff
+  metadata.gz: 9bd6d663c941e4d49cc8c8b7446f15c2c4cff0ac7025e85144dd72706ac24220
+  data.tar.gz: 68949c187f57eb403d1f890a236fd9caa671beeaf395225c8100d4da22ce8fac
 SHA512:
-  metadata.gz: 5d9ccb478cbf809a1c747b578946a08e0fb30a0e8d968d5eda0cd8389f7c572825e37772c1f0ec590b8ba618d47748262de8a65c773e8fa0ade262b0cb75d471
-  data.tar.gz: c43112450a0ffebabf538bb3a0cb50d2a74be3c0de0712f8e138a34d226b8eb7710f6e9b78560280412d2001e7860439ef3c1d1975559f67519df0bb6d838262
+  metadata.gz: b6c58655e6ea4657b358e0edf1c147226d2fdbb5b55b64a18b88f3f5b5d821b58b78ef29ccb2f505554998c6ef9e8183afa45f5dcf01fcf9f9eb3fb5dea28d75
+  data.tar.gz: 8ab2b5635e0b7edc16bfd1202e92ed24e0a189a115d2f37ea26e2c21c9b7780e5f9f942a50edda29cdb6645e2b7a61228bcea014abe08c44f20f5565f31f1057

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.11.1 (2026-10-15)
+
+- `get_image` docs: the API now returns the image's original bytes and format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...) instead of a PNG re-encode. The method still returns a binary `String`.
+- A missing image (the host's own 404/410) raises `TargetNotFoundError`; a URL that is not an image raises `UnsupportedContentError`. The 415 message no longer says "does not serve HTML", which was wrong for `get_image`.
+
+## 0.11.0 (2026-10-13)
+
+- Removed four plugin methods added in 0.10.0: `youtube_comments`, `telegram_channel`, `quora_question` and `quora_profile`. Those plugins are no longer offered through the SDKs; `plugin(path, params)` still reaches any endpoint your account can use.
+
 ## 0.10.1 (2026-10-13)
 
 - `tiktok_shop_search` takes its keyword as optional, so a category-only search works (`category` option).
@@ -9,7 +18,7 @@
 
 ## 0.10.0 (2026-10-13)
 
-- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Telegram, Quora, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
+- 104 new plugin methods on `ScrapeUnblocker::Client`, one for every public plugin endpoint: e-commerce (Walmart, Target, Best Buy, Home Depot, Etsy, Temu, SHEIN, Zalando, AliExpress, Alibaba, 1688, TikTok Shop, Amazon Best Sellers and reviews), groceries (Woolworths, Coles, ALDI Australia), travel (Booking.com reviews, Airbnb listings and reviews, Tripadvisor), Google (Trends, Shopping, Jobs, Maps place details), Bing / DuckDuckGo / Yahoo / Brave search and images, real estate (Zillow, Redfin, Rightmove, Idealista, ImmobilienScout24), vehicles (mobile.de, AutoScout24), jobs (Indeed, Glassdoor, Naukri, Wellfound, company career sites), reviews and companies (Trustpilot, G2, Capterra, Product Hunt, Crunchbase), Yelp, Naver Place, Google Play, the App Store, YouTube, Spotify, Similarweb, Ahrefs and Semrush. The README lists them by category; every parameter is on the docs plugin pages.
 - Each method takes the plugin's main input positionally and the rest as keyword options; options left `nil` are not sent, so the API's defaults apply. Methods for one mode of a multi-mode endpoint set the mode themselves (`woolworths_product("888137")`, `spotify_artist(...)`).
 - New `plugin(path, **params)` calls any plugin endpoint by path. Array values are sent as a repeated query parameter.
 

data/README.md

@@ -145,7 +145,7 @@ jobs = su.career_site_jobs("https://jobs.ashbyhq.com/ramp", keyword: "engineer")
 | Reviews & Companies | `trustpilot_reviews`, `g2_reviews`, `g2_search`, `capterra_reviews`, `capterra_search`, `product_hunt`, `crunchbase_company`, `crunchbase_search` |
 | Local & Maps | `yelp_search`, `yelp_business`, `yelp_reviews`, `naver_place_search`, `naver_place_details` |
 | Apps | `google_play_app`, `google_play_search`, `app_store_app`, `app_store_search`, `app_store_reviews` |
-| Video, Music & Communities | `youtube_channel`, `youtube_search`, `youtube_comments`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist`, `telegram_channel`, `quora_question`, `quora_profile` |
+| Video & Music | `youtube_channel`, `youtube_search`, `youtube_transcript`, `spotify_search`, `spotify_artist`, `spotify_album`, `spotify_track`, `spotify_playlist` |
 | SEO & Ads | `similarweb_overview`, `ahrefs`, `semrush_overview`, `meta_ad_library` |
 <!-- plugin-methods:end -->
 
@@ -242,6 +242,8 @@ bytes = su.get_image("https://example.com/photo.jpg")
 File.binwrite("photo.jpg", bytes)
 ```
 
+The bytes come back exactly as the image host serves them, in the original format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...); nothing is re-encoded. Pass the image URL itself, not the page it sits on.
+
 ## Skyscanner plugins
 
 Flights, hotels and car hire as JSON:
@@ -287,10 +289,10 @@ end
 | `BudgetExceededError` | 402 | This billing period's spend reached the monthly budget limit you set |
 | `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
-| `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
+| `NotFoundError` | 404 | What you asked for does not exist - a plugin lookup found nothing |
 | `TargetNotFoundError` | 404 / 410 | The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed) |
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
-| `UnsupportedContentError` | 415 | The URL serves something other than HTML |
+| `UnsupportedContentError` | 415 | The URL serves something other than HTML (`get_page_source`) or other than an image (`get_image`) |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |

data/lib/scrapeunblocker/client.rb

@@ -215,7 +215,11 @@ module ScrapeUnblocker
                 proxy_country: proxy_country)
     end
 
-    # Fetch an image URL through the bypass chain and return its raw bytes.
+    # Fetch an image and return its bytes exactly as the image host serves them -
+    # the original format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...), nothing
+    # re-encoded. +url+ must point at the image itself: a page URL raises
+    # UnsupportedContentError, and an image the host answers 404 or 410 for
+    # raises TargetNotFoundError.
     def get_image(url, proxy_country: nil)
       request("/getImage", url: url, proxy_country: proxy_country)[:body]
     end

data/lib/scrapeunblocker/errors.rb

@@ -94,16 +94,15 @@ module ScrapeUnblocker
 
   # Something the call asked for does not exist (HTTP 404).
   #
-  # #get_image raises it when the page rendered but held no <img> tag, and
-  # plugin methods raise it when the item they look up does not exist. When the
+  # Plugin methods raise it when the item they look up does not exist. When the
   # target page itself answered 404 or 410, the more specific
   # TargetNotFoundError subclass is raised instead.
   class NotFoundError < APIError; end
 
   # The target page itself does not exist (HTTP 404 or 410).
   #
-  # Raised by #get_page_source, #get_parsed and #get_page_with_cookies when the
-  # site you asked for answered 404 or 410 on its own. The API passes that
+  # Raised by #get_page_source, #get_parsed, #get_page_with_cookies and
+  # #get_image when the site you asked for answered 404 or 410 on its own. The API passes that
   # status through and marks it with the X-Origin-Status header, which is how
   # this is told apart from an API-side 404. It is the target's final answer,
   # not a block, so it is never retried - and the call is billed, because the
@@ -130,8 +129,10 @@ module ScrapeUnblocker
   # the API answered - it just could not render the page in time.
   class BrowserTimeoutError < APIError; end
 
-  # The URL serves something other than HTML (HTTP 415).
-  # The message names the content type found. For images, use #get_image.
+  # The URL serves a content type the call cannot return (HTTP 415): something
+  # other than HTML for #get_page_source (for images, use #get_image), or
+  # something other than an image for #get_image. The message names the content
+  # type found.
   class UnsupportedContentError < APIError; end
 
   # A request parameter is missing or has the wrong type (HTTP 422).
@@ -183,7 +184,7 @@ module ScrapeUnblocker
     403 => "Target blocked by bot protection on every bypass path",
     404 => "Requested element not found on the page",
     408 => "Browser run timed out before the page was ready",
-    415 => "URL does not serve HTML",
+    415 => "URL serves a content type this call cannot return",
     422 => "Validation error - see the detail array in the response body",
     429 => "Rate limited - too many requests",
     503 => "Upstream origin returned a server-side outage page",

data/lib/scrapeunblocker/plugins.rb

@@ -3302,30 +3302,6 @@ module ScrapeUnblocker
       })
     end
 
-    # Scrape a YouTube video's comments.
-    #
-    # Returns a YouTube video's comments as JSON. For each comment: `commentId`, `author`,
-    # `authorChannelId`, `authorThumbnail`, `text`, `likeCount`, `publishedTimeText`,
-    # `replyCount`, and the `isPinned`, `isHearted`, `isAuthorVerified` and `isUploader` flags,
-    # plus the video `title` and total `commentCount`. Sort by `top` or `newest` and collect up
-    # to `max_results` across pages.
-    #
-    # @param video [String] Video id (dQw4w9WgXcQ) or any YouTube video URL.
-    # @param sort [String] Comment order: top, newest. API default: top.
-    # @param max_results [Integer] How many comments to collect across pages (~20 per page). API
-    # default: 20.
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE') for the watch page. Defaults to US.
-    # Calls POST /content/youtube-comments.
-    def youtube_comments(video, sort: nil, max_results: nil, proxy_country: nil)
-      post_json("/content/youtube-comments", {
-        video: video,
-        sort: sort,
-        max_results: max_results,
-        proxy_country: proxy_country,
-      })
-    end
-
     # Get a YouTube video transcript.
     #
     # Returns a YouTube video's captions as JSON: timestamped `segments` (`start`, `dur`,
@@ -3566,146 +3542,6 @@ module ScrapeUnblocker
       })
     end
 
-    # Scrape a public Telegram channel: info and posts.
-    #
-    # Reads a public Telegram channel's web preview. Returns `channel` (`username`, `title`,
-    # `description`, `avatar`, `verified`, `subscribers` and the other `counters`) and `posts`,
-    # newest first: `id`, `url`, `date` (ISO-8601 UTC), `text` (plain), `textHtml`, `views`,
-    # `author` (signature), `edited`, `forwardedFrom`, `replyTo`, `media` (photo / video /
-    # animation / round_video / voice / audio / document / sticker, with CDN `url`, `thumbnail`,
-    # `duration`), `poll`, `linkPreview`, `reactions` (a custom-emoji reaction has its
-    # `customEmojiId`, plus the standard `emoji` and `image` Telegram falls back to when the
-    # page shows that emoji anywhere), `buttons`, `links`, `hashtags` (the tags Telegram links;
-    # a bare '#7' is text) and `mentions`. One page (about 20 posts) by default; `max_posts` (up
-    # to 500) pages back through history; `before`, `after` and `since` bound the range, `q`
-    # searches inside the channel and `post_id` returns one post. `nextBefore` continues a
-    # capped answer. Only channels have a public preview: users, bots, groups and private
-    # channels answer 404. Billed one request per page fetched (`pagesFetched`).
-    #
-    # @param channel [String] Public channel: username ('durov'), '@durov', or a t.me link
-    # ('https://t.me/durov', 'https://t.me/s/durov'). A post link ('https://t.me/durov/400')
-    # returns that one post.
-    # @param max_posts [Integer] Posts to collect, newest first (1-500), paging back through the
-    # channel's history; each page fetched is billed as one request. Omit it for one page,
-    # billed once: the newest posts (a page has 20 message slots and an album fills one per
-    # photo, so it can hold fewer posts).
-    # @param before [Integer] Only posts with an id below this one. Pass `nextBefore` of a
-    # previous response to continue where it stopped.
-    # @param after [Integer] Stop at this post id: only posts with a higher id are returned.
-    # @param since [String] Stop at this date: only posts published on or after it (YYYY-MM-DD
-    # or an ISO-8601 datetime, UTC when no offset is given).
-    # @param q [String] Search inside the channel: only posts matching this text.
-    # @param post_id [Integer] Return only this post (a post id of the channel).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank to let the API pick.
-    # Calls POST /social/telegram-channel.
-    def telegram_channel(channel, max_posts: nil, before: nil, after: nil, since: nil, q: nil, post_id: nil, proxy_country: nil)
-      post_json("/social/telegram-channel", {
-        channel: channel,
-        max_posts: max_posts,
-        before: before,
-        after: after,
-        since: since,
-        q: q,
-        post_id: post_id,
-        proxy_country: proxy_country,
-      })
-    end
-
-    # Scrape Quora questions, answers and profiles.
-    #
-    # Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
-    # `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
-    # `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
-    # merged into this one, `related` with `include_related=true`), the answered `question`,
-    # `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
-    # `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
-    # `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
-    # are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
-    # within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
-    # every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
-    # `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
-    # `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
-    # follower / following / answer / question / post counts, `contentViews`, `credentials`,
-    # `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
-    # only what Quora shows an anonymous visitor - Quora's search is not available that way, and
-    # Quora lists only a profile's newest few dozen answers to logged-out visitors
-    # (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
-    # ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
-    # (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
-    #
-    # @param question [String] Quora question URL (https://www.quora.com/What-is-the-best-way-
-    # to-learn-programming), an answer URL on it, or the question slug (What-is-the-best-way-to-
-    # learn-programming). Language editions such as es.quora.com are accepted as URLs. Required
-    # with mode=question.
-    # @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
-    # 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
-    # first few; each page is one billed unit.
-    # @param include_related [Boolean] mode=question: also return the answers Quora shows from
-    # related questions (its 'All related' view, `relation`='related'). By default only answers
-    # to this question and to questions merged into it. API default: false.
-    # @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
-    # each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
-    # can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
-    # ask for a larger `max_answers` in one call (up to 300).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank for the countries chosen for the site.
-    # Calls POST /social/quora.
-    def quora_question(question, max_answers: nil, include_related: nil, cursor: nil, proxy_country: nil)
-      post_json("/social/quora", {
-        mode: "question",
-        question: question,
-        max_answers: max_answers,
-        include_related: include_related,
-        cursor: cursor,
-        proxy_country: proxy_country,
-      })
-    end
-
-    # Scrape Quora questions, answers and profiles.
-    #
-    # Given a Quora `question` (URL or slug), returns the `question` (`qid`, `title`, `url`,
-    # `answerCount`, `allRelatedCount`, `followerCount`, `commentCount`, `topics`) and its
-    # `answers`: `id`, `url`, `relation` (`answer`, `merged` for an answer to a question Quora
-    # merged into this one, `related` with `include_related=true`), the answered `question`,
-    # `author` (`name`, `uid`, `profileUrl`, `imageUrl`, `credential`, `isVerified`,
-    # `isAnonymous`), the full `text` with `images` and `links`, `upvotes`, `views`, `shares`,
-    # `comments`, `createdAt`, `updatedAt` and `isAiGenerated`; plus `relatedQuestions`. Answers
-    # are in Quora's own order, up to `max_answers` (10 per page after the first few), unique
-    # within a call. `nextCursor` as `cursor` continues, best-effort: Quora re-ranks the list on
-    # every visit, so a continuation can repeat or skip answers of earlier calls (dedupe by
-    # `id`); for a complete list ask for a larger `max_answers` in one call (up to 300).
-    # `mode=profile&profile=` returns a public `profile` (`name`, `headline`, `description`,
-    # follower / following / answer / question / post counts, `contentViews`, `credentials`,
-    # `knowsAbout`, `spaces`, `topWriterYears`, `joinedAt`) with its newest `answers`. No login:
-    # only what Quora shows an anonymous visitor - Quora's search is not available that way, and
-    # Quora lists only a profile's newest few dozen answers to logged-out visitors
-    # (`pagingStopped`: `quora_logged_out_limit`). `pagingStopped` otherwise names why paging
-    # ended early (`blocked`, `time`, `unavailable`). Billed one unit per page fetched
-    # (`pagesFetched`); a `cursor` continuation bills only its GraphQL pages.
-    #
-    # @param profile [String] Quora profile URL (https://www.quora.com/profile/Adam-DAngelo) or
-    # profile slug (Adam-DAngelo). Required with mode=profile.
-    # @param max_answers [Integer] Answers to return: 1-300 with mode=question (default 20),
-    # 0-200 with mode=profile (default 10; 0 = profile only). Answers come 10 per page after the
-    # first few; each page is one billed unit.
-    # @param cursor [String] Continue after a previous call: pass its `nextCursor`. Best-effort:
-    # each call is a new Quora visit and Quora re-ranks the list per visit, so a continuation
-    # can repeat or skip answers of earlier calls - dedupe by answer `id`. For a complete list
-    # ask for a larger `max_answers` in one call (up to 300).
-    # @param proxy_country [String] Proxy country the request is sent from (ISO-2, e.g. 'US',
-    # 'DE'). Leave blank for the countries chosen for the site.
-    # Calls POST /social/quora.
-    def quora_profile(profile, max_answers: nil, cursor: nil, proxy_country: nil)
-      post_json("/social/quora", {
-        mode: "profile",
-        profile: profile,
-        max_answers: max_answers,
-        cursor: cursor,
-        proxy_country: proxy_country,
-      })
-    end
-
     # Get a website's Similarweb traffic overview.
     #
     # Returns Similarweb's public traffic overview for `domain` as JSON: `globalRank`,

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.10.1"
+  VERSION = "0.11.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.10.1
+  version: 0.11.1
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-10-13 00:00:00.000000000 Z
+date: 2026-10-15 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and 100+ ready-made site plugins (Amazon, Walmart, Zillow, Google