scrapeunblocker 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2668bf966d35f46fd8c65d4d78cdc0d93d989a54b9d3f257f57e7924cc4ad6f6
-  data.tar.gz: 7ec5e5166b99712828dd771597ba0db89a09f398a1c5c0988af3256f077139f1
+  metadata.gz: 941309503bf1a937b76c503a811aab8e7e738e4ec775360ad7f9f6cfc40b6259
+  data.tar.gz: 68753623ec6ad6623632c520e688a5ba2f7697abf6588b8351cdd33e911f473c
 SHA512:
-  metadata.gz: 98330917b1ddce4f2d24f3938a97ce719ef84e31a712a3e774ac5d5821b1df164d8defd3984d4732d736f374f357701b59bae7b3a3154539b06543b92b8ecc5a
-  data.tar.gz: 0fb21ac2f6c04e979bb21415a9d393fd2853bdb1eb793e6ddf1d52cd9a38ab976d9ab403352f15bbe3b24ae64972c60df972942e231d85ff5e2ac805bf76bc34
+  metadata.gz: 14f6f077622d3669a8ef06944d4cb810b89d693d28eead00b1b5789242945478e0e83529ede25aed9a474ee79351ca5202a3c91c03a174fd0cc811a031a48e11
+  data.tar.gz: 23314b94ae64e4ce3ade4bbef52278df0639681801981865e3faa8914fd71f6dec7f3047fbad96e4954dc70b8d603ef35487b5a3d5a9a65623be6e7de41071f7

data/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.7.0 (2026-09-30)
+
+- New `ScrapeUnblocker::TargetNotFoundError` (a subclass of `NotFoundError`): raised by `get_page_source`, `get_parsed` and `get_page_with_cookies` when the target page itself answers 404 or 410. The API now passes the target's own status through instead of a 200, marked with the `X-Origin-Status` header. The error carries `origin_status`, the not-found page on `html` and `destination_url`. It is never retried, and the call is billed like any delivered page.
+- A custom `transport:` may return `headers:` alongside `status:` and `body:`; `ScrapeUnblocker.error_for_status` takes them as an optional third argument.
+
+Behaviour change: a dead target URL used to return its not-found page as a normal String; it now raises `TargetNotFoundError`. `rescue ScrapeUnblocker::NotFoundError` still catches it. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
+
+## 0.6.0 (2026-09-24)
+
+- `google_images` takes `pages:` (1-5): fetch up to five Google Images result pages of ~100 results each in one call. Each page fetched is billed as one request; the response's `pagesFetched` says how many. The Google market now follows `proxy_country` automatically, so `gl` is only an optional override, and `max_results` is an optional cap up to 500.
+
+No breaking changes.
+
 ## 0.5.0 (2026-09-17)
 
 - Added `google_images(q, ...)` for the new Google Images plugin (`POST /images/google-search`). Given a keyword `q` it returns Google Images results as a Hash - each with the full-size `imageUrl` and its `sourceDomain`, plus the source page URL, title, source name, thumbnail URL, pixel dimensions and file size. Optional `gl` (ISO-2 lowercase market), `max_results` (1-100) and `proxy_country` (ISO-2) refine the search.

data/README.md

@@ -131,7 +131,7 @@ local["results"].each { |biz| puts "#{biz['name']} #{biz['rating']} #{biz['addre
 ## Google Images
 
 ```ruby
-images = su.google_images("golden retriever puppy", proxy_country: "US", gl: "us")
+images = su.google_images("golden retriever puppy", proxy_country: "US", pages: 3)
 images["results"].each { |img| puts "#{img['imageUrl']} #{img['sourceDomain']} #{img['title']}" }
 ```
 
@@ -277,7 +277,8 @@ end
 | `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
 | `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
-| `NotFoundError` | 404 | Page loaded but held no image (`get_image` only) |
+| `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
+| `TargetNotFoundError` | 404 / 410 | The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed) |
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
 | `UnsupportedContentError` | 415 | The URL serves something other than HTML |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
@@ -287,6 +288,21 @@ end
 | `TimeoutError` | - | This client gave up locally before the API answered |
 | `ConnectionError` | - | Could not reach the API |
 
+### The target page does not exist (404 / 410)
+
+When the site you scrape answers 404 or 410 itself, the API passes that status through with an `X-Origin-Status` header, and the client raises `ScrapeUnblocker::TargetNotFoundError`. It is the target's final answer, so it is never retried, and it is billed like any delivered page. The not-found page is on `#html`:
+
+```ruby
+begin
+  html = su.get_page_source("https://example.com/removed-listing")
+rescue ScrapeUnblocker::TargetNotFoundError => e
+  puts e.origin_status   # 404 or 410
+  puts e.html            # the target's own not-found page (can be empty)
+end
+```
+
+`TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
+
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 
 ### Billing errors (402)

data/lib/scrapeunblocker/client.rb

@@ -57,6 +57,9 @@ module ScrapeUnblocker
     # +list_elements+, when true, makes the API return a JSON summary of the
     # matched elements ({"url", "count", "elements"}) instead of HTML. This method
     # then returns that parsed Hash rather than an HTML String.
+    #
+    # When the target page itself answers 404 or 410 this raises
+    # ScrapeUnblocker::TargetNotFoundError (billed; the not-found page is on #html).
     def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
                         steps: nil, list_elements: nil)
       body = request("/getPageSource",
@@ -111,11 +114,13 @@ module ScrapeUnblocker
     # Returns image results, each with the full-size +imageUrl+ and its
     # +sourceDomain+, plus the source page URL, title, source name, thumbnail
     # URL, pixel dimensions and file size. +q+ is the search keyword. Set
-    # +proxy_country+ (and optionally +gl+) to target a market, and
-    # +max_results+ (1-100) to cap the count.
-    def google_images(q, proxy_country: nil, gl: nil, max_results: nil)
+    # +proxy_country+ to target a market (the Google market follows it; +gl+
+    # is an optional override), +pages+ (1-5, ~100 results each; each page
+    # fetched is billed as one request, see +pagesFetched+) and +max_results+
+    # (1-500) to cap the count.
+    def google_images(q, proxy_country: nil, pages: nil, gl: nil, max_results: nil)
       post_json("/images/google-search",
-                q: q, gl: gl, max_results: max_results, proxy_country: proxy_country)
+                q: q, pages: pages, gl: gl, max_results: max_results, proxy_country: proxy_country)
     end
 
     # Fetch an advertiser's Meta (Facebook) Ad Library ads and return them as a Hash.
@@ -282,7 +287,8 @@ module ScrapeUnblocker
 
         return { status: status, body: body } if status >= 200 && status < 300
 
-        raise ScrapeUnblocker.error_for_status(status, body)
+        response_headers = (result[:headers] || {}).to_h { |k, v| [k.to_s.downcase, Array(v).join(", ")] }
+        raise ScrapeUnblocker.error_for_status(status, body, response_headers)
       end
     end
 
@@ -313,7 +319,7 @@ module ScrapeUnblocker
         raise ConnectionError, "Could not reach the API: #{e.message}"
       end
 
-      { status: response.code.to_i, body: response.body }
+      { status: response.code.to_i, body: response.body, headers: response.each_header.to_h }
     end
   end
 end

data/lib/scrapeunblocker/errors.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "json"
+
 module ScrapeUnblocker
   # Base class for every error raised by this library.
   class Error < StandardError; end
@@ -78,10 +80,38 @@ module ScrapeUnblocker
   # use instead.
   class InvalidRequestError < APIError; end
 
-  # The page loaded but the requested element was absent (HTTP 404).
-  # Only #get_image raises this: the page rendered and held no <img> tag.
+  # Something the call asked for does not exist (HTTP 404).
+  #
+  # #get_image raises it when the page rendered but held no <img> tag, and
+  # plugin methods raise it when the item they look up does not exist. When the
+  # target page itself answered 404 or 410, the more specific
+  # TargetNotFoundError subclass is raised instead.
   class NotFoundError < APIError; end
 
+  # The target page itself does not exist (HTTP 404 or 410).
+  #
+  # Raised by #get_page_source, #get_parsed and #get_page_with_cookies when the
+  # site you asked for answered 404 or 410 on its own. The API passes that
+  # status through and marks it with the X-Origin-Status header, which is how
+  # this is told apart from an API-side 404. It is the target's final answer,
+  # not a block, so it is never retried - and the call is billed, because the
+  # page was fetched and delivered.
+  #
+  # +origin_status+ is the status the target answered with (404 or 410),
+  # +html+ the target's own not-found page as served (can be empty; nil when
+  # the body is a parsed-data JSON payload), and +destination_url+ the URL the
+  # target answered for, when the API sent X-Destination-URL.
+  class TargetNotFoundError < NotFoundError
+    attr_reader :origin_status, :html, :destination_url
+
+    def initialize(message, status_code:, origin_status:, body: nil, html: nil, destination_url: nil)
+      super(message, status_code: status_code, body: body)
+      @origin_status = origin_status
+      @html = html
+      @destination_url = destination_url
+    end
+  end
+
   # The browser run did not finish in time on our side (HTTP 408).
   #
   # Distinct from TimeoutError, which is this client giving up locally. Here
@@ -153,8 +183,35 @@ module ScrapeUnblocker
   end
   private_class_method :billing_error_class
 
-  # Build a typed error from an HTTP status code and response body.
-  def self.error_for_status(status, body)
+  # The API passes a target's "page does not exist" answer through with its
+  # status and an X-Origin-Status header. A 404 without that header is the
+  # API's own (a plugin lookup, a missing element) and returns nil so the
+  # general NotFoundError applies.
+  def self.target_not_found_error(status, body, headers)
+    origin = headers["x-origin-status"]
+    return nil unless [404, 410].include?(status) && origin && !origin.to_s.empty?
+
+    origin_status = Integer(origin.to_s, exception: false) || status
+    html = body
+    begin
+      data = JSON.parse(body.to_s)
+      html = data["html"].is_a?(String) ? data["html"] : nil if data.is_a?(Hash)
+    rescue JSON::ParserError
+      # Not JSON: the body is the target's own page.
+    end
+    message = "Target page does not exist (HTTP #{origin_status}). This is the " \
+              "target's own answer, not a block; the call is billed."
+    TargetNotFoundError.new(message, status_code: status, body: body, origin_status: origin_status,
+                                     html: html, destination_url: headers["x-destination-url"])
+  end
+  private_class_method :target_not_found_error
+
+  # Build a typed error from an HTTP status code, response body and headers
+  # (a Hash with lowercase names).
+  def self.error_for_status(status, body, headers = {})
+    target_error = target_not_found_error(status, body, headers || {})
+    return target_error if target_error
+
     snippet = (body || "").strip.gsub(/\s+/, " ")
     snippet = "#{snippet[0, 200]}..." if snippet.length > 200
     base = BASE_MESSAGES.fetch(status, "API returned HTTP #{status}")

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.5.0"
+  VERSION = "0.7.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.7.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-17 00:00:00.000000000 Z
+date: 2026-09-30 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.