scrapeunblocker 0.6.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3076debfbde8d8f4aac22c77b69c7d9117d2050743f7c5794e856b4c8f0618f4
-  data.tar.gz: 19dd437fccd00ae77927c1863695bfd0599f20d6a424e61ead7aff0abffbfd04
+  metadata.gz: 250fdc0480188972b4735ad8a308cd16df5702a07f98de0d1ce024db61907fe5
+  data.tar.gz: 82f6efb47746aabab6045e47776709aa56de498e32265e635eb0e258c0514af1
 SHA512:
-  metadata.gz: e0e42653b97107a96cd99ef480633c455b5833d38b57b1e18942232ff46684c48a7d616a3953c9ba887bf5eef848fa17cdafcba912b3014618d9dc45775570b2
-  data.tar.gz: c7809f130d23a492dd1faec14b243a77b8d1618f796998fc2aad9c05fa6d3c681f175fc54444e59eed818c5fb0906596daa697260f3b4c29b4bde25b689f5257
+  metadata.gz: fb3abf429337da70c37605c7c37d13976da20d69c6b53226bd0eed943bf85a032c8f6c401cb0daa8a432e8b8394a239217b8f0c73d8beda15da648d1724f97e7
+  data.tar.gz: 95717ff391f8fbf0a5297141edef98a17cea5fcc051dbf0f1a05ce463c8ac7a17d2f89134dfbf48b6f451703ee9ca97042e30053e0ba8b4c6085580aadd83596

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.7.1 (2026-09-30)
+
+- New `ScrapeUnblocker::NoDataExtractedError` (a subclass of `ValidationError`): raised by `get_parsed` when the page rendered but no structured data could be extracted from it. The API answers 422 with `{"error": "no_data_extracted", "detail": ...}`; the error carries `detail`. The call is not billed and is never retried - use `get_page_source` for the HTML. Before, this came back as a billed 200 with empty `data`.
+- A dead URL fetched with `get_parsed` raises `TargetNotFoundError` with `html` nil (the body is the parsed-data JSON, on `body`).
+
+## 0.7.0 (2026-09-30)
+
+- New `ScrapeUnblocker::TargetNotFoundError` (a subclass of `NotFoundError`): raised by `get_page_source`, `get_parsed` and `get_page_with_cookies` when the target page itself answers 404 or 410. The API now passes the target's own status through instead of a 200, marked with the `X-Origin-Status` header. The error carries `origin_status`, the not-found page on `html` and `destination_url`. It is never retried, and the call is billed like any delivered page.
+- A custom `transport:` may return `headers:` alongside `status:` and `body:`; `ScrapeUnblocker.error_for_status` takes them as an optional third argument.
+
+Behaviour change: a dead target URL used to return its not-found page as a normal String; it now raises `TargetNotFoundError`. `rescue ScrapeUnblocker::NotFoundError` still catches it. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
+
 ## 0.6.0 (2026-09-24)
 
 - `google_images` takes `pages:` (1-5): fetch up to five Google Images result pages of ~100 results each in one call. Each page fetched is billed as one request; the response's `pagesFetched` says how many. The Google market now follows `proxy_country` automatically, so `gl` is only an optional override, and `max_results` is an optional cap up to 500.

data/README.md

@@ -277,16 +277,50 @@ end
 | `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
 | `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
-| `NotFoundError` | 404 | Page loaded but held no image (`get_image` only) |
+| `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
+| `TargetNotFoundError` | 404 / 410 | The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed) |
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
 | `UnsupportedContentError` | 415 | The URL serves something other than HTML |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
+| `NoDataExtractedError` | 422 | `get_parsed`: the page rendered but held no structured data; carries `detail` (subclass of `ValidationError`, not billed) |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
 | `ServerError` | 5xx | Unexpected server error, including a 504 upstream timeout |
 | `TimeoutError` | - | This client gave up locally before the API answered |
 | `ConnectionError` | - | Could not reach the API |
 
+### The target page does not exist (404 / 410)
+
+When the site you scrape answers 404 or 410 itself, the API passes that status through with an `X-Origin-Status` header, and the client raises `ScrapeUnblocker::TargetNotFoundError`. It is the target's final answer, so it is never retried, and it is billed like any delivered page. The not-found page is on `#html`:
+
+```ruby
+begin
+  html = su.get_page_source("https://example.com/removed-listing")
+rescue ScrapeUnblocker::TargetNotFoundError => e
+  puts e.origin_status   # 404 or 410
+  puts e.html            # the target's own not-found page (can be empty)
+end
+```
+
+`TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
+
+With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
+
+### No structured data on the page (422)
+
+When `get_parsed` renders the page but can extract no structured data from it, the API answers 422 and the client raises `ScrapeUnblocker::NoDataExtractedError`. The call is **not billed**, and retrying gives the same result - fetch the HTML with `get_page_source` instead:
+
+```ruby
+begin
+  page = su.get_parsed("https://example.com/some-page")
+rescue ScrapeUnblocker::NoDataExtractedError => e
+  puts e.detail          # the API's explanation
+  html = su.get_page_source("https://example.com/some-page")
+end
+```
+
+`NoDataExtractedError` subclasses `ValidationError`, so `rescue ScrapeUnblocker::ValidationError` catches it too.
+
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 
 ### Billing errors (402)

data/lib/scrapeunblocker/client.rb

@@ -57,6 +57,9 @@ module ScrapeUnblocker
     # +list_elements+, when true, makes the API return a JSON summary of the
     # matched elements ({"url", "count", "elements"}) instead of HTML. This method
     # then returns that parsed Hash rather than an HTML String.
+    #
+    # When the target page itself answers 404 or 410 this raises
+    # ScrapeUnblocker::TargetNotFoundError (billed; the not-found page is on #html).
     def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
                         steps: nil, list_elements: nil)
       body = request("/getPageSource",
@@ -70,6 +73,11 @@ module ScrapeUnblocker
     end
 
     # Fetch a URL and return structured JSON instead of HTML.
+    #
+    # Raises ScrapeUnblocker::NoDataExtractedError (not billed) when the page
+    # rendered but held no structured data - use #get_page_source for the HTML -
+    # and ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the
+    # target page itself answered 404 or 410.
     def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
       body = request("/getPageSource",
                      url: url, parsed_data: true, proxy_country: proxy_country,
@@ -284,7 +292,8 @@ module ScrapeUnblocker
 
         return { status: status, body: body } if status >= 200 && status < 300
 
-        raise ScrapeUnblocker.error_for_status(status, body)
+        response_headers = (result[:headers] || {}).to_h { |k, v| [k.to_s.downcase, Array(v).join(", ")] }
+        raise ScrapeUnblocker.error_for_status(status, body, response_headers)
       end
     end
 
@@ -315,7 +324,7 @@ module ScrapeUnblocker
         raise ConnectionError, "Could not reach the API: #{e.message}"
       end
 
-      { status: response.code.to_i, body: response.body }
+      { status: response.code.to_i, body: response.body, headers: response.each_header.to_h }
     end
   end
 end

data/lib/scrapeunblocker/errors.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "json"
+
 module ScrapeUnblocker
   # Base class for every error raised by this library.
   class Error < StandardError; end
@@ -78,10 +80,38 @@ module ScrapeUnblocker
   # use instead.
   class InvalidRequestError < APIError; end
 
-  # The page loaded but the requested element was absent (HTTP 404).
-  # Only #get_image raises this: the page rendered and held no <img> tag.
+  # Something the call asked for does not exist (HTTP 404).
+  #
+  # #get_image raises it when the page rendered but held no <img> tag, and
+  # plugin methods raise it when the item they look up does not exist. When the
+  # target page itself answered 404 or 410, the more specific
+  # TargetNotFoundError subclass is raised instead.
   class NotFoundError < APIError; end
 
+  # The target page itself does not exist (HTTP 404 or 410).
+  #
+  # Raised by #get_page_source, #get_parsed and #get_page_with_cookies when the
+  # site you asked for answered 404 or 410 on its own. The API passes that
+  # status through and marks it with the X-Origin-Status header, which is how
+  # this is told apart from an API-side 404. It is the target's final answer,
+  # not a block, so it is never retried - and the call is billed, because the
+  # page was fetched and delivered.
+  #
+  # +origin_status+ is the status the target answered with (404 or 410),
+  # +html+ the target's own not-found page as served (can be empty; nil when
+  # the body is a parsed-data JSON payload), and +destination_url+ the URL the
+  # target answered for, when the API sent X-Destination-URL.
+  class TargetNotFoundError < NotFoundError
+    attr_reader :origin_status, :html, :destination_url
+
+    def initialize(message, status_code:, origin_status:, body: nil, html: nil, destination_url: nil)
+      super(message, status_code: status_code, body: body)
+      @origin_status = origin_status
+      @html = html
+      @destination_url = destination_url
+    end
+  end
+
   # The browser run did not finish in time on our side (HTTP 408).
   #
   # Distinct from TimeoutError, which is this client giving up locally. Here
@@ -98,6 +128,22 @@ module ScrapeUnblocker
   # each problem field. Read it from #body.
   class ValidationError < APIError; end
 
+  # The page rendered but no structured data came out of it (HTTP 422).
+  #
+  # Raised by #get_parsed when the API loaded the page but could not extract any
+  # structured fields from it. The API answers 422 with a JSON body of
+  # {"error": "no_data_extracted", "detail": ...}; +detail+ holds the API's
+  # explanation. The call is not billed and retrying returns the same answer;
+  # call #get_page_source for the HTML.
+  class NoDataExtractedError < ValidationError
+    attr_reader :detail
+
+    def initialize(message, status_code:, body: nil, detail: nil)
+      super(message, status_code: status_code, body: body)
+      @detail = detail
+    end
+  end
+
   # The target site blocked every available bypass path (HTTP 403).
   # Blocked calls are not billed.
   class BlockedError < APIError; end
@@ -153,8 +199,58 @@ module ScrapeUnblocker
   end
   private_class_method :billing_error_class
 
-  # Build a typed error from an HTTP status code and response body.
-  def self.error_for_status(status, body)
+  # The API passes a target's "page does not exist" answer through with its
+  # status and an X-Origin-Status header. A 404 without that header is the
+  # API's own (a plugin lookup, a missing element) and returns nil so the
+  # general NotFoundError applies.
+  def self.target_not_found_error(status, body, headers)
+    origin = headers["x-origin-status"]
+    return nil unless [404, 410].include?(status) && origin && !origin.to_s.empty?
+
+    origin_status = Integer(origin.to_s, exception: false) || status
+    html = body
+    begin
+      data = JSON.parse(body.to_s)
+      html = data["html"].is_a?(String) ? data["html"] : nil if data.is_a?(Hash)
+    rescue JSON::ParserError
+      # Not JSON: the body is the target's own page.
+    end
+    message = "Target page does not exist (HTTP #{origin_status}). This is the " \
+              "target's own answer, not a block; the call is billed."
+    TargetNotFoundError.new(message, status_code: status, body: body, origin_status: origin_status,
+                                     html: html, destination_url: headers["x-destination-url"])
+  end
+  private_class_method :target_not_found_error
+
+  # parsed_data answers 422 with {"error": "no_data_extracted", "detail"} when
+  # the page rendered but held no structured data. Anything else returns nil so
+  # the general ValidationError applies.
+  def self.no_data_extracted_error(status, body)
+    return nil unless status == 422
+
+    data = begin
+      JSON.parse(body.to_s)
+    rescue JSON::ParserError
+      nil
+    end
+    return nil unless data.is_a?(Hash) && data["error"] == "no_data_extracted"
+
+    detail = data["detail"].is_a?(String) && !data["detail"].empty? ? data["detail"] : nil
+    message = detail || "The page was rendered, but no structured data could be extracted from it. Not billed."
+    message = "#{message} Not billed." unless message.downcase.include?("not billed")
+    NoDataExtractedError.new(message, status_code: status, body: body, detail: detail)
+  end
+  private_class_method :no_data_extracted_error
+
+  # Build a typed error from an HTTP status code, response body and headers
+  # (a Hash with lowercase names).
+  def self.error_for_status(status, body, headers = {})
+    target_error = target_not_found_error(status, body, headers || {})
+    return target_error if target_error
+
+    no_data_error = no_data_extracted_error(status, body)
+    return no_data_error if no_data_error
+
     snippet = (body || "").strip.gsub(/\s+/, " ")
     snippet = "#{snippet[0, 200]}..." if snippet.length > 200
     base = BASE_MESSAGES.fetch(status, "API returned HTTP #{status}")

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.6.0"
+  VERSION = "0.7.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.1
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-24 00:00:00.000000000 Z
+date: 2026-09-30 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.