scrapeunblocker 0.7.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 941309503bf1a937b76c503a811aab8e7e738e4ec775360ad7f9f6cfc40b6259
-  data.tar.gz: 68753623ec6ad6623632c520e688a5ba2f7697abf6588b8351cdd33e911f473c
+  metadata.gz: 250fdc0480188972b4735ad8a308cd16df5702a07f98de0d1ce024db61907fe5
+  data.tar.gz: 82f6efb47746aabab6045e47776709aa56de498e32265e635eb0e258c0514af1
 SHA512:
-  metadata.gz: 14f6f077622d3669a8ef06944d4cb810b89d693d28eead00b1b5789242945478e0e83529ede25aed9a474ee79351ca5202a3c91c03a174fd0cc811a031a48e11
-  data.tar.gz: 23314b94ae64e4ce3ade4bbef52278df0639681801981865e3faa8914fd71f6dec7f3047fbad96e4954dc70b8d603ef35487b5a3d5a9a65623be6e7de41071f7
+  metadata.gz: fb3abf429337da70c37605c7c37d13976da20d69c6b53226bd0eed943bf85a032c8f6c401cb0daa8a432e8b8394a239217b8f0c73d8beda15da648d1724f97e7
+  data.tar.gz: 95717ff391f8fbf0a5297141edef98a17cea5fcc051dbf0f1a05ce463c8ac7a17d2f89134dfbf48b6f451703ee9ca97042e30053e0ba8b4c6085580aadd83596

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.7.1 (2026-09-30)
+
+- New `ScrapeUnblocker::NoDataExtractedError` (a subclass of `ValidationError`): raised by `get_parsed` when the page rendered but no structured data could be extracted from it. The API answers 422 with `{"error": "no_data_extracted", "detail": ...}`; the error carries `detail`. The call is not billed and is never retried - use `get_page_source` for the HTML. Before, this came back as a billed 200 with empty `data`.
+- A dead URL fetched with `get_parsed` raises `TargetNotFoundError` with `html` nil (the body is the parsed-data JSON, on `body`).
+
 ## 0.7.0 (2026-09-30)
 
 - New `ScrapeUnblocker::TargetNotFoundError` (a subclass of `NotFoundError`): raised by `get_page_source`, `get_parsed` and `get_page_with_cookies` when the target page itself answers 404 or 410. The API now passes the target's own status through instead of a 200, marked with the `X-Origin-Status` header. The error carries `origin_status`, the not-found page on `html` and `destination_url`. It is never retried, and the call is billed like any delivered page.

data/README.md

@@ -282,6 +282,7 @@ end
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
 | `UnsupportedContentError` | 415 | The URL serves something other than HTML |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
+| `NoDataExtractedError` | 422 | `get_parsed`: the page rendered but held no structured data; carries `detail` (subclass of `ValidationError`, not billed) |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
 | `ServerError` | 5xx | Unexpected server error, including a 504 upstream timeout |
@@ -303,6 +304,23 @@ end
 
 `TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
 
+With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
+
+### No structured data on the page (422)
+
+When `get_parsed` renders the page but can extract no structured data from it, the API answers 422 and the client raises `ScrapeUnblocker::NoDataExtractedError`. The call is **not billed**, and retrying gives the same result - fetch the HTML with `get_page_source` instead:
+
+```ruby
+begin
+  page = su.get_parsed("https://example.com/some-page")
+rescue ScrapeUnblocker::NoDataExtractedError => e
+  puts e.detail          # the API's explanation
+  html = su.get_page_source("https://example.com/some-page")
+end
+```
+
+`NoDataExtractedError` subclasses `ValidationError`, so `rescue ScrapeUnblocker::ValidationError` catches it too.
+
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 
 ### Billing errors (402)

data/lib/scrapeunblocker/client.rb

@@ -73,6 +73,11 @@ module ScrapeUnblocker
     end
 
     # Fetch a URL and return structured JSON instead of HTML.
+    #
+    # Raises ScrapeUnblocker::NoDataExtractedError (not billed) when the page
+    # rendered but held no structured data - use #get_page_source for the HTML -
+    # and ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the
+    # target page itself answered 404 or 410.
     def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
       body = request("/getPageSource",
                      url: url, parsed_data: true, proxy_country: proxy_country,

data/lib/scrapeunblocker/errors.rb

@@ -128,6 +128,22 @@ module ScrapeUnblocker
   # each problem field. Read it from #body.
   class ValidationError < APIError; end
 
+  # The page rendered but no structured data came out of it (HTTP 422).
+  #
+  # Raised by #get_parsed when the API loaded the page but could not extract any
+  # structured fields from it. The API answers 422 with a JSON body of
+  # {"error": "no_data_extracted", "detail": ...}; +detail+ holds the API's
+  # explanation. The call is not billed and retrying returns the same answer;
+  # call #get_page_source for the HTML.
+  class NoDataExtractedError < ValidationError
+    attr_reader :detail
+
+    def initialize(message, status_code:, body: nil, detail: nil)
+      super(message, status_code: status_code, body: body)
+      @detail = detail
+    end
+  end
+
   # The target site blocked every available bypass path (HTTP 403).
   # Blocked calls are not billed.
   class BlockedError < APIError; end
@@ -206,12 +222,35 @@ module ScrapeUnblocker
   end
   private_class_method :target_not_found_error
 
+  # parsed_data answers 422 with {"error": "no_data_extracted", "detail"} when
+  # the page rendered but held no structured data. Anything else returns nil so
+  # the general ValidationError applies.
+  def self.no_data_extracted_error(status, body)
+    return nil unless status == 422
+
+    data = begin
+      JSON.parse(body.to_s)
+    rescue JSON::ParserError
+      nil
+    end
+    return nil unless data.is_a?(Hash) && data["error"] == "no_data_extracted"
+
+    detail = data["detail"].is_a?(String) && !data["detail"].empty? ? data["detail"] : nil
+    message = detail || "The page was rendered, but no structured data could be extracted from it. Not billed."
+    message = "#{message} Not billed." unless message.downcase.include?("not billed")
+    NoDataExtractedError.new(message, status_code: status, body: body, detail: detail)
+  end
+  private_class_method :no_data_extracted_error
+
   # Build a typed error from an HTTP status code, response body and headers
   # (a Hash with lowercase names).
   def self.error_for_status(status, body, headers = {})
     target_error = target_not_found_error(status, body, headers || {})
     return target_error if target_error
 
+    no_data_error = no_data_extracted_error(status, body)
+    return no_data_error if no_data_error
+
     snippet = (body || "").strip.gsub(/\s+/, " ")
     snippet = "#{snippet[0, 200]}..." if snippet.length > 200
     base = BASE_MESSAGES.fetch(status, "API returned HTTP #{status}")

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.7.0"
+  VERSION = "0.7.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.7.1
 platform: ruby
 authors:
 - ScrapeUnblocker