scrapeunblocker 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 941309503bf1a937b76c503a811aab8e7e738e4ec775360ad7f9f6cfc40b6259
-  data.tar.gz: 68753623ec6ad6623632c520e688a5ba2f7697abf6588b8351cdd33e911f473c
+  metadata.gz: e8096ca55e9f93c01077d947e60e36ec71c595e7ee92d1405a413a0746c303f0
+  data.tar.gz: 28c0667c78459edf026b26effd0209ea0635409eb078f152b8e4963be2312dd3
 SHA512:
-  metadata.gz: 14f6f077622d3669a8ef06944d4cb810b89d693d28eead00b1b5789242945478e0e83529ede25aed9a474ee79351ca5202a3c91c03a174fd0cc811a031a48e11
-  data.tar.gz: 23314b94ae64e4ce3ade4bbef52278df0639681801981865e3faa8914fd71f6dec7f3047fbad96e4954dc70b8d603ef35487b5a3d5a9a65623be6e7de41071f7
+  metadata.gz: f1b51dee641104d7d77fd600c5cb2299f30d6dcdf3e155686befc2d3024c981fb0c904d2aa8d5d1f360797a9cdbd331356ec2c99f1a3832907704198689306c6
+  data.tar.gz: 8274485fa1644d6dfb1512bc33e93fc034f3c01a0248e450869315f3d0669244761ef146db9e11cac0ccfb3fe6361148d5b3ef64a4c706d967d825ef23fba9e6

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.8.0 (2026-10-06)
+
+- `get_parsed` on a page that rendered but held no structured data now returns a normal `ParsedPage` instead of raising `NoDataExtractedError`. The API answers this case with a billed 200: `data` is empty, the new `data_extracted?` is `false`, the new `detail` explains it and the new `html` carries the rendered page, so no second call is needed. On a successful parse `data_extracted?` is `true` and `html` / `detail` are `nil`.
+- `NoDataExtractedError` is deprecated: the API no longer sends the 422 `no_data_extracted`. The class stays so existing `rescue` clauses keep loading.
+
+Behaviour change: code that rescued `NoDataExtractedError` should check `page.data_extracted?` instead.
+
+## 0.7.1 (2026-09-30)
+
+- New `ScrapeUnblocker::NoDataExtractedError` (a subclass of `ValidationError`): raised by `get_parsed` when the page rendered but no structured data could be extracted from it. The API answers 422 with `{"error": "no_data_extracted", "detail": ...}`; the error carries `detail`. The call is not billed and is never retried - use `get_page_source` for the HTML. Before, this came back as a billed 200 with empty `data`.
+- A dead URL fetched with `get_parsed` raises `TargetNotFoundError` with `html` nil (the body is the parsed-data JSON, on `body`).
+
 ## 0.7.0 (2026-09-30)
 
 - New `ScrapeUnblocker::TargetNotFoundError` (a subclass of `NotFoundError`): raised by `get_page_source`, `get_parsed` and `get_page_with_cookies` when the target page itself answers 404 or 410. The API now passes the target's own status through instead of a 200, marked with the `X-Origin-Status` header. The error carries `origin_status`, the not-found page on `html` and `destination_url`. It is never retried, and the call is billed like any delivered page.

data/README.md

@@ -303,6 +303,22 @@ end
 
 `TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
 
+With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
+
+### No structured data on the page
+
+When `get_parsed` renders the page but can extract no structured data from it, the API still answers 200 and the client returns a normal `ParsedPage`: `data` is empty, `data_extracted?` is `false`, `detail` says why, and `html` carries the rendered page. The call is billed like any delivered page, so you already have the HTML - no second call needed:
+
+```ruby
+page = su.get_parsed("https://example.com/some-page")
+unless page.data_extracted?
+  puts page.detail       # the API's explanation
+  html = page.html       # the rendered page, parse it yourself
+end
+```
+
+On a successful parse `data_extracted?` is `true` and `html` is `nil`. Up to 0.7.1 this case raised `NoDataExtractedError` (a 422); the API no longer sends that, and the class stays only so existing `rescue` clauses keep loading.
+
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 
 ### Billing errors (402)

data/lib/scrapeunblocker/client.rb

@@ -73,6 +73,12 @@ module ScrapeUnblocker
     end
 
     # Fetch a URL and return structured JSON instead of HTML.
+    #
+    # A page that rendered but held no structured data comes back as a
+    # ParsedPage with #data_extracted? false and the rendered page on #html
+    # (billed like any delivered page). Raises
+    # ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the target
+    # page itself answered 404 or 410.
     def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
       body = request("/getPageSource",
                      url: url, parsed_data: true, proxy_country: proxy_country,

data/lib/scrapeunblocker/errors.rb

@@ -128,6 +128,22 @@ module ScrapeUnblocker
   # each problem field. Read it from #body.
   class ValidationError < APIError; end
 
+  # Deprecated: the API no longer sends this 422.
+  #
+  # A page that rendered but held no structured data now comes back from
+  # #get_parsed as a normal ParsedPage with #data_extracted? false and the page
+  # on #html (a billed 200). The class stays so existing rescue clauses keep
+  # loading; it is raised only for a legacy 422 body of
+  # {"error": "no_data_extracted", "detail": ...}.
+  class NoDataExtractedError < ValidationError
+    attr_reader :detail
+
+    def initialize(message, status_code:, body: nil, detail: nil)
+      super(message, status_code: status_code, body: body)
+      @detail = detail
+    end
+  end
+
   # The target site blocked every available bypass path (HTTP 403).
   # Blocked calls are not billed.
   class BlockedError < APIError; end
@@ -206,12 +222,35 @@ module ScrapeUnblocker
   end
   private_class_method :target_not_found_error
 
+  # Maps a legacy 422 {"error": "no_data_extracted", "detail"} body; the API now
+  # answers an empty parse with a 200 (data_extracted: false). Anything else
+  # returns nil so the general ValidationError applies.
+  def self.no_data_extracted_error(status, body)
+    return nil unless status == 422
+
+    data = begin
+      JSON.parse(body.to_s)
+    rescue JSON::ParserError
+      nil
+    end
+    return nil unless data.is_a?(Hash) && data["error"] == "no_data_extracted"
+
+    detail = data["detail"].is_a?(String) && !data["detail"].empty? ? data["detail"] : nil
+    message = detail || "The page was rendered, but no structured data could be extracted from it. Not billed."
+    message = "#{message} Not billed." unless message.downcase.include?("not billed")
+    NoDataExtractedError.new(message, status_code: status, body: body, detail: detail)
+  end
+  private_class_method :no_data_extracted_error
+
   # Build a typed error from an HTTP status code, response body and headers
   # (a Hash with lowercase names).
   def self.error_for_status(status, body, headers = {})
     target_error = target_not_found_error(status, body, headers || {})
     return target_error if target_error
 
+    no_data_error = no_data_extracted_error(status, body)
+    return no_data_error if no_data_error
+
     snippet = (body || "").strip.gsub(/\s+/, " ")
     snippet = "#{snippet[0, 200]}..." if snippet.length > 200
     base = BASE_MESSAGES.fetch(status, "API returned HTTP #{status}")

data/lib/scrapeunblocker/parsed_page.rb

@@ -11,12 +11,25 @@ module ScrapeUnblocker
     attr_reader :data
     # @return [Hash] the full JSON payload as returned by the API
     attr_reader :raw
+    # @return [String, nil] the rendered page when #data_extracted? is false
+    attr_reader :html
+    # @return [String, nil] the API's explanation when #data_extracted? is false
+    attr_reader :detail
 
-    def initialize(page_type:, source:, data:, raw:)
+    def initialize(page_type:, source:, data:, raw:, data_extracted: true, html: nil, detail: nil)
       @page_type = page_type
       @source = source
       @data = data
       @raw = raw
+      @data_extracted = data_extracted
+      @html = html
+      @detail = detail
+    end
+
+    # @return [Boolean] false when the page rendered but no structured data
+    #   could be extracted from it (#data is then empty, #html holds the page)
+    def data_extracted?
+      @data_extracted
     end
 
     def self.from_hash(payload)
@@ -25,7 +38,10 @@ module ScrapeUnblocker
         page_type: inner["page_type"],
         source: inner["source"],
         data: inner["data"],
-        raw: payload
+        raw: payload,
+        data_extracted: payload["data_extracted"] != false,
+        html: payload["html"],
+        detail: payload["detail"]
       )
     end
   end

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.7.0"
+  VERSION = "0.8.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-30 00:00:00.000000000 Z
+date: 2026-10-06 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.