scrapeunblocker 0.7.1 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 250fdc0480188972b4735ad8a308cd16df5702a07f98de0d1ce024db61907fe5
-  data.tar.gz: 82f6efb47746aabab6045e47776709aa56de498e32265e635eb0e258c0514af1
+  metadata.gz: e8096ca55e9f93c01077d947e60e36ec71c595e7ee92d1405a413a0746c303f0
+  data.tar.gz: 28c0667c78459edf026b26effd0209ea0635409eb078f152b8e4963be2312dd3
 SHA512:
-  metadata.gz: fb3abf429337da70c37605c7c37d13976da20d69c6b53226bd0eed943bf85a032c8f6c401cb0daa8a432e8b8394a239217b8f0c73d8beda15da648d1724f97e7
-  data.tar.gz: 95717ff391f8fbf0a5297141edef98a17cea5fcc051dbf0f1a05ce463c8ac7a17d2f89134dfbf48b6f451703ee9ca97042e30053e0ba8b4c6085580aadd83596
+  metadata.gz: f1b51dee641104d7d77fd600c5cb2299f30d6dcdf3e155686befc2d3024c981fb0c904d2aa8d5d1f360797a9cdbd331356ec2c99f1a3832907704198689306c6
+  data.tar.gz: 8274485fa1644d6dfb1512bc33e93fc034f3c01a0248e450869315f3d0669244761ef146db9e11cac0ccfb3fe6361148d5b3ef64a4c706d967d825ef23fba9e6

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.8.0 (2026-10-06)
+
+- `get_parsed` on a page that rendered but held no structured data now returns a normal `ParsedPage` instead of raising `NoDataExtractedError`. The API answers this case with a billed 200: `data` is empty, the new `data_extracted?` is `false`, the new `detail` explains it and the new `html` carries the rendered page, so no second call is needed. On a successful parse `data_extracted?` is `true` and `html` / `detail` are `nil`.
+- `NoDataExtractedError` is deprecated: the API no longer sends the 422 `no_data_extracted`. The class stays so existing `rescue` clauses keep loading.
+
+Behaviour change: code that rescued `NoDataExtractedError` should check `page.data_extracted?` instead.
+
 ## 0.7.1 (2026-09-30)
 
 - New `ScrapeUnblocker::NoDataExtractedError` (a subclass of `ValidationError`): raised by `get_parsed` when the page rendered but no structured data could be extracted from it. The API answers 422 with `{"error": "no_data_extracted", "detail": ...}`; the error carries `detail`. The call is not billed and is never retried - use `get_page_source` for the HTML. Before, this came back as a billed 200 with empty `data`.

data/README.md

@@ -282,7 +282,6 @@ end
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
 | `UnsupportedContentError` | 415 | The URL serves something other than HTML |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
-| `NoDataExtractedError` | 422 | `get_parsed`: the page rendered but held no structured data; carries `detail` (subclass of `ValidationError`, not billed) |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
 | `ServerError` | 5xx | Unexpected server error, including a 504 upstream timeout |
@@ -306,20 +305,19 @@ end
 
 With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
 
-### No structured data on the page (422)
+### No structured data on the page
 
-When `get_parsed` renders the page but can extract no structured data from it, the API answers 422 and the client raises `ScrapeUnblocker::NoDataExtractedError`. The call is **not billed**, and retrying gives the same result - fetch the HTML with `get_page_source` instead:
+When `get_parsed` renders the page but can extract no structured data from it, the API still answers 200 and the client returns a normal `ParsedPage`: `data` is empty, `data_extracted?` is `false`, `detail` says why, and `html` carries the rendered page. The call is billed like any delivered page, so you already have the HTML - no second call needed:
 
 ```ruby
-begin
-  page = su.get_parsed("https://example.com/some-page")
-rescue ScrapeUnblocker::NoDataExtractedError => e
-  puts e.detail          # the API's explanation
-  html = su.get_page_source("https://example.com/some-page")
+page = su.get_parsed("https://example.com/some-page")
+unless page.data_extracted?
+  puts page.detail       # the API's explanation
+  html = page.html       # the rendered page, parse it yourself
 end
 ```
 
-`NoDataExtractedError` subclasses `ValidationError`, so `rescue ScrapeUnblocker::ValidationError` catches it too.
+On a successful parse `data_extracted?` is `true` and `html` is `nil`. Up to 0.7.1 this case raised `NoDataExtractedError` (a 422); the API no longer sends that, and the class stays only so existing `rescue` clauses keep loading.
 
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 

data/lib/scrapeunblocker/client.rb

@@ -74,10 +74,11 @@ module ScrapeUnblocker
 
     # Fetch a URL and return structured JSON instead of HTML.
     #
-    # Raises ScrapeUnblocker::NoDataExtractedError (not billed) when the page
-    # rendered but held no structured data - use #get_page_source for the HTML -
-    # and ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the
-    # target page itself answered 404 or 410.
+    # A page that rendered but held no structured data comes back as a
+    # ParsedPage with #data_extracted? false and the rendered page on #html
+    # (billed like any delivered page). Raises
+    # ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the target
+    # page itself answered 404 or 410.
     def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
       body = request("/getPageSource",
                      url: url, parsed_data: true, proxy_country: proxy_country,

data/lib/scrapeunblocker/errors.rb

@@ -128,13 +128,13 @@ module ScrapeUnblocker
   # each problem field. Read it from #body.
   class ValidationError < APIError; end
 
-  # The page rendered but no structured data came out of it (HTTP 422).
+  # Deprecated: the API no longer sends this 422.
   #
-  # Raised by #get_parsed when the API loaded the page but could not extract any
-  # structured fields from it. The API answers 422 with a JSON body of
-  # {"error": "no_data_extracted", "detail": ...}; +detail+ holds the API's
-  # explanation. The call is not billed and retrying returns the same answer;
-  # call #get_page_source for the HTML.
+  # A page that rendered but held no structured data now comes back from
+  # #get_parsed as a normal ParsedPage with #data_extracted? false and the page
+  # on #html (a billed 200). The class stays so existing rescue clauses keep
+  # loading; it is raised only for a legacy 422 body of
+  # {"error": "no_data_extracted", "detail": ...}.
   class NoDataExtractedError < ValidationError
     attr_reader :detail
 
@@ -222,9 +222,9 @@ module ScrapeUnblocker
   end
   private_class_method :target_not_found_error
 
-  # parsed_data answers 422 with {"error": "no_data_extracted", "detail"} when
-  # the page rendered but held no structured data. Anything else returns nil so
-  # the general ValidationError applies.
+  # Maps a legacy 422 {"error": "no_data_extracted", "detail"} body; the API now
+  # answers an empty parse with a 200 (data_extracted: false). Anything else
+  # returns nil so the general ValidationError applies.
   def self.no_data_extracted_error(status, body)
     return nil unless status == 422
 

data/lib/scrapeunblocker/parsed_page.rb

@@ -11,12 +11,25 @@ module ScrapeUnblocker
     attr_reader :data
     # @return [Hash] the full JSON payload as returned by the API
     attr_reader :raw
+    # @return [String, nil] the rendered page when #data_extracted? is false
+    attr_reader :html
+    # @return [String, nil] the API's explanation when #data_extracted? is false
+    attr_reader :detail
 
-    def initialize(page_type:, source:, data:, raw:)
+    def initialize(page_type:, source:, data:, raw:, data_extracted: true, html: nil, detail: nil)
       @page_type = page_type
       @source = source
       @data = data
       @raw = raw
+      @data_extracted = data_extracted
+      @html = html
+      @detail = detail
+    end
+
+    # @return [Boolean] false when the page rendered but no structured data
+    #   could be extracted from it (#data is then empty, #html holds the page)
+    def data_extracted?
+      @data_extracted
     end
 
     def self.from_hash(payload)
@@ -25,7 +38,10 @@ module ScrapeUnblocker
         page_type: inner["page_type"],
         source: inner["source"],
         data: inner["data"],
-        raw: payload
+        raw: payload,
+        data_extracted: payload["data_extracted"] != false,
+        html: payload["html"],
+        detail: payload["detail"]
       )
     end
   end

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.7.1"
+  VERSION = "0.8.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.7.1
+  version: 0.8.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-30 00:00:00.000000000 Z
+date: 2026-10-06 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.