scrapeunblocker 0.7.1 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 250fdc0480188972b4735ad8a308cd16df5702a07f98de0d1ce024db61907fe5
-  data.tar.gz: 82f6efb47746aabab6045e47776709aa56de498e32265e635eb0e258c0514af1
+  metadata.gz: f20529c692d1de01f3f02acca1accaf677cd5df2ec9d075bfb949b4b230272d2
+  data.tar.gz: b0eff5160bbd56e6e45700609ab17966d061d74537909c7a1a06df9242cddb7a
 SHA512:
-  metadata.gz: fb3abf429337da70c37605c7c37d13976da20d69c6b53226bd0eed943bf85a032c8f6c401cb0daa8a432e8b8394a239217b8f0c73d8beda15da648d1724f97e7
-  data.tar.gz: 95717ff391f8fbf0a5297141edef98a17cea5fcc051dbf0f1a05ce463c8ac7a17d2f89134dfbf48b6f451703ee9ca97042e30053e0ba8b4c6085580aadd83596
+  metadata.gz: 2421c59b2bc7c32d769d01546b4f796b7f71237edf500af4ee21b56f0a1393532b6361cc4207264c6072c5510bd31dc840986adda94dfbc18e7d2fdbf03a1df8
+  data.tar.gz: 98532dd9a066ef4e52f1123a958e36f1f40df7f603a3055b78784337bedfb256a237343d011e086ba8d0feb94cd17100e2501d5317055b35bae15fc57bea27ea

data/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.9.0 (2026-10-07)
+
+- New `ScrapeUnblocker::BudgetExceededError` (a subclass of `PaymentRequiredError`): raised when the API answers 402 with `User set budget exceeded`, meaning this billing period's spend reached the monthly budget limit (EUR, excluding VAT) set in the [profile](https://app.scrapeunblocker.com/dashboard/profile). It clears at the start of the next billing period, or within about a minute after the limit is raised or removed. Like the other 402s it is never retried and not billed. Before, this body surfaced as a plain `PaymentRequiredError`.
+
+No breaking changes: `rescue ScrapeUnblocker::PaymentRequiredError` still catches it.
+
+## 0.8.0 (2026-10-06)
+
+- `get_parsed` on a page that rendered but held no structured data now returns a normal `ParsedPage` instead of raising `NoDataExtractedError`. The API answers this case with a billed 200: `data` is empty, the new `data_extracted?` is `false`, the new `detail` explains it and the new `html` carries the rendered page, so no second call is needed. On a successful parse `data_extracted?` is `true` and `html` / `detail` are `nil`.
+- `NoDataExtractedError` is deprecated: the API no longer sends the 422 `no_data_extracted`. The class stays so existing `rescue` clauses keep loading.
+
+Behaviour change: code that rescued `NoDataExtractedError` should check `page.data_extracted?` instead.
+
 ## 0.7.1 (2026-09-30)
 
 - New `ScrapeUnblocker::NoDataExtractedError` (a subclass of `ValidationError`): raised by `get_parsed` when the page rendered but no structured data could be extracted from it. The API answers 422 with `{"error": "no_data_extracted", "detail": ...}`; the error carries `detail`. The call is not billed and is never retried - use `get_page_source` for the HTML. Before, this came back as a billed 200 with empty `data`.

data/README.md

@@ -108,7 +108,6 @@ result["elements"].each { |el| p el }
 ```ruby
 result = su.get_parsed("https://www.walmart.com/ip/12345")
 puts result.page_type # e.g. "product"
-puts result.source    # how it was extracted
 p result.data         # the fields
 
 # If a parse ever comes back wrong, force a fresh set of rules:
@@ -259,7 +258,7 @@ begin
 rescue ScrapeUnblocker::BlockedError
   # 403: the target blocked every bypass path (not billed)
 rescue ScrapeUnblocker::PaymentRequiredError
-  # 402: quota, credit limit, or a failed payment - fix billing
+  # 402: quota, credit limit, your budget limit, or a failed payment - fix billing
 rescue ScrapeUnblocker::RateLimitError
   # 429: slow down
 rescue ScrapeUnblocker::UpstreamOutageError
@@ -272,9 +271,10 @@ end
 | `InvalidRequestError` | 400 | Bad URL, unsupported scheme, or the API key header was not sent |
 | `AuthenticationError` | 401 | Key not recognised - typo, stray whitespace, or a rotated key |
 | `NoSubscriptionError` | 401 | Key is fine, but the account has no active plan |
-| `PaymentRequiredError` | 402 | Billing block - base class for the three below |
+| `PaymentRequiredError` | 402 | Billing block - base class for the four below |
 | `QuotaExceededError` | 402 | The plan's requests for this period are used up |
 | `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
+| `BudgetExceededError` | 402 | This billing period's spend reached the monthly budget limit you set |
 | `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
 | `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
@@ -282,7 +282,6 @@ end
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
 | `UnsupportedContentError` | 415 | The URL serves something other than HTML |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
-| `NoDataExtractedError` | 422 | `get_parsed`: the page rendered but held no structured data; carries `detail` (subclass of `ValidationError`, not billed) |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
 | `ServerError` | 5xx | Unexpected server error, including a 504 upstream timeout |
@@ -306,26 +305,25 @@ end
 
 With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
 
-### No structured data on the page (422)
+### No structured data on the page
 
-When `get_parsed` renders the page but can extract no structured data from it, the API answers 422 and the client raises `ScrapeUnblocker::NoDataExtractedError`. The call is **not billed**, and retrying gives the same result - fetch the HTML with `get_page_source` instead:
+When `get_parsed` renders the page but can extract no structured data from it, the API still answers 200 and the client returns a normal `ParsedPage`: `data` is empty, `data_extracted?` is `false`, `detail` says why, and `html` carries the rendered page. The call is billed like any delivered page, so you already have the HTML - no second call needed:
 
 ```ruby
-begin
-  page = su.get_parsed("https://example.com/some-page")
-rescue ScrapeUnblocker::NoDataExtractedError => e
-  puts e.detail          # the API's explanation
-  html = su.get_page_source("https://example.com/some-page")
+page = su.get_parsed("https://example.com/some-page")
+unless page.data_extracted?
+  puts page.detail       # the API's explanation
+  html = page.html       # the rendered page, parse it yourself
 end
 ```
 
-`NoDataExtractedError` subclasses `ValidationError`, so `rescue ScrapeUnblocker::ValidationError` catches it too.
+On a successful parse `data_extracted?` is `true` and `html` is `nil`. Up to 0.7.1 this case raised `NoDataExtractedError` (a 422); the API no longer sends that, and the class stays only so existing `rescue` clauses keep loading.
 
 Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
 
 ### Billing errors (402)
 
-The three billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
+The four billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
 
 ```ruby
 begin
@@ -334,12 +332,16 @@ rescue ScrapeUnblocker::QuotaExceededError
   # plan quota (plus any overage allowance) is used up for this period
 rescue ScrapeUnblocker::CreditLimitExceededError
   # unpaid balance passed the account credit limit
+rescue ScrapeUnblocker::BudgetExceededError
+  # this period's spend reached the monthly budget limit set in your profile
 rescue ScrapeUnblocker::PaymentFailedError
   # card declined three times - update the payment method
 end
 ```
 
-When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota. All three lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
+When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota, which outranks your own budget limit. All four lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
+
+`BudgetExceededError` (body `User set budget exceeded`) means this billing period's spend reached the monthly budget limit you set in your [profile](https://app.scrapeunblocker.com/dashboard/profile?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk) (EUR, excluding VAT). The key works again at the start of the next billing period, or within about a minute after you raise or remove the limit.
 
 Full details for every status code: [docs.scrapeunblocker.com/errors](https://docs.scrapeunblocker.com/errors).
 

data/lib/scrapeunblocker/client.rb

@@ -74,10 +74,11 @@ module ScrapeUnblocker
 
     # Fetch a URL and return structured JSON instead of HTML.
     #
-    # Raises ScrapeUnblocker::NoDataExtractedError (not billed) when the page
-    # rendered but held no structured data - use #get_page_source for the HTML -
-    # and ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the
-    # target page itself answered 404 or 410.
+    # A page that rendered but held no structured data comes back as a
+    # ParsedPage with #data_extracted? false and the rendered page on #html
+    # (billed like any delivered page). Raises
+    # ScrapeUnblocker::TargetNotFoundError (billed, #html nil) when the target
+    # page itself answered 404 or 410.
     def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
       body = request("/getPageSource",
                      url: url, parsed_data: true, proxy_country: proxy_country,

data/lib/scrapeunblocker/errors.rb

@@ -36,15 +36,16 @@ module ScrapeUnblocker
   # The account has a billing problem (HTTP 402).
   #
   # Credentials are fine - the request was stopped for a billing reason. There
-  # are three, each raised as a dedicated subclass: QuotaExceededError,
-  # CreditLimitExceededError and PaymentFailedError. Rescue this base class to
-  # handle all three.
+  # are four, each raised as a dedicated subclass: QuotaExceededError,
+  # CreditLimitExceededError, BudgetExceededError and PaymentFailedError.
+  # Rescue this base class to handle all four.
   #
   # When more than one applies, the most serious wins: failed payment outranks
-  # credit limit, which outranks quota. All three lift by themselves once the
-  # billing state changes - access returns within roughly a minute, with no key
-  # change needed. Like a 401, a 402 is refused before anything is scraped, so
-  # it is never billed. Retrying is pointless; fix the billing state first.
+  # credit limit, which outranks quota, which outranks your own budget limit.
+  # All four lift by themselves once the billing state changes - access returns
+  # within roughly a minute, with no key change needed. Like a 401, a 402 is
+  # refused before anything is scraped, so it is never billed. Retrying is
+  # pointless; fix the billing state first.
   class PaymentRequiredError < APIError; end
 
   # Every request the plan allows this period has been used (HTTP 402).
@@ -64,6 +65,17 @@ module ScrapeUnblocker
   # usually clears itself within about a minute.
   class CreditLimitExceededError < PaymentRequiredError; end
 
+  # This billing period's spend reached the monthly budget limit you set
+  # (HTTP 402).
+  #
+  # The limit is set in your profile (EUR, excluding VAT), and spend is counted
+  # the way the invoice is: the plan's fixed monthly fee, if any, plus the
+  # requests billed on top of it. Requests paid from coupon credit do not count.
+  # The key works again at the start of the next billing period, or within
+  # about a minute after you raise or remove the limit at
+  # https://app.scrapeunblocker.com/dashboard/profile.
+  class BudgetExceededError < PaymentRequiredError; end
+
   # A card payment has been declined three times (HTTP 402).
   #
   # Those attempts are the payment provider's automatic retries spread over
@@ -128,13 +140,13 @@ module ScrapeUnblocker
   # each problem field. Read it from #body.
   class ValidationError < APIError; end
 
-  # The page rendered but no structured data came out of it (HTTP 422).
+  # Deprecated: the API no longer sends this 422.
   #
-  # Raised by #get_parsed when the API loaded the page but could not extract any
-  # structured fields from it. The API answers 422 with a JSON body of
-  # {"error": "no_data_extracted", "detail": ...}; +detail+ holds the API's
-  # explanation. The call is not billed and retrying returns the same answer;
-  # call #get_page_source for the HTML.
+  # A page that rendered but held no structured data now comes back from
+  # #get_parsed as a normal ParsedPage with #data_extracted? false and the page
+  # on #html (a billed 200). The class stays so existing rescue clauses keep
+  # loading; it is raised only for a legacy 422 body of
+  # {"error": "no_data_extracted", "detail": ...}.
   class NoDataExtractedError < ValidationError
     attr_reader :detail
 
@@ -167,7 +179,7 @@ module ScrapeUnblocker
   BASE_MESSAGES = {
     400 => "Invalid request (bad URL, unsupported scheme, or missing API key header)",
     401 => "Authentication failed - key not recognised, or account has no active plan",
-    402 => "Billing block - quota exceeded, credit limit exceeded, or a failed payment",
+    402 => "Billing block - quota exceeded, credit limit exceeded, budget limit reached, or a failed payment",
     403 => "Target blocked by bot protection on every bypass path",
     404 => "Requested element not found on the page",
     408 => "Browser run timed out before the page was ready",
@@ -187,12 +199,13 @@ module ScrapeUnblocker
   end
   private_class_method :auth_error_class
 
-  # The three billing blocks share a status code and differ only in their
+  # The four billing blocks share a status code and differ only in their
   # plain-text body. An unrecognised body falls back to PaymentRequiredError.
   def self.billing_error_class(body)
     text = (body || "").downcase
     return QuotaExceededError if text.include?("quota exceeded")
     return CreditLimitExceededError if text.include?("credit limit exceeded")
+    return BudgetExceededError if text.include?("user set budget exceeded")
     return PaymentFailedError if text.include?("payment failed")
 
     PaymentRequiredError
@@ -222,9 +235,9 @@ module ScrapeUnblocker
   end
   private_class_method :target_not_found_error
 
-  # parsed_data answers 422 with {"error": "no_data_extracted", "detail"} when
-  # the page rendered but held no structured data. Anything else returns nil so
-  # the general ValidationError applies.
+  # Maps a legacy 422 {"error": "no_data_extracted", "detail"} body; the API now
+  # answers an empty parse with a 200 (data_extracted: false). Anything else
+  # returns nil so the general ValidationError applies.
   def self.no_data_extracted_error(status, body)
     return nil unless status == 422
 

data/lib/scrapeunblocker/parsed_page.rb

@@ -11,12 +11,25 @@ module ScrapeUnblocker
     attr_reader :data
     # @return [Hash] the full JSON payload as returned by the API
     attr_reader :raw
+    # @return [String, nil] the rendered page when #data_extracted? is false
+    attr_reader :html
+    # @return [String, nil] the API's explanation when #data_extracted? is false
+    attr_reader :detail
 
-    def initialize(page_type:, source:, data:, raw:)
+    def initialize(page_type:, source:, data:, raw:, data_extracted: true, html: nil, detail: nil)
       @page_type = page_type
       @source = source
       @data = data
       @raw = raw
+      @data_extracted = data_extracted
+      @html = html
+      @detail = detail
+    end
+
+    # @return [Boolean] false when the page rendered but no structured data
+    #   could be extracted from it (#data is then empty, #html holds the page)
+    def data_extracted?
+      @data_extracted
     end
 
     def self.from_hash(payload)
@@ -25,7 +38,10 @@ module ScrapeUnblocker
         page_type: inner["page_type"],
         source: inner["source"],
         data: inner["data"],
-        raw: payload
+        raw: payload,
+        data_extracted: payload["data_extracted"] != false,
+        html: payload["html"],
+        detail: payload["detail"]
       )
     end
   end

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.7.1"
+  VERSION = "0.9.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.7.1
+  version: 0.9.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-30 00:00:00.000000000 Z
+date: 2026-10-07 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.