scrapeunblocker 0.1.2 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4d9a2366dce1f1f9586ccb5d9d0f5b9c667a1a9573f811093f017984118d138a
-  data.tar.gz: 527960e09c0159162825fe7ef7a75dcb071bafaf0872f847bcd25d6fd65df813
+  metadata.gz: ab6bd524121d205b90ef9cabe0315d1185772ee313a6e40d1bf9057dad3b535b
+  data.tar.gz: 82bc7674e38f9bfa125320956e440a66ad535d3aac8ac923e086d0b0ec3fcba2
 SHA512:
-  metadata.gz: e71d1368c7be85c86736955c25f39043457ae20e733d7ebee33416717981588eb8f35f82a16c10c9dd18b2d961f920121b984417e3dfb42aa450ecab5b2ace77
-  data.tar.gz: 58cb22b93d148eb3cdc342cf184268c4826f2567380909bb964cb8ebb37a26644995df0dcbcb183e8d1266d5f50fd1f0bf21ff075ddc69c3ef6b20ed5f7da39c
+  metadata.gz: 7d8ecc218ec1814f6cdca5681e156488b43302a935a7e8c8fe3d2d73cd98f45558c979c555c33c03db0a43e5a8cf440d2edc8d3b259854844ee7cf9083399fa3
+  data.tar.gz: ad8a15c080c559e9d6d5927351e8336788266a65f07971c0c49c41e69ffbee98b8a3d2a661c62d325365d9abdd716b08241a3afbe1a8a2f47234da48d162894e

data/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## 0.1.7 (2026-07-27)
+
+- Registry and README links to scrapeunblocker.com now carry UTM parameters so traffic from package registries is attributable. No functional changes.
+
+## 0.1.6 (2026-07-23)
+
+Version jumps from 0.1.2 to 0.1.6 so all four official SDKs (Python, Node.js, Ruby, PHP) share one version number from here on. Nothing was skipped - 0.1.3 to 0.1.5 were never released for Ruby.
+
+- Added `PaymentRequiredError` for HTTP 402, which previously surfaced as a bare `APIError` with no explanation. The three billing blocks now each get their own subclass, picked from the response body: `QuotaExceededError` (`Quota exceeded`), `CreditLimitExceededError` (`Credit limit exceeded`) and `PaymentFailedError` (`Payment failed - update payment method`). Rescue `PaymentRequiredError` to handle all three.
+- Added `NoSubscriptionError`, a subclass of `AuthenticationError`, for the 401 that means "the key is fine, the account has no active plan" (`No valid subscription`) as opposed to an unrecognised key.
+- Added typed errors for the remaining documented status codes: `NotFoundError` (404), `BrowserTimeoutError` (408), `UnsupportedContentError` (415) and `ValidationError` (422). All previously raised a bare `APIError`.
+- Error messages now describe every documented status code accurately - notably 400, which also covers a missing `x-scrapeunblocker-key` header, not just a bad URL.
+- Documented the full error hierarchy in the README, including which errors are retried, which are billed, and how each 402 clears.
+- Fixed the README claim that Oopbuy brand keywords return HTTP 422. They return a successful `200` with `keywordRejected: true` and an empty `results` array.
+
+No breaking changes: every new class inherits from `APIError`, so existing `rescue ScrapeUnblocker::APIError` / `rescue ScrapeUnblocker::Error` handlers keep working unchanged.
+
 ## 0.1.2 (2026-07-22)
 
 - Added `oopbuy_search(keyword, ...)` for the new Oopbuy goods search plugin (`POST /goods/oopbuy-search`) - searches 1688, Taobao or the official channel and returns matched products (spu, title, price, monthSold, image, url) as a Hash.

data/README.md

@@ -1,6 +1,6 @@
 # ScrapeUnblocker Ruby client
 
-Official Ruby client for the [ScrapeUnblocker](https://scrapeunblocker.com) web scraping API.
+Official Ruby client for the [ScrapeUnblocker](https://scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk) web scraping API.
 
 Every request is fully JavaScript-rendered in a real browser and routed through premium proxies, so it bypasses Cloudflare, DataDome, PerimeterX, Akamai, Kasada and similar anti-bot systems - from one simple call. You are only billed for successful requests.
 
@@ -38,7 +38,7 @@ puts product.page_type # "product"
 p product.data
 ```
 
-Get your API key at [app.scrapeunblocker.com](https://app.scrapeunblocker.com). The free trial does not require a credit card.
+Get your API key at [app.scrapeunblocker.com](https://app.scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk). The free trial does not require a credit card.
 
 ## Authentication
 
@@ -94,7 +94,7 @@ goods = su.oopbuy_search("running shoes", channel: "1688", page: 1, page_size: 2
 goods["results"].each { |item| puts "#{item['title']} #{item['price']} #{item['url']}" }
 ```
 
-`channel` is one of `"1688"` (default), `"taobao"` or `"official"`. `sort` is one of `"default"`, `"price_asc"`, `"price_desc"` or `"best_selling"`. `page_size` max is 60. Brand keywords return HTTP 422.
+`channel` is one of `"1688"` (default), `"taobao"` or `"official"`. `sort` is one of `"default"`, `"price_asc"`, `"price_desc"` or `"best_selling"`. `page_size` max is 60. Oopbuy trademark-blocks brand keywords at its own backend: those come back as a successful `200` with `keywordRejected: true` and an empty `results` array, not an error.
 
 ## Cookies and the serving proxy
 
@@ -130,13 +130,15 @@ cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:0
 
 ## Error handling
 
-Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`. Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff.
+Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`.
 
 ```ruby
 begin
   html = su.get_page_source("https://example.com")
 rescue ScrapeUnblocker::BlockedError
   # 403: the target blocked every bypass path (not billed)
+rescue ScrapeUnblocker::PaymentRequiredError
+  # 402: quota, credit limit, or a failed payment - fix billing
 rescue ScrapeUnblocker::RateLimitError
   # 429: slow down
 rescue ScrapeUnblocker::UpstreamOutageError
@@ -146,15 +148,46 @@ end
 
 | Error | Status | Meaning |
 |---|---|---|
-| `InvalidRequestError` | 400 | Bad URL or unsupported scheme |
-| `AuthenticationError` | 401 | Missing or invalid API key |
+| `InvalidRequestError` | 400 | Bad URL, unsupported scheme, or the API key header was not sent |
+| `AuthenticationError` | 401 | Key not recognised - typo, stray whitespace, or a rotated key |
+| `NoSubscriptionError` | 401 | Key is fine, but the account has no active plan |
+| `PaymentRequiredError` | 402 | Billing block - base class for the three below |
+| `QuotaExceededError` | 402 | The plan's requests for this period are used up |
+| `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
+| `PaymentFailedError` | 402 | A card payment was declined three times |
 | `BlockedError` | 403 | Blocked by bot protection on every path |
+| `NotFoundError` | 404 | Page loaded but held no image (`get_image` only) |
+| `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
+| `UnsupportedContentError` | 415 | The URL serves something other than HTML |
+| `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
-| `ServerError` | 5xx | Unexpected server error |
-| `TimeoutError` | - | Request exceeded the timeout |
+| `ServerError` | 5xx | Unexpected server error, including a 504 upstream timeout |
+| `TimeoutError` | - | This client gave up locally before the API answered |
 | `ConnectionError` | - | Could not reach the API |
 
+Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff. A 401 or 402 is never retried - it clears when the key or the billing state changes, not on another attempt. Neither is billed or counted against your quota, because the request is refused before anything is scraped.
+
+### Billing errors (402)
+
+The three billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
+
+```ruby
+begin
+  html = su.get_page_source("https://example.com")
+rescue ScrapeUnblocker::QuotaExceededError
+  # plan quota (plus any overage allowance) is used up for this period
+rescue ScrapeUnblocker::CreditLimitExceededError
+  # unpaid balance passed the account credit limit
+rescue ScrapeUnblocker::PaymentFailedError
+  # card declined three times - update the payment method
+end
+```
+
+When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota. All three lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
+
+Full details for every status code: [developers.scrapeunblocker.com/errors](https://developers.scrapeunblocker.com/errors).
+
 ## Configuration
 
 ```ruby
@@ -168,9 +201,9 @@ ScrapeUnblocker::Client.new(
 
 ## Links
 
-- Documentation: https://developers.scrapeunblocker.com
-- Website: https://scrapeunblocker.com
-- Dashboard: https://app.scrapeunblocker.com
+- Documentation: [developers.scrapeunblocker.com](https://developers.scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk)
+- Website: [scrapeunblocker.com](https://scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk)
+- Dashboard: [app.scrapeunblocker.com](https://app.scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk)
 
 ## License
 

data/lib/scrapeunblocker/errors.rb

@@ -15,12 +15,89 @@ module ScrapeUnblocker
     end
   end
 
-  # The API key is missing, malformed, or not recognised (HTTP 401).
+  # The API key was rejected (HTTP 401).
+  #
+  # Two cases produce a 401: an unrecognised key ("Unauthorized" - a typo,
+  # trailing whitespace, an empty value, or a key rotated in the dashboard),
+  # and a valid key on an account with no plan, which raises the
+  # NoSubscriptionError subclass. Omitting the key header entirely is a 400,
+  # not a 401. Nothing is scraped for a 401, so it is not billed.
   class AuthenticationError < APIError; end
 
-  # The request was rejected as invalid, e.g. a malformed URL (HTTP 400).
+  # The key is valid but the account has no active plan (HTTP 401).
+  #
+  # Raised when the API answers a 401 with "No valid subscription". Pick a plan
+  # at https://app.scrapeunblocker.com - access resumes within about a minute,
+  # and the key does not change.
+  class NoSubscriptionError < AuthenticationError; end
+
+  # The account has a billing problem (HTTP 402).
+  #
+  # Credentials are fine - the request was stopped for a billing reason. There
+  # are three, each raised as a dedicated subclass: QuotaExceededError,
+  # CreditLimitExceededError and PaymentFailedError. Rescue this base class to
+  # handle all three.
+  #
+  # When more than one applies, the most serious wins: failed payment outranks
+  # credit limit, which outranks quota. All three lift by themselves once the
+  # billing state changes - access returns within roughly a minute, with no key
+  # change needed. Like a 401, a 402 is refused before anything is scraped, so
+  # it is never billed. Retrying is pointless; fix the billing state first.
+  class PaymentRequiredError < APIError; end
+
+  # Every request the plan allows this period has been used (HTTP 402).
+  #
+  # On plans that permit overages this only fires past the quota *plus* the
+  # overage allowance; inside that band requests still succeed and the extra
+  # usage is invoiced. Active coupon credit is spent before plan quota. The
+  # counter resets on the subscription's anniversary day, not the first of the
+  # month.
+  class QuotaExceededError < PaymentRequiredError; end
+
+  # The unpaid balance has passed the account's credit limit (HTTP 402).
+  #
+  # The balance counted here is the amount remaining on open invoices plus
+  # metered usage already consumed but not yet invoiced. Outstanding invoices
+  # are charged automatically when this triggers, so with a working card it
+  # usually clears itself within about a minute.
+  class CreditLimitExceededError < PaymentRequiredError; end
+
+  # A card payment has been declined three times (HTTP 402).
+  #
+  # Those attempts are the payment provider's automatic retries spread over
+  # several days, so a card has been failing for a while. Subscribing to a new
+  # plan does NOT clear this: the old unpaid invoice stays open, and the block
+  # stays until that specific invoice is paid.
+  class PaymentFailedError < PaymentRequiredError; end
+
+  # The request was rejected as invalid (HTTP 400).
+  #
+  # Raised for a malformed URL or unsupported scheme, for a missing
+  # x-scrapeunblocker-key header ("Missing x-scrapeunblocker-key"), and for a
+  # URL that belongs to a dedicated plugin - the response names the endpoint to
+  # use instead.
   class InvalidRequestError < APIError; end
 
+  # The page loaded but the requested element was absent (HTTP 404).
+  # Only #get_image raises this: the page rendered and held no <img> tag.
+  class NotFoundError < APIError; end
+
+  # The browser run did not finish in time on our side (HTTP 408).
+  #
+  # Distinct from TimeoutError, which is this client giving up locally. Here
+  # the API answered - it just could not render the page in time.
+  class BrowserTimeoutError < APIError; end
+
+  # The URL serves something other than HTML (HTTP 415).
+  # The message names the content type found. For images, use #get_image.
+  class UnsupportedContentError < APIError; end
+
+  # A request parameter is missing or has the wrong type (HTTP 422).
+  #
+  # Unlike the other errors the body is JSON, with a "detail" array pinpointing
+  # each problem field. Read it from #body.
+  class ValidationError < APIError; end
+
   # The target site blocked every available bypass path (HTTP 403).
   # Blocked calls are not billed.
   class BlockedError < APIError; end
@@ -32,6 +109,7 @@ module ScrapeUnblocker
   class UpstreamOutageError < APIError; end
 
   # ScrapeUnblocker returned an unexpected 5xx error.
+  # Also covers the 504 returned when a SERP fetch times out upstream.
   class ServerError < APIError; end
 
   # The request did not complete within the configured timeout.
@@ -40,24 +118,58 @@ module ScrapeUnblocker
   # The client could not reach the ScrapeUnblocker API.
   class ConnectionError < Error; end
 
+  BASE_MESSAGES = {
+    400 => "Invalid request (bad URL, unsupported scheme, or missing API key header)",
+    401 => "Authentication failed - key not recognised, or account has no active plan",
+    402 => "Billing block - quota exceeded, credit limit exceeded, or a failed payment",
+    403 => "Target blocked by bot protection on every bypass path",
+    404 => "Requested element not found on the page",
+    408 => "Browser run timed out before the page was ready",
+    415 => "URL does not serve HTML",
+    422 => "Validation error - see the detail array in the response body",
+    429 => "Rate limited - too many requests",
+    503 => "Upstream origin returned a server-side outage page",
+    504 => "Fetch timed out upstream"
+  }.freeze
+  private_constant :BASE_MESSAGES
+
+  # A 401 is either an unknown key or a recognised key on an account without a
+  # plan, and only the body tells them apart. Anything unrecognised stays on
+  # the general AuthenticationError rather than guessing.
+  def self.auth_error_class(body)
+    (body || "").downcase.include?("no valid subscription") ? NoSubscriptionError : AuthenticationError
+  end
+  private_class_method :auth_error_class
+
+  # The three billing blocks share a status code and differ only in their
+  # plain-text body. An unrecognised body falls back to PaymentRequiredError.
+  def self.billing_error_class(body)
+    text = (body || "").downcase
+    return QuotaExceededError if text.include?("quota exceeded")
+    return CreditLimitExceededError if text.include?("credit limit exceeded")
+    return PaymentFailedError if text.include?("payment failed")
+
+    PaymentRequiredError
+  end
+  private_class_method :billing_error_class
+
   # Build a typed error from an HTTP status code and response body.
   def self.error_for_status(status, body)
     snippet = (body || "").strip.gsub(/\s+/, " ")
     snippet = "#{snippet[0, 200]}..." if snippet.length > 200
-    base = {
-      400 => "Invalid request (bad URL or unsupported scheme)",
-      401 => "Authentication failed - check your API key",
-      403 => "Target blocked by bot protection on every bypass path",
-      429 => "Rate limited - too many requests",
-      503 => "Upstream origin returned a server-side outage page"
-    }.fetch(status, "API returned HTTP #{status}")
+    base = BASE_MESSAGES.fetch(status, "API returned HTTP #{status}")
     message = snippet.empty? ? base : "#{base}: #{snippet}"
 
     klass =
       case status
       when 400 then InvalidRequestError
-      when 401 then AuthenticationError
+      when 401 then auth_error_class(body)
+      when 402 then billing_error_class(body)
       when 403 then BlockedError
+      when 404 then NotFoundError
+      when 408 then BrowserTimeoutError
+      when 415 then UnsupportedContentError
+      when 422 then ValidationError
       when 429 then RateLimitError
       when 503 then UpstreamOutageError
       else status >= 500 ? ServerError : APIError

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.1.2"
+  VERSION = "0.1.7"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.7
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-22 00:00:00.000000000 Z
+date: 2026-07-27 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.
@@ -27,13 +27,13 @@ files:
 - lib/scrapeunblocker/parsed_page.rb
 - lib/scrapeunblocker/skyscanner.rb
 - lib/scrapeunblocker/version.rb
-homepage: https://scrapeunblocker.com
+homepage: https://scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk
 licenses:
 - MIT
 metadata:
-  homepage_uri: https://scrapeunblocker.com
+  homepage_uri: https://scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk
   source_code_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby
-  documentation_uri: https://developers.scrapeunblocker.com
+  documentation_uri: https://developers.scrapeunblocker.com?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk
   changelog_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby/blob/main/CHANGELOG.md
   bug_tracker_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby/issues
   rubygems_mfa_required: 'true'