scrapeunblocker 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +9 -5
- data/lib/scrapeunblocker/errors.rb +22 -9
- data/lib/scrapeunblocker/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f20529c692d1de01f3f02acca1accaf677cd5df2ec9d075bfb949b4b230272d2
|
|
4
|
+
data.tar.gz: b0eff5160bbd56e6e45700609ab17966d061d74537909c7a1a06df9242cddb7a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2421c59b2bc7c32d769d01546b4f796b7f71237edf500af4ee21b56f0a1393532b6361cc4207264c6072c5510bd31dc840986adda94dfbc18e7d2fdbf03a1df8
|
|
7
|
+
data.tar.gz: 98532dd9a066ef4e52f1123a958e36f1f40df7f603a3055b78784337bedfb256a237343d011e086ba8d0feb94cd17100e2501d5317055b35bae15fc57bea27ea
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.9.0 (2026-10-07)
|
|
4
|
+
|
|
5
|
+
- New `ScrapeUnblocker::BudgetExceededError` (a subclass of `PaymentRequiredError`): raised when the API answers 402 with `User set budget exceeded`, meaning this billing period's spend reached the monthly budget limit (EUR, excluding VAT) set in the [profile](https://app.scrapeunblocker.com/dashboard/profile). It clears at the start of the next billing period, or within about a minute after the limit is raised or removed. Like the other 402s it is never retried and not billed. Before, this body surfaced as a plain `PaymentRequiredError`.
|
|
6
|
+
|
|
7
|
+
No breaking changes: `rescue ScrapeUnblocker::PaymentRequiredError` still catches it.
|
|
8
|
+
|
|
3
9
|
## 0.8.0 (2026-10-06)
|
|
4
10
|
|
|
5
11
|
- `get_parsed` on a page that rendered but held no structured data now returns a normal `ParsedPage` instead of raising `NoDataExtractedError`. The API answers this case with a billed 200: `data` is empty, the new `data_extracted?` is `false`, the new `detail` explains it and the new `html` carries the rendered page, so no second call is needed. On a successful parse `data_extracted?` is `true` and `html` / `detail` are `nil`.
|
data/README.md
CHANGED
|
@@ -108,7 +108,6 @@ result["elements"].each { |el| p el }
|
|
|
108
108
|
```ruby
|
|
109
109
|
result = su.get_parsed("https://www.walmart.com/ip/12345")
|
|
110
110
|
puts result.page_type # e.g. "product"
|
|
111
|
-
puts result.source # how it was extracted
|
|
112
111
|
p result.data # the fields
|
|
113
112
|
|
|
114
113
|
# If a parse ever comes back wrong, force a fresh set of rules:
|
|
@@ -259,7 +258,7 @@ begin
|
|
|
259
258
|
rescue ScrapeUnblocker::BlockedError
|
|
260
259
|
# 403: the target blocked every bypass path (not billed)
|
|
261
260
|
rescue ScrapeUnblocker::PaymentRequiredError
|
|
262
|
-
# 402: quota, credit limit, or a failed payment - fix billing
|
|
261
|
+
# 402: quota, credit limit, your budget limit, or a failed payment - fix billing
|
|
263
262
|
rescue ScrapeUnblocker::RateLimitError
|
|
264
263
|
# 429: slow down
|
|
265
264
|
rescue ScrapeUnblocker::UpstreamOutageError
|
|
@@ -272,9 +271,10 @@ end
|
|
|
272
271
|
| `InvalidRequestError` | 400 | Bad URL, unsupported scheme, or the API key header was not sent |
|
|
273
272
|
| `AuthenticationError` | 401 | Key not recognised - typo, stray whitespace, or a rotated key |
|
|
274
273
|
| `NoSubscriptionError` | 401 | Key is fine, but the account has no active plan |
|
|
275
|
-
| `PaymentRequiredError` | 402 | Billing block - base class for the
|
|
274
|
+
| `PaymentRequiredError` | 402 | Billing block - base class for the four below |
|
|
276
275
|
| `QuotaExceededError` | 402 | The plan's requests for this period are used up |
|
|
277
276
|
| `CreditLimitExceededError` | 402 | Unpaid balance is past the account's credit limit |
|
|
277
|
+
| `BudgetExceededError` | 402 | This billing period's spend reached the monthly budget limit you set |
|
|
278
278
|
| `PaymentFailedError` | 402 | A card payment was declined three times |
|
|
279
279
|
| `BlockedError` | 403 | Blocked by bot protection on every path |
|
|
280
280
|
| `NotFoundError` | 404 | What you asked for does not exist - no image on the page (`get_image`), or a plugin lookup found nothing |
|
|
@@ -323,7 +323,7 @@ Transient failures (429, 502, 503, 504 and network errors) are retried automatic
|
|
|
323
323
|
|
|
324
324
|
### Billing errors (402)
|
|
325
325
|
|
|
326
|
-
The
|
|
326
|
+
The four billing blocks share a status code and differ only in their message, so the client raises a dedicated error for each:
|
|
327
327
|
|
|
328
328
|
```ruby
|
|
329
329
|
begin
|
|
@@ -332,12 +332,16 @@ rescue ScrapeUnblocker::QuotaExceededError
|
|
|
332
332
|
# plan quota (plus any overage allowance) is used up for this period
|
|
333
333
|
rescue ScrapeUnblocker::CreditLimitExceededError
|
|
334
334
|
# unpaid balance passed the account credit limit
|
|
335
|
+
rescue ScrapeUnblocker::BudgetExceededError
|
|
336
|
+
# this period's spend reached the monthly budget limit set in your profile
|
|
335
337
|
rescue ScrapeUnblocker::PaymentFailedError
|
|
336
338
|
# card declined three times - update the payment method
|
|
337
339
|
end
|
|
338
340
|
```
|
|
339
341
|
|
|
340
|
-
When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota. All
|
|
342
|
+
When more than one applies, the most serious wins: failed payment outranks credit limit, which outranks quota, which outranks your own budget limit. All four lift by themselves once the billing state changes - access returns within about a minute, and the API key stays the same. One catch worth knowing: subscribing to a new plan does **not** clear `PaymentFailedError`, because the old unpaid invoice stays open until it is paid.
|
|
343
|
+
|
|
344
|
+
`BudgetExceededError` (body `User set budget exceeded`) means this billing period's spend reached the monthly budget limit you set in your [profile](https://app.scrapeunblocker.com/dashboard/profile?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk) (EUR, excluding VAT). The key works again at the start of the next billing period, or within about a minute after you raise or remove the limit.
|
|
341
345
|
|
|
342
346
|
Full details for every status code: [docs.scrapeunblocker.com/errors](https://docs.scrapeunblocker.com/errors).
|
|
343
347
|
|
|
@@ -36,15 +36,16 @@ module ScrapeUnblocker
|
|
|
36
36
|
# The account has a billing problem (HTTP 402).
|
|
37
37
|
#
|
|
38
38
|
# Credentials are fine - the request was stopped for a billing reason. There
|
|
39
|
-
# are
|
|
40
|
-
# CreditLimitExceededError and PaymentFailedError.
|
|
41
|
-
# handle all
|
|
39
|
+
# are four, each raised as a dedicated subclass: QuotaExceededError,
|
|
40
|
+
# CreditLimitExceededError, BudgetExceededError and PaymentFailedError.
|
|
41
|
+
# Rescue this base class to handle all four.
|
|
42
42
|
#
|
|
43
43
|
# When more than one applies, the most serious wins: failed payment outranks
|
|
44
|
-
# credit limit, which outranks quota
|
|
45
|
-
#
|
|
46
|
-
# change needed. Like a 401, a 402 is
|
|
47
|
-
# it is never billed. Retrying is
|
|
44
|
+
# credit limit, which outranks quota, which outranks your own budget limit.
|
|
45
|
+
# All four lift by themselves once the billing state changes - access returns
|
|
46
|
+
# within roughly a minute, with no key change needed. Like a 401, a 402 is
|
|
47
|
+
# refused before anything is scraped, so it is never billed. Retrying is
|
|
48
|
+
# pointless; fix the billing state first.
|
|
48
49
|
class PaymentRequiredError < APIError; end
|
|
49
50
|
|
|
50
51
|
# Every request the plan allows this period has been used (HTTP 402).
|
|
@@ -64,6 +65,17 @@ module ScrapeUnblocker
|
|
|
64
65
|
# usually clears itself within about a minute.
|
|
65
66
|
class CreditLimitExceededError < PaymentRequiredError; end
|
|
66
67
|
|
|
68
|
+
# This billing period's spend reached the monthly budget limit you set
|
|
69
|
+
# (HTTP 402).
|
|
70
|
+
#
|
|
71
|
+
# The limit is set in your profile (EUR, excluding VAT), and spend is counted
|
|
72
|
+
# the way the invoice is: the plan's fixed monthly fee, if any, plus the
|
|
73
|
+
# requests billed on top of it. Requests paid from coupon credit do not count.
|
|
74
|
+
# The key works again at the start of the next billing period, or within
|
|
75
|
+
# about a minute after you raise or remove the limit at
|
|
76
|
+
# https://app.scrapeunblocker.com/dashboard/profile.
|
|
77
|
+
class BudgetExceededError < PaymentRequiredError; end
|
|
78
|
+
|
|
67
79
|
# A card payment has been declined three times (HTTP 402).
|
|
68
80
|
#
|
|
69
81
|
# Those attempts are the payment provider's automatic retries spread over
|
|
@@ -167,7 +179,7 @@ module ScrapeUnblocker
|
|
|
167
179
|
BASE_MESSAGES = {
|
|
168
180
|
400 => "Invalid request (bad URL, unsupported scheme, or missing API key header)",
|
|
169
181
|
401 => "Authentication failed - key not recognised, or account has no active plan",
|
|
170
|
-
402 => "Billing block - quota exceeded, credit limit exceeded, or a failed payment",
|
|
182
|
+
402 => "Billing block - quota exceeded, credit limit exceeded, budget limit reached, or a failed payment",
|
|
171
183
|
403 => "Target blocked by bot protection on every bypass path",
|
|
172
184
|
404 => "Requested element not found on the page",
|
|
173
185
|
408 => "Browser run timed out before the page was ready",
|
|
@@ -187,12 +199,13 @@ module ScrapeUnblocker
|
|
|
187
199
|
end
|
|
188
200
|
private_class_method :auth_error_class
|
|
189
201
|
|
|
190
|
-
# The
|
|
202
|
+
# The four billing blocks share a status code and differ only in their
|
|
191
203
|
# plain-text body. An unrecognised body falls back to PaymentRequiredError.
|
|
192
204
|
def self.billing_error_class(body)
|
|
193
205
|
text = (body || "").downcase
|
|
194
206
|
return QuotaExceededError if text.include?("quota exceeded")
|
|
195
207
|
return CreditLimitExceededError if text.include?("credit limit exceeded")
|
|
208
|
+
return BudgetExceededError if text.include?("user set budget exceeded")
|
|
196
209
|
return PaymentFailedError if text.include?("payment failed")
|
|
197
210
|
|
|
198
211
|
PaymentRequiredError
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: scrapeunblocker
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.9.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- ScrapeUnblocker
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-10-
|
|
11
|
+
date: 2026-10-07 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
|
|
14
14
|
plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.
|