scrapeunblocker 0.1.8 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '038dc9fc1bd496b441430ed4f1def53055dfb98db2cf21d40ec0242bcd66f344'
-  data.tar.gz: d8fb2745df9da6ac37db17308670f87b1c4131e2b9dfde9e5107ee4fe9e9c260
+  metadata.gz: d921e3453928431b4e1715efd3f8ae53fa1175f5b4fdbf4a67db12c0409cab18
+  data.tar.gz: 59bcc76323226693b74aa96d8a3f969118f6df95328a6e3424bed1ec8dfa78e0
 SHA512:
-  metadata.gz: ebc1b6eec94316dfb50c56b6109212eaaec54e5b8a0087f8e11f9cc9c634c751c1fed8bafe0bad51d8cf45ba5cd4d978f46feb40fb9c0d64de18aee98712ec46
-  data.tar.gz: 444e12bb48cfcf2e3b1cee4d76676c4334d234cb76a8642917845e4ac3b9a179786abac31c77e7251cf2eca95c1bf3e16ae12d4ace9c6aa131581617d01c12ad
+  metadata.gz: 4d0791186d046ebe36897a987c7fd08fdfa98c1c78754fc7a32bc9552ac782a903ec1e10bde22fa7d1b442972586ab8a85d6777a00dc5d310e1c9cfe40f9e372
+  data.tar.gz: b132825a15081702a1e76ab431ae9835f88ffa7a589241775688f1820ee58c14d438e0a9150afa995e70fee3ee97df63de6f0dd6b9ba78c17adbacd0ba00510b

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.2.0 (2026-08-29)
+
+- `get_page_source` now accepts `steps:` - an ordered Array of browser-action Hashes the API runs in the real browser after the page loads (`wait_for`, `wait_for_text`, `wait`, `click`, `type`, `select`, `press_key`, `scroll`). The array is JSON-encoded into the `steps` query parameter. Steps run once and are not idempotent; a failed step returns HTTP 422 and raises `ScrapeUnblocker::ValidationError`, whose `body` names the failed step (`step_index`, `action`, `reason`, `selector`, `html`).
+- `get_page_source` now accepts `list_elements:` - pass `true` to get a JSON summary of the matched elements (`{"url", "count", "elements"}`) instead of HTML. When set, the method returns the parsed Hash rather than an HTML String, mirroring `get_parsed`.
+
+No breaking changes.
+
+## 0.1.9 (2026-08-28)
+
+- Added `amazon_product` and `amazon_search` for the new Amazon plugin. `amazon_product(asin:/url:)` returns one product - title, brand, numeric price and currency, list price and savings, availability, rating, review count, seller, feature bullets, categories and images. `amazon_search(keyword, ...)` returns a keyword search's cards - asin, title, price, list price, rating, review count, a clean product URL, image and the sponsored/prime flags - on any of 20 regional marketplaces.
+- Prices come back in the right currency automatically: `proxy_country` defaults to the marketplace's home country (amazon.com -> US, amazon.de -> DE), pinning the exit over our ISP pool.
+
 ## 0.1.8 (2026-07-31)
 
 - Added `ebay_search` for the new eBay Search plugin: listings from any of the 19 regional eBay marketplaces as structured JSON - title, numeric price and currency, condition with a normalised `conditionCode`, seller username and feedback, shipping cost, sold/watcher/bid counts, image and a clean item URL.

data/README.md

@@ -62,6 +62,47 @@ html = su.get_page_source(
 )
 ```
 
+## Browser steps
+
+Drive the page after it loads with an ordered list of browser actions - fill a form, click through, wait for content, then capture the rendered result. Steps run once in the real browser, in order.
+
+```ruby
+html = su.get_page_source(
+  "https://example.com/search",
+  steps: [
+    { action: "type", selector: "input#q", value: "web scraping", clear: true },
+    { action: "press_key", value: "Enter" },
+    { action: "wait_for", selector: "#results" },
+    { action: "scroll", value: "bottom" }
+  ]
+)
+```
+
+Supported actions and their fields:
+
+| Action | Fields |
+|---|---|
+| `wait_for` | `selector`, `selector_type` (`css` default, `xPath`, `className`, `tagName`), `timeout_ms` |
+| `wait_for_text` | `value` (text to await), `timeout_ms` |
+| `wait` | `value` (milliseconds) |
+| `click` | `selector`, `selector_type`, `timeout_ms` |
+| `type` | `selector`, `selector_type`, `value`, `clear` (bool), `timeout_ms` - types like a human |
+| `select` | `selector`, `selector_type`, `value`, `timeout_ms` |
+| `press_key` | `value` - one of `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `Space`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `Home`, `End`, `PageUp`, `PageDown` |
+| `scroll` | `value` - `"bottom"` or a pixel count |
+
+Steps are not idempotent (they submit forms, click buttons), so a call is not safe to blindly retry. If a step fails, the API answers HTTP 422 and the client raises `ScrapeUnblocker::ValidationError`; its `body` is the JSON describing which step failed (`error: "step_failed"`, `step_index`, `action`, `reason`, `selector`, `html`).
+
+## List elements
+
+Pass `list_elements: true` to get a JSON summary of the matched elements instead of the full HTML. The method then returns a parsed Hash (`{"url", "count", "elements"}`) rather than an HTML String.
+
+```ruby
+result = su.get_page_source("https://example.com", list_elements: true)
+puts result["count"]
+result["elements"].each { |el| p el }
+```
+
 ## Get parsed JSON
 
 ```ruby
@@ -96,6 +137,24 @@ goods["results"].each { |item| puts "#{item['title']} #{item['price']} #{item['u
 
 `channel` is one of `"1688"` (default), `"taobao"` or `"official"`. `sort` is one of `"default"`, `"price_asc"`, `"price_desc"` or `"best_selling"`. `page_size` max is 60. Oopbuy trademark-blocks brand keywords at its own backend: those come back as a successful `200` with `keywordRejected: true` and an empty `results` array, not an error.
 
+## Amazon
+
+Product and search data as a Hash, priced in the marketplace's own currency:
+
+```ruby
+# One product by ASIN (or url: "https://www.amazon.de/dp/B0BSHF7WHW")
+product = su.amazon_product(asin: "B0BSHF7WHW", marketplace: "amazon.com")
+puts [product["title"], product["price"], product["currency"], product["rating"]].join(" | ")
+
+# Keyword search
+results = su.amazon_search("wireless headphones", sort: "price_asc")
+results["results"].each do |item|
+  puts [item["title"], item["price"], item["currency"], item["asin"]].join(" | ")
+end
+```
+
+`proxy_country` defaults to the marketplace's home country (`amazon.com` -> US, `amazon.de` -> DE), so prices come back in the right currency with no configuration.
+
 ## eBay search
 
 ```ruby

data/lib/scrapeunblocker/client.rb

@@ -37,10 +37,31 @@ module ScrapeUnblocker
     end
 
     # Fetch a URL and return the fully rendered HTML.
-    def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil)
-      request("/getPageSource",
-              url: url, proxy_country: proxy_country, time_sleep: time_sleep,
-              method: method, value: value, method_timeout: method_timeout)[:body]
+    #
+    # +steps+ is an ordered Array of browser-action Hashes the API runs in the
+    # real browser after the page loads (each Hash carries an +action+ and its
+    # fields): +wait_for+ {selector, selector_type?, timeout_ms?}, +wait_for_text+
+    # {value, timeout_ms?}, +wait+ {value}, +click+ {selector, selector_type?,
+    # timeout_ms?}, +type+ {selector, selector_type?, value, clear?, timeout_ms?},
+    # +select+ {selector, selector_type?, value, timeout_ms?}, +press_key+ {value},
+    # +scroll+ {value}. +selector_type+ is one of "css" (default), "xPath",
+    # "className" or "tagName". The steps run once and are not idempotent; if a
+    # step fails the API answers HTTP 422 with a JSON body naming the failed step,
+    # which surfaces here as a ScrapeUnblocker::ValidationError.
+    #
+    # +list_elements+, when true, makes the API return a JSON summary of the
+    # matched elements ({"url", "count", "elements"}) instead of HTML. This method
+    # then returns that parsed Hash rather than an HTML String.
+    def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
+                        steps: nil, list_elements: nil)
+      body = request("/getPageSource",
+                     url: url, proxy_country: proxy_country, time_sleep: time_sleep,
+                     method: method, value: value, method_timeout: method_timeout,
+                     steps: (steps ? JSON.generate(steps) : nil),
+                     list_elements: (list_elements ? true : nil))[:body]
+      return JSON.parse(body) if list_elements
+
+      body
     end
 
     # Fetch a URL and return structured JSON instead of HTML.
@@ -121,6 +142,35 @@ module ScrapeUnblocker
                 proxy_country: proxy_country)
     end
 
+    # Scrape one Amazon product by ASIN or URL and return it as a Hash.
+    #
+    # Returns title, brand, numeric price and currency, list price and savings,
+    # availability, rating, review count, seller, feature bullets, categories
+    # and images. Prices come back in the marketplace's own currency:
+    # +proxy_country+ defaults to the marketplace's home country
+    # (amazon.com -> US), pinning the exit over the ISP pool. Pass either
+    # +asin+ (with +marketplace+) or a full product +url+.
+    def amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil)
+      post_json("/marketplace/amazon-product",
+                asin: asin, url: url, marketplace: marketplace,
+                proxy_country: proxy_country)
+    end
+
+    # Search Amazon and return the result cards as an Array of Hashes.
+    #
+    # Each card carries asin, title, numeric price and currency, list price,
+    # rating, review count, a clean product URL, image and the sponsored /
+    # prime flags. +sort+ is "featured" (default), "price_asc", "price_desc",
+    # "avg_review" or "newest". Prices are in the marketplace's own currency;
+    # +proxy_country+ defaults to the marketplace's home country.
+    def amazon_search(keyword, marketplace: "amazon.com", page: 1, sort: "featured",
+                      min_price: nil, max_price: nil, proxy_country: nil)
+      post_json("/marketplace/amazon-search",
+                keyword: keyword, marketplace: marketplace, page: page,
+                sort: sort, min_price: min_price, max_price: max_price,
+                proxy_country: proxy_country)
+    end
+
     # Fetch an image URL through the bypass chain and return its raw bytes.
     def get_image(url, proxy_country: nil)
       request("/getImage", url: url, proxy_country: proxy_country)[:body]

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.1.8"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.1.8
+  version: 0.2.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-31 00:00:00.000000000 Z
+date: 2026-08-29 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.